▲ 83 points
back
69 comments
I have a giant storage RAID for my home server, with a bunch of 16TB drives. I bought each of the drives used about three years ago, and they cost about $120 each. They have been working fine until last night.
One of them appears to be broken [1]. No big deal, this is what RAIDs are for, I go and try to find one and now they're going anywhere between 2-4x that price, for a used one! It's not going to bankrupt me (and having a home server is a privilege in the first place, that's not lost on me), but I really hope that the others survive, at least until this storage crunch is over. If it ever does end...sigh.
I guess I didn't realize that even relatively slow storage like spinner drives was going to be affected too.
[1] I think, I am really hoping it's just a bad connection or something but I haven't fully diagnosed it yet.
ETA: Looks like at least in my case it was actually just a bad SATA cable. The drive is reading properly and resilvering now. Phew.
I believe many sellers an eBay are illegally manipulating the market, and eBay is tacitly helping them by ignoring and removing feedback.
For example this seller: https://www.ebay.com/str/disctechllc
“Accidentally miss-priced” a bunch of drives, and then instead of canceling the orders, refunded everyone, but still shipped packages: https://forums.servethehome.com/index.php?threads/enterprise...
I believe they intentionally did this, causing people huge import fees in some cases, in order to not remove the “26” sold on their listings that are now astronomically priced: https://ebay.us/m/mGRdiT
Edit: They also lied on their customs declarations (!)
Yeah, totally agree. I believe that the axing of several anti-monopoly enforcement departments and regulations in the largest market in the world (the US) is effectively a very big wink wink, nudge nudge to market participants; Trump basically got to play as Oprah for Big Businesses everywhere: "You get to be a cartel! You get to be a cartel! Everybody gets to be a cartel!"
He's basically created a sort of one-sided economic "Ferry Ordeal" (like the Joker on The Dark Knight [1]), basically leaving us consumers to not be exploited only if there are decent men at the helm of big businesses. It could be asymmetrical instead of one-sided if you consider that the people can only tolerate so much squeezing before they start clamoring for guillotines [2].
[1]: https://batman.fandom.com/wiki/Ferry_Ordeal_and_Skyscraper_B...
Holy shit, they want three grand for a 3.84TB drive. That's absurd.
I have a home array too, though based on 4Tb drives (6x, two lots of RAID10, one 3x4T one 3x6T, linked as an LVM volume as I extended it by adding the second array some time ago).
I was planning to downsize anyway as most of the media I don't need to keep and I plan to replace that server with a much lower power one with a bunch of smaller SSDs. Luckily I bought the SSDs (and the other parts) before the recent price hikes, I just haven't got around to building the machine! Hopefully they all work when I do finally get around to it…
Three years seems ridiculously low lifetime - I'd hope that was covered by warranty.
As I said, they were used, so I knew that a drive breaking was kind of an inevitability. As far as I'm aware there's no warranty, I certainly didn't pay for an extended one.
Good news though, since writing this I just started playing with dmesg and smartctl, it actually might be something with the SATA connector. At least those are still pretty cheap.
I've recently bought 3x32TB (new) and 2x28TB (recertified) drives for a new NAS as my old one started running out of space on some drives (local LLMs and datasets or media for dataset preparations are huge these days) as I expect the prices to go up considerably due to most HDD manufacturers being booked years in advance already. Some drives don't even make it to retail at all (44TB Seagate).
also bought a handful of 14 to 16tb drives. They sold for such a low price last year I thought it can't be wrong to grab them.
It's odd mechanical disks also surged, I thought it was only transistor based memory that are becoming rarity.
Or does it work like with fuel, gas and electricity goes up when oil spikes ?
If I need storage now and I can't get flash disks, I will buy mechanical. Production will not ramp up overnight (or at all - there is no point investing in scaling up if I think the situation is only temporary), so an increase in demand inevitably will result in prices going up.
Btw gas goes up when oil goes up not just because it can replace it in some applications, but also because it is often produced and transported in the same areas as oil, by the same companies involved in oil, so it is typically affected in similar ways.
Customer demands (or hoarding) sentiments kept us busy on the production lines of racks with 1K+ drives and I think we've already shipped out more than 5K of them since early last year. No supply constraints on cpu and memory parts due to the way this sku was designed but damn did we have so many defective hdd's; I'd ballpark about 20% from the vendor and 10% from mishandling.
Yeah I have no idea the direct cause. I didn't think that the SATA controllers for a hard drive took that much.
It could be a secondary effect; SSDs have gotten so expensive that people are willing to put up with spinners and thus there's an increased demand. No idea, I'm sure an economist or something will do a write up of the downstream effects of the RAM crunch causes eventually.
I've seen lots of articles on HN of AI startups building massive drive arrays for mass storage.
AI runs on data above all else. Gotta feed the compute.
GenAI and/or smart glasses video? WD already sold their entire 2026 production of nearline drives for data centers.
If you buy (hypothetically) a lot of disks and don't use them for, say 10 years, will they degrade?
Why isn't production scaling to meet demand?
Shouldn't the market address this.
Because:
1. Factories take time to build.
2. Building factories requires capital to be invested now.
3. The return-on-capital will only be obtained in the next n years.
4. But if demand goes down, we'll have much more supply than demand, leading to a cutthroat price competition, which could prevent the factory costs from ever being recouped.
No 4 is why nobody is willing to really move on new fascilities.
China is doing it but that was for merely creating tech independence.
5. There are only limited number of suppliers and they’re making a killing from all of this.
Didn’t these companies collude before? I’m sorry but I don’t believe that they’re also using this opportunity to make as much money as possible as well.
And to add to that 4th point, there is a very real chance that many the “AI” companies that are buying all this hardware could go bankrupt at anytime and in rapid succession, leaving drive/memory manufactures with a bunch of unpaid orders and a second hand market now saturated with liquidated equipment.
Much like the GPU/crypto boom/bust some years ago, but on an even larger scale.
> 4. But if demand goes down, we'll have much more supply than demand, leading to a cutthroat price competition, which could prevent the factory costs from ever being recouped.
Why not normalize that, if things don't work out, the government steps in, buys the business, and offers a better regulated version of the business?
Capitalism is generally touted as useful for innovation; but haven't we paid collectively for the initial investment for many businesses many times over now that there's little innovation done? By definition there's no innovation once the only thing happening in a business sector is consolidation.
So, I'd conclude that capitalism isn't useful to us at that point, and actually harmfully inefficient (too costly for consumers), so we'd be wise to do away with it by that point in favor of a more efficiently run operational/economic system... That would actually be innovative! What that system looks like, I'm not sure, but I imagine it's something like a cooperative.
Because when the AI customers explode N months down the line, you don't want to be on the hook for a new factory.
Building production facilities isn't like flipping a switch.
Production was close to maximum anyway because of existing demand and how expensive new fabs are to bring up. The boom in AI use wasn't sufficiently planned for as it wasn't expected at the scale we are seeing, so to scale up means building more fabs - a long and expensive task. The situation is not helped by one of the AI companies successfully negotiating huge deals for a year worth of production with the two biggest providers at the same time and keeping it secret enough that neither bumped up prices in the deal as a result of knowing what the play was.
Building these factories goes way beyond "nontrivial". We may see a few come online in a couple of years though, but time will tell if the extra capacity alleviates the crunch.
Most players are rational actors and this isn't their first time in this sort of market. Getting more production online if you do not have slack like downtime takes quite a while. And there is no guarantee that they can sell the new production even at previous rates if they end up in over supply.
Generally people just don't understand how long ramping up new production facilities can take and what is the realistic pay off period for them.
It is the law of inertia, it applies to many more situations than we think. Markets is one of the, especially large-scale ones.
Adding to what others have said, several hdd, ssd and ram manufactures have said that they don't believe demand will be sustained in time for long enough to warrant extra investments in increasing production.
Because the old guard is at it again.
Production is actually falling this year compared to last for Samsung and Hynix (there was an article on here a few days ago?).
Because the manufacturers of storage (and RAM) consider it to be an AI bubble, too. The surge in demand is a short burst, not a sustained one. Hence it makes no sense to throw a whole lot of ressources into scaling up production, when the demand won't be there anymore in 1-2 years when the factories will be ready.
A chip fab costs a billion dollars to build.
Modern cutting edge ones, yes. Not all fabs, by a long shot.
Free markets, yes
Why isn't magic just doing the magic things the capitalists always tell us is magic?
I picked the worst era to build a NAS.
I bought 8x 4TB NAS RED 5400RPM HDD and paid in total AUD$1.4k, I am not even joking. My pool is only 8TB mirrorer and I am keeping the other 16TB as spare because things will get worse.
I bought them brand new because used ones cannot be trusted but even that brings risks.
WD has been selling 7200RPM labelled as 5400RPM disks.
WD was selling SMR disks instead of CMR disks.
Seagate disks used into crypto farms are being sold as brand new, so folks are finding our their brand new disks have 65k hours.
Likewise.
I was planning on upgrading my disks about a year ago, but help holding off for discounts. The AI race to the bottom, driving up prices only really started happening within the last 12 months (at least for HDDs) and now I'm too late.
Some years ago, Modi announced that he was going to make India go all-in on semiconductors. When I read that the first facility to begin commercial production was going to be Micron with memory chips, I did an eyeroll. Memory chips? And just an assembly and testing facility? To me that seemed like an easy cop-out. To me (my admittedly naive self), wafer fabs and CPUs seemed like the real game.
Now, with what has happened with memory chip prices, it almost seems like they got lucky (the Micron facility is doing commercial shipments now).
Obama used to talk about having "spooky" good luck. I think Modi has some of that too.
I would think if you're a developing country looking to build some domestic semiconductor manufacturing expertise, it'd probably be best to start with something on the easier side. Something with closer to well-known and standard tech that can still be sold on the open market.
Just opened this year, apparently: https://investors.micron.com/news-releases/news-release-deta...
It's just an attempt to copy Taiwan and the Philippines by starting with the labor intensive low value add part of semiconductor manufacturing.
Guess I'll find the old ones at the back of my cupboard for the time being ... oh wait. A 16MB SD card. Those were the days.
Earlier today I discovered the existence of 2TB microsd cards
Who cares guys, soon food shortages will start. In Europe they started rationing fuels. In Australia gas stations are out of diesel.
We are trully doomed.
> In Europe they started rationing fuels
They are rationing in Slovenia to prevent 'gasoline tourism' (people taking advantage of lower taxes), not due to shortages.
far be it for me to question your prophetic capacities. i wonder, do you have any historical examples or logic-based arguments that our doom of nigh?
A bit tangential: it’s narratives like this which can create sudden crashes on the stock market.
why are chefs baking bread? there's buildings to construct.
> In Australia gas stations are out of diesel.
This is incorrect.
Not sure why you are getting downvoted. The scariest one is the fertilizer situation which means less productive harvests, which means hunger.