back
95 comments
I always love the Backblaze hard drive stats, even if I'm not in the market.

> During this quarter 4 (four) drive models, from 3 (three) manufacturers, had 0 (zero) drive failures. None of the Toshiba 4TB and Seagate 16TB drives failed in Q1, but both drives had less than 10,000 drive days during the quarter. As a consequence, the AFR can range widely from a small change in drive failures. For example, if just one Seagate 16TB drive had failed, the AFR would be 7.25% for the quarter. Similarly, the Toshiba 4TB drive AFR would be 4.05% with just one failure in the quarter.

Backblaze should consider adding interval estimates in addition to point estimates. It might help the reader understand the uncertainty of the point estimate.

More important than interval estimates would be survival curves to know how the drives fail as they age, which could give you a sense of when they should be replaced.
I look forward to these reports with the same zeal and enthusiasm as I used to reserve for comic strips in Sunday newspapers.

This data is significant and appreciated. Thank-you so much, and please keep up the good work.

While I don't _quite_ share your enthusiasm. I sure do share your appreciation. Nobody else at their level appears to be sharing this data. This is not something they had any obligation to share with us.

It certainly is useful and interesting.

Every time I read through their update articles I'm reminded that Backblaze is a service I could possibly buy! I only buy a hard drive once every few years, but I enjoy seeing the data. The time it takes to write up and share what they are no doubt tracking anyway is probably quite worthwhile, I'd guess this is an effective piece of marketing. I'd be quite interested to see how many new sign-ups they get each time they post these stats.
.. it is however, very good cheap marketing
Some argue this data has mainly historical value. By the time it's available vendors may have changed many aspects of respective models, from geometry to controller chip to motors, let alone firmware. You might end up bying a very different product under the same SKU, to which this data has no relation at all.
At this point, Backblaze is a serious "influencer", and HDD companies would be smart to make sure they get the best possible drives, even if the rest of us get junk.

I mean "doing it well for an influencer and shafting everyone else" is far cheaper and easier than just making a solid and reliable product consistently. That's such old-fashioned thinking these days. :)

You might end up bying a very different product under the same SKU, to which this data has no relation at all.

Exactly, as we just saw with the Western Digital debacle.

The unfortunate reality is, only a fortuneteller can say what the best hard drive to buy right now is.

I agree. I circulate it every quarter in my company's Slack channel to show the importance of publishing open data and reports, not just as a steward of technology, but because it is actually useful marketing.
I concur - I used to work with a set of clusters totaling ~45k drives and it was always fun to compare failure rates (especially on a per model basis).
Did you end up with significally different values sometimes ?
Another Backblaze stats release, another time I wish someone on the inside at Seagate would drop in and tell us what the deal is with their drives.

I have a lot of old Hitachis that still work, but every last one of my Seagates died years ago. Yes, they've gotten better since then, but they're still reliably trounced by Hitachi.

Where are the failures coming from? What did they cheap out on? Has the C-suite decided it's not financially worth increasing reliability? What are the internal feelings on being the outlier, year in and year out, in these stats? Does anyone care?

They will get sued if they say why. Hence they will never admit why.
They sell a lot of drives, do they have to care? Someone has a spreadsheet that tells them when it is time to care.
Interesting data. Overall surprised at how few failures they saw.

From an interview last year, it sounds like they don't use SMR drives[1]. I would be interested if there is any good source for failure rates of those.

[1] https://www.backblaze.com/blog/how-backblaze-buys-hard-drive...

Yev from Backblaze here -> Yea, we've tested them but found they didn't play nice so we aren't deploying them in droves. If you find a good place for data on SMRs send it our way, would be fun to read up on it!
SMR performs like crap and doesn't save much money on density, so it's a poor play for anything except tape-replacement.
I recently bought 4x WD HGST WUH721414ALE6L4 512e 14TB. The 4Kn's were the same price but on lengthy backorder/dropship from WD, so that wasn't going to work. Also, I absolutely refuse to buy the WD Gold (WD141KRYZ) that is effectively the same product but at a much higher price ($480 vs. $346). Marketing people can take a long walk from a short pier.
I'm (unfortunately) boycotting WD and their subsidiaries; https://news.ycombinator.com/item?id=22935563
I was actually looking at WD Gold for the next workstation. Do you have any good resource / introduction-page about their HDDs price/performance/misc (e.g. which drives are just rebranded). Very hard to compare specs for us uninitialized =)
Still wondering what happen to those HDD roadmaps. HAMR and MAMR and those 40TB promised by 2023. And as far as I am aware even the 18TB and 20TB coming in late 2020 / early 2021 are still CMR and SMR.

If it wasn't for Helium Sealed tech moving more platters into HDD, we would have zero capacity improvement in the past 4-5 years.

I'm expecting a resurgence of quantum bigfoot style drives.

So, while the idea was ridiculed in the press, the advent of SSD's and the usecase for harddrives these days actually means it makes a lot of sense given the area (and therefor the capacity) increases with the square. Combined with the slower spin rates also increasing the density, and you get a device which is far more generally useful for bulk storage than SMR.

The reason I asked is because we are not driving down cost. And judging from how HDD maker react they are going to milk this for as long as possible, all while our Data usage and requirement continues to grow rapidly. i.e Our total cost for Data is actually increasing.

NAND prices fluctuate up and down and the current lows are not very sustainable.

It seems the days of computing getting cheaper every two or three years are gone.

I've heard they are actually closer to production, but yeah.. HAMR's been "a couple years away" for over a decade at this point, I think.
I'm thankful for these reports as well, despite them being trailing edge/etc. For a while they mirrored some problems we were having at work with a particular vendor (and helped to justify switching products a year or two into the nightmare).

What I really wish is that they would make an effort to go beyond just reporting their experiences and see what they are doing as also providing a service in the form of model reliability data. AKA toss a few pods of WD's/etc in there even if they are slightly more expensive/whatever. If the data were broken out by production location and manufacture date, its likely it would be something that they could sell on the open market for a small fee. I know I would have gotten the company I worked for to pay such a fee for a somewhat scientific look at the failure rates of certain models/etc.

AKA, pay a bit more for a broader set of drives, summarize the data for free, and then get people to pay for the detail data. If you work for a company buying a thousand or so drives a year, avoiding a 10% AFR is going to be worth a lot of money. AKA too small to have a good view of the state of things, but big enough that buying 1k bad drives and fighting daily RAID rebuilds for the next three years is real nightmare. Think of it as a bit of insurance, or at least validation of a problem when things start to go south.

Disclaimer: I work at Backblaze.

> AKA toss a few pods of WD's/etc in there even if they are slightly more expensive/whatever.

If you see "low numbers" of one particular drive model in the stats, that is usually us trying out that drive model because at some point in the future the price might drop making it worth purchasing it in bulk worth it, or the price is ALREADY worth it but we're being careful in the rollout to make sure that drive model performs in our particular application - nobody else's application, just for us.

> make an effort to go beyond just reporting their experiences

We are VERY careful to report what we are seeing in our datacenter for our particular application, and no more. This isn't a scientific study, we aren't Consumer Reports, we're just publishing data we would collect whether or not we released it. We have a core business we super happy focusing on, somebody else is WELCOME to sell drive testing and drive predictions and we won't compete with them or get in their way. Heck, we would totally subscribe to that service!

When there were still stats being published for WD, they were always significantly worse than HGST, even long after WD bought HGST.

I'm wondering why they keep maintaining two product lines that are sufficiently different to result in one basically being consistently the best, the other being consistently the worst, in terms of reliability.

I actually knew the company from those reports. Migrated from S3 to b2, not the same featureset, but good enough for me.

Over the same features I find b2 to be way simpler and cheaper.

Thanks

Yev from Backblaze here -> Nice! Welcome aboard :)

Edit -> what are the feature's your missing most? We're collecting feedback at: b2feedback@backblaze.com if you want to send some notes over!

It's surprising to me that they use actual boot drives and don't just boot their machines off of USB sticks or via PXE or like those fancy Dell dual redundant internal sdcard modules.

But perhaps the boot drives are just leftovers that are too small to be used any more for storage?

Yev here -> The boot drives are helpful for log storage/collection so the extra capacity is nice - plus they're not "large" drives so the costs associated are pretty minimal!
Might just be convenient for logs and other random junk
Once again Seagate putting on a poor show reliability wise - although, with only three companies really making hard drives these days I guess Backblaze doesn't have much choice but to also include their drives in their service.
Why doesn't Backblaze then switch to more of the models of the other companies. It almost seems like the more of a particular model they have the higher the associated failure rates.
Little offtopic, but the new S3 integration works for you guys?

I tried to connect with s3cmd but it wasn't working. Unfortunately their support was not very helpful neither...

Yev here -> we're working on it! We've heard a lot from folks using s3cmd and I believe we have some changes coming shortly. If you want to chat with the PM team directly, we're grabbing feedback at: b2feedback@backblaze.com!
I love these. It'd be quite nice if they made some attempt to publish confidence intervals.
Andy at Backblaze here: Good to know. We used to do this with the lifetime stats, but that got lost somewhere along the way. I'll look at getting them back. Thanks.
Unrelated, but anyone use B2 in SEA? How is the dl speed? I'm on the market to try to move away from cloudfront + s3 (I use cdn77 in front of cloudfront now, but have like 10% cache misses in a key area everyday that would theoretically hit b2)
For anyone that comes across this: they only have 4 data-centers, and the first out out side of the US was in Europe asof EOM august 2019[0].

[0] https://www.backblaze.com/blog/announcing-our-first-european...

I wonder how long it will be before backblaze moves to SSDs. Obviously a lot more expensive right now, but with the lower power consumption/heat and ability to cram way more in, might be sooner than we think?
Andy at Backblaze here. We use SSDs in our core servers and more recently in boot drives as they both need to speed. To store data in our case we don't need the speed so its not worth it yet. Given the amount of data growth, most predictions have HDDs still with about 50% of the storage market in 2025.
When SSDs are comparable in price to HDDs. Which is... not soon.
I see Seagate is still tops the list of failures after all these years. I wonder what we would see if we put all those stats together which company actually comes out as the worst.
One company will always be worst. Using the worst drives often makes sense for a business. It all depends on price, availability, and deployment plans.

Seagate's drive longevity has improved a lot. Compare today's report to Backblaze's 2015Q1 report:

https://www.backblaze.com/blog/hard-drive-reliability-q1-201...

Is there a SSD version of this sort of report?
And if so, would the TL,DR version of it still be, "Buy Intel?"