[HN Gopher] The drive stats of Backblaze storage pods
___________________________________________________________________
The drive stats of Backblaze storage pods
Author : leiferik
Score : 133 points
Date : 2024-01-03 18:07 UTC (4 hours ago)
(HTM) web link (www.backblaze.com)
(TXT) w3m dump (www.backblaze.com)
| MeteorMarc wrote:
| >>>All 40 of the Dell servers which make up these two Vaults were
| relocated to the top of 52U racks, and it appears that initially
| they did not like their new location.
|
| Or the way in which they were handled during the relocation?
| newsclues wrote:
| Maybe they moved them again to confirm location was the issue
| lostlogin wrote:
| "All 40 of the Dell servers which make up these two Vaults
| were relocated to the top of 52U racks, and it appears that
| initially they did not like their new location. _Recent data
| indicates they are doing much better, and we'll publish that
| data soon._ " Italics mine.
|
| Seems like they changed something.
| willcodeforfoo wrote:
| Looks like ~3,087 Petabytes of total storage!
| stackskipton wrote:
| Is there any information what lead to Backblaze to stop making
| their own hardware and going with COTS?
| mbrameld wrote:
| > Over the last few years, we began using storage servers from
| Dell and, more recently, Supermicro, as they have proven to be
| economically and operationally viable in our environment.
|
| Sounds like it was money.
| wannacboatmovie wrote:
| Yes, building your own shit is expensive.
|
| Tale as old as time.
|
| Before someone uses Google's servers as an example, I would say
| that strapping together consumer grade components with zip ties
| and no case isn't what I'd consider a 'server'; rather, a loose
| collection of parts.
| hinkley wrote:
| Google has 3 orders of magnitude more servers than I'll ever
| need. With that sort of difference it's not only a change of
| solution but also a change of rules.
|
| We simply do not witness the same bottlenecks.
| lesuorac wrote:
| IIRC, parts for some of the garage servers were found from
| literally dumpster diving.
|
| However, it does cause you to write some very fault tolerant
| code.
| Analemma_ wrote:
| They may have just reached a scale where it was viable, where
| it wasn't before. The bigger you are, the bigger discounts you
| can negotiate. They also certainly have the leverage to walk
| away from a deal, since they have proof "hey, we'll just build
| our own if we don't like your offer."
| ericbarrett wrote:
| My sense is that storage server prices have had to come down to
| compete with both cloud (like S3 archival tiers) and roll-your-
| own solutions. The former are relatively recent and the latter
| have become more feasible as companies--Backblaze among them--
| have open sourced well-engineered designs. Storage server
| margin used to be insane, over 80% after COGS back in the 00s
| and only slightly less last decade, and there's still room to
| compete.
| hinkley wrote:
| I hope history will eventually read something like this:
|
| In the aftermath of the Dot Com Bust, manufacturers became
| conservative, and lost their ability to dream big. Into this
| power vacuum stepped the so called Cloud Providers, who in
| some cases made their own hardware and tools to solve their
| problems.
|
| Over time manufacturing caught up, missing tools were
| written, and the Cloud providers went back to solving the
| main problem nearly none of their customers of suppliers
| could ever solve: the speed of light (locality).
| zdw wrote:
| I would imagine that in the early days there was nothing that
| met the requirements. Early on very few systems had the drive
| density needed - products like Sun's Thumper with 48 drives
| (https://en.wikipedia.org/wiki/Sun_Fire_X4500) were few and far
| between.
|
| Today there are lots of high storage density devices, so no
| point to build your own if you can get it already engineered
| and with a warranty from somewhere else.
| louis-paul wrote:
| Here is another post by the same author:
| https://www.backblaze.com/blog/the-storage-pod-story-innovat...
| dijit wrote:
| I love these so I'm really sad to say this: What's missing is an
| AFR based on deployed time in increments, not total time.
|
| For example: there should be a sliding window (or a histogram)
| relating to AFR after deployment for a time frame.
|
| IE: AFR between 0-300 days, AFR between 301-600 days, AFR between
| 601-900 days (etc).
|
| Otherwise we're looking at failures historically for the entire
| period, which might hide a spat of failures that consistently
| occur 3 years in, giving a relatively unfair advantage in numbers
| to newer drives.
|
| That said, I really do love these and I hope they continue.
| ahazred8ta wrote:
| Backblaze's SSD bathtub curve shows higher failure rates < 6mo
| and > 3yr, with a sweet spot around 1.5yr where the failure
| rate is 4-5 times lower. -
| https://regmedia.co.uk/2023/09/26/bathtub.jpg
|
| The HDD afr is under 2% for the first 3.5 years. Due to factory
| testing, 'the bathtub now has no left side, which makes it hard
| to take a bath'. https://i.gzn.jp/img/2021/12/21/black-blaze-
| how-long-disk-dr...
| okbrown wrote:
| If there is one company that could give accurate stats on what
| drives actually perform better and what drives are the worst and
| fail more often it is Black Blaze, but yet they don't. What a
| missed opportunity to provide some insight.
| jannyfer wrote:
| Like this?
|
| https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-...
| okbrown wrote:
| My first Facepalm!! For 2024!!
| okbrown wrote:
| No.... I take it back, I am a fool! They do publish the stats!!
| https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
|
| My bad!!
| okbrown wrote:
| One thing for sure though, these stats have confirmed my
| disappointment with Seagate all these years. Utter garbage!
| ranting-moth wrote:
| Yep, I never buy a drive for myself or recommend to other
| without having first looked the their stats.
| apocalyptic0n3 wrote:
| They've been publishing these stats for years and years now,
| too. It's a wealth of data they've provided and has helped me
| pick the dozens of drives I've ordered over the years
| autoexec wrote:
| Where are you ordering your drives from these days? In the
| past I went from physical shops -> new egg -> amazon and
| now amazon and new egg aren't reliable sellers and the
| physical shops are either gone, under-stocked, or buy their
| stuff on amazon too.
| pqdbr wrote:
| I really wished Backblaze B2 worked better to be a real
| alternative to S3. The pricing is a middle finger to AWS and
| other providers.
|
| However, we were having 1 outage per month with B2, and in the
| middle of 2023 we decided to go back to AWS and only use S3 for
| replication.
|
| We still monitor both services simultaneously (S3 and B2), and
| every other month we are still having an episode where the
| latency rises from 15ms to something like 25 seconds for each
| write/read operation.
| nerdponx wrote:
| It works pretty well for low-volume ad-hoc data storage at
| least. I use it for personal data science projects.
| eropple wrote:
| Have you looked at Cloudflare R2? I've had good luck there, but
| am exercising it only lightly.
| tempest_ wrote:
| It has been a while since I looked but CFs R2 was cheaper than
| s3 the last I looked, never used it though.
| jiripospisil wrote:
| The problem for us was the latency - sometimes it would take
| ~2s to serve an image. We ended up putting Cloudflare in front
| of it and that worked but might have actually been against
| their ToS (I know there have been some changes to it since
| then).
| phpisthebest wrote:
| I have tried several S3 Alternative so far Wasabi has been the
| best...
|
| I had some Latency and Bandwidth problems when trying to push
| alot of data into BackBlaze not terrible but slower than Wasabi
| it however was usable. Some of the other vendors I tried were
| not even usable
| atesti wrote:
| Does anybody know which Dell product they use? Dell also has JBOD
| enclosures like Powervault MD2460
| mmmooo wrote:
| they mention servers, so possibly not jbods, but R740xd2 (or
| newer R760xd2). The former are 26x3.5" so would match what they
| describe.
| gorkish wrote:
| The most interesting thing here is the discussion about using
| vendor hardware instead of their own bespoke stuff.
|
| My takeaway is that it's still essentially impossible to
| negotiate anywhere close to a fair price with commodity server
| vendors until you are buying hundreds of machines, and then only
| if you are capable of demonstrating that you are willing to
| design and build them yourself. And yet despite this, it's still
| cheaper than cloud even paying advertised prices.
|
| Where do regular people buy servers without insane markup if you
| need say 10-20? Used to be I could actually buy supermicro
| barebones, but that ended a really long time ago.
| prirun wrote:
| > Where do regular people buy servers without insane markup if
| you need say 10-20?
|
| IMO, for a small number of servers, the insane markup isn't a
| huge cost of business so doesn't matter that much.
| lostlogin wrote:
| It's a pity there isn't a way for several buyers to pool
| their purchasing power.
| lazide wrote:
| 2-3x of 1 is still in the same order of magnitude and
| unlikely to matter.
| aaronax wrote:
| You get two VARs, one with Dell and one with HP. Then a couple
| back and forth negotiation rounds and you will be paying 40-50%
| of MSRP. Or even try to do it direct with your Dell/HP reps,
| since VARs are of dubious value.
| mynameisnoone wrote:
| Boooo! No drive manufacturer names and model numbers. This is
| totally useless data for anyone so it's absolutely pointless to
| blog about.
|
| https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
| contains the necessary data, but it seems old.
|
| I don't know why they're buying COTS server gear because their
| whole premise was the economy-of-scale of having their own pods
| made. I don't understand why they don't go to Quanta or Foxxconn
| to build them whatever they need because Dell is really just a
| marketing front like CDW that relies heavily on third-party
| contract designers and major component manufacturing, and then
| only does final assembly itself.
| Sohcahtoa82 wrote:
| > https://www.backblaze.com/blog/backblaze-drive-stats-
| for-q3-... contains the necessary data, but it seems old.
|
| ...is this a joke?
|
| That's Q3 2023. Assuming the quarters are calendar quarters, Q4
| 2023 ended literally 3 days ago. They're not going to have a
| report out right away.
| joshfee wrote:
| I use B2 as the backend for my personal backups using restic
| (which I would highly recommend
| https://github.com/restic/restic). I don't have a ton of data to
| backup, so even with hourly backups (restic only backs up when
| there are changes) I have ~100GB and it runs me a whopping
| $0.60/month. I almost feel guilty when I get the bill. But the
| minute I need to pick a storage platform in a professional
| context I know what my first choice will be.
|
| (I am _not_ affiliated with Backblaze in anyway. Just a happy
| user)
| cvalka wrote:
| Rustic is better
| antx wrote:
| Better in what way? Restic is well written, and is "mature".
| Rustic, is still considered "beta" software.
| cvalka wrote:
| It supports append only(which is a MUST for backup
| software) and is declarative.
| dsissitka wrote:
| restic has supported append-only repositories since
| before rustic was a thing.
| autoexec wrote:
| What plan are you using? Their website suggests that their
| plans for a single user costs are ~$6-$10 a month.
|
| https://www.backblaze.com/cloud-backup/pricing
|
| It's still a pretty good deal though...
| phpisthebest wrote:
| B2 is not Cloud Backup, B2 is there S3 compatible storage
| that you pay by the GB
|
| your links are to their "backup" service which is only for
| Windows and Mac computers, and is limited to their backup app
| which many people report having throttling and other issues,
| it is "unlimited" in the sense that it should only be used
| for a single computer, which is why they never support Linux
| on it, because they believe (probably correctly) that linux
| support would mean most people will install it to NAS devices
| and ruin the business model for everyone else
|
| Historically that has been the case for all of the backup
| solutions that offered "unlimited" data for a fixed price
| monthly, I think BackBlaze is the only remaining vendor in
| the game that does
| MertsA wrote:
| Backblaze B2 is their generic object storage platform similar
| to S3. You pay what you use and it scales into petabytes.
| There's no minimums so if you're only using small amounts,
| you get a small bill at the end of the month. It's not backup
| software, just the underlying cloud storage platform.
| artimaeis wrote:
| Cloud backup != B2.
|
| https://www.backblaze.com/cloud-storage/pricing
|
| $6/TB/Month. Very manageable egress-fees. If you're using it
| for cold-backup, hopefully you rarely have to pay them.
| cweagans wrote:
| No egress fees if you aren't downloading more than 3x what
| you've stored!
| sudhirkhanger wrote:
| Is their any difference in storing to B2 via Restic or
| Duplicity vs BackBlaze Computer Backup?
|
| Do you have access to your file via mobile access, sharing
| feature, etc.? https://www.backblaze.com/cloud-backup
| qwertox wrote:
| We're so lucky that Backblaze creates these reports and shares
| them openly. That's not normal for companies to do and it's
| valuable for us consumers to get these insights.
|
| Actually I'm referring to their drive stats and not their storage
| pod stats like in this report.
|
| Thank you Backblaze.
| agumonkey wrote:
| Seconded, it's one of the links that I will automatically read
| in great details.
|
| Thanks indeed.
| gadders wrote:
| Yeah yeah. Just make your client software suck less. Crashplan is
| much better but went professional only, sadly.
___________________________________________________________________
(page generated 2024-01-03 23:00 UTC)