[HN Gopher] 12 years of Backblaze data center storage drives, vi...
___________________________________________________________________
12 years of Backblaze data center storage drives, visualized
Author : bddicken
Score : 118 points
Date : 2025-02-18 19:55 UTC (3 days ago)
(HTM) web link (benjdd.com)
(TXT) w3m dump (benjdd.com)
| RedShift1 wrote:
| What exactly is this visualizing? Does each dot represent a chunk
| of data?
| Piraty wrote:
| > 1 small node -> 100 drives
| halper wrote:
| 1 small node is a 100 drives. So the small circles represent
| 100 drives each, I think. Not sure what they ... do, though.
| RedShift1 wrote:
| Maybe the amount of drives they purchased?
| florbo wrote:
| I'm assuming it's the acquisition and removal of drives
| over time.
| sebmellen wrote:
| Doesn't appear to work on mobile safari
| jrimbault wrote:
| Or firefox and chromium ubuntu desktop
| patchtopic wrote:
| or Chrome, Brave on Linux ?
| Tijdreiziger wrote:
| Doesn't appear to work in Firefox or Chrome on Android
| either.
| fluidcruft wrote:
| Works fine for me in Firefox Android.
| alwyn wrote:
| Works fine for me on Firefox Linux. Interestingly took a lot
| longer to load in Chromium and Brave but they all work.
| codecraze wrote:
| It works on my iphone with safari
| mvanbaak wrote:
| It does, it just takes some time to download all the data
| NKosmatos wrote:
| Nice one, we can see the (logical) shift to bigger drives. One
| small comment if I may, after 2020-2021 it gets really crowded
| with the dots and the number of drives leading to a loss of
| overall picture ;-)
| bddicken wrote:
| Thats good feedback. It definitely gets crowded, and sluggish
| due to the number of dom nodes. I didn't spend much time
| optimizing performance.
| joshka wrote:
| there's a few places where all the dots seem to drop (guessing
| there's some discontinuities in the data?)
|
| There's also a few places where there's duplicate labels (e.g.
| Hitachi 3TB)
|
| Would be great to group by manufacturer somehow (e.g. color) and
| make the size more prominent.
|
| Very cool visualization regardless.
| Damogran6 wrote:
| I took that to be massive decomming of drives
| bddicken wrote:
| Yeah, when drives fall off the bottom that represents
| decommissioning.
| joshka wrote:
| No, it's an obvious data error. The same drives reappear in
| future months.
|
| Pause and look at 2013-08-01 through 2013-11-01, or
| 2015-09-01 through 2016-01-01 to confirm this.
| amelius wrote:
| Why doesn't it show a pile of failed drives?
| FlyingAvatar wrote:
| Yeah, I think it would be a cooler visualization if the drives
| were in a line instead of a circle and new drives are added on
| to the right. The failed drives pile up on the bottom.
|
| Still fun to watch as it is, though.
| toss1 wrote:
| YES
|
| And I'd want to see failed drives somehow organized by
| TimeInService and maybe origin...
|
| We of course expect their drive usage to grow, but what would
| be surprising (& provide more info) is how the drives fail or
| age-out. None of us without huge data centers can get that kind
| of info
| bobbob1921 wrote:
| Exactly, I'm. I was hoping to see all time drive failure data
| as well.
| bddicken wrote:
| I'd like to do another vis that includes failed drives. Please,
| keep helping me brainstorm.
| sega_sai wrote:
| It would be nice if it was just a conventional graph...
| ziddoap wrote:
| Data from https://www.backblaze.com/cloud-
| storage/resources/hard-drive... if you want to make one.
| bddicken wrote:
| [OP]
|
| Backblaze writes about these data sets, and includes some more
| conventional graphs. For example:
|
| https://www.backblaze.com/blog/backblaze-drive-stats-for-202...
|
| I wanted to do something more fun!
| vladde wrote:
| The data takes a while to load - a dozen megabytes of data or so.
| After a while the visualization loaded.
|
| (It loaded about 0.3MB/s for me)
| nrh wrote:
| needs the ball sizes to represent the storage capacity!
| pinoy420 wrote:
| That's not true. Don't shame me like that
| bddicken wrote:
| Great suggestion.
| xbmcuser wrote:
| I personally thought that ssd density and cost sizes would have
| crossed hdd prices by now. And we would start seeing them in
| these stats from blackblaze but hdd manufacturers seem to have
| stayed ahead of them so far.
| datadrivenangel wrote:
| Density wins but not cost. Maybe in another 5-10 years we'll
| get to parity.
| pcurve wrote:
| As a data guy, I'm not sure how useful this chart is.
|
| But it sure is fun to look at. I enjoyed it ;-)
|
| Not sure why it would intermittently redraw the whole scene
| though.. could be a Chrome thing.
| pmontra wrote:
| It's empty in Firefox Android. Play button, slider and nothing
| else.
| bddicken wrote:
| [OP] Interesting! I just tested it on Android/FF and it
| works. Could be a version-specific thing. Could also be just
| taking awhile to load (it has to download a 32MB json file).
| mlry wrote:
| I had to allow d3.js in NoScript for Fennec. Had the same
| intermittant redraw of the entire screen mentioned above.
| BadJo0Jo0 wrote:
| Fun to look at! Since I also had a mini project, that utilized
| this data. Sadly, haven't maintained it in awhile. It's a Show HN
| on my profile if you're curious.
|
| I hope you had a better time with ingesting the data than I did
| :)
| bddicken wrote:
| I actually built this primarily with chatGPT o1.
|
| One of the things LLMS are _really_ good at is writing scripts
| for processing and pairing down data. I wanna do a blog post
| talking about how did some of this, maybe coming up!
| fencepost wrote:
| Probably wouldn't be that difficult to organize down the y axis
| based on drive capacity, and the amount of pointless jostling
| around of small nodes makes it noticeably bog down as the years
| go by.
|
| Interesting visualization though.
| atYevP wrote:
| Yev from Backblaze here -> Y'all this is one of our favorite
| things and I've spread it internally. So cool <3
| bddicken wrote:
| OP here - thank you! I'd actually like to more in the future
| with your data sets. It's so cool that you publish this.
| atYevP wrote:
| Love that! I think I re-shared your tweet about it a few days
| ago! I've been kinda staring at it on a loop on my monitor,
| fun to watch the balls bouncing around :D
___________________________________________________________________
(page generated 2025-02-21 23:01 UTC)