[HN Gopher] Magic Pocket: Dropbox's exabyte-scale blob storage s...
___________________________________________________________________
Magic Pocket: Dropbox's exabyte-scale blob storage system
Author : rbanffy
Score : 70 points
Date : 2023-05-16 06:49 UTC (1 days ago)
(HTM) web link (www.infoq.com)
(TXT) w3m dump (www.infoq.com)
| dymk wrote:
| > 99.99% availability
|
| That seems... really bad, for the Core Core Service that
| Everything Depends On.
| abeppu wrote:
| S3 standard storage class also says it's 99.99% availability so
| ... I wonder how hard it is to get beyond that?
|
| https://aws.amazon.com/s3/faqs/
| sosodev wrote:
| It's extremely hard to get availability higher than that.
| Possible but not something companies are willing to promise
| in an SLA.
| awill wrote:
| I'd say it's not so much that it's really hard, it's that
| the compromises aren't worth it.
|
| note, I worked on S3 2015-2017
| abeppu wrote:
| So ... e.g. for 5 nines, the erasure code configuration
| would demand that do more writes in more locations and
| the cost impact is too high?
| skrtskrt wrote:
| or replication if you're not erasure coding.
|
| So yeah you need more storage-optimized server racks and
| all the associated manpower and maintenance, you also
| need them to be distributed across different datacenters
| and zones which of course also impacts latency and your
| ability to provide some appearance of consistency, then
| you also need the same distribution for stateless
| services serving the data.
|
| On and on and on and you might be nearly doubling cost to
| get to an extra 9 that almost all of your customers won't
| care about.
| jl6 wrote:
| I imagine that as you add more nines, you start to hit
| more problems that are more out of your control. Like,
| how many nines do your backup diesel generators have?
| sosodev wrote:
| That's a fair point. It would be quite possible if the
| customers were willing to pay a lot more.
|
| I worked on DigitalOcean's object storage for about a
| year not long ago. Makes sense that those of us who have
| been in this space would be interested in this article
| haha.
| gumby wrote:
| By comparison the old US POTS phone system was required to
| maintain downtime of less than five minutes per decade. I have
| no idea what rules, if any, apply to the modern wireless
| systems.
|
| Note: The consent decree, the actual legal obligation of the
| Bell system, actually specified the percentage of times someone
| would pick up the phone and not get a dial tone or operator. It
| was surprisingly high -- IIRC something like 2%, which is why
| you can see people toggling the hook in old movies. The five
| minutes per decade constraint I know because some of my old
| customers made digital phone switches and _they_ had to provide
| that SLA to their phone company customers, or else not get the
| order.
| [deleted]
| biofunsf wrote:
| I'm having trouble finding any reference for the "five
| minutes per decade" downtime limit. That would be "6 9s" or
| 99.9999% uptime which is just crazy in today's world. Even
| today VOIP providers only claim 99.999%, though it seems like
| in practice many fail to even get close too that. I think I
| found the 756 page consent decree between Bell and the US you
| mentioned but I can't find a reference there either, though
| it is quite a massive doc with so-so OCR: https://www.google.
| com/books/edition/Consent_Decree_Program_...
|
| Stepping back a bit, it seems like the FCC would be in charge
| of establishing rules like this but that consent decree I
| found (which might be the wrong one), is with the house
| Antitrust committee. (After my own Googling failed I also
| asked GPT-4 and it isn't aware of this either)
| gumby wrote:
| Well as I noted in my comment the consent decree itself
| IIRC specified availability rather than uptime.
|
| However the 5 min figure comes from my customers like DSC
| (R.I.P), Ericsson and Nokia building POTS switches. These
| guys were deadly serious, like the folks who made
| spacecraft and medical devices, plus the Ericsson and Nokia
| folks were nice too.
|
| In DSC's case they were so paranoid that they paid us a
| massive amount to maintain a special tool chain for just
| for them. It was frozen in time (no upgrades) and when they
| reported a bug and we sent them an updated tool chain they
| diffed the binaries and made sure that every delta was due
| to the big fix and nothing else (that the dev hadn't snuck
| in some other patch for some reason)! They did some other
| headstands with their hardware and software, but in the end
| it didn't save them.
|
| It's cool that the consent decree is online. That
| arrangement with the Bell system was very clever, though it
| led to a lot of weird anomalies and distortions but I think
| it did end up with a better phone system than the PTT
| model. It's also been a better model than what's happened
| with the power and water utilities.
|
| The FCC back then was a better regulator for phone
| customers than the (now obsolete) ICC had been.
| didacticBuffalo wrote:
| We kept a data closet in Manhattan in a sister building
| to 33 Thomas. Customers would ask for a BRP. We had N+2
| for data level stuff and higher but some customers would
| get hung up on not having second location. We took the
| stance if the facility fails we all have bigger issues.
|
| We had one 18 minute issue in the ten years of use. It
| was the best facility we ever used.
| gumby wrote:
| I remember in the late 90s/early 00s a lot of cases when
| sites would go down (everybody self-hosted) due to
| backhoe events and the like.
|
| These companies weren't idiots: they had replication for
| their databases and leased redundant transmission service
| because they knew this kind of thing could happen. The
| problem is you'd buy transmission from two
| providers...whose fiber turned out to be in the same
| conduit, or even both had rented bandwidth on the _same_
| fiber.
|
| At least people are smarter these days, and with
| widespread cloud service there are fewer people who need
| to keep track of this stuff.
| sosodev wrote:
| I think they got 5 nines mixed up with 6 nines. The 5 nines
| is 5 minutes per year. I asked ChatGPT and it claims they
| had a target of 5 nines which makes sense.
| zeroxfe wrote:
| That's 5 minutes a year, not decade. (I worked in telecom for
| a very long time.)
|
| Telco basically set the bar for "5 9s."
| _trampeltier wrote:
| That was before we had to update every system every day :-)
| petters wrote:
| Not really bad. It's less than an hour per year. Most users
| would not notice.
| sosodev wrote:
| How is that bad? 99.99% means the system is down for less than
| an hour each year. https://uptime.is/99.99
| colesantiago wrote:
| Oof.
|
| You're not wrong, this is really really bad, especially for
| Dropbox, storage is their business so I expected _way_ better.
|
| These stats are no different to S3 at all. All of this
| engineering and moving away from AWS and for so few gain in
| availability.
|
| I was initially excited when they moved away from AWS and
| expected industry leading higher availability when they moved
| away, but I was wrong.
|
| This is disappointing for Dropbox which this is their main
| business, storing files without any minor hiccups or outages
| for years.
| awill wrote:
| this is an overreaction. Dropbox is mostly storing files that
| are already stored on the customer's device, so customers
| usually won't notice an outage.
| colesantiago wrote:
| How is this an overreaction?
|
| I would expect Dropbox, a file storage company that proudly
| invests heavily in tech and infrastructure to achieve a
| better availability than what they already were on AWS
| (99.99%)
|
| In terms of availability the change is pretty much 0 and as
| a business / enterprise customer I might as well choose a
| different service with similar or higher 9s or (if my needs
| are complex) choose S3.
| prmoustache wrote:
| I'd rather have maximum consistency and integrity than
| maximum availability.
| nmjohn wrote:
| Can you provide an example of an alternative service
| which will give higher than 4 nines for availability that
| an enterprise customer would pick instead if that < 1hr
| of downtime per year was too high?
| colesantiago wrote:
| AWS (Architected Correctly) which I am sure Dropbox has
| experience in.
|
| https://aws.amazon.com/blogs/publicsector/achieving-five-
| nin...
|
| Here is a service that has managed to achieve 5x9s of
| availability:
|
| https://ably.com/
| jazzyjackson wrote:
| > achieve 5x9s of availability:
|
| Guaranteed availability is a bet they're willing to make,
| a gamble they've been on top of so far, a risk that,
| should something fail, they will pay out on according to
| their SLA.
| rrdharan wrote:
| > These stats are no different to S3 at all. All of this
| engineering and moving away from AWS and for so few gain in
| availability.
|
| What makes you think a gain in availability matters or is
| necessarily a motivation for the project?
|
| If they can achieve the same availability at far lower cost,
| it's a win for them, which is why they would (and did) do it.
| colesantiago wrote:
| > What makes you think a gain in availability matters or is
| necessarily a motivation for the project?
|
| This isn't a win for enterprise / business / mission
| critical customers. Governments and public services cannot
| use this at all.
| CaveTech wrote:
| They can, and do.
| jazzyjackson wrote:
| You think 9-1-1 doesn't experience downtime?
| alex_lav wrote:
| The propaganda is wild to make people believe governments
| and public services have even 99% uptime, let alone
| 99.99%.
| sosodev wrote:
| Do you know of any companies that actually provide better
| reliability in their consumer product? The ones that lie or
| skew their uptime calculation don't count.
| rsync wrote:
| "... storage is their business so I expected way better."
|
| Storage is our business and we target an _even worse_ 99.95%
| availability[1].
|
| Availability has a cost. That cost is complexity.
|
| We would very much prefer to have boring outages more often
| than have _fascinating outages_ very rarely.
|
| [1] https://www.rsync.net/resources/notices/sla.html
| rbanffy wrote:
| When you build anything on it, you know it may not be available
| for a couple hours a year and architect around it. 99.99 isn't
| bad at all. Your ISP probably has worse SLAs.
| varenc wrote:
| Fun bit of trivia: The name "Magic Pocket" comes from the very
| first Dropbox demo video. Back when the site was still on
| getdropbox.com. I believe the video was on the homepage when
| Dropbox was launched on HN. Here's a copy of it:
| https://www.youtube.com/watch?v=xy9nSnalvPc
| bj-rn wrote:
| Thought it might have come from the bitmap brothers' game:
| https://en.m.wikipedia.org/wiki/Magic_Pockets
| LtdJorge wrote:
| Or Doraemon
| CobrastanJorji wrote:
| This is an impressive product, and I apologize, but I'm gonna go
| on a bit of a rant about the PR language.
|
| I hate the phrase "Our system has over twelve 9s of durability."
| Amazon was the first motherfucker to claim this, but the other
| cloud storage folks are also culpable, but at least they mostly
| had the modesty to add some weasel words like "designed for" and
| didn't just straight up claim there was less than a 1 in a
| trillion chance of a durability failure.
|
| You don't have twelve 9s of durability. Your collection of copies
| of data on the hard drives do, assuming they exist in a vacuum
| and nothing bad happens to them except the normal sorts of things
| that cause hard drive failures that are nice and completely
| independent. But it completely ignores all other sources of
| problem, and those are so many orders of magnitude more common
| that you might as well claim "God-given, perfect durability"
| because it'd be just as accurate.
| laluser wrote:
| Of course, these systems are always designed for that. Just
| like every system is designed for a certain amount of
| availability. Also, even in a vacuum, things would degrade over
| time due to bit rot, etc. That's why the article mentions
| protections with verifiers and due to other things such
| accidental deletions due to potential bugs.
| sujayakar wrote:
| magic pocket's tech lead (disclaimer: my cofounder at convex)
| has a whole talk on this concept of "durability theater" [1]!
|
| the tldr is that those numbers of 9s are just table stakes. no
| system should ever lose data due to routine disk failures. so
| then, as you mention, there's another whole art to mitigating
| those other sources of problems.
|
| [1]
| https://www.facebook.com/atscaleevents/videos/17416916227706...
| dang wrote:
| Related:
|
| _Optimizing Magic Pocket for cold storage_ -
| https://news.ycombinator.com/item?id=19841887 - May 2019 (13
| comments)
|
| _Dropbox Extending Magic Pocket with SMR Drive Deployment_ -
| https://news.ycombinator.com/item?id=17300661 - June 2018 (1
| comment)
|
| _Inside the Magic Pocket_ -
| https://news.ycombinator.com/item?id=11645536 - May 2016 (29
| comments)
|
| _Scaling to exabytes and beyond_ -
| https://news.ycombinator.com/item?id=11283064 - March 2016 (6
| comments)
|
| _Dropbox's Exodus from the Amazon Cloud_ -
| https://news.ycombinator.com/item?id=11282948 - March 2016 (240
| comments)
| eatonphil wrote:
| Magic Pocket has been around for a while [0]! I'm curious if
| there is anything especially new that prompted the QCon talk and
| this blog post or if it was just a good time?
|
| [0] https://dropbox.tech/infrastructure/inside-the-magic-pocket
| [deleted]
| bityard wrote:
| Sounds interesting, but I can't find the github link...
| dilyevsky wrote:
| So is this thing using parts of Ceph? Haven't really seen OSD
| term used anywhere else...
| pinewurst wrote:
| Lustre used that nomenclature long before Ceph.
| MuffinFlavored wrote:
| https://www.lustre.org/
|
| > The Lustre(r) file system is an open-source, parallel file
| system that supports many requirements of leadership class
| HPC simulation environments. Whether you're a member of our
| diverse development community or considering the Lustre file
| system as a parallel file system solution, these pages offer
| a wealth of resources and support to meet your needs.
|
| Hadn't heard of it.
| LtdJorge wrote:
| It's normally used for supercomputers
___________________________________________________________________
(page generated 2023-05-17 23:01 UTC)