[HN Gopher] Amazon S3's 15th Birthday: 5,475 Days and 100T Objects
___________________________________________________________________
Amazon S3's 15th Birthday: 5,475 Days and 100T Objects
Author : yarapavan
Score : 140 points
Date : 2021-03-15 14:17 UTC (8 hours ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| codegeek wrote:
| One of the best things that made AWS what it is today. Sure it
| can be a little pricey compared to competitors but it was
| revolutionary when it came out and still is a great product. Also
| a lot of its competitors still lack certain key features that
| makes it hard to move away from s3. aws s3 sync
| ...
|
| And you are in business.
| astuyvenberg wrote:
| When people talk about avoiding vendor lock in with AWS, they
| tend to talk in terms of using k8s or another compute
| abstraction layer.
|
| But I think the real lock-in lies in services like S3, making
| so many workflows simple and intuitive (while controlling your
| data, which is where the real lock-in lies)
| selfhoster11 wrote:
| Minio and other S3-compatibles make moving off of AWS that
| much simpler, though obviously it might be difficult if
| you're using Amazon-only S3 features extensively.
| astuyvenberg wrote:
| Especially if you layer on additional functionality, like
| triggering a lambda function to resize an image when it
| hits the S3 bucket.
| luhn wrote:
| Yeah, even if you don't use any vendor-specific features,
| there's data gravity--It's hard to move your data from one
| location to another. AWS and others increase this cost even
| further by making egress so expensive.
| astuyvenberg wrote:
| Precisely - vendor lock-in generally happens at the data
| level. For example, AWS finally turned off their last
| Oracle DB server only a year or two ago.
| PaulHoule wrote:
| I am worried.
|
| We will start having trouble with random UUID collisions when we
| have made 2^64 of them, which is a huge number. It's about as
| many iron atoms as are in an iron filing.
|
| It is also about 200,000 times 100 trillion, so AWS is just 18
| doubling times away from the UUID system breaking down.
| gabagool wrote:
| Is there any specific reason one could not just put two
| UUID-4's alongside one another?
|
| Also as another commenter mentioned, if they are tied to a
| namespace prefix, that delays collisions.
| pradn wrote:
| Is this a real problem? It should be possible for Amazon to
| update their code to generate IDs a different way, and for them
| to update their client libraries well ahead of a time when this
| becomes a problem. Yes, code that relies on the ID being a
| particular format will break. But users have to deal with
| deprecated client library features all the time. It's
| reasonable to give a few year grace period before old APIs are
| shut down.
|
| UUIDs are usually not simply random numbers. So it's possible
| to tell when a particular standard of UUID is used. Amazon's
| code can branch based on this detection, or even something
| simpler like "UUID-type", a new field that would be initialized
| by default to the old UUID type.
|
| I simply don't see this as a problem for anyone.
| herodoturtle wrote:
| At this rate, won't AWS run out of storage space before they
| run out of UUIDs?
| PaulHoule wrote:
| or the waste heat boils the oceans... or something
| gonzo41 wrote:
| All AWS need to do is add something like a namespace. Shazzam!
| problem solved for your lifetime.
| leesalminen wrote:
| > That's almost 13,000 objects for each person in the world, or
| 50 objects for every one of the roughly two trillion galaxies
| (according to this 2021 estimate) in the Universe.
|
| That's mind boggling to think about. I wonder how much paper
| would've been needed to store 100T objects on paper. It's like
| the new Library of Alexandria.
| TedDoesntTalk wrote:
| > It's like the new Library of Alexandria.
|
| Presumably the Library of Alexandria didn't have access control
| lists
| jcims wrote:
| And look what happened...
| herodoturtle wrote:
| There's a joke in here about OVH...
| leesalminen wrote:
| For sure ;) but I also imagine that the number of objects
| with a public-read ACL (potentially mis-configured) would
| still dwarf Alexandria.
| koolba wrote:
| If it did then Caesar wouldn't have had permission to burn
| it.
| tmotwu wrote:
| > It's like the new Library of Alexandria.
|
| Would be very unfortunate if the S3 story ends like the Library
| of Alexandria.
| jliptzin wrote:
| What's way more mind boggling to me is that there are at least
| 260 galaxies per person in the world. I am not shocked at all
| that S3 is home to any number of cat pictures many of which are
| probably duplicates.
| anderspitman wrote:
| Found the IPFS user.
| iamgopal wrote:
| I don't know, but to me 260 per person seems less than what I
| would guess.
| intsunny wrote:
| Can we update the title? 100T reads as "100 terabytes" instead of
| "100 trillion" (as per the source).
| globular-toast wrote:
| It should be read as "100 tera-objects", meaning 100 trillion
| objects. Writing "K" seems to be so widespread these days
| (Youtube likes/dislikes are written as 1.5K, for example), I'm
| surprised there is any confusion here.
| sneak wrote:
| 100 terabytes is 100TB. The T prefix without a unit means 100
| trillion, just as 100TB means 100 trillion bytes.
|
| I think it's clear as-is.
| woudsma wrote:
| I read it as 100 trillion. 100TB would be the 'right' suffix
| for terabytes IMHO.
| alpaca128 wrote:
| > 100T reads as "100 terabytes"
|
| No, it doesn't. Not more than it reads terameters or teragrams.
| SI prefixes were not invented specifically for computer memory.
| nrmitchi wrote:
| 100 terabytes would be '100TB', not '100T'.
|
| Further, '100 Terabyte Objects' wouldn't really make sense as a
| sentence anyways.
| TedDoesntTalk wrote:
| > S3 is designed to provide "11 9's" (99.999999999%) of
| durability
|
| Has anyone lost data on S3 or know anyone who has?
| pauldino wrote:
| Not to me, but it's happened.
|
| https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanent...
| gnrlst wrote:
| There's different storage tiers, with different redundancy. So
| if anyone has an anecdote, would be curious to know which
| storage tier was used (e.g. single availability zone)
| rhacker wrote:
| I imagine that in most cases it's a giant archive - data that
| no one really cares about (I mean they do in principal). Then
| there's probably like 0.1% that's probably operationally
| important data - which is actually huge.
|
| This is just a guess based on my own usage of it over the last
| 15 years at different companies.
| qrush wrote:
| Yes. Once provided extensive documentation and debugging info
| they did refund portions of bills too.
| ZiiS wrote:
| If it works as designed I guess 1000 have; but how would you
| know. Did your perfect code running on an perfect CPU and RAM
| connected though a perfect network upload the file with 11 9's
| certainty?
| donavanm wrote:
| This is a real problem, but it is partially accounted for:
|
| > To ensure that data is not corrupted traversing the
| network, use the Content-MD5 header. When you use this
| header, Amazon S3 checks the object against the provided MD5
| value and, if they do not match, returns an error.
| Additionally, you can calculate the MD5 while putting an
| object to Amazon S3 and compare the returned ETag to the
| calculated MD5 value.
|
| It's not as obvious with multipart uploads where each chunk
| is (iirc) individually checksumd and stored, but only a
| single md5 is returned in the "etag"
| rootsudo wrote:
| Wow, has it really been 15 years?
| bluedino wrote:
| Do they de-dupe at all?
| mathgladiator wrote:
| no
| jgalt212 wrote:
| oops, I think you found their secret sauce for cost control and
| economies of scale.
| weird-eye-issue wrote:
| There is no way
| jgalt212 wrote:
| prove me wrong.
| babelfish wrote:
| You have the burden of proof for your claim.
| tpmx wrote:
| It's obviously technically _possible_.
| weird-eye-issue wrote:
| "Technically possible" that they do it, sure. But I would
| disagree it is possible that it is S3's "secret sauce".
| tpmx wrote:
| Because?
| weird-eye-issue wrote:
| ... this is on you to prove it is their secret sauce.
| Throwing out some random feature and saying it is their
| secret sauce and putting the burden of proof on somebody
| else makes no sense.
|
| But it would be because 1) they probably don't have many
| duped objects to begin with 2) the system to de-dupe
| items would be complex, error prone, increase latency,
| and just not worth it
| tpmx wrote:
| > 1) they probably don't have many duped objects to begin
| with
|
| That's crap reasoning. You have no way of knowing that.
|
| > 2) the system to de-dupe items would be complex, error
| prone, increase latency, and just not worth
|
| Would it really be that complex? I kinda doubt that.
|
| Read latencies wouldn't be affected - except being
| improved.
|
| Write latencies - the deduplication can happen after the
| write has been confirmed, so no extra latency added.
|
| Whether or not it's worth it - I think it's very hard to
| to estimate the deduplication factor. My guess is 1.5x.
| At that point it would save 33% storage. That would
| instanly make a whole lot of extra complexity worth it.
| weird-eye-issue wrote:
| It's like you are living in an alternate reality
| tpmx wrote:
| ...because?
|
| I dunno, I kinda think you might be out of your depth on
| this one, based on that response.
|
| Also: none of your few previous posts are tech-related.
| They're all about personal finance.
| weird-eye-issue wrote:
| So just to sum up your argument: You think 33% of objects
| in S3 are dupes and that removing those dupes would be
| S3's secret sauce?
| butt_hugger wrote:
| Doubtful since they have an option for at-rest encryption.
| tpmx wrote:
| It's just that, an option, not the default. People tend to
| use the defaults.
| amthrwy wrote:
| Yes. The contents of zero-byte objects are deduplicated. (The
| metadata, of course, are not.)
| simonebrunozzi wrote:
| If you're curious about some "S3 internals", this is a talk I
| gave in 2009 [0], based on a ton of interesting stuff that S3's
| then-GM Alyssa Henry [1] shared with me.
|
| [0]: https://vimeo.com/7330740
|
| [1]: http://itc.conversationsnetwork.org/shows/detail5273.html
| jodrellblank wrote:
| This recent interview with Werner Vogels[1] touches on S3
| design priorities as well:
|
| https://cacm.acm.org/magazines/2021/3/250706-a-second-conver...
|
| https://news.ycombinator.com/item?id=26365873
|
| [1] which you commented on, but others might not have seen.
| tw04 wrote:
| >Most of our customers, if they have on-premises systems--if
| they're lucky--can store two objects in the same data center,
| which gives them four 9s. If they're really good, they may
| have two data centers and actually know how to replicate over
| two data centers, and that gives them five 9s. But eleven
| 9's, in terms of durability, is just unparalleled
|
| I have a tough time taking him seriously reading this - I'm
| not sure if he's being intentionally ignorant or
| intentionally misleading.
|
| I don't know of a single enterprise customer that doesn't
| have their data replicated to two datacenters, and then
| backed up to some other medium (tape or disk-based backup
| appliance) for anything business critical.
|
| Their goal is 5-9's of UPTIME - the expectation is 100% data
| durability. You would get fired if you architected a solution
| keeping two copies of data in one datacenter.
|
| Object storage absolutely has a place and I'm happy amazon
| was able to push a quasi-standard for the industry. Object
| storage prior to S3 was a hodge-podge of proprietary plays
| (EMC Centera, Bytecast, etc) and sort-of standards that
| nobody really used (CDMI). I just wish they did a better job
| of fairly representing it vs. turning every opportunity into
| a sales pitch spreading FUD about the alternatives.
| ctvo wrote:
| Giving him the benefit of the doubt that he knows his own
| customers here.
|
| He said:
|
| > Most of our customers, if they have on-premises systems--
| if they're lucky--can store two objects in the same data
| center
|
| You said:
|
| > I don't know of a single enterprise customer that doesn't
| have their data replicated to two datacenters
|
| His customers don't equal your enterprise customers. You
| are both right most likely.
| tw04 wrote:
| Given that part of what I do is help enterprise customers
| with cloud adoption, almost all of whom are fortune 500
| (and on his radar), with an AWS presence, I strongly
| doubt it.
| ignoramous wrote:
| > _I don 't know of a single enterprise customer that
| doesn't have their data replicated to two datacenters, and
| then backed up to some other medium (tape or disk-based
| backup appliance) for anything business critical._
|
| You snipped out the part of the quote that answers your
| question.
|
| "Most of our customers, if they have on-premises systems--
| if they're lucky--can store two objects in the same data
| center, which gives them four 9s. _If they 're really good,
| they may have two data centers and actually know how to
| replicate over two data centers, and that gives them five
| 9s._ But eleven 9's, in terms of durability, is just
| unparalleled. And it trumps everything."
|
| > _I don 't know of a single enterprise customer that
| doesn't have their data replicated to two datacenters..._
|
| Except, in AWS' case, each AZs (Availability Zones) is made
| up of upto 8 DCs (Data Centers), and each full-region has
| at least 3 AZs and 2 Transit Centers. Amazon S3 replicates
| data to 3 different AZs (which, I am guessing, is in
| addition to replicating it across DCs in a single AZ for
| 'eleven 9s').
|
| With S3 cross-region replication durability may shoot up to
| 'sixteen 9s'? 100% durability, if it exists, is something
| the major Cloud providers are yet to offer?
|
| ref: https://maisonbisson.com/post/object-storage-prior-
| art-and-l...
| tw04 wrote:
| >You snipped out the part of the quote that answers your
| question.
|
| Replicating across two datacenters isn't "really good"
| though, that's considered table stakes.
|
| >Except, in AWS' case, each AZs (Availability Zones) is
| made up of upto 8 DCs (Data Centers), and each full-
| region has at least 3 AZs and 2 Transit Centers. Amazon
| S3 replicates data to 3 different AZs (which, I am
| guessing, is in addition to replicating it across DCs in
| a single AZ for 'eleven 9s').
|
| In AWS' case, the 8 DCs in an AZ are directly adjacent
| which isn't really useful for anything beyond metro
| (active/active) availability. I haven't seen a DR plan
| that doesn't have a hard requirement for the secondary
| copy of data to be outside of the metro loop/blast radius
| generally in a different state.
| txlpo78 wrote:
| > I don't know of a single enterprise customer that doesn't
| have their data replicated to two datacenters, and then
| backed up to some other medium (tape or disk-based backup
| appliance) for anything business critical.
|
| Longtime consultant here with BCP/DR insight into 20+ large
| F500 companies... I think you would be seriously surprised
| at how common it is for even enterprise customers to not
| bother with multi-DC or even offsite backups. And even
| among the ones that do, many of the ones that are non-
| cloud-based are so immature at it that I would not put
| money on their backups being restorable if needed. And this
| is even more true for your "we're a startup, we don't have
| time to worry about backing up our data!" companies, which
| are in abundance.
|
| For a small peak, go look at how many people were freaking
| out about losing their entire business due to the loss of a
| single OVH data center.
|
| Of course, as a consultant I do naturally skew towards
| customers that need help with this stuff, so my perspective
| is probably biased towards the companies that are worse off
| in this regard. But they're definitely out there.
| corban1 wrote:
| I am really getting afraid of this centralization of data.
| klodolph wrote:
| Eh... the thing about cloud storage is that it can be efficient
| in ways that smaller players cannot. A system like Amazon S3
| can pack its hard drives to the brim with data and pay a fairly
| small overhead for replicas & unused drives.
|
| If you try to decentralize the data, or keep your data
| yourself, you end up with something that uses more resources,
| has worse durability, or is worse along some other axis.
|
| I am not so worried about centralization yet. There is still
| healthy competition between different cloud storage vendors. My
| impression is that cheap storage is used to make other cloud
| services more attractive--which means that Amazon's incentives
| here are somewhat aligned with mine (as a customer). Personally
| I am much more worried about the centralization of the
| _shopping_ experience.
| ghgdynb1 wrote:
| I think it's fine as long as people know what they're getting
| themselves into. I'm sure I would've found whatever was
| happening on Parler abhorrent, but in the same way Merkel and
| Macron were concerned by the tech deplatforming that followed
| the capitol riots, so am I.
|
| I don't think filecoin will ever demolish a fantastic business
| like S3 based on stealing away customers. The interesting value
| proposition from the protocol is probably more about censorship
| resistance and adjacent issues.
| stingraycharles wrote:
| At the same time, the S3 api is so ubiquitous that there
| implementations for it in multiple clouds, it frequently comes
| with hardware storage vendors as well, and there are projects
| like minio that provide a solution to host your own s3
| compatible data architecture.
|
| All things considered, I'm much more concerned about the
| centralization of eg browser technologies than this.
| dehrmann wrote:
| I wonder how many objects have been lost and how many have been
| corrupted.
| dna_polymerase wrote:
| > All of those decisions remain valid and code written on launch
| day will still work just fine today.
|
| And this is why GCP will never come close to AWS.
| astuyvenberg wrote:
| S3 may be my favorite AWS product. It's the epitome of "It just
| works" design.
| ignoramous wrote:
| One of the tenets for the S3 team is/was "indistinguishable
| from perfect." I guess they're living up to it.
|
| https://www.allthingsdistributed.com/2010/05/amazon_s3_reduc...
| TheGuyWhoCodes wrote:
| S3 is a great service but it's a double edge sword. It's easy to
| start with and "cheap" until you scale and then you come to
| realize you pay a ton for writes and you have a latency of 100ms+
| to retrieve data. Even S3 select which on face value seems great
| is absurdly slow. Then they offer you Fsx or elastic cache as a
| proxy.
| throwawaysea wrote:
| > tens of millions of requests per second
|
| How does something like that get handled? At some point isn't
| contacting this service really just contacting a handful of IP
| addresses? Are those just "regular" computers? Specialized
| hardware?
|
| Also with the amount of centralization here, presumably they are
| the target of bad actors. How do they defend against things like
| DDOS's to provide their claimed uptimes?
| zokier wrote:
| > At some point isn't contacting this service really just
| contacting a handful of IP addresses?
|
| "Handful". S3 has 283 subnets allocated to it, biggest one
| being a /15, so there are some few hunderd thousand IP
| addresses reserved for S3. While impossible to say how many of
| them are actively in use, I imagine large fraction of them.
|
| The AWS IP ranges are documented in https://ip-
| ranges.amazonaws.com/ip-ranges.json
| ttul wrote:
| I'm wildly guessing here, but I assume that there are many
| tiers of load balancing to turn your initial request into the
| message that reaches the thing that gives back your file data.
| They also likely do have specialized hardware in various parts
| of the AWS infrastructure.
|
| A service that can handle tens of millions of requests per
| second can probably outscale most DDoS attacks without further
| engineering.
___________________________________________________________________
(page generated 2021-03-15 23:03 UTC)