[HN Gopher] AWS switch from gzip to zstd - about 30% reduction i...
___________________________________________________________________
AWS switch from gzip to zstd - about 30% reduction in compressed S3
storage
Author : luu
Score : 417 points
Date : 2022-08-20 05:34 UTC (17 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| obikeyz wrote:
| togs wrote:
| Zstd, it's okay. 600 .rvz files = 450 Gb -> 163 GB. Wanted it
| more like 10 GB.
| seydor wrote:
| Which of course will not be passed to the consumer
| [deleted]
| eru wrote:
| That's what competition is for.
| NavinF wrote:
| Well yeah. S3 is ~$23/TB/month vs $10/TB HDD NRC +
| $0.20/TB/month electricity for a data hoarder NAS. I can't
| imagine S3 users are very price sensitive. Most of them store
| <1PB.
|
| Of course there are cheaper cloud services like B2 at $5/TB/mo,
| but even that can't compete on price with physical infra when
| you have more than a handful of drives.
| Dylan16807 wrote:
| Not that it changes the math much, but:
|
| $15/TB is hard enough for a home buyer to reach for hard
| drives, excluding server cost. Where did you get $10?
| cmeacham98 wrote:
| Not them, but searching for "8TB HDD" on Amazon (huh,
| ironic) shows several options for $90, which is almost
| $10/TB. And this was 60 seconds of searching, no real
| effort put in.
| NavinF wrote:
| $15/TB is extremely expensive if you're building a raidz2
| array.
|
| $9.70/TB 10TB: https://www.ebay.com/itm/275400447467
|
| $8.31/TB 4x8TB: https://www.ebay.com/itm/125132232253
|
| You can get even cheaper if you buy in bulk.
|
| Server cost is $1/TB with reasonable density
| booi wrote:
| they should at least etch our names into the yacht that the
| savings will buy
| cloudymeatballs wrote:
| As a cloud employee I am willing to make the public
| commitment that should I ever be in the position of buying a
| yacht, I will etch "booi" into the hull for you.
| ddorian43 wrote:
| Google/Azure should compress and pass the savings back to the
| customer..
| seydor wrote:
| I'm not holding my breath
| midasuni wrote:
| Well if they figure they will get more customers to offset
| lower profits they'll reduce the price. On the other hand if
| they think they can double the price but only lose 10% of
| customers they'll do that
|
| It's econ101, the cost of providing the service doesn't
| really factor into the price charged, what does factor in is
| the cost of alternatives (storage from azure, backblaze, on
| prem, not doing it), and the risks (increase s3 costs and
| people might realise they are being ripped off on ec2)
| eru wrote:
| Yes. The cost of providing the service only factors
| indirectly into the price charged: via competition.
| 404mm wrote:
| Random shallow rant, I really wish they came up with a different
| abbreviation than zstd. Not only it sounds strange but it's also
| too long for abbreviation, imo. It could have been "zs" or "zd".
| hdante wrote:
| https://github.com/facebook/zstd/issues/1333
| abotsis wrote:
| But they still only support gzip or bzip2 for S3 Select'ing json
| files :(
| wging wrote:
| As a semi-educated guess, I wonder if he might just be referring
| to the compression of log files via standard Amazon-internal
| mechanisms, which use S3 as the backing storage, for many
| services.
| charcircuit wrote:
| I think he means a 30% reduction in used storage and not that 30%
| of objects were no longer being stored.
|
| It would be a reduction in data, but not compressed data.
| alrlroipsp wrote:
| But what about the % increase in power consumption?
| akmittal wrote:
| Zstd is very efficient in decompression
| https://engineering.fb.com/2016/08/31/core-data/smaller-and-...
| markdog12 wrote:
| Good article on zstd:
| https://gregoryszorc.com/blog/2017/03/07/better-compression-...
| scarface74 wrote:
| Standard Disclaimer: I work at AWS. Not only are all opinion my
| own. I'm so far away removed from the service teams, that I might
| as well be a just like any other customer - I work in the
| consulting department.
|
| That being said, looking at things from the outside, I tend to
| think that objects are not stored in a compressed form. It would
| make many operations much slower.
|
| 1. There are APIs to fetch byte ranges from objects. Would that
| work without decompressing the entire object?
|
| https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing...
|
| 2. You can query over multiple objects using SQL either via
| Athena or S3 Select.
|
| https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-gla...
|
| Edit:
|
| Someone has already explained how byte range requests would work.
| bhedgeoser wrote:
| > Someone has already explained how byte range requests would
| work.
|
| Where?
| xtreak29 wrote:
| Probably a reference to
| https://news.ycombinator.com/item?id=32531141
| latchkey wrote:
| > There are APIs to fetch byte ranges from objects. Would that
| work without decompressing the entire object?
|
| I wonder if you could store the objects in chunks and then
| individually compressed. Create an index of the byte ranges for
| those chunked objects so they can be looked up easily.
|
| For example:
|
| range 0-4000 -> objectA, objectB, objectC
|
| Then just return those 3 objects, decompressed.
| jl6 wrote:
| Is there some nuance in the phrase "compressed storage" here?
| Like, is that a subset of overall storage, with some objects not
| being compressed? I imagine that a great proportion of S3 objects
| are not amenable to zstd compression, either because they are
| natively in a compressed format (e.g. JPEG) or because the
| customer has already compressed them to realise the saving
| themselves.
| hyuijk wrote:
| If the customers stored .gz/.zip/... files you could
| transparently transcompress them to .zstd and back (with an
| added test of course)
| sgtnoodle wrote:
| It seems like that would cause problems. Right off the bat,
| any integrity hashes like md5 or sha256 for the original
| compressed files would likely be corrupted. Also, the
| compressed archive could have been structurally baked in a
| specific way that's meaningful to the customer. Zip archives
| in particular can have arbitrary data pretended to them.
|
| I suppose you could speculatively decompress and then re-
| compress and see if you get the original compressed file
| back, and maybe most people happen to use the same
| compression implementations with default settings.
| Cyberdog wrote:
| > Also, the compressed archive could have been structurally
| baked in a specific way that's meaningful to the customer.
|
| EPUB is an example of this. They're mostly bog-standard ZIP
| archives, but in order for the file to be valid, the
| "first" file in the archive, linearly speaking, must be
| named "mimetype" and stored with no compression (each file
| in a ZIP archive can have a different compression level).
| If you just unzip an EPUB and then just dumbly zip it back
| up again, your end file will not be a valid EPUB.
| spockz wrote:
| You would only apply the compression on the internally
| stored file and then decompress when retrieving it for the
| customer again. That way all the hashes and original
| structure of the user are retained.
| klauspost wrote:
| You would need to be able to reconstruct the input file
| bit-by-bit. S3 Clients expect to get back what they sent,
| exactly.
|
| This puts a serious limitation on your compression. You
| would only be able to re-do the entropy coding part of
| DEFLATE, which is actually pretty good.
|
| You would still need to store the original Huffman tables
| for each block, so you can reconstruct the entropy coding
| exactly.
|
| I doubt this would even gain you a single percentage.
| lifthrasiir wrote:
| There are tools like preflate [1] or precomp [2] that
| guarantees a bitwise identical reconstruction, of course
| modulo bugs.
|
| [1] https://github.com/deus-libri/preflate
|
| [2] https://github.com/schnaader/precomp-cpp/ (which
| internally makes use of preflate)
| mkup wrote:
| Besides ZIP metadata, there may be flushes in specific
| points in the deflate stream (which applies to .gz files as
| well). These flushes reset the compression dictionary and
| make further compressed data independent from previous data
| (at the expense of losing some compression efficiency). So:
| AWS S3 customer may have injected these flushes to their
| .gz files (for whatever reason, e.g. steganography), and
| after gzip-to-zstd-to-gzip transcompression this
| steganographic data will be be lost (and of course sha256
| and other similar hashes will be different, as you already
| said).
| sgtnoodle wrote:
| Exactly. I've built several logging systems over the
| years that intentionally flush compression streams (or
| concatenate gz streams) for robustness reasons.
| mike_hock wrote:
| That wastes a lot of CPU compared to just running it
| through zstd.
| scrame wrote:
| It's written through a virtual file system which stores in
| blocks and not files, the implementation of that could happily
| run zstd on the way to the multiple physical disks.
|
| This isn't about them shrinking your individual files, its
| compression on their internal transport.
| badpun wrote:
| Certainly, if used as a data store for a data-lake-like
| solution, it's much more convenient to store data in
| uncompressed form, knowing that the underlying storage solution
| will take care of that for you. This way, you can just dump
| Json files in without it feeling like a collosal waste (or
| having to do extra work of converting them to a compressed
| format, such as Avro).
| paranoidrobot wrote:
| Storage level compression usually benefits whomever owns the
| storage.
|
| I doubt Amazon intends to bill on compressed size.
| infogulch wrote:
| I believe zstd can detect incompressible data quickly and falls
| back to a low-overhead passthrough mode in that case. So you
| can just run everything through zstd and it handles those cases
| pretty optimally already.
| le-mark wrote:
| Hmm this made me chuckle, is that not equivalent to a test
| for randomness, which is very very hard indeed?
| hexxagone wrote:
| It is not. Just look at the frequencies in the data
| (histogram) and compute the order 0 entropy. Then you know
| if your data is compressible.
| Cyberdog wrote:
| I think a useful test for this case could be pretty simple.
| Compress maybe 10% of the file (up to some reasonable
| maximum, like maybe a few megabytes) and check the
| compression ratio. If it's not something better than maybe
| 1.1x, give up, throw out the compressed data, and just pass
| through the file. Otherwise keep going.
| dudeinjapan wrote:
| Everything counts in large amounts.
| ksec wrote:
| At the scale of AWS S3, exabyte scale, I assume that could be
| close to a billion dollar in cost savings?
| jankeymeulen wrote:
| Amazon will pay perhaps $50 for a 10TB disk, 10x it to cover
| for redundancy and the servers and datacenters to put the disks
| in, and you're looking at "only" 16 mil savings for an exabyte.
| social_quotient wrote:
| Anyone by chance know how many exabytes s3 is?
|
| Did some searches but came up with some guesses like
| https://maisonbisson.com/post/how-big-is-s3/ but they are
| dated and still just guesses.
| kukx wrote:
| Does it mean that gzip is a legacy compression than should be
| replaced with zstd where possible?
| wolf550e wrote:
| Yes, if you control both sides, always use zstd instead of
| deflate/gzip/zip.
| wongarsu wrote:
| Yes. Gzip used to be a decent middle ground when you needed
| neither super fast compression/decompression nor super high
| compression ratio, but zstd beats it by a good margin over the
| whole range from gzip -1 to gzip -9, while offering you a wider
| range of compression ratios to choose from. And at settings
| that provide compression ratios comparable to gzip it's
| blazingly fast.
|
| gzip will stay with us because of ubiquity, just as we mostly
| use image formats from the dot-com era. But if you control both
| sides, zstd is a big upgrade.
| Beltalowda wrote:
| > the whole range from gzip -1 to gzip -9
|
| gzip compression levels are almost useless and typically
| result in very little actual compression ratio differences,
| but typically with massive CPU usage differences.
|
| zstd doesn't do _everything_ better than gzip though; I
| believe gzip still offers somewhat faster compression for
| equal compression ratios last time I looked (but
| decompression is much faster). I mostly replaced gzip with
| zstd myself, but there are still scenarios where gzip might
| be preferable.
| linuxhansl wrote:
| In all the testing I have been doing across a variety of
| use cases, from BigData, to databases, to games, I have not
| come across a scenario where Zstd is doing worse than Zstd.
|
| Compression time CPU is lower for equivalent ratios.
| Maximum compression is better. Decompression CPU is _much_
| lower (i.e. faster), and that is independent of the level
| used at compression time.
| Beltalowda wrote:
| I did a bunch of testing about five years ago and in some
| scenarios gzip was a bit faster, but I don't recall which
| exactly and it was five years ago so things may have
| changed. I thought I had saved the results somewhere, but
| I can't find them right now.
|
| Overall, zstd still came out as the clear winner (also
| compared some other compression tools), so I went with
| that.
| funstuff007 wrote:
| Not be that guy, but we never send an object to S3 without
| compressing it on our end first.
| flatiron wrote:
| We never send anything unencrypted to S3. I have no clue why
| someone would not want to encrypt their data before putting it
| in the cloud...
| cmeacham98 wrote:
| For people also using other AWS services (in particular
| compute ones like EC2) this doesn't do anything meaningful.
| flatiron wrote:
| We use other aws services. We decrypt the files on those
| services.
| thecleaner wrote:
| Because there is already server side encryption. Which I
| guess means data on disk is always encrypted not sure about
| in-memory.
| flatiron wrote:
| We also use server side encryption. But honestly encrypting
| the files before putting on s3 has given us 0 grief and
| gives us a feeling even if we somehow are "hacked" the
| files are useless to the hackers.
| yunohn wrote:
| Where is your decryption key stored?
| moltar wrote:
| Post-it as a Service
| barkingcat wrote:
| Would you encrypt files that are supposed to be publicly
| readable and accessible?
|
| I get hashing it and providing hashes, but encrypting public
| files seems excessive.
| 101011 wrote:
| There's a very common data/ETL pattern wherein raw
| (unencrypted) data is stored into S3 at the very beginning of
| any pipeline. Adding encryption adds a layer for failure,
| which can grind your pipeline to a halt.
|
| I've seen a pattern of: drop raw data into an S3 bucket that
| has a very restrictive policy with a long retention policy.
| Then, process that data asynchronously (encrypt, transform,
| filter, etc) and drop it into a different bucket/area that is
| accessed by other consumers.
|
| Then, if any part of your ETL fails (encryption included),
| you can fix your bug and reprocess from your raw data without
| writers seeing any impact.
| renewiltord wrote:
| The only way to get the savings yourself is to zstd yourself in
| the first place, which is what we do.
| IMSAI8080 wrote:
| There's a balance between compression ratio and CPU utilisation.
| Zstd is several times faster at compressing than gzip to produce
| a file of similar size (it's good, give it a try if you haven't
| already). I guess he means by using zstd they were able to crank
| up the compression ratio and maintain the same CPU usage maybe?
|
| I don't think AWS routinely compress customer data that I've
| noticed. I guess he must mean for their internal products that
| use S3 perhaps?
| notimetorelax wrote:
| I doubt there's a single byte stored to disk that is not
| compressed and encrypted at AWS. It's transparent to the
| customer.
| uluyol wrote:
| I'm sure it's encrypted, but I doubt that they compress
| everything. Images and video tend not too compress well since
| they've typically already been aggressively compressed with
| specialized algorithms. It would just be throwing CPU cycles
| away.
| smueller1234 wrote:
| It's not quite that simple. If you have customer data that's
| already encrypted, then compression won't do much because it
| looks random. But of course by the time you get to your
| infrastructure layers, that'll be the case (or you really
| messed up your security story!). Which means you'd have to
| compress right at the edge. They might be doing that (which
| would basically mean it's the customer compressing it before
| they encrypt it with their keys because AWS has no business
| seeing the clear text), but then you get to compress each
| item separately, which might not be very effective for small
| values.
|
| tl;dr:There's a real efficiency/security/insider risk trade-
| off here.
|
| Edit: I should disclose that I work for a competitor. Don't
| intend any astroturfing.
| notimetorelax wrote:
| I agree with you, there could be scenarios where customers
| supply their own keys and compress the data on their own.
| My original statement is still true though, the data at
| rest ends up being compressed and encrypted.
|
| That said, of course, customers can upload encrypted blobs
| of uncompressed data. But I'd call it an exception that
| proves the rule. Here service simplicity should win and
| those blobs may end up recompressed.
| alexchamberlain wrote:
| +1 if you are storing objects uncompressed, I'd be amazed if
| AWS doesn't compress them and charge you for the full space
| anyway
| sitkack wrote:
| If this true, there is possibly a side channel one could
| run against object storage to determine if someone else in
| the content-addressable-store has the same files.
|
| Like when it was easy to file share on dropbox by having
| the correct hashes. A GUID could summon a 1GB file.
| alexchamberlain wrote:
| A couple of comments across the thread have made similar
| points, but if I were implementing this, the "client
| metadata" like the incoming sha256 etc would be
| implemented a layer higher than the actual byte storage,
| so the byte storage could be compressed without any
| impact on that sort of thing.
| staticassertion wrote:
| That assumes cross-tenant compression.
| eurg wrote:
| Compression and content-addressing are two separate
| things.
|
| Content addressing across accounts on private, AWS
| encrypted S3 buckets would run counter to their claims.
| IMSAI8080 wrote:
| I don't think it's so clear cut. They have to pay to compress
| it. If the data the customer stores is short lived it may not
| be worth it to them. They don't know if the customer already
| compressed it so they might be wasting their CPU. They also
| have to pay to decompress it on every access. They allow you
| to slice an arbitrary byte range out of an object which is
| technically harder to implement on a compressed file. They
| charge by the GB and are not exactly super cheap so if the
| customer wants to store a big fat file of easily compressible
| zeros then whatever, they got their money.
|
| It might make more sense on their "deep archive" product
| maybe where the customer has to commit to a minimum storage
| retention and also pay a retrieval charge which scales with
| the amount of data recovered (hence paying for the CPU to
| decompress).
| paulsutter wrote:
| Amazon has millions of idle cpus available 24 hours a day
| (they can use all the idle time for all customer instances
| for whatever they want)
| eru wrote:
| That doesn't make it completely free. They still have
| opportunity costs.
| oogali wrote:
| I think the point of the different storage tiers of AWS S3
| is to get customers to classify their own data, then AWS
| can pick the right mix of hardware, software, and compute
| that satisfies AWS's requirements for availability and
| COGS.
|
| If the difference between standard S3 and S3 Glacier was
| just slower disk, then rate limiting the customer would
| suffice.
|
| But if there's a significant amount of compute thrown at
| data de-duplication, compression, and indexing, then it
| starts to clarify why there's a pricing penalty for using
| Glacier with the same access patterns as one would use on
| standard storage.
| Spooky23 wrote:
| I don't work at AWS, but storage at scale is a funny beast,
| usually you're constrained by IOPS, and if anything you
| have a surplus of CPU.
|
| If you can stuff more bits in an IO operation, you're
| winning.
| natmaka wrote:
| Moreover zstd quite unusual '--adapt' parameter enables
| it to "dynamically adapt compression level to perceived
| I/O conditions". Works for me (albeit the manpage states
| that "it can remain stuck at low speed when combined with
| multiple worker threads").
| thecleaner wrote:
| Is there a paper on how it "perceives" the I/O
| conditions?
| flaviut wrote:
| I'd guess by using backpressure.
|
| Modify the compression level to try and keep the output
| buffer at 60% full.
| [deleted]
| metadat wrote:
| Too bad the flag doesn't come with detection for this
| environmental condition and then coordinate accordingly
| across processes.
| blibble wrote:
| > They allow you to slice an arbitrary byte range out of an
| object which is technically harder to implement on a
| compressed file.
|
| this is pretty easy, you flush the compression buffer every
| megabyte or so and maintain an index
|
| maybe 50 lines of code
| jeffffff wrote:
| Sure, but now you've added an extra layer of indirection
| which can have a significant impact on performance
| klauspost wrote:
| It doesn't really have to impact performance. The index
| is generated easily as a side-effect of compression. And
| the index is only needed if you need to seek.
|
| I implemented this as part of the MinIO server. See
| "Seeking Compressed Files" here:
| https://blog.min.io/transparent-data-compression/
|
| We choose a compressor without literal compression for a
| faster baseline, but the concept remains the same.
| jeffffff wrote:
| But if you do need to seek, which is really common in
| data warehouse workloads for example, unless you keep the
| index in ram you have to do an extra IO on every seek to
| read the index
| blibble wrote:
| there's always going to be some metadata for the file
| that needs to be looked up before you can start seeking
| (ACLs, sector/extent/cluster location, etc)
|
| the index goes in there, no extra seek needed
| notimetorelax wrote:
| This was true a few years back, nowadays it's cheaper and
| faster to compress the data at rest as the bottleneck is
| frequently IO and storage space. Both, in terms of capacity
| and cost.
| LinAGKar wrote:
| It would still produce some CPU overhead, and thus some
| energy usage.
| IntelMiner wrote:
| Presumably it's the tradeoff of CPU overhead versus disk
| and bandwidth (larger files take longer to copy into
| memory, which is also energy usage. And more bandwidth to
| shunt around Amazon's own network)
| SuperQue wrote:
| There's also a latency component.
|
| Since CPUs are fast enough to deflate in real-time now,
| your bottleneck for a read is your storage/network.
|
| Reducing the bytes read from storage improves the IO
| latency.
| [deleted]
| danieldk wrote:
| _This was true a few years back_
|
| Only temporarily with SSDs. With spinning rust, it also
| often paid off to compress data. We'd store large
| treebanks compressed, because decompression was much
| faster than disk reads.
| [deleted]
| erk__ wrote:
| They could be using hardware compression which can be
| orders of magnitude faster than doing it on the CPU.
|
| Hardware compression is sadly not widely available, I think
| the only consumer product I know with it is the PlayStation
| 5.
|
| The mainframes from IBM have had hardware zlib since Z14
| iirc and in my small tests it is very fast compared to the
| CPU implementation
| estebarb wrote:
| Hardware compression IS available in Graviton 2: "1Tbit/s
| of compression accelerators * 2xlarge and larger
| instances will have a compression device * DPDK and Linux
| kernel drivers will be available ahead of GA * Data
| compression at up to 15GB/s and decompression at up to
| 11GB/"
| sexy_panda wrote:
| I wonder if this could be done on a FPGA..
| vasco wrote:
| FPGA for development, AWS can afford to order custom ICs.
| pclmulqdq wrote:
| How much would someone pay for this? I have a half-
| written zstd core, but I doubt the market for $100-150
| FPGA-based compression accelerators is all that large.
| tgsovlerkhgsel wrote:
| Probably not worth it as a FPGA solution or even in
| general as an add-on card (the overhead of dealing with
| such extra hardware means that the threshold for "worth
| it" is very high).
|
| I would expect this to become part of newer generations
| of CPUs once it becomes popular.
| wmf wrote:
| Intel sells a gzip PCIe card called QAT. Not many seem to
| be sold.
| pclmulqdq wrote:
| QAT is actually likely to end up inside new server CPUs
| from Intel - at least according to the advertising
| material. Also, it is in their new SmartNICs. At least
| somebody is using it.
| zxcvbn4038 wrote:
| It has been done before, if you offload gzip you turn
| your PCI bus into a choke point. Most of the time you do
| better keeping it on the main processor.
| NelsonMinar wrote:
| I'd never heard of this! The full name is QuickAssist and
| it does encryption too. They advertise 100Gb/s symmetric
| crypto, 70Gb/s compression (or roughly 100x faster than
| ZStandard on a single CPU). Seems to retail for about
| $650 for a card.
| Frost1x wrote:
| With enough gates you can do anything on an FPGA
| sigmoid10 wrote:
| I think a lot if datacenter SSDs already come with in-
| drive hardware compression these days, since it not only
| increases speed but also longevity. So it would actually
| save money anyways.
| snoopy_telex wrote:
| They do not. It would be difficult to plan correctly if
| your free disk space is... variable.
|
| Example:
|
| You have an existing 40 gigabyte file
|
| It happened to compress well
|
| You delete it and your free disk space goes up by 4
| gigabytes.
|
| You then write a new 40 gigabyte file that doesn't
| compress well
|
| Replacing an existing file of the same size just ate an
| extra 36 gigabytes.
|
| How would you plan around that? SSDs should store the
| bytes given and don't play fancy games.
| anamexis wrote:
| Yes they do.
|
| https://www.intel.com/content/www/us/en/support/articles/
| 000...
| wmf wrote:
| Note that these are pre-2017 consumer SSDs. I think SSD
| compression fell out of favor due to the rise of FDE.
| ChrisLomont wrote:
| FDE can be done on the SSD after compression, and often
| is.
|
| Adding life to SSDs is a terribly useful feature
| ChrisLomont wrote:
| SSDs could still do it for speed and to write less pages,
| making the drive last longer, and simply report the
| uncompressed space as used. They already do all sorts of
| tricks on pages such as moving them logically, having
| more internally than they report to use as pages wear
| out, and so on.
|
| Given that, it's good sense to compress if at all
| possible simply to make the drive live longer.
|
| And guess what - I just googled, tons of hits, and this
| has been done for a long time :)
|
| So it makes sense, is done, and is important for modern
| SSD behavior.
| tgsovlerkhgsel wrote:
| None of this works if the data is encrypted.
|
| I'm surprised (and shocked) that letting unencrypted data
| hit the disk is still common enough to make such
| optimizations worth it.
|
| Even if you just stick the key in the server's TPM
| without any sealing, an encrypted disk makes it much
| easier to deal with e.g. drive returns (for warranty or
| fault analysis) or disposal.
| ChrisLomont wrote:
| Plenty of SSDs implement strong encryption, and there's
| an entire standard API for it. Why waste cpu on tasks
| your hardware already does?
|
| Bitlocker supports on drive hardware encryption, and I'd
| be surprised if other major file systems didn't.
|
| If I recall, it's a FIPS requirement for data at rest
| now.
| goodpoint wrote:
| > I'm surprised (and shocked) that letting unencrypted
| data hit the disk is still common enough
|
| There's very, very little benefit in encrypting data at a
| filesystem level in a datacenter if you think about it.
| staticassertion wrote:
| Unfortunately that's going to really depend. For example,
| if your threat model is "hard drive gets stolen" there's
| no point. If your threat model is "attacker can access my
| database" encrypting the data at the DB level does make
| sense. But it obviously breaks compression.
|
| And unfortunately compression and encryption are
| seemingly at odds fundamentally :c
| sp332 wrote:
| It's not great, but it's also not unheard of. Tape
| capacities are often quoted at double the actual storage
| space, with fine print that says "assuming 50%
| compression". Also, if compression makes IOPs faster or
| reduces wear on SSDs, people might not complain so much.
| collegeburner wrote:
| how is this any different than e.g. fs level compression
| with zstd on btrfs? seems enough people find that useful.
| natmaka wrote:
| Isn't it solved by adding an abstraction layer between
| the storage and the filesystem, the way "vdo" (
| https://github.com/dm-vdo/vdo ) does it?
| yunohn wrote:
| What do you mean? Even without compression, you have half
| the problem. Obviously, at scale, it's all just
| statistics and planning on trends instead of individual
| files.
| mekster wrote:
| Why do they have to either compress it all or not. They
| must be smart like, have the files split in pieces (just
| like some network file systems/backups do) and if those
| blocks are untouched for a while, compress what's
| compressible and leave them as is during active uses.
| Havoc wrote:
| >. They charge by the GB and are not exactly super cheap so
| if the customer wants to store a big fat file of easily
| compressible zeros then whatever, they got their money.
|
| Maybe they charge at uncompressed rate but store it at
| compressed? Then they got even more money!
| treffer wrote:
| I think it is a clear cut, mostly because I do not think
| that compression compromises any of those features, all
| while making the user experience better.
|
| For any storage system like this you usually have a few
| bottlenecks. IO and Network are the obvious ones, followed
| by tiering (cache, fast io, slow io, ...) and at the very
| end CPU.
|
| Now let's say network is your bottleneck. If you can send
| the data to the client in a compressed for then you get the
| compression ratio as additional bandwidth. And the user
| would get the data quicker! So compression to the network
| is a clear win.
|
| But the common bottleneck is often IO, a high end SSDs with
| 1M IOPS at 4KB would _theoretically_ serve 4GB/s, a 40GBit
| link. That's without any redundancy over other overhead.
|
| Again compression to the storage layer would decrease the
| total amount of IOs, thus making sure a customer gets data
| quicker.
|
| Ok, let's say both are not the issue. The fastest
| compression algorithms compete with memcopy. So if you need
| just one copy of your data you might have been faster by
| compressing it.
|
| Especially fast compression algorithms (zstd, lz4, snappy,
| lzo, ...) are worth the CPU cost with virtually no
| downsides. The problem is finding the right sweet spot that
| reduces the current bottleneck without creating a CPU
| bottleneck, but zstd offers the greatest flexibility there,
| too.
|
| Oh for range requests.... Those large objects are likely
| split anyway, for easier error recovery (imagine 100MB into
| a 1GB transfer you notice that the file data was corrupted
| - not good). Once you work on blocks it's easy to do
| somewhat efficient range requests again.
| dylan604 wrote:
| How clear cut is it when I'm storing a bunch of
| compressed video files? It's totally a waste at that
| point to even attempt to compress these files.
| lazide wrote:
| What I've personally implemented is trial compression
| with heuristics (you eagerly compress chunks, and if
| enough chunks don't compress, stop trying). It does
| require low level input/output control and per-
| chunk/block compression.
|
| That said, a surprising number of video sources use
| sparse file type setups, and I've gotten pretty good
| compression (up to 60%) using LZ4 with NVR files from
| some brands.
| miohtama wrote:
| Video files have already entropy coding applied to them
| and thus any compression gains with reapplying a generic
| entropy coding like zstd are unlikely:
|
| https://en.wikipedia.org/wiki/Entropy_coding
| sicp-enjoyer wrote:
| That's exactly what the poster is saying.
| goodpoint wrote:
| No, most compressed storage systems do not waste any CPU
| on data that is not compressible.
| eru wrote:
| > They charge by the GB and are not exactly super cheap so
| if the customer wants to store a big fat file of easily
| compressible zeros then whatever, they got their money.
|
| That's not a good argument: they could lower their costs
| with compression, still charge the same, and make more
| profit.
| HiJon89 wrote:
| How would that work for something like S3 range requests?
| Rather than reading an entire object sequentially (which
| would work fine with transparent compression) you can also
| ask to read an arbitrary byte range (give me bytes
| 1,000,000,000-1,000,001,000 from the original file). I guess
| you could maybe store the compressed file in chunks with
| metadata about the original byte range inside each chunk.
| klauspost wrote:
| For MinIO (an S3 compatible server), we add an index for
| each part, which contains uncompressed -> compressed offset
| pairs.
|
| Since we already used a Snappy-derived method, each 1MB
| block is stored without backreferences. With this we only
| have to decode at most 1MB-1 extra bytes to respond with a
| specific range offset.
| rcxdude wrote:
| Generally with filesystem-level compression you don't
| compress an entire multi-GB file: you compress segments of
| maybe a few 100k. This gives you a very slightly worse
| compression ratio but allows random seeks to still be
| efficient.
| [deleted]
| [deleted]
| zxcvbn4038 wrote:
| They do compress for most log delivery types like load
| balancers, cdn, cloudtrail, etc. and it makes a huge difference
| over raw logs - compression ratios are in the 90s. One AWS
| specific trick is individual log files below some size
| threshold are passed through raw, so you end up with a mix of
| compressed and uncompressed files in S3, and the only reliable
| way to distinguish between them is to receive them and look for
| a zlib header - you can't depend on the file name or any other
| metadata to tell you ahead of time if an individual file is
| compressed or not. (I think Cloudtrail does set metadata
| correctly for uncompressed files, but other types do not, best
| to spend the time developing a abstraction that deals with
| both)
| usefulcat wrote:
| It's also faster to decompress. So it would likely reduce net
| CPU use for read-mostly resources.
| havblue wrote:
| Quick question, for personal file storage, such as family photos,
| would zstd be preferable now to, say, tar.gz?
| antisthenes wrote:
| For archiving RAW files, yes.
|
| For RAW --> lossy format, storing them in JPEG-XL would be
| preferable, and I doubt most compression algos would do better
| than JPEG-XL.
| kevincox wrote:
| Photos generally won't get much compression anyways since the
| image data is already compressed. However in general yes, zstd
| is almost always preferable to gzip these days.
| londons_explore wrote:
| I want to know why a company as big as Amazon was still using
| gzip in 2022....
|
| Sure - it's a handy tool. But at scale surely someone had enough
| time to research better options? And at Amazon scale, it probably
| even pays to hire a team to write a custom compression algorithm
| perfectly tuned to your compute and storage?
| Theodores wrote:
| Gzip was written extremely well. I have been trying to get
| static compression with Brotli working for compressing web
| assets, along the way of getting that worked out, you realise
| that few people can be bothered with it, hence you are having
| to compile your own Nginx modules.
|
| I found out the way to do it was to have Nginx run a proxy
| server that caches the output of another Nginx 'server' that
| does the Brotli dialled up to 11. So you are making your own
| CDN.
|
| Nothing is difficult when you know it, but Gzip has scratched
| the compression itch so well that people just do not have a
| problem with it and therefore do not seek to change it.
| silvestrov wrote:
| There is also the cost of a bug. Imagine that AWS corrupted
| customers data due to a compression bug in your home-grown
| compression code.
|
| It is almost like encryption in this regard.
| bick_nyers wrote:
| I'm surprised they aren't using an FPGA solution
| hbbio wrote:
| Knowing such implementation exists, is open source, and since
| they don't state what they're using... you may just be right
|
| https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jiany.
| ..
| pclmulqdq wrote:
| FPGAs are really expensive in volume compared to other
| options, so it's not surprising to me.
| londons_explore wrote:
| But on an FPGA you can probably implement super fast and
| effective compression.
|
| If it is far enough above the curve of speed Vs ratio, then
| it will pay for itself in saved storage.
|
| Remember there is no need for said FPGA to be in every
| machine - in a data center with ten's of gigabits of
| bandwidth to every node, you can send data to another
| machine for compression and receive back the compressed
| data to store.
| Someone wrote:
| If I were Amazon, researching whether zstd can be used reliably
| could take quite some time. It would require a thorough code
| review, inspection of the assembly produced by compilers, and a
| test set of a few terabytes.
|
| More importantly, it would require researching whether
| something even better might be just around the corner. I don't
| think Amazon wants to switch compression every few months.
| unboxingelf wrote:
| Huh. If an IC had a compelling argument for cost savings like
| this, I'd recommend they pursue the idea as a 10% style
| project. First verify the basics: licensing, health of the
| community, etc. Then build a standalone pipeline using the
| new protocol. Process data, collect benchmarks and metrics.
| Present your findings to the team and manager. If the manager
| thinks it's still viable, they go up the chain and involve
| principals, architects and so on.
|
| In summary, lower the bar for trying new things. This is how
| we innovate.
| antisthenes wrote:
| That's literally what the person above you said. On
| Amazon's scale the process will take time.
| [deleted]
| swaranga wrote:
| Amazon may be big but even then resources are limited.
| Individual teams owning the roadmap for services are still
| small. And there are always competing priorities. Do you
| prioritize these security and availability improvements or do
| you pickup some operational improvements to make our engineers
| oncall easier? Oh there is also this long line of customer
| features you need to deliver. And re:Invent is also
| approaching.
|
| When you look at it like this, not very surprising that
| initiatives like cost savings optimizations may take a back
| seat for periods of time.
| alas44 wrote:
| He did not say in the tweet zstd was implemented at AWS in
| 2022, could have been implemented years ago
| bastawhiz wrote:
| How much does it cost to switch? How much effort are you
| putting into making sure there are no pathological cases while
| encoding and decoding that have an outsized effect on resource
| use? How much risk are you willing to take that you don't lose
| any customer data? If you recompress stored data, how much are
| you willing to pay in compute? Recompressing at AWS scale is
| enough IO to burn through _many thousands_ of disks in
| aggregate--how much does all that wear cost? How much tooling
| do you need to rewrite?
|
| Let's say you save a few hundred million dollars a year after
| switching. The cost to switch is almost certainly more than a
| few hundred million dollars. When you make that kind of
| investment--especially when you're moving _away_ from a very
| boring technology--you want to be damn sure you know exactly
| what you 're getting yourself into.
| zxcvbn4038 wrote:
| The beauty of gzip is that it is effective, free, resource
| efficient, and it works with everything.
| tgsovlerkhgsel wrote:
| Because finishing the next product you can sell can produce
| more revenue than the savings from switching to zstd.
| donavanm wrote:
| The tweet doesnt mention dates or the coverage.
|
| Second I believe youre underestimating the scope of the
| problem. Amazon has thousands of teams, even more services,
| each with their own priorities, and innumerable different
| access patterns, data types, etc. For all practical purposes
| there is no single "they."
|
| To get an idea of the scope how long would it take you to
| remove gz from oh ... 50,000 projects? Each with many
| deployments and up to 15 years of active data.
| gooeywave wrote:
| When I was there last over a year ago, there was a big push to
| switch logs stored in the internal log service from gzip to
| zstd. But gzip is extremely well integrated into essentially
| everything, that it's a little annoying to switch. For example,
| the base OS images didn't even have zstd installed. It's still
| worth it for the benefits (at least once all the internal tools
| started supporting it), but it's not without _some_ downsides.
| Which is maybe why it took so long to happen in earnest.
| Twirrim wrote:
| > I want to know why a company as big as Amazon was still using
| gzip in 2022....
|
| The short answer to "Why still using gzip in 2022" is almost
| always answered by "Because it made no business or financial
| sense to spend head count on it".
|
| Amazon is usually fairly smart about the ways it spends head
| count, particularly on things that could notably reduce costs.
| Managers/directors/VPs/SVPs obsess over what the value is from
| various work, and routinely adjust priorities and reallocate
| head count. They track not just what is happening within
| individual services, but also the overall business strategies,
| what's on the horizon etc.
|
| One simple example that most folks outside the industry won't
| be seeing is that every major cloud has been working their
| collective arses off for well over a year on meeting JWCC
| contract needs. There's a lot of work involved in that
| contract, because unsurprisingly it's really hard to build and
| run an entirely air-gapped cloud region, but the payoff on
| being selected as a vendor is phenomenal. Far more than saving
| an additional 30% of storage, even at AWS S3 scale. _It's not
| the only major business and engineering initiative_ that will
| be taking place across AWS.
|
| I've got a long list of "We should do x, y, z" that applies to
| my service in the cloud I work for, a number of which will see
| notable performance improvements. They make zero sense
| allocating head count to, though, because I've got at least a
| dozen other more important things going on from a business
| roadmap perspective to get solved that will make us way more
| money and/or save way more engineering resources down the line,
| be it automation stuff, or new features.
|
| What does happen, though, is that list is kept in mind any time
| new business priorities come about. If there is any way bits on
| the list can be tied in to something the business wants, it'll
| get hooked in to it. The business gets what it wants, the
| service gets what I want, everyone is happy.
|
| (To repeat something others have pointed out, there's also zero
| indication of timeline, it could have happened any time in the
| last several years that zstd has been a thing, they may well
| have switched to it almost as soon as Facebook removed that
| ridiculous "You can't sue us" clause)
| wooptoo wrote:
| A parallel implementation like pigz is still pretty good and
| widely backwards compatible.
| tamsaraas wrote:
| zstd not good for everything. Yes, it's fast in terms of
| compression. Easy & fast you can make compressed archive. But if
| you need to use the zstd on daily basis - i have bad news. ZSTD
| not suitable for general purpose archiver.
|
| Depends on the data - you can have totally different output
| compressed archive if compare to winrar for example.
|
| Example: I doing backups time to time, and archive important
| files into compressed archive containers (rar / 7zstd).
|
| I've noticed, that my dev folder with tons of repositories,
| images, and different work related fines - vary damn too much.
|
| /dev/ size = ~11GB
|
| rar output (normal compression) = ~2.1 GB zstd archive output
| (normal compression) = ~4.7GB
|
| Why? linked files, same files not treated as a 1 file + links to
| these files. Instead these files compressed each 1 by 1 instead
| of copy 1 identical, and compress the file. And many things like
| that.
|
| Suggestion for 7z-zstd -> add ability to save links to files, not
| treat them as separate files, and adding an option like in winrar
| to search for identical files first and re-link all of them and
| remove duplicates, instead of compression each.
| CodesInChaos wrote:
| Your criticisms seem to be about 7z, not zstd. Ztd doesn't have
| an archive concept, it's just compresses a sequence of bytes.
| tamsaraas wrote:
| i'm totally correct in what am i writting. zstd not wide
| range suitable compression. And all current implementation
| that used non for 1 file compression - awful.
| e63f67dd-065b wrote:
| No, your critiques are of how 7zip implements an archival
| format built on top of zstd. The zstd algorithm has no
| concept of files, only bytes.
|
| Archival software then has to build a file format on top of
| the compression algorithm, and there are multiple ways to
| slice the problem. For example, a tar.gz will first tar
| everything into a big archive file, then feed it into gzip
| for compression. zip, on the other hand, feeds each file
| individually into the chosen algorithm (DEFLATE for most
| implementations).
|
| Your critique is that the 7zip _archive format_ is not
| suitable for use with zstd in the case of many small yet
| identical files. zstd is doing its job, just the archival
| format is not playing along.
| lifthrasiir wrote:
| Zstandard doesn't accept multiple files at all, so it's an
| archiving format's job to convert files into byte sequences
| and compress them accordingly. It looks like that 7z wasn't
| able to deduplicate entirely or partially identical files,
| in the other words zstd could never know that it is
| compressing almost identical files over and over.
| vbezhenar wrote:
| No support in CompressionStream, that's too bad.
| andrewmcwatters wrote:
| Not too long ago I started compressing statistics data I was
| collecting from GitHub[1] with ZStandard after noticing that
| GitHub Actions CI caching used it.
|
| It's really remarkable for my use case. I compress gigabyte-sized
| SQLite databases under the GitHub 100 MB file limit since Git LFS
| is a non-starter. My tables are collections of follower and
| stargazer counts to measure user and repository popularity and
| understand where one might be on a distribution compared to a
| subset of other GitHub data points.
|
| [1]: https://github.com/andrewmcwattersandco/github-statistics
| fpoling wrote:
| gzip has rsync-friendly mode allowing for much smaller deltas
| between files during rsync transfer or backups that use rsync-
| compatible algorithms. No other compressor implements that as
| far as I know.
| kevinoid wrote:
| zstd 1.3.8 and later also has an rsync-friendly mode (via the
| --rsyncable command-line option). See
| https://github.com/facebook/zstd/issues/1155
| fpoling wrote:
| Nice to know, thanks for pointing this out.
| tester756 wrote:
| It uses some black magic wizardy
|
| >FSE, short for Finite State Entropy, is an entropy codec based
| on ANS. FSE encoding/decoding involves a state that is carried
| over between symbols, so decoding must be done in the opposite
| direction as encoding. Therefore, all FSE bitstreams are read
| from end to beginning. Note that the order of the bits in the
| stream is not reversed, we just read the elements in the reverse
| order they are written.
|
| https://en.wikipedia.org/wiki/Asymmetric_numeral_systems
| shargs wrote:
| For what it's worth, shameless plug, I've written a whole
| series on how FSE works: https://shargs.github.io/data-
| compression/2022/07/28/entropy...
| Dylan16807 wrote:
| It's not that crazy if you separate it into two parts.
|
| A) Use arithmetic coding to encode symbols as fractions instead
| of using an integer number of bits for each symbol.
| https://en.wikipedia.org/wiki/Arithmetic_coding
| http://www.ws.binghamton.edu/fowler/fowler%20personal%20page...
|
| B) Shuffle the bits around so that you can use faster math and
| lookup tables instead of directly operating on fractions.
| fourfourthree wrote:
___________________________________________________________________
(page generated 2022-08-20 23:01 UTC)