[HN Gopher] Small File Archives in MinIO
___________________________________________________________________
Small File Archives in MinIO
Author : edogrider
Score : 41 points
Date : 2022-02-16 18:54 UTC (4 hours ago)
(HTM) web link (blog.min.io)
(TXT) w3m dump (blog.min.io)
| toomuchtodo wrote:
| I have wanted this forever from S3 (cherry-picking from stored
| object archives) without having to implement reading/caching the
| zip index and doing a range request. This is awesome, solid
| feature.
| laurencerowe wrote:
| While not zip S3 does now support SELECT into a number of
| datatypes.
| https://docs.aws.amazon.com/AmazonS3/latest/userguide/select...
| HyperRational wrote:
| gopalv wrote:
| So this looks a lot like what Hadoop did with .har files[1] on
| HDFS (like storing GPS tiles on HDFS without blowing through the
| 1 M files per-dir limit).
|
| > It is not possible to update individual files inside the ZIP
| file. Therefore this should only be used for data that isn't
| expected to change.
|
| I've actually done file-replaces on .zip files on HDFS, because
| .ZIP files are actually written with a directory in a footer, you
| can go to the end and append new data without having to "modify"
| existing files.
|
| This doesn't conflict with the block level immutability, though
| the entire write has to be a single commit to avoid leaving the
| file in a bad way.
|
| I'd say that the best case use-case for this is the storage of
| log files (like if you had fluentd writing .zip files by
| appending to it rather than a diff object for each 5 minute
| window).
|
| When it comes to stuff which compresses well but full of small
| objects, ZIP is pretty bad because each file in the zip
| independently contains a dictionary (look at the .xlsx file
| inside to know how MSFT solved that, but in way which makes you
| hate it - a strings directory for shared strings across all
| files).
|
| [1] -
| https://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html#H...
| heipei wrote:
| Backblaze, S3 and R2 listen up: This is a great feature and one
| which I would immediately use to store related data in one zip
| file. Not only does this cut down on the number of objects in
| your storage, it also makes deletion atomic since you don't have
| to ensure that all types of objects for the same ID have been
| deleted.
| speedgoose wrote:
| Would it work in gateway mode ?
| y4m4b4 wrote:
| No, gateway mode is EOL'ed and it is not taking any new
| features
| jaytaylor wrote:
| Would this approach also work for Tar archives? Transparent
| support for sub-files from a .tar would be badass.
| rmgraham wrote:
| Tar doesn't use any sort of index like zip does, so to extract
| the specified file the server side would need to parse through
| possibly the entire file just to see if the requested file is
| there, and then start streaming it. Requests for files that
| aren't in the tar archive would be prohibitively expensive.
|
| There are definitely ways to do it without those problems,
| though. They just wouldn't be quite as simple as the approach
| done for supporting zip.
| remram wrote:
| You could pre-index them I suppose. Though even that would
| only work with a subset of compression methods or no
| compression.
| [deleted]
| blacha wrote:
| This is basically exactly what we do we have created a
| cloud optimised tar (cotar)[1] by creating a hash index of
| the files inside the tar.
|
| I work with serving tiled geospatial data [2] (Mapbox
| vector tiles) to our users as slippy maps where we serve
| millions of small (mostly <100KB) files to our users, our
| data only changes weekly so we precompute all the tiles and
| store them in a tar file in s3.
|
| We compute a index for the tar file then use s3 range
| requests to serve the tiles to our users, this means we can
| generally fetch a tile from s3 with 2 (or 1 if the index is
| cached) requests to s3 (generally ~20-50ms).
|
| To get full coverage of the world with map box vector tiles
| it is around 270M tiles and a ~90GB tar file which can be
| computed from open street map data [3]
|
| > Though even that would only work with a subset of
| compression methods or no compression.
|
| We compress the individual files as a work around, there
| are options for indexing a compressed (gzip) tar file but
| the benefits of a compressed tar vs compressed files are
| small for our use case
|
| [1] https://github.com/linz/cotar (or wip rust version
| https://github.com/blacha/cotar-rs) [2]
| https://github.com/linz/basemaps or
| https://basemaps.linz.govt.nz [3]
| https://github.com/onthegomap/planetiler
___________________________________________________________________
(page generated 2022-02-16 23:00 UTC)