[HN Gopher] Speeding up LXC container pull by up to 3x
___________________________________________________________________
Speeding up LXC container pull by up to 3x
Author : ctecte
Score : 75 points
Date : 2022-02-01 17:34 UTC (5 hours ago)
(HTM) web link (databricks.com)
(TXT) w3m dump (databricks.com)
| scintill76 wrote:
| GNU tar seems to auto-detect compression type as well. I don't
| bother passing a flag to say which type to use, when I'm
| extracting.
| ctecte wrote:
| Author here, it's true that GNU tar auto-detects compression
| when specifying a local tarball to extract. However when piping
| a file stream to tar to extract it actually doesn't auto-detect
| and fails if you don't explicitly specify a compression type.
| Should have made that more clear in the post.
| scintill76 wrote:
| Oh, that makes sense. Nice addition.
| jarpineh wrote:
| This is interesting and seems general purpose. Not merely for
| container images.
|
| There's this option for OCI containers which I don't pretend to
| understand: https://github.com/containerd/stargz-snapshotter
|
| It is used by containerd and nerdctl. You do have to build the
| image with it. Images work in OCI compatible registry. By
| fetching most used files first container can be started before
| loading is finished. Or so I gather.
| tyingq wrote:
| That's pretty interesting. One downside might be losing the
| ability to do checksums on what's been downloaded, as a
| safeguard. I wonder if there's a way to preserve that without
| ruining the parallel downloads and un-tarring.
| ctecte wrote:
| I did look into performing stuff like md5 inline while
| streaming the download, however it tended to slow down the
| extraction significantly. Other less rigorous checks like crc32
| are still possible inline though with minimal performance
| reduction.
|
| It should also be possible to split off and write the raw
| download to disk in the background with something like an
| io.TeeReader object [0]. You could then checksum it after the
| fact like you normally would.
|
| [0]: https://pkg.go.dev/io#TeeReader
| fdr wrote:
| A long time ago, I wrote something similar (
| https://github.com/htcat/htcat) to assist with Heroku's efforts
| to speed up moving the tar formatted application releases around.
| It's pretty old, it doesn't integrate tar archival itself, it
| probably can stand improvement. Or given its small size, even a
| rewrite.
|
| My favorite hack in there (that also made it work with pre-signed
| S3 urls) was not using the HEAD method as is customary to
| determine object sizes, but instead doing a regular "GET" that,
| for small files, would execute on its own...but for larger would
| simply be abruptly closed by htcat once it reached the bytes that
| had since been fetched in parallel by a range-based request sent
| immediately afterwards. The goal was to have htcat not offer a
| penalty on small files so it could be used on blended workloads
| without thinking.
|
| It also found a bug in S3's range implementation. We had a
| problem with some object or other, I wrote in about it, and was
| told that upon investigation a bug had been fixed. No more
| problem.
|
| Some kind soul packaged it for Debian, so it's easy to get if you
| need it.
| ctecte wrote:
| Oh nice! Yeah this is basically the same design/intuition that
| this project also started out with (I also just wanted some way
| to combine the benefits of aria2c with piping). Wish I found
| this earlier lol.
|
| I only later decided to look into also parallelizing the tar
| extraction as well. This originally started with looking into
| making a change to the GNU tar source code itself. However upon
| cloning I realized there were a lot of design choices in tar
| that made the assumption of single threaded execution (global
| state, etc). At this point I decided it would just be easier to
| re-implement the tar extraction myself using golang's
| "archive/tar" package.
| stabbles wrote:
| How about squashfs files for every layer and overlay mount those.
| At the end of the day you access probably .1% of the data, so
| with this you'd defer decompression to runtime and only
| decompress what you need
| ctecte wrote:
| Had not considered that. Reading about squashfs I see that it's
| readonly. Do you know if that's fine when mounting the
| directory for an LXC container? Also I wonder how much
| performance impact (if any) there might be on spark jobs due to
| the decompression at runtime.
|
| To be completely honest this all started as a quick hackathon
| project just to speed up downloading before piping to GNU tar.
| Only later did I consider also re-implementing the tar
| extraction in a parallel manner. If considering changing the
| overall packaging method from tar to something else there's a
| lot more ways to consider going about this (including
| squashfs).
|
| I'd say one of the nice things about this is that tar is fairly
| ubiquitous, not just in container images. For example, we also
| have tarball build artifacts in Jenkins jobs that could benefit
| from this tool.
| tarasglek wrote:
| You can cut down the startup time to milliseconds
|
| 1) either mount tar directly
|
| 2) or use squashfs and mount that...both would be mounted
| with some sort of rw overlay
|
| 3) underneath do some lazy httpfs type filesystem, so
| filesystem can be mounted while it's downloading
|
| 4) parallelize the underlying download ala aria2c
|
| 5) provide metadata to aria2c-alike downloader with file
| boundaries, so extract further latency/bandwidth savings
| ctecte wrote:
| 1) Are there any ways to do this in a performant manner?
| Afaik, since the tarball is still sequential with no way to
| jump around, interacting with this filesystem would be
| fairly slow right?
|
| 2) Agreed, still need to look more into this, although it's
| more involved and entails changing more of our pipeline for
| packaging and distributing these images.
|
| 3) I need to look into how this would handle files that
| haven't been downloaded yet. From what I know about httpfs
| filesystems, I'm not sure how much would need to be done to
| let this block until file needed is downloaded vs the
| normal behavior of calling out to get the file being
| requested.
|
| 5) Could you go into more detail here? Not sure I
| understand how file boundaries, etc can help with
| latency/bandwidth.
| hawski wrote:
| Ad 3. With squashfs via httpfs it would probably be fast
| enough. I remember years ago httpfs capable of booting
| livecd iso and it was performant enough. I'm not current
| with this knowledge, but is there a httpfs, that would
| download increasingly cache the file while doing range
| requests as it gets read requests for certain parts of
| the file?
| tarasglek wrote:
| 3) it just blocks and prioritizes that part of download
| 5) if you can chuck the file so chunks that get
| downloaded are whole files..less likely to do multiple
| requests for a single file
|
| 6)I forgot to do the best part of the optimization..do
| feedback-guided-optimization. You can then repack the
| squashfs/tar file to have all the frequently-accessed
| files together
| ctecte wrote:
| Ahh got it, this makes a lot of sense, maybe something to
| try next hackathon :)
| ffk wrote:
| Something to consider, if you are IO constrained, compression
| may speed up reads because you shift some of the cost of IO
| to the CPU.
|
| Ultimately, you'll need to measure this to know for sure, and
| those results will likely only be valid on a given hardware
| configuration.
|
| OverlayFS also has a "copy_up" function, where the file is
| copied at the initial write. Once the copy is done, I'd
| expect write access to be fast. Again, you'll need to measure
| this.
|
| The setup could probably look like:
|
| container read/write -> OverlayFS([mutable fs as overlay] ->
| [squashfs layer as underlay] -> [squashfs layer as underlay])
| nhoughto wrote:
| Came here to say exactly that, squashfs with overlay mount for
| writes beats this easily.
|
| Hard to change the entire container ecosystem away from
| tarballs tho so this is still useful. Is it available anywhere?
| I didn't see a link?
| ctecte wrote:
| Company wanted to see if there was any interest in it before
| putting in effort to open source :/
|
| Hoping this is enough to release it on Github.
| nhoughto wrote:
| Have thought about writing the same thing, would love to
| see it OSS.
|
| Real value for me would be to integrate it with containerd
| or similar to speed up image layer pull times, I imagine
| that isn't shelling out to a separate binary tho.
|
| What language is it implemented in?
| ctecte wrote:
| This was written in golang. Specifically channels and
| io.Pipe objects made it pretty straight forward to "glue"
| different phases of download and extraction together into
| the final solution.
___________________________________________________________________
(page generated 2022-02-01 23:01 UTC)