[HN Gopher] The Road to OCIv2 Images: What's Wrong with Tar? (2019)
___________________________________________________________________
The Road to OCIv2 Images: What's Wrong with Tar? (2019)
Author : pcr910303
Score : 72 points
Date : 2022-02-01 02:49 UTC (1 days ago)
(HTM) web link (www.cyphar.com)
(TXT) w3m dump (www.cyphar.com)
| gilgad13 wrote:
| I agree with many of the concerns the author raises, but I'm left
| with the question:
|
| Given all this, what does layering give us?
|
| It gives some deduplication, but only a crude form. It gives some
| reproducibility from building off a well-known base and tag, but
| not full reproducibility. It gives some security benefit from
| building off a well-known base, but not as large a benefit as
| standard package managers provide.
|
| I would be excited to see a image distribution system based off
| of something like casync, maybe with an initial rootfs formed
| through image-focused distributions like yocto[1]. The embedded
| device ecosystem has been concerned with reproducibility, image
| signing, and incremental updates for awhile and I think their
| approaches are very applicable to container images.
|
| [1]: https://www.yoctoproject.org/
| nonameiguess wrote:
| Apparently, not a whole lot for image transfer and portability.
| But layering still gives you something at runtime if a single
| organization is using the same base image for all of its own
| containers. And, in practice, I think layer-level deduplication
| does still save on transfer costs. I'm not sure if this author
| just wasn't considering or realizing the state of where
| industry was heading, but with projects that are rebuilt on
| every commit, the change frequency of upper layers is still a
| lot greater than the rate of change on distro base images. They
| may be patched daily and you need to re-download the whole
| thing every day, but if you're building 40 times a day, that's
| still better than downloading 40 times a day. It's just a lot
| worse than we could be doing if we could only download diffs
| instead of the entire layer when a single bit changes.
|
| It would be nice to see what, if anything, ever came of the
| ending tease. Something like git but also for binary files is
| what is called for. Arguably, ClearCase offered this exact
| feature 27 years ago, but being proprietary and expensive
| limited its adoption among modern web tooling.
| nix23 wrote:
| Hmm thinking about it...a ZFS-filesystem on a file could solve
| ~all the problems with the exception of a good performing dedup
| and shrinking the file/device.
|
| But really good article, quite interesting tar-history.
| Datagenerator wrote:
| This is possible already with losetup?
| nonameiguess wrote:
| Machine independent representation? One of the explicit goals
| is a format that works no matter the underlying filesystem it
| is copied onto. ZFS is supported as a storage driver by most
| container runtimes, but it isn't desirable to have it be the
| only supported storage driver.
| justinludwig wrote:
| The original title, "The Road to OCIv2 Images: What's Wrong with
| Tar?", is a lot better IMHO. Otherwise, a great history lesson on
| the tar format, and why it's not a great fit for OCI images.
| Kim_Bruning wrote:
| Also a bit mean! The original author explicitly considered and
| rejected the "x considered harmful" title.
| pid-1 wrote:
| Since I learned that OCI images are pretty much archives with
| metadata, I've been thinking about applying that to other stuff.
|
| Like, it would be cool to have all my python virtualenvs saved in
| bundles so they could be reused by other others, without all the
| container bloat.
| tyingq wrote:
| Sounds like a Java jar/war file...
| nonameiguess wrote:
| And apk, deb, rpm, whl, msi. Virtually all packaging formats
| come down to "archive plus metadata."
| tyingq wrote:
| Yes, though narrowed down to "packaging for a python
| virtual environment" draws closer to jar/war.
| ecnahc515 wrote:
| OCI is starting to be used for more non-container things like
| your describing actually. One example I'm aware of is the Helm
| tool in Kubernetes, which has support for using OCI as a method
| of storing/transporting helm charts (which are effectively just
| tarballs of Go templates/yaml). I
|
| I think your challenge for virtualenvs would be that
| virtualenvs generally are not relocatable, as they hardcode
| absolute paths within the generated scripts (eg:
| venv/bin/activate`).
| [deleted]
| rektide wrote:
| Great post, but:
|
| > _How Do We Get It?_
|
| > _I'm afraid to find that out, you'll need to wait until the
| next instalment. I hope to get it complete in a few weeks (I was
| hoping to have a PoC available with the next instalment but
| that's just a silly goal at this point)._
|
| > _If you want a taste though, the general idea is that we can
| resolve most of the issues I've listed and gain most of the
| properties we want by creating our own format that is built on
| top of the OCI content-addressable store, and is a Merkle tree
| with content-defined chunking of file contents. The basic idea is
| very similar to backup tools like restic or borgbackup. Since OCI
| has smart pointers, we can define a few new media-types, and then
| our new format would be completely transparent to OCI image tools
| (as opposed to opaque tar archives)._
|
| > _But you'll learn all about that next time. Thanks for reading,
| and happy hacking!_
|
| Any follow-up on this 3 year old post? This appears to be the
| final entry on cyphar's blog. Did this ever happen? I've done
| some hunting but can't find any stated plans for OCIv2 from
| Cyphar.
|
| There was a hack-session in march 2020 on this topic:
| https://hackmd.io/@cyphar/ociv2-brainstorm . Really fascinating
| document. It has a section for each problem, & does a great job
| citing & discussing prior work in the area. Was a delight to
| stumble into this. But still seems very formative. It felt like
| cyphar was already onto a plan, well before this date. I don't
| see any evidence of v2 forming in the official repo,
| https://github.com/opencontainers/image-spec/tree/main/specs... .
| Cyphar does still seem active in the project.
|
| Seems there was a come-together call on the mailing list last
| June, calling for trying to extend-their-way-forwards rather than
| do a big revamp/rewrite:
| https://groups.google.com/a/opencontainers.org/g/dev/c/6DGtH...
| dang wrote:
| Discussed at the time:
|
| _Road to OCIv2 Images: What 's Wrong with Tar?_ -
| https://news.ycombinator.com/item?id=18965881 - Jan 2019 (33
| comments)
___________________________________________________________________
(page generated 2022-02-02 23:02 UTC)