[HN Gopher] Tiny Container Challenge: Building a 6kB Containeriz...
___________________________________________________________________
Tiny Container Challenge: Building a 6kB Containerized HTTP Server
Author : kiyanwang
Score : 352 points
Date : 2021-04-22 06:06 UTC (16 hours ago)
(HTM) web link (devopsdirective.com)
(TXT) w3m dump (devopsdirective.com)
| dvfjsdhgfv wrote:
| Using asm is a bit extreme and can lead to many problems. The toy
| project used is quite limited anyway; you could use one of the
| tiny C httpds and still get the size under 15k for similar
| functionality.
|
| Would anyone use this in production? I doubt so. You want a
| reliable battle-tested solution. Nginx can be compiled to 500kb
| without OpenSSL; if you need https then it's just a few
| megabytes, I find it acceptable for non-embedded solutions like
| containers.
| hardwaresofton wrote:
| One thing I'm wondering -- why are people obsessed with container
| size? A lot of enterprise is Java shops which deal with huge
| dependencies, huge distributions, and generally pretty heavy
| software -- with internet speeds what they are today, it's not
| hard to throw 100-500MB around internal networks (building the
| code closer to the deployment area is also a solution). Even
| gigabyte size containers are not impossible to deploy these days
| with the right caching (lord help you if you invalidate a bunch
| of layers though).
|
| In a world of xTB thumbdrives, why are we so worried about
| containers being <1MB? Security is one thing, minimalism is
| another, but I do sometimes wonder about this.
|
| Anyway, if you want to build minimal containers and want to do it
| with a bit more backing than this one (pretty well written) blog
| post, probably also check out distroless:
|
| https://github.com/GoogleContainerTools/distroless
|
| I personally find most of the time alpine-based containers are
| more than small enough and good enough with nice ergonomics
| (sometimes you just want to shell in to the container), but maybe
| it makes sense to build containers with and without
| debugging/inspection tools (a `project:vx.x.x` and a
| `project:vx.x.x-debug`) and switch them out when problems arise.
| thrwaeasddsaf wrote:
| I don't know about you but I work on a laptop, and I work on
| multiple projects, and each project might use dozens of
| containers for builds, testing, releases, etcetra. If every
| container were a gigabyte in size, majority of the laptop's
| internal storage would be wasted on them, nevermind the time
| wasted pushing and pulling updates.
| hardwaresofton wrote:
| I do too (my laptop is pretty beefy though), and yes you're
| right this _would_ be a problem, if containers weren 't built
| to combat this with Copy on Write filesystems.
|
| That said, dependencies _do_ take up a lot of space, but
| machines these days also come with a lot of space. People
| also run prune and develop caching strategies (ex. NPM) as a
| result. It 's not like _every_ container will be a GB in
| size, as it 's not reasonable to have every container have
| that much content, but how many people are out there using
| 950MB dependency-filled base containers with 50MB of actual
| relevant application code? Most people learned the run-it-on-
| alpine lesson like 5+ years ago at this point.
| lanstin wrote:
| My new laptop has 3T of disk. I have the old feeling of
| what can I possibly do with it all. I have two copies of
| all my photos and text files etc. and dual boot Arch and
| Ubuntu but still it is so Empty. I don't think even a very
| aggressive go get campaign can fill it.
| paxys wrote:
| Plus, figuring out optimal sizing for Docker images is in
| general complicated and often counterintuitive. A company I
| worked with mandated that every image must be written from
| scratch to reduce size on disk and save on transfer
| bandwidth/time. The end result though was that there were no
| sharable layers, and so every server had to redownload every
| image wherever there was the tiniest update. Moving to a set of
| common 1GB+ base images with all dependencies included made the
| entire system smaller and faster.
| blowski wrote:
| I wonder the same thing. I can see the value in shrinking a 1GB
| container by 90%, but I suspect there are diminishing returns
| by the time you're shrinking a 5MB container below 1MB. In most
| organisations, there are probably bigger opportunities for
| improving security, performance, etc.
| viraptor wrote:
| It's worth remembering the MBs add up in multiple places.
| Sure, it may not be worth for 5MB. But once you're in
| tens/hundreds, they add up in storage (how many old versions
| do you keep), transfer times (how quickly can a fresh
| instance pull all tasks and how quickly can a developer (at
| home rather than in us-east-1) start with a project),
| transfer costs (how many times does a CI node need to
| download/upload the last layer).
|
| Saving 10% in size may actually translate to a non-trivial
| cost saving - both obvious and cost-as-in-paid-waiting-time.
| lanstin wrote:
| Also all those dependencies are security risks. A lot
| quicker and easier to scan a container with just one go
| executable than even the most benign distribution of core
| utils and so on. And if your containers are just one binary
| and you can read the logs then you don't need to go into
| the container any more than you need to go into your
| binary. Then your k8s node or other container runner host
| is just like your old host running a bunch of processes but
| with much better process isolation and file system
| isolation.
| blowski wrote:
| There are some teams where this will be the best use of
| their time, and I'd be well out of my depth suggesting to
| them otherwise.
|
| But the vast majority of teams in my experience have
| spent ages farting around with these kind of micro-
| optmisations, whilst there were far better opportunities
| to improve the infrastructure. And in the process, they
| actually made it more fragile and harder to debug.
|
| Simply put, this is a niche skill with niche value. Most
| people trying to do this probably can't and shouldn't do
| it.
| emptyparadise wrote:
| Why create bloat when it can be avoided? The 6kB server might
| be overkill but the other options in that article seem pretty
| good.
| tluyben2 wrote:
| A lot of companies that sell products (non open source ones
| anyway) actually try to bloat the binaries and deployments to
| make them look more significant. Many gigabyte installers are
| not that weird for some products, where I found (for fun as
| corp wise we cannot really care) the relevant software we are
| paying for to be a few mb and it doesn't need or even touch
| the rest the rest in the package for 99% of cases. But it
| seems/feels like you are paying those $100000s for some
| serious big software.
|
| When software was still delivered in physical boxes, I
| noticed the same thing and I even found products in the past
| where the .exe had megabytes of random data attached to it to
| make it look bigger.
|
| With Docker images it's all a lot easier to bloat things up
| and 'hide things'; I used to pack up our main product in a
| chroot tgz (this is early 2000s) with an installer, so I did
| not have to actually think about different linux distro's or
| installers; I know for a fact that some people bought our
| product over competitors because the installer was so large
| in size so it looks like better value. People told me when I
| asked them afterwards why they did not go for the
| competition. And this was not intentional; it just had a full
| Debian install in a chroot.
| daniellarusso wrote:
| This was something I never considered, but reminds me of
| the Larry Ellison tactic of never naming anything v1, or
| the concept that because something costs the most, it
| therefore must be the best.
| tluyben2 wrote:
| Well, Oracle was (not sure if they still are) one of
| these companies. Some of their products we used in the
| past where artificially bloated and packed over multi CDs
| etc to make it all look bigger.
| dolmen wrote:
| Oracle was cofounded by... Larry Ellison.
| lanstin wrote:
| If all the readers don't just know this, I am old and
| full of useless knowledge. A good read to understand
| Ellison's persona in the 90s is here:
| https://philip.greenspun.com/wtr/dead-trees/53011.htm
|
| Particularly flinging $100 bills at peasants and how
| RDBMS set pricing for their products. If you do have an
| implicit awareness of Oracle and their tactics, you will
| find this an amusing piece of nostalgia when people were
| starting to understand just how to build efficient and
| nice web sites in the 1997 time frame.
| daniellarusso wrote:
| Speaking of 90s, remember the Greenspun AOLserver
| courses?
| goodpoint wrote:
| > why are people obsessed with container size
|
| Good question! Security should be a priority instead.
|
| A lot of containers on docker hub have serious vulnerabilities,
| as proven by various research papers. Removing bloat is good
| for security.
|
| However, using a well-tested distro and receiving security
| updates from its security team is more important than having a
| tiny footprint.
|
| This is a good reason to use Debian stable in containers and
| avoid alpine.
| El_RIDO wrote:
| I'm working at a shop where we have to use extremely large
| images (10 - 40 GiB) on a platform of 60+ medium sized blades
| (4 - 12 cores, 32 - 256 GiB RAM, 200 - 1000 GiB SSDs). Startup
| time for running such huge images often is 5 - 10 minutes. They
| contain prepackaged test data so that they can be run in
| parallel against newly built versions of our software - there
| are about 30ish of these images.
|
| You not only store these images once in the registry, but on
| all container hosts that they might run on. And you need to
| transmit them, if they aren't cached there, yet. And you may
| want to redeploy new images several times a day, as they get
| updated, new ones added, etc.
|
| To be fair, the above is not the use case containers were
| originally intended for. And if your 10 GiB container image
| only runs on 3 or 5 nodes and only gets updated once a week,
| you obviously don't have to worry about the overhead too much.
|
| But at scale (beyond 10 nodes) and/or on very fast development
| cycles (=more then one deployment per day), size IMHO starts to
| matter.
| daniellarusso wrote:
| So, is there a point where it makes sense to use virtual
| machines instead of containers?
| El_RIDO wrote:
| For this particular use case: We previously used VMs and
| snapshots for that workload. The problems we encountered
| were: - snapshots aren't really intended to be portable (we
| used ESXi and also KVM on LVM backed volumes), hence we had
| to write and maintain tooling to have a "snapshot
| repository", versioning for these and distributing them to
| the target nodes - that did all work, but was even slower
| (startup times, which can include shipping and setting up
| the snapshot, was 15 - 60 minutes) - a VM will duplicate
| all of the services and the kernel, so all of that has to
| be started as well, where as the containers only start the
| services under test. - Using VMs makes it much more
| difficult for our developers and QA to retreive a
| particular image and replicate a failing test on a given
| version of the tested software locally, especially when
| considering a wide range of client OSs we see in our not
| that large group (MacOS, various Linux distros, even
| Windows)
|
| A general observation on trade-offs: With containers
| layering and COW you get fast development, but have to pay
| the performance bill when you happen to download and apply
| an image for the first time. Similarly, taking a snapshot
| of a LV under LVM or on a ESXi VM is fast, but applying the
| snapshot is slow.
|
| We had therefore at one point considered using Ceph RDB and
| their COW clone snapshots[0]. It would let us do "cheap"
| restores of snapshots. Our initial tests showed that the
| network bandwidth requirements[1] would have needed some
| serious infrastructure re-engineering in order to keep up
| with our I/O expectations. And again, the containers slot
| in nicely with commonly available local resources and allow
| working offline, to a degree.
|
| [0] https://docs.ceph.com/en/latest/rbd/rbd-snapshot/ [1]
| the environment in question uses 10 - 40 Gb/s network
| links, 40 on the gitlab git and container registry side, 10
| on the blade side.
| daniellarusso wrote:
| I appreciate the response and you answered my next
| question about the networking specs!
| tarasglek wrote:
| What if you had a way to use Docker to create your VM
| images? Then you'd have no registry, be able to mount the
| vm storage over network(ondemand fetching), boot quickly.
| hardwaresofton wrote:
| I can totally see this starting to matter, but I do want to
| point out that it hasn't ground your business to a halt _just
| yet_ and is very very unlikely to. From what I 've seen most
| ops pain comes from a lack of automation when deploying -- as
| in if the process is 5-10 or even 15 minutes, no one really
| cares (outside of an emergency) if it's completely automated.
| Do you find that is true with your work as well?
|
| I can certainly see that it can be a painful problem
| (10-40GiB is huge), but I expect most smaller shops that
| aren't running on their own servers/colo as such these image
| sizes were never even an option (I can't imagine trying to
| upload 40GB across the public internet to launch _one_
| instance!).
| ethbr0 wrote:
| I've worked in the general automation space for awhile, and
| agree that time-to-result is rarely the limiting factor.
|
| It's also not usually total-person-clock-time.
|
| It's number-of-distinct-manual-steps.
|
| (5 minutes manual) + (15 minutes automation) + (5 minutes
| manual) + (30 minutes automation) = everyone hates doing it
|
| (10 minutes manual) + (45 minutes automation) = everyone's
| happy and can use their time productively
| js8 wrote:
| Because every manual step means things can go wrong. You
| might also need to look it up if you don't do it often.
| All these increase the overhead of manual steps compared
| to clock time.
| ethbr0 wrote:
| Partly. I think it's also because manual steps require
| synchronization.
|
| Not only do I have to _do_ something, I also have to set
| an alarm / watch the clock to know when to do something.
| tiagod wrote:
| And the CI with many-GB Docker images is very painful...
| Turning on layer caching usually makes the process even
| slower as it needs to pull and unpack the previous image
| before starting, and if you turn it off you're downloading a
| ton of deps on every build.
|
| If you separate the heavy stuff into a base-image you still
| have to load it on the beginning of CI, which without beefy
| machines with local SSD caching can take a loooong time.
| nfrankel wrote:
| This.
|
| I found that most people focus on size and forget about reuse.
|
| Sure, each layer adds size to your final image, but you can
| reuse them, so that you don't need to transfer the complete
| image but just the layers on top of the one that changed.
|
| Flatten everything and you need to ship the whole stuff.
| zokier wrote:
| Container size generally impacts startup/deploy times. 500MB
| container still takes almost a second to copy over gigabit
| connection, and typically you need to copy it at least twice:
| from build server to repository and from repository to
| whereever its going to run.
| AdrianB1 wrote:
| A Gigabit Ethernet connection means 1Gbps minus some
| overhead, effective rate is around 117-120 MB/sec, so the 500
| MB container takes at least 4 seconds to copy. 10 Gbps gets
| you to less than a second, but that works usually between
| servers, not from the regular laptops.
| chrisandchris wrote:
| Which also means that _if_ you pull at full speed, this node
| has no bandwith at all for serving users (more or less).
| AdrianB1 wrote:
| I am working in Manufacturing Execution Systems area and the
| internal sites are spread around the globe on all continents;
| speed does matter, there are dozens of places hundreds of
| kilometers away from a place with Gigabit Internet (for WAN).
| Taking backups off site is faster by physically shipping hard
| drives than copying over the WAN, I don't like any large
| software.
|
| In one place in Africa driving to another site 3 hours away
| gets network speed 10x better, so we do this from time to time.
| bob1029 wrote:
| > building the code closer to the deployment area is also a
| solution
|
| This is exactly what we do. Once the first build of our
| application is installed on our customer's servers, it rebuilds
| itself from source each time. This is also really good at
| firewalling CI/CD compromises away from your customers, as long
| as you don't merge cryptolockers/miners into your private
| github repos.
| ldng wrote:
| Wasted energy and bandwith do matter. Now, there is a gap
| between a few kB and multiple GB.
| geek_at wrote:
| oh god I fell in love with everthing related to Alpine Linux.
| Found out about it when I started playing around with docker
| but then ported all my home servers and Raspberry Pis to
| Alpine. It's so damn small and has little overhead. Need
| docker? `apk add docker`, done!
|
| I've written guides on how to boot Alpine (even with GUI) over
| PXE [1] and how to set up a perfect file server that runs from
| a RAMdisk USB Thumb drive with full disk encryption [2]
|
| [1] https://blog.haschek.at/2019/build-your-own-datacenter-
| with-... [2] https://blog.haschek.at/2020/the-perfect-file-
| server.html
| Annatar wrote:
| https://rom-o-matic.eu/ is a broken link.
|
| https://blog.haschek.at/2020/the-perfect-file-server.html is
| made obsolete by SmartOS, which does the exact same thing, in
| addition to offering Triton (enterprise web GUI for virtual
| systems), OpenZFS (end-to-end data protection, ease of
| administration), zones (full-blown, running-at-the-speed-of-
| metal virtual UNIX servers), DTrace (for production-safe,
| real-time, deep machine state instrumentation and
| inspection), Bardiche (for building firewalls), FireEngine /
| Crossbow (high-performance TCP/IP stack for building virtual
| switches and routers) and imgadm(1M) / vmadm(1M) (for virtual
| server provisioning and software management), and last but
| not least, a reference implementation of the NFS V4 protocol.
| geek_at wrote:
| > is made obsolete by SmartOS
|
| or by proxmox or by any other OS. That's not what it's
| about. If it comes with a web gui, it's already bloated for
| me. This project was about the minimal perfect setup for my
| needs and I only need SSH access
| Annatar wrote:
| SmartOS does not come with a web GUI; Triton is
| completely optional and isn't required at all, but if one
| is running 100,000 servers or more, it's there as a
| gratis option if one needs one.
|
| As far as I am aware, and please correct me if I am
| wrong, but "proxmox" does not have nearly the same list
| of capabilities as SmartOS, all the while running from a
| ~650 MB, read-only RAMDisk.
| capableweb wrote:
| You seem to have jumped the gun a bit. The question was "why
| are people obsessed with container size?" while you're
| answering some question like "What's a good and small
| container OS?"
| hardwaresofton wrote:
| Yes kind of but also not really -- this answer was really
| delightful to read because Alpine linux literally _was_ the
| first answer to the first concerns with container size.
| Before docker multi stage builds existed (people we doing
| the builder pattern adhoc before that anyway), the usual
| way to get drastic size reductions in container size was to
| run your container on alpine linux.
|
| Base Ubuntu/Fedora used to be absolutely huge, and no one
| was smart/patient enough to pick out all the dynamic libs
| that you needed with your JAR or script to go into
| production so you usually just shipped the whole fat image.
| capableweb wrote:
| Yes yes, but you also seem to be missing the point. Here
| you are answering the question "How did Alpine appear and
| what came before it?" but again missing the original
| question which was "why are people obsessed with
| container size?"
| hardwaresofton wrote:
| Alpine is one of the unsung heroes of the container world.
| It's insane how much value Natanael Copa has created &
| shepherded over the years (decades?). Recently came upon an
| interview with him and it was the first time I saw the
| creator behind Alpine Linux[0]. Similarly, the musl libc
| project, just chugging along, giving most projects that build
| on top of it an easy out for portable static binaries.
|
| Thank you for those links, I am about to gobble up those blog
| posts -- I recently went on a benchmarking kick[1] and
| diskless alpine instantly struck me as the perfect server
| setup. ECC memory + running from ram would give me full use
| (to put in RAID/whatever else) of the NVMe drives, it's
| something I'm going to try out as soon as I get a chance to.
|
| I am so interested in the infrastructure space, I know
| exactly two hosting providers that will give me PXE level
| access (so I could use something like tinkerbell[2]):
|
| - OVH [3][4]
|
| - Vultr[5]
|
| - LeaseWeb[6]
|
| Unfortunately my personal favorite hosting provider,
| Hetzner[7] (I fell in love the moment I came across the robot
| marketplace) does not offer it yet, though I've automated
| going through their rescue system at this point so it's OK.
|
| [0]: https://www.tfir.io/meet-the-creator-of-alpine-linux-
| natanae...
|
| [1]: https://vadosware.io/post/k8s-storage-provider-
| benchmarks-ro...
|
| [2]: https://docs.tinkerbell.org/
|
| [3]: https://github.com/gmasse/ovh-ipxe-customer-script
|
| [4]: https://geekgonecrazy.com/2020/09/07/tinkerbell-or-ipxe-
| boot...
|
| [5]: https://www.vultr.com/docs/ipxe-boot-feature
|
| [6]: https://kb.leaseweb.com/products/dedicated-
| server/installing...
|
| [7]: https://www.hetzner.com
| harpratap wrote:
| At scale every tiny bit helps. Especially now that Edge
| computing is becoming a reality, it's much better to pull and
| run a 1MB container image than a 100MB one on limited hardware
| and bandwidth
| hardwaresofton wrote:
| Agreed -- edge computing is a different world, and the
| difference between 100kb-1mb-10mb-100mb is huge. What I was
| discussing is definitely not a fit at that scale/usecase --
| I'd argue that most infrastructure using containers (and mos
| t people who see this blog post) are not built for this
| scale/use-case, and I'm not sure containers even make sense
| in the edge computing paradigm just yet.
| fouc wrote:
| Wonder what the smallest containerized c10k web server would look
| like.
| ttfkam wrote:
| About 50KB, but supports more than just AMD64.
|
| I submitted a patch to darkhttpd so folks can build the image
| themselves, but you can also download the images here.
| https://hub.docker.com/r/mileselam/darkhttpd/tags
| ttfkam wrote:
| darkhttpd really is awesome.
| https://github.com/emikulic/darkhttpd
| [deleted]
| [deleted]
| sigmonsays wrote:
| I did this challenge a long time ago and found a much smaller
| http server in assembly.
|
| This container ends up being 436 bytes
| https://github.com/sigmonsays/smallest-docker-httpd
| stevefolta wrote:
| I tried using Nix to build the Node version
| ("pkgs.dockerTools.buildImage"). That brought the image size down
| from 943M to 205M. Using a musl64 "crossSystem" (which required
| rebuilding everything) brought it down to 188M.
| El_RIDO wrote:
| To provide a real-life use-case for such a tiny container image:
| With Kubernetes one may want to run the service container as a
| random or at least non-root user id and store data in an attached
| volume. That volume will usually be attached as owned by root. So
| often an init-container is used, which runs once, before the
| service container is started, to change the permissions.
|
| Here is an image with a statically linked chown binary from the
| busybox tools cowering this use case:
| https://hub.docker.com/r/privatebin/chown (I am the author of
| that image)
|
| Benefits of using a small image are less attack surface and
| faster operation (less data to download, lower startup time, less
| memory, etc.).
| spalas wrote:
| Hi! Author here!
|
| My (somewhat silly) use case for the tiny container was trying
| to cram 10,000 pods onto a k8s cluster without breaking the
| bank: https://www.youtube.com/watch?v=1y2nRNexRVk
| xiaodai wrote:
| Amazing! Thank you.
| pantuza wrote:
| Interesting discussion. Just a comment about the size o Golang
| binary. In the post example it uses standard fmt library. fmt by
| itself, adds several megabytes to a binary when compiled.
| Removing it would reduce binary size.
| friseurtermin wrote:
| I love blog posts like these: they don't go after real-world
| problems directly but rather explore what's even possible.
|
| paulfurtado in this comment section mentioned UPX[0] as a way to
| get binaries even smaller. Funnily enough UPX even managed to
| decrease the size of the asmttpd web server for me:
| ### build stage ### FROM ubuntu:18.04 as builder
| RUN apt update RUN apt install -y make yasm as31 nasm
| binutils git RUN git clone
| https://github.com/nemasu/asmttpd.git asmttpd RUN mv
| asmttpd/* . RUN rm -rd asmttpd RUN make release
| # i have the upx binary on my machine in this case COPY
| ./upx-3.96-amd64_linux/upx upx RUN ./upx --brute asmttpd
| ### run stage ### FROM scratch COPY
| --from=builder /asmttpd /asmttpd COPY ./index.html
| /web_root/index.html CMD ["/asmttpd", "/web_root",
| "8080"]
|
| With a 4 byte index.html my total image size is now 5.3kB!
|
| [0]: https://github.com/upx/upx
| spalas wrote:
| Hi, author here!
|
| Thanks for the kind words. I didn't know about UPX.
|
| I should add an update to the end with this trick!
| e12e wrote:
| Might be possible to extend asmhttpd to serve gzipped files
| directly (possibly breaking compliance with rfcs by _forcing_
| compression)? Ie send index.html.gz?
| throwaway4good wrote:
| What is in "scratch"? Couldn't you run a C-program there or a
| JVM?
| arafalov wrote:
| For JVM, you have to bend a bit backwards and use still-
| evolving toolchain, but it is actually possible to create
| native-image of a Java program that will run on top of SCRATCH:
| https://jamesward.com/2020/05/07/graalvm-native-image-tips-t...
| onei wrote:
| If you imagine a container as a tarball, then scratch is an
| empty tarball. Each time you copy a file or set of files into
| the container during the build, you get a new tarball or layer.
| Some base containers provided by OS vendors, e.g. CentOS,
| Debian, are a single layer with the entire OS filesystem inside
| including tooling, libraries etc.
|
| Scratch has no tooling or libraries or anything. It's empty.
| You typically add a binary into it and then just run that
| binary, but due to the lack of libraries, it just be statically
| compiled. So in answer to your question, you can run statically
| compiled C. You can't run Java due to the lack of JVM which I
| believe has its on set of filesystem dependencies, but I'm not
| a Java dev, so I'm not entirely sure.
| ulzeraj wrote:
| So Go binaries should also work right?
| onei wrote:
| Go binaries that are statically linked work great in
| scratch images. If you or your dependencies start
| dynamically linking against libraries then they don't so
| much, but that's relatively unusual in my experience.
| raesene9 wrote:
| yeah I've seen statically compiled go binaries in scratch
| base images before, seems to work fine.
|
| If you need some basic OS files, Google's distroless is
| another option.
| throwaway4good wrote:
| I meant copying the JVM unix tar ball in there and use it to
| run whatever.
|
| Why aren't everyone just using scratch?
| NavinF wrote:
| I'm pretty sure the default build of openjdk dynamically
| links glibc. If you build it statically, that should work.
| throwaway4good wrote:
| Why is it not best practice to do so?
|
| The naive solution using the golang image is nearly 1GB.
| Why carry this extra complexity around?
| NavinF wrote:
| It is best practice and I wish it was common practice,
| but there's a lot of friction in the way and some people
| will actively work against you.
|
| Static linking glibc is painful because the former
| maintainer has some strong opinions:
| https://www.akkadia.org/drepper/no_static_linking.html
|
| Most distros force you to dynamically link every
| dependency if you want them to package it. So the default
| build for most projects is dynamic
|
| You've stumbled on a holy war between distros and guys
| like you, me, and Linus Torvalds[1] that want to deploy a
| binary and just have it work everywhere.
|
| 1: https://youtu.be/5PmHRSeA2c8?t=295 (Highly recommend
| watching Linus's answer ~6 minutes in)
| throwaway4good wrote:
| Sure this is understandable when we are building an OS.
|
| But here we are using Docker so we have full control over
| the application we are building. Why this craziness of
| having these huge images containing who knows what?
|
| Is it just because we can? And the cloud providers like
| us when we do it?
| ratww wrote:
| It mostly happened because the we carried over the
| previous assumptions, practices and limitations when
| moving into containers.
|
| I agree with you and the parent commenter, this should be
| the default, but some people are against static-linking,
| even in cases dynamic-linking provides no advantages.
| onei wrote:
| There's downsides to statically linking, particularly
| around what to do when vulnerabilities are found in your
| dependencies. If you use Debian for example, you can scan
| packages to detect versions with known vulnerabilities and
| rebuild the container to upgrade them if it's a static
| binary, that moves the detection into scanning your
| dependencies. In modern GitHub (other VCS hosts are
| available with similar features) this is relatively easy
| assuming you depend exclusively on things in your language
| of choice. Outside of that it gets harder and more awkward.
| That said, using a distro's image also means you may have a
| bunch of false positives in your scans, who it's a matter
| of taste imo.
| prova1 wrote:
| test111
| zwischenzug wrote:
| Reminds me of this related article:
|
| https://zwischenzugs.com/2018/05/22/a-docker-image-in-less-t...
|
| TL;DR can you get an image to run for test purposes that can fit
| in a screenful of base64?
| ourmandave wrote:
| Whatever it takes to stay on the AWS free tier.
| paulfurtado wrote:
| FWIW, if you wanted to get the static from-scratch go image a lot
| smaller, try running the binary through UPX. It typically
| improves compiled go binary sizes by absurd amounts:
| https://upx.github.io/
|
| Also, https://busybox.net is a 1.0MB static binary that provides
| all of the standard posix-compliant tools (sh, grep, tail, etc)
| and actually comes with an httpd implementation. So if you wanted
| a more useful container at about 1MB, you could use it.
|
| Edit: here's a previous thread on UPX that has 80 comments
| talking about the tradeoffs it makes. You should definitely
| consider these if you intend to use UPX:
| https://news.ycombinator.com/item?id=15456980
| fouc wrote:
| 1.0MB is quite large compared to 6.34kB :)
| whizzter wrote:
| Certainly, I think this is more about getting over the
| developer pain threshold. In terms of computer speed vs human
| perception I don't see either one mattering too much
| (compared to a 900meg base for node-debian on Docker,
| although I guess there is some incremental stuff for that?).
| paulfurtado wrote:
| It definitely is, although a cool thing about busybox is that
| you can rebuild it with any applets disabled. I'm not sure
| just how small it can get though.
|
| But yeah, the tradeoff of that 1MB is that you get a very
| functional shell environment vs just an http server. It's
| impressive just how small ut is, given everything that it is
| capable of
| shakna wrote:
| > It definitely is, although a cool thing about busybox is
| that you can rebuild it with any applets disabled. I'm not
| sure just how small it can get though.
|
| They do distribute statically linked individual binaries
| [0]. For just HTTPD and nothing else is about 85kb. But it
| is a lot more than just a static file server - it comes
| with CGI support.
|
| [0] https://www.busybox.net/downloads/binaries/1.30.0-i686/
| js8 wrote:
| > But yeah, the tradeoff of that 1MB is that you get a very
| functional shell environment vs just an http server. It's
| impressive just how small ut is, given everything that it
| is capable of
|
| Maybe.. I suspect a well-designed Forth image could pack
| similar capability in less than 100kB. So it might seem
| low, but not a miracle.
| prova1 wrote:
| holaa
| endgame wrote:
| If you only want the sh, you can get it even smaller. This
| doesn't UPX the busybox bin, though:
|
| $ ls -shL `nix-build -E '(import <nixpkgs>
| {}).pkgsCross.musl64.busybox.override { enableStatic = true;
| enableMinimal = true; }'`/bin/sh
|
| 128K /nix/store/qkq9cj81mcvnfr5y04jmf6nqvnrfg2z1-busybox-1.31.1
| -x86_64-unknown-linux-musl/bin/sh
| codetrotter wrote:
| Beware that at least historically some antivirus software would
| wrongly classify any UPX-packed binary as malware. Don't know
| if this is still the case but something to keep in mind.
| account42 wrote:
| "some antivirus software would wrongly classify X as malware"
| is true for any X (except malware, then would not be wrong)
| dolmen wrote:
| About busybox: if you want a small container, you don't need
| the posix tools at all, but just your binary. No shell.
| TonyTrapp wrote:
| Keep in mind that UPX needs to decompress the whole executable
| to memory and thus cannot make use of OS-level memory sharing.
| Normally, when you load the same executable twice, its base
| image will only consume memory once. But it with UPX, you
| cannot make use of that. So in the end, it may use more
| resources than you thought!
| paulfurtado wrote:
| Definitely important to note, I wonder if upx could solve
| this via execing an memfd_create in some mode. Also wish upx
| came with a trivial extraction tool so you could use it for
| distribution but not runtime.
| prova1 wrote:
| ok.
| mikepurvis wrote:
| Aren't your container layers compressed in transit anyway?
| paulfurtado wrote:
| UPX does significantly more compression on binaries than
| gzip can on the layers. It's not just simple compression,
| it makes a lot of modifications to the binary to make it
| more compressible.
| dividuum wrote:
| I was initially a bit skeptical if the page cache would work
| with all layering docker does. But OverlayFS, perhaps
| unsurprisingly, does indeed support that: [1] "OverlayFS
| supports page cache sharing. Multiple containers accessing
| the same file share a single page cache entry for that file.
| This makes the overlay and overlay2 drivers efficient with
| memory and a good option for high-density use cases such as
| PaaS." TIL
|
| [1] https://docs.docker.com/storage/storagedriver/overlayfs-
| driv...
| Hackbraten wrote:
| UPX creates anonymous pages that contain the decompressed
| payload, right? Those aren't file-backed pages so how would
| the filesystem even know those pages exist?
| dividuum wrote:
| That's correct and I guess only page deduplication might
| help in that case. What I meant in my post was just to
| state that docker can use the page cache despite the
| massive indirection caused by overlays. I was wondering
| if that was even possible and it indeed is.
| synergy20 wrote:
| that's true, plus on low-end devices upx will take forever to
| decompress and is essentially unusable.
| paulfurtado wrote:
| Is UPX actually that much slower than extracting a gzipped
| docker layer? I didn't realize UPX decompression was much
| slower than other compression algorithms
| amscanne wrote:
| The base would consume "up to once". You will only load pages
| touched, and they are all reclaimable under memory pressure.
| Memory used by UPX for decompression is guaranteed to be
| used, and is not reclaimable!
|
| So in fact, it's not _might_ use more resources, it's
| guaranteed to use more resources.
|
| The only benefit of UPX is saving disk space, which is
| unlikely to be super useful to most people these days.
| paulfurtado wrote:
| The other use case is to distribute images more quickly
| over the network and improve image pull times, at the
| expense of slightly more memory use.
|
| Similar to how saving disk is less useful to most people
| these days, RAM on servers in big docker/kubernetes
| clusters is also likely not much of a concern for binaries
| that are well under 100MB if you're not running a ton of
| copies on each server. In particular, it seems like this
| tradeoff is most useful for static go binaries, which don't
| benefit from shared system libraries and aren't likely to
| have many copies per server, but of course that definitely
| depends on the use case and I definitely do not recommend
| blindly using UPX without considering the tradeoffs.
|
| It would be an amazing integration though if it were
| possible to integrate UPX with docker's layer compression
| such that the binary is extracted to the FS before running.
| But given how it all works, that's pretty challenging to
| implement and the network is only getting faster...
| shawnz wrote:
| Are you sure UPX actually beats the standard layer
| compression which is already provided?
|
| After applying UPX, the result should be nearly
| uncompressable, so if UPX performs worse than layer
| compression then it might be better for network usage to
| not use it at all.
|
| In this scenario where you are unpacking it in advance,
| that eliminates the main benefit of UPX which is that it
| is self-extracting.
| spalas wrote:
| Hi, author of the article here!
|
| I figured there were a bunch of ways to shrink that one, and I
| had a C based image that was in between my GO and ASM images,
| but decided to cut them for the sake of brevity!
| dnautics wrote:
| stealing vexu's routez: https://github.com/Vexu/routez
|
| gives you a multiprocess http server in 612K uncompressed if
| you compile -Drelease-small
| davidjfelix wrote:
| If you're going for size, why not just sidecar all of the
| "standard posix-compliant tools" but ones that you would want
| normally. For example, bash instead of ash/dash. You don't need
| those tools all the time right?
| dolmen wrote:
| Well, you don't even need a shell.
| davidjfelix wrote:
| I mostly meant to sidecar the shell and treat it as a
| debugging utility. While I get the point of ash being
| smaller as a point for busybox, if size stops being a
| concern because the shell is in an inactive-sidecar, then
| you can use a shell that has less surprising edge cases for
| users, like bash.
| okamiueru wrote:
| I did exactly this a few years back.
|
| It added 18.5 kB on top of busybox, so, not as good as the 6kB
| ASM server code. With busybox included, the whole image is 1.18
| MiB, or 751 kB compressed.
|
| If anyone cares to take a look:
| https://hub.docker.com/r/swalog/tiny-hello/
| daniellockard wrote:
| Be sure to check out the switches, --ultra-brute is what I
| usually use.
| prova1 wrote:
| eeee
| prova1 wrote:
| 222
| alpb wrote:
| Why do you need a httpd server in front of Node.js? Isn't the
| builtin server capable of handling http requests concurrently and
| routing?
| pjmlp wrote:
| Or just ship a statically compiled binary, with resources inside,
| without having to mess with anyhing docker related.
| Annatar wrote:
| Or just do the work like a professional, link dynamically and
| build OS packages for the operating systems which you have
| qualified and can guarantee that your software will work on.
| And take pride in the level of quality you are capable of
| delivering. Few in this day and age are capable of writing
| software at such a high level of quality as to be able to make
| guarantees that it will JustWork(SM); creating operating system
| packages out of one's work makes it possible to deliver
| software like a professional, perform integration testing and
| provide guarantees to the consumers, while minimizing the load
| on the users of said software.
| tbrock wrote:
| The golang example where he uses scratch as the base image is
| essentially this.
| pjmlp wrote:
| Except he packages it on a container.
| dijit wrote:
| The container, if it's `from SCRATCH` is basically just a
| gzip/zip archive.
|
| https://github.com/opencontainers/image-
| spec/blob/master/ima...
| regularfry wrote:
| Which is probably unnecessary if you've got the deployment
| platform to yourself and can decide that just-run-this-
| binary is suitable, safe, and manageable. Personally I
| prefer "just install this .deb" because it wraps up
| dependency management along with everything else, but
| that's not been a popular opinion for a long time.
| secondcoming wrote:
| Maybe because creating custom deb packages can be a hair-
| pulling experience?
|
| But this is our flow too. We have a jenkins job that
| builds our master branch and uploads deb packages to
| jfrog. And then another to create a GCP image with these
| psckages, and a third to actually deploy these images to
| GCE. Containers are used for the various jenkins jobs,
| but not on the actual GCE instances.
| regularfry wrote:
| It depends which tooling layer you're using, really. If
| you're just saying "bundle up these files and put them on
| that part of the filesystem" it can be extremely simple.
| If you're trying to provide the full build-from-source
| chain... less so. Being able to ignore the Debian
| Packaging Guidelines is the path to sanity.
|
| In a previous incarnation I wrote
| https://github.com/regularfry/au to take the grunt-work
| out of packaging up ruby apps, and pretty much all it
| does is generate the minimum set of files that `dpkg-deb`
| needs to spit out an archive.
| debiandev wrote:
| Debian Developer here.
|
| Reminder: the Debian Packaging Guidelines are meant for
| official Debian packages.
|
| Making a package with dpkg-deb -b <dir> <packagename> is
| very easy and gives all the features of APT (dependency
| tracking, config management, atomic deployment).
| regularfry wrote:
| Yep. If you're packaging _for Debian_ , the packaging
| guidelines make sense. If you're packaging something
| nobody outside your org is ever going to see, you can be
| a lot more flexible.
| secondcoming wrote:
| Yes, dpkg-deb is a recent discovery. Out of interest,
| what am I losing by doing it this way rather than if I
| followed the official guidelines?
| goodpoint wrote:
| Or just ship a .deb
|
| It contains everything you need, can provide sandboxing,
| integrates well in the OS, can do deploy and rollback, manages
| configuration changes.
| hardwaresofton wrote:
| And don't forget to include making sure to create the right
| resource and namespace constraints so that the one binary
| doesn't gobble up memory endlessly or use up the file system or
| put your infrastructure completely at risk if/when it's
| compromised -- whether you do that with systemd units, raw
| cgroup/namespace finagling, or regular tried-and-true linux
| user-based resource segregation.
|
| Containers do provide benefits.
| SahAssar wrote:
| So ship the static binary and a systemd unit file like a lot
| of packages in the repos of your linux distro do.
| hardwaresofton wrote:
| Yep, and you'd generally expect to spend time understanding
| the complexity that systemd brings, the subsystems that
| power it, and how you can separate resources there! There
| is no free lunch.
|
| Restated, my point here is that while containers can be
| more complex and have pitfalls (a lot of which have been
| worked out somewhat at this point), there is no complexity
| free lunch -- `[docker|podman|crictl] run --rm --cpus 2
| --memory 500mb ...` is _pretty darn easy_ , and more so
| than writing properly portable and well-considered systemd
| unit files (and putting them in the right place, with the
| right permissions, under the right slice, etc). It's easier
| than most of the options out there (including the old
| methods of per-user resource segregation).
| SahAssar wrote:
| Let's say you are installing this from a normal package
| manager and want to limit it:
|
| To install: apt install mypackage
|
| To edit the systemd unit (automatically creating the file
| in the right place): systemctl edit
| mypackage
|
| Then add these lines to limit to 2 cpus and 500mb memory:
| [Service] CPUAccounting=true
| CPUQuota=200% MemoryAccounting=true
| MemoryHigh=500M
|
| And then to start it now and on boot:
| systemctl enable --now mypackage
|
| This is now integrated with package updates, starts on
| boot, logs to the same place that most other system
| utilities do and so on.
| aequitas wrote:
| But there is a lot of work to be done before you can do
| the simple apt install. I (gladly) don't know how it is
| nowadays but before Dockerfiles/Docker creating your own
| packages according to the various standards was a pita.
| Most companies needed a 'packaging specialist/release
| engineer' role as most developers where not up to the
| task. Solutions like FPM[0] did help somewhat, but it was
| still hard when dealing with non-homogeneous
| environments. Containers solved that problem universally
| for all distributions.
|
| [0] https://github.com/jordansissel/fpm
| hardwaresofton wrote:
| I'm not about to argue against systemd, it's great
| software, and it's in every distro right now for a
| reason. Understanding apt (+/- how to package for it),
| systemctl, and the options you've laid out in your unit
| file are not trivial, and I would argue that they are
| _less_ trivial (or harder) than understanding what 's
| happening with containers, _especially_ if you 're
| running rootless containers, and/or using a container
| tool like podman which does without the daemon.
| --cpus 2 --memory 500M
|
| is easier, and gets you the same results though they may
| not be as permanent or as well managed -- the management
| and external stuff is an orthogonal concern, and that's
| not the situation I was addressing. The original point
| was insinuating that throwing up a binary and getting it
| to run. your filesystem is also not available to the
| container _by default_ , and in this way docker sort of
| fails closed. If you're running a rootless container, the
| story is even better.
|
| One thing you have not covered is filesystem isolation,
| which docker also does very easily. There is _a lot_ to
| configure on the systemd side[0] and the parts that are
| overlapping are just easier to configure and run with
| docker. Systemd is the better tool to build repeatable
| installs for pet processes, but again, there is a lot of
| knowledge underneath that is related. People to this day
| still complain that systemd does too much (I personally
| like it a lot, and it 's great to have everything in one
| place).
|
| [EDIT] Just to make myself clear, systemd is an amazing
| tool -- I like it, I run it, I'm not smart enough to
| administer a more complicated setup -- but docker is
| easier, for a large part of the small subset of systemd's
| capabilities that docker covers.
|
| [0]: https://www.redhat.com/sysadmin/mastering-systemd
| c0l0 wrote:
| I don't see how appending --cpus 2
| --memory 500mb
|
| to the argument vector liberates these options' definer
| from "understanding [...] the subsystems that power it",
| and reflect on their potential impact. And I'd argue that
| THIS is the real complexity incurred by these kinds of
| resource constraints that seem _so_ simple on the surface
| - not the specific syntax or location you have to use to
| introduce them. All of which makes systemd unit files and
| their settings ' implications (which are amazingly well-
| documented btw) as good as any other option, imho.
| hardwaresofton wrote:
| Systemd is great software -- it is useful, and powerful.
| A systemd unit file, the idea of what a unit is, when
| they run, _how_ they run, what shells they use, what
| permissions they run under, and lots of other
| complexities are expressed as options in the unit file,
| configuration files on the system, and in other places.
|
| If I want to run a useful piece of software like redis
| let's say, but I want to run it with a resource
| constraint to make sure that it doesn't take more than
| 2CPUs and 500MB of memory, it is _far_ easier to do that
| with the following command line: docker
| run --rm redis --cpus 2 --memory 500mb -p 6379:6379
|
| Than to write the equivalent systemd unit file, set up
| the isolated filesystems that docker would let you easily
| bind mount in, etc. This is like comparing systemd-nspawn
| to systemd -- if systemd-nspawn isn't simpler than
| systemd then what are we even doing.
|
| Docker won because of it's developer ergonomics
| (containers weren't new), systemd won because of it's
| feature set, convenience and sturdiness. They're
| different tools with different primary use-cases.
| slt2021 wrote:
| can anyone explain why you might need to limit memory
| consumption of your service?
|
| I thought the kernel has virtual memory and if you
| consume more it will swap some memory and thats it.
|
| Couldnt you just manage memory from inside your app? if
| its redis, then check the db size and shutdown/cleanup
| gracefully, and not crash redis with OOM?
| hardwaresofton wrote:
| > can anyone explain why you might need to limit memory
| consumption of your service?
|
| > I thought the kernel has virtual memory and if you
| consume more it will swap some memory and thats it.
|
| Well just to make sure the right memory goes to the right
| places -- if someone uploads a large file and you've made
| a mistake in your code that tries to hold it all in
| memory instead of buffering it straight to disk for
| example, you'd want that process to crash, and not your
| machine.
|
| Also you generally don't want to swap, so much so that
| Kubernetes disables it immediately[0]. Not that Google is
| the only group with the right answer but they seem to
| think nothing can come of a machine having to swap. Maybe
| they're right. Even if they're not, A world where _one_
| service swaps[1] (I 've never done this with docker to
| try it though) is probably better than one where it uses
| all the memory and _everything_ swaps.
|
| > Couldnt you just manage memory from inside your app? if
| its redis, then check the db size and shutdown/cleanup
| gracefully, and not crash redis with OOM?
|
| You'd be surprised -- some languages just don't have a
| way to very easily get feedback from GC[2]. It's also
| something that I don't think most people think about,
| messing with the -XmXx<setting>s in Java is definitely
| year 2/3/4 java development for most people.
|
| [0]:
| https://github.com/kubernetes/kubernetes/issues/53533
|
| [1]: https://docs.docker.com/config/containers/resource_c
| onstrain...
|
| [2]: https://github.com/golang/go/issues/29696
| slt2021 wrote:
| I think it is just different thinking: DEV where software
| is a pet, where I want full control of my memory and take
| care of it carefully, and dont want anyone shutting down
| my service violently.
|
| contrast it to Ops mentality of software is a cattle
| (here is your memory quota and if its OOM, just
| kill/restart the service and hope next run it wont run
| OOM)
| slt2021 wrote:
| Also thank you very much for kubernetes link.
|
| Wow, given how expensive RAM is it is no surprise they
| will never allow kubernetes work with swap, becausr it
| directly translates to $$$ for GCP and other cloud
| providers. Also since everyone loves using
| Java/Spring/Node and other memory hungry frameworks - it
| print enormous $ for cloud providers to require users
| overallocate RAM and disable swap.
|
| or am I just spitballing conspiracy theory here and there
| is no conflict btw decisions like these and vendors'
| revenue streams?
| hartmel wrote:
| Then you will want to manage the configuration file and
| to browse logs, and you ll have to mount files to you
| container. And your deployment start to reimplement (a
| subset to your needs) what distros did System integration
| complexity didn't disappear thanks to containers. It was
| just moved to another place. I dd add that, being
| convenient for developers doesn't mean it's convenient
| for hosting management. At the end, when one side
| convenience is taken in consideration, it often translate
| to complexity and pain at the other side. K8S become in
| some companies more a mandatory runtime requirement than
| a hosting commodity/facility.
|
| As a (former) developer I d rather ran redis in a
| container during development, but for production I d
| rather rely on boring VMs unless some scaling is
| required. (Managed k8s case set apart)
| hardwaresofton wrote:
| > Then you will want to manage the configuration file and
| to browse logs, and you ll have to mount files to you
| container. And your deployment start to reimplement (a
| subset to your needs) what distros did System integration
| complexity didn't disappear thanks to containers.
|
| Agree, but my view on this is that the implicit answer of
| _how_ you do all that (i.e. the file system, syslog) is
| now gone. There will be pain (complexity) in the short
| term, but at the end of the day, we 're going to be able
| to build much better orchestration and systems. To kind
| of restate that, before you had to worry _where_ a
| process wrote out it 's output (stdout?
| /var/log/<program>? /etc/<program>/logs?
| /home/<user>/<program>/logs? syslog?), now you know want
| to get the non-stdout/stderr logs of the thing you're
| running, you'd better give it a volume to write to (which
| may be fake, and _actually_ write everything to some
| remote storage or something), and I think that 's a step
| forward.
|
| Of course, I'm not saying containers should go
| _everywhere_ -- relying on boring VMs over containers is
| fine too -- but I think rich world of functionality
| available to container-driven workflows is popular for
| good _and_ bad reasons, and the good reasons are worth
| exploring /beneficial to me.
| jeroenhd wrote:
| I don't understand why this isn't being done more. systemd
| has pretty much every convenience Docker has these days,
| without any of the ridiculous overhead and cognitive load
| that comes with managing a Docker daemon.
|
| Yes, you'd need to learn how to make a systemd unit file
| but honestly, that shouldn't be a problem at all. I've had
| much worse headaches fighting Docker's networks and
| firewall-overruling network configuration in the past.
| Every time I forget to specify a restart option, I have to
| dig through my shell history again to see how I launched
| the image so I can kill and recreate it, or I have to look
| up that oneliner that shows you the docker run command for
| a running container.
|
| I run stuff in Docker for one simple reason: I have it
| installed and I'm too lazy to think about software
| sometimes. If you're packaging software, that laziness
| isn't a reason to pick a distribution tool.
|
| Static binaries have other problems, perhaps most
| importantly package management. How do you distribute your
| app? Traditional deb/rpm packages? curl-to-bash installers?
| Snap? Flatpak? Binaries that people flog into /usr/bin?
| What about automated updates, do you set up a repository,
| do you include a self-updater in your code? The list goes
| on. Docker solves that by having one general source of
| packages with the option of adding a repo of your own (take
| note, Canonical, your shitty Snap Store is practically
| useless without that last bit).
|
| For services I run on servers, I much prefer traditional
| packages with dynamically linked executables, for a simple
| reason: automatically fixing security issues and bugs
| across applications with a single update command, instead
| of having to wait for every maintainer of their statically-
| linked tool to update their dependencies.
|
| It's one of the big problems I have with Rust; many
| dependencies and applications embed their own versions of
| dependencies with sometimes very specific versions, and
| when there will eventually be a massive security problem in
| one of the TLS packages I'm dependent on recompiles and
| code modifications from random open source maintainers.
| goodpoint wrote:
| > I don't understand why this isn't being done more.
|
| I can only offer one explanation: marketing.
| IshKebab wrote:
| Most servers are already running in a VM that takes care of
| all that.
| hardwaresofton wrote:
| yeah but that bit of complexity wasn't mentioned :). If
| every process you need to run in it's own VMs/per-instance
| then you've got best-in-class isolation already, no need to
| bother with containers, and you can get your resource and
| namespace isolation. Not quite the same as just throwing a
| static binary over the wall though, a little bit more
| complexity there.
| krageon wrote:
| > compromised
|
| Containers are not a security measure. They're acceptable for
| some resource constraint considerations, but you should not
| ever use them because of security concerns. If that's
| something to worry about, use VMs at the very least.
| hardwaresofton wrote:
| > Containers are not a security measure. They're acceptable
| for some resource constraint considerations, but you should
| not ever use them because of security concerns. If that's
| something to worry about, use VMs at the very least.
|
| This is repeatedly really commonly, but is bordering on
| wrong more and more every year. At the lowest level it's
| not even completely correct. Containers are resource +
| namespace isolation -- cgroups and namespaces -- isolation
| and kernel-assisted resource hiding/access control is
| absolutely a facet of good defence-in-depth. Enhanced
| security is not a core feature of containerization as a
| whole but it can absolutely help with your security
| posture.
|
| For what % that satement is wrong, it's even more so
| because it fails to consider the container runtime.
| Container runtimes these days (containerd is the best one
| IMO) _include_ ways to run containers at different levels
| of isolation (VMs and as micro kernels), so a move to
| containers can absolutely help security posture, because
| now you don 't have to pull out packer/ansible/etc to build
| your VM, you can just take the same container you were
| running and change the runtime. Time to ship/launch
| improved security features is important, and containers can
| reduce that burden for teams.
|
| Containers are a process isolation and resource control
| tool -- they can absolutely aid in security posture. Would
| you say the same about BSD Jails?
|
| Just so there's some more value here, here are some
| projects that are somewhat close to cutting edge (though
| it's been a while) in this space that I like:
|
| - nabla-containers.github.io/
|
| - https://github.com/firecracker-microvm/firecracker-
| container...
|
| - https://github.com/kata-containers/runtime (see
| https://github.com/kata-
| containers/documentation/blob/master...)
|
| - https://rootlesscontaine.rs/
|
| - https://www.kernel.org/doc/html/latest/admin-
| guide/cgroup-v2...
|
| - https://starkandwayne.com/blog/the-capable-kernel-an-
| introdu...
|
| The container ecosystem surfaces a lot of really cool
| resource limiting and isolation techniques and puts them in
| very close reach.
| dundarious wrote:
| GP is noting the security limitations of Linux
| containers, the mechanism, not some management daemon
| like containerd or some API or workflow. firecracker-
| containerd is not a container, per-se, as stated in their
| README.
|
| > Like traditional containers, Firecracker microVMs offer
| fast start-up and shut-down and minimal overhead. Unlike
| traditional containers, however, they can provide an
| additional layer of isolation via the KVM hypervisor.
|
| I think the security mechanism of note there is the
| hypervisor -- a VM technology. The fact that it is
| manageable via a daemon/API that was created for
| containers, does not mean it provides security via Linux
| containers, the mechanism.
|
| A similar story can be told regarding capabilities, a
| mechanism that is orthogonal to containers -- all
| combinations of with and without capabilities and
| containers have their uses.
|
| I do think GP is overly glib and assumes a particular
| threat model in order to dismiss Linux containers so
| thoroughly, and you are right to note "they can
| absolutely aid in security posture", etc. But GP is
| correct to note that VMs (though also imperfect) provide
| certain kinds of security/isolation that Linux
| containers, the mechanism, cannot yet match.
| jeffbee wrote:
| I'll double down on what the GP said: containers provide
| less security than people imagine, and contrary to your
| comments Linux control groups provide no meaningful
| resource isolation.
| hardwaresofton wrote:
| > containers provide less security than people imagine
|
| Well sure, this heavily depends on the people you're
| talking to and what their imaginations are like. If it's
| the "containers are lightweight VMs" crowd then sure --
| that view is fundamentally wrong.
|
| > Linux control groups provide no meaningful resource
| isolation.
|
| So this is a pretty bold claim -- as far as I can see
| processes certainly get OOMkilled when they use more
| resources than allowed for by their cgroup.
|
| Do you have a link you could share? Is some fundamental
| part of cgroups (v1? v2?) broken in some way I haven't
| heard of up until now such that everyone has patched
| around it and it does what it says on the tin but despite
| the code in the kernel?
| jeffbee wrote:
| The incompressible resources, including memory, have the
| best isolation story, but it is by no means perfect. A
| process in a given control group can easily cause
| external memory consumption that either escapes
| accounting altogether, or is charged to another control
| group. An example is the way kernel slabs are assigned to
| control groups. There were some patches in late 2020 from
| Facebook to help fix this, but I am not sure if they are
| merged and released yet. Those patches are also just
| better, not perfect.
|
| Another example is a control group that is chronically
| out of memory. It may page vigorously, which will have
| external effects on other control groups through
| unaccounted resources like nvme controller time, memory
| fragmentation, global vmscan, etc.
|
| A third popular way to abuse the resources of other
| containers is through networking. The way Linux handles
| network traffic is frankly hostile to proportional
| resource sharing. A control group with zero CPU quota can
| still easily cause external CPU time consumption.
| hardwaresofton wrote:
| Thanks for explaining, while I still think what you said
| was a bit hyperbolic, I definitely didn't know about
| these... externalities, I guess you could call them?
| Learned something today.
| jeffbee wrote:
| I think of control groups as being adequate for what
| their inventor intended: approximate sharing of resources
| between non-antagonistic, first-party processes. Outside
| of that zone they can't be relied on for much of
| anything.
| raesene9 wrote:
| I'm going to have to disagree with that statement.
| Containers (as implemented by Docker/containerD/CRI-O) do
| add security controls. Specifically, separate namespaces,
| restrictions on capabilities, seccomp filters and apparmor
| profiles.
|
| If you can easily break out of any containerized
| enviornment, I'd suggest you could register for bug bounty
| programmes and make quite a lot of money also you can get a
| reward from Jessie Frazelle by escaping from
| https://contained.af/
|
| Obviously containers have a larger attack surface than say
| a VM hypervisor, but security is not an absolute and both
| containers and VMs have suffered from breakout issues in
| the past. No security measure in isolation provides perfect
| protection.
| ex_amazon_sde wrote:
| No, you are confusing containers with sandboxing and they
| are completely orthogonal.
| [deleted]
| emteycz wrote:
| Containers as understood in context of LXC or Docker are
| a strong variant of sandbox.
| ex_amazon_sde wrote:
| You are describing sandboxing and it's completely orthogonal
| to containers.
| pjmlp wrote:
| My first container experience goes back to HP-UX Vaults.
|
| I let you research when that was, and wonder what container
| technologies might have used since then.
|
| I know when they make sense and when they happen to be
| fashion.
| hardwaresofton wrote:
| Didn't doubt your experience (I read and enjoy your
| comments all the time on HN), just pointed out that the
| feature set offered by single static binary (which arguably
| is not easy to build depending on what language you're
| using, etc) is not the same as what's offered by linux
| containerization in the current day.
|
| Half the "static" binaries that used to float around
| weren't even fully static because of glibc, until building
| with musl (and getting bit by getaddrinfo) became
| widespread -- the situation is not as simple as you made it
| seem.
| pjmlp wrote:
| I agree with the glibc issue, but that is a problem of
| GCC specifically.
|
| My point is that I see containers being overused
| nowadays, like "big data" that fits on a USB pen kind of
| scenario.
| hardwaresofton wrote:
| Agreed -- containers are certainly being overused these
| days, I find a big indicator of when people know what
| they're talking about is when they mention/realize that
| docker _is_ containerd these days (docker is simply a
| shim over containerd which uses runc underneath), but it
| is absolutely still very buzzword-y.
|
| I think we've basically stumbled into a very effective
| and widespread packaging paradigm though -- now you don't
| even have to pick the right language/toolset to get
| static binaries easily -- just throw a container over the
| wall and your filesystem requirements (mounts), network
| requirements, etc will be made pretty obvious to the
| person doing the deployment
| robertlagrant wrote:
| Yeah cool, now just need to spin up 50 of then evenly across my
| servers, pop SSL termination and round robin routing in front
| and pull the logs out.
___________________________________________________________________
(page generated 2021-04-22 23:02 UTC)