[HN Gopher] Anti-patterns when building container images
___________________________________________________________________
Anti-patterns when building container images
Author : henrik_w
Score : 148 points
Date : 2021-11-30 13:30 UTC (2 days ago)
(HTM) web link (jpetazzo.github.io)
(TXT) w3m dump (jpetazzo.github.io)
| kator wrote:
| When I read people complaining about "Small Images" like
| mentioned here where the author seems to warn against it and even
| proposes making sure enough "tooling" is in the container.
|
| I'm always a bit curious why more people don't talk about using
| nsenter in the rare situation they need to inspect stuff in a
| running container?
|
| For a production microservice container, why have a bunch of
| bloat in the container that increases the surface area to manage
| while also only actually being used in edge cases?
| xuhu wrote:
| There is probably no nsenter equivalent for "kubectl exec
| thepod -- /bin/sh" if you can't ssh into the node.
| mook wrote:
| For non-production clusters, you may be able to play with
| ephemeral containers+ if turning on the feature gate seems
| reasonable.
|
| + https://kubernetes.io/docs/tasks/debug-application-
| cluster/d...
| gizdan wrote:
| Minimal containers shouldn't need that though. Your logs
| should tell you enough about the application it's running.
| K8s (get/describe) should tell you about the lifecycle of the
| pod. Lastly something like "dive" should tell you all you
| need about the image's file layout.
| gouggoug wrote:
| > Minimal containers shouldn't need that though.
|
| In theory. In practice this is only rarely the case.
| hvgk wrote:
| I shall look into nsenter. I have never actually used it. It
| may cover this use case.
|
| As for the status quo, I spend a lot of time extracting dumps
| from things in containers and debugging weird issues which is
| really really difficult without the ability to install any
| tools on the container at will. When done, we blast the
| container and let kubernetes reschedule a new one, disposing of
| all tools and state.
|
| On bloat, mine are sitting on top of a debian container layer,
| mostly because it's a hell of a lot easier dealing with some
| dynamic libraries than fecking around with static linking stuff
| and copying dependencies into a minimalist container.
|
| Edit: There is overlap between these two things which nsenter
| may not be able to solve looking at what I am reading.
| foxfluff wrote:
| How do you use nsenter efficiently for running more than one
| program? You might need dozens of tools (and all their
| dependencies) to diagnose and fix issues.
| yjftsjthsd-h wrote:
| If I understand your question correctly, you just make sure
| that the first command is nsenter bash
| Lhiw wrote:
| I aim for "small images" because downloading a full install of
| Ubuntu is rediculous on anything but gigabit fibre.
|
| But I don't aim for minimal images. Using alpine and installing
| what ever you like without worrying about all the bullshit to
| make caches stay empty still results in images only a few megs,
| don't need to do much better than that.
|
| Though I tend to agree, if you're talking about a compiled
| language. Static binary and there is little need for anything
| other than `from scratch`.
| machinerychorus wrote:
| This is a great article, I just have one tiny nitpick: please add
| the publication date!
|
| Information like this is very dependent on the context of when it
| was written, and something that is "best practice" or "an anti-
| pattern" will not always be that way. Adding a publication date
| and/or a "last updated" date to the top of the article is very
| helpful for readers to align the information in their minds.
|
| EDIT: I realized just after posting this comment that the
| publication date is in the url. Time for that second cup of
| coffee :)
| ithkuil wrote:
| You can run a small sidecar with a shell and if you enable shared
| process namespace you can access the filesystem of the main
| "distroless" container via /proc (e.g. /proc/123/root)
| BossingAround wrote:
| I think the difficulty arises in the fact that you can't spawn
| the sidecar without terminating a pod, as far as I know. So you
| either create a sidecar for every pod, "just in case", or you
| have to restart your pod for debugging purposes (and modify
| your YAML files).
|
| That is, if we're talking Kubernetes of course.
| ithkuil wrote:
| There is an alpha feature: https://kubernetes.io/docs/concept
| s/workloads/pods/ephemeral...
|
| Otherwise you have to prepare in advance and add a small
| sidecar to all pods that need it
| staticassertion wrote:
| The small image problem seems like it may be best solved through
| volumes.
|
| Consider that busybox is just under 1MB when dynamically linked
| and installed. That's roughly 50% larger than a distroless
| container.
|
| Further, distroless doesn't just target size for size's sake -
| it's also to ensure that an attacker in the container doesn't
| have tooling that they like to use.
|
| Instead if your containers are mounting a volume, say
| `/var/run/debug-tools/`, you could copy the tools into that mount
| before entering your containers. This probably has a good dev UX
| in that you can enter containers and have your tools, it
| amortizes the cost across your containers on a given host, and it
| isn't container specific ie: you perform one copy and now all
| containers on that host have it.
| bob332 wrote:
| This website ui is an anti pattern
| IshKebab wrote:
| Good list.
|
| I would add that using Docker in the first place is an anti-
| pattern. If you're using software you wrote yourself that can be
| made self-contained, e.g. if it's written in Go or Rust, then you
| probably don't even need Docker.
|
| People always say "but how can you isolate the network?" and use
| Kurbernetes and .. and sure Docker is the easiest way to do that.
| But in my experience people rarely use Docker for that feature
| anyway. 99% of Docker use is because most Linux software is
| written in such a way that it is an absolute pain in the arse to
| distribute so shipping a whole OS is the bodgy solution that
| everyone accepts.
| returningfory2 wrote:
| RE small images: advice like this always seems to ignore that
| Docker images are cached incrementally. If you build, say, a Go
| binary and then put it in a popular stock Debian or Alpine Docker
| image, pushing the final image to a registry will only transfer
| _the diff_ over the network. The size of the diff is basically
| the compressed Go binary and doesn't depend on the base image
| you're using!
|
| Then when you pull the Docker image onto a machine it's likely
| the base image is already in cache so again it's only the diff
| that is transferred.
|
| It's sometimes said this Alpine stuff is a premature
| optimization, but in most cases it's not even an optimization.
| justin_oaks wrote:
| While that's true, you or your company have to standardize on a
| base image to get benefit of shared image layers.
|
| In my job, I rarely have an exact match on the base layer
| because I'll use images from Docker Hub. Although many of those
| images use the same small set of distros (Alpine, Debian, or
| Ubuntu), any given image is likely from a different base
| version of that distro, thus there is no common base image.
|
| There may be differences in the base image even if you use the
| same tag since the same image may change over time. Thus if you
| pull debian:bullseye today, and then pull again tomorrow, you
| may have two different images.
|
| Generally I don't bother trying to get a common base image and
| I'm happy enough just attempt to lower the overall size of my
| images.
| nickjj wrote:
| > It's sometimes said this Alpine stuff is a premature
| optimization, but in most cases it's not even an optimization.
|
| I do use Debian Slim over Alpine without thinking but image
| size does matter in a lot of cases once you leave your dev box
| or start to use Kubernetes.
|
| For example if you're using AWS Fargate images aren't
| cached[0]. That means every time you deploy your app a new pod
| is going to get created and it has to wait for your image to
| download.
|
| Fargate already has a 30-40ish second spin up time inherit to
| the platform. Having to wait X seconds or even minutes more to
| download your app's image delays things quite a lot, especially
| if you have something like a pre-sync hook running to do
| something like running a database migration. Now every deploy
| pays this penalty twice.
|
| [0]: https://github.com/aws/containers-roadmap/issues/696
| varikin wrote:
| > It's sometimes said this Alpine stuff is a premature
| optimization, but in most cases it's not even an optimization.
|
| Not always. If you have thousands of containers running, the
| network cost* for changing the base image is massive because
| each instance needs possibly all new layers, not just the last
| layer or 2. Having a smaller base image is very practical at
| this point.
|
| * And for network cost, there is both the literal cost with
| egress from dockerhub, or another registry, but also the load
| on your network infra.
| p_l wrote:
| Actually using that caching in a way that benefits you is,
| well, very hard. Especially since a lot of the time you have a
| lot of stuff included in order to start the final program, _and
| every step above the base image will result in different layer
| that will be separate and probably not cacheable_
|
| Also, Alpine images are used for being, well, small in general
| - there's not much to exploit on such image, nor is there going
| to be a lot of unplanned-for data, and especially it will speed
| up deployment when you upload a new image.
|
| It takes special, dedicated CI/CD work to have common base
| images to exploit sharing of layers properly.
| BiteCode_dev wrote:
| Years ago, freshly out of school, I ended up in a job at a french
| small shop that required me to learn a lot of things.
|
| They had a brillant sysadmin, that I bothered to explain them to
| me. Sometimes, he'll come to my desk, looking at my screen,
| typing the solution while explaining the problem to me with
| simple words.
|
| Standing up, typing with one hand, and working on what was
| supposed to be my specialty, he was doing a better job than I
| was, despite me interrupting his workflow.
|
| I left the job, and levelled up. One day, I read an article about
| docker, and the name of the dev caught my eyes: it was the
| friendly former sysadmin colleague, Jerome Petazzoni, who went on
| becomming docker's biggest evangelist.
|
| Thank you Jerome, for making sure I started my carrer by not
| being the smartest guy in the room. I kept it as a habit.
|
| Or I'm just dumb :)
| klibertp wrote:
| > for making sure I started my carrer by not being the smartest
| guy in the room
|
| That's indeed invaluable. That should be the first advice to
| anyone starting out: if you find yourself being the smartest in
| a room, go find a new room. Right now.
| throwaway894345 wrote:
| I graduated college and started working at a local tractor
| manufacturer. In hindsight, this company had no idea how to
| make software.
|
| For instance, they required that all "software engineers"
| were from an accredited engineering program--at the time,
| none of the universities in the area had an accredited
| software engineering program, so they would hire electrical
| engineering majors who had no desire to write software but
| were looking for a way to get their foot in the door. Most of
| the EE majors had a single semester of C and that was their
| entire software qualification; however, I double majored in
| EE and CS and my passion was software. That said, when I
| graduated I didn't understand any of this--I just figured
| there was this successful company and there was so much I
| didn't know about software, so I thought I would learn loads
| there--and I did learn loads, but it was mostly about what
| not to do.
|
| One of my first projects was to take over an embedded C++/Qt
| project that had been bid out to some contractors in the
| Ukraine. The software was super buggy and the code was
| immensely convoluted. Test coverage was part of the contract,
| so it had tests, but even still, no one could change anything
| without breaking _everything_. It took me weeks of plotting
| the code paths to figure out that there were many paths
| through the code that were only exercised by tests--literally
| they were just trying to boost their code coverage metrics by
| adding code that was unreachable at production. I guess it
| didn 't occur to me that _professional software engineers_
| would be so bad at writing code, and that a large and well-
| known software company would hire them.
|
| Thankfully, after eventually gaining my entirely-nontechnical
| manager's trust, we were able to bring in a consultancy who
| helped me demonstrate that it would be _no more than half the
| cost_ to rewrite the code rather than try to fix all of the
| existing bugs _even if we never found other bugs or
| introduced a single bug of our own_. We did the rewrite and
| it was successful, but I was beginning to realize that in
| some overly simplistic sense, I was the "smartest" one in
| the room as it pertained to software development anyway.
|
| Ten years later, when I talk to my peers who stayed, many of
| them have been writing embedded C programs and various MatLab
| models and they still don't understand pointers or dynamic
| memory or git or anything outside of their very local niche.
| Which would be fine if they were happy, but now many want to
| move to another company but they lack the skills and
| confidence to get anything more than an entry level job at
| many other software companies--so much of their compensation
| is derived from their knowledge of their employer's business
| and processes rather than their technical acumen, so they
| would likely have to take a significant pay cut to move to
| another employer.
|
| Anyway, this is already a very long post despite
| oversimplifying many things, but I hope this illustrates what
| it looks like to be the smartest person in the room and why
| it's not good.
| arch-ninja wrote:
| I left the company I was part-timing at because after college
| I realized I was the smartest one in the room; my boss
| thought some of the code I wrote was "brilliant" and for this
| reason did not let me improve/fix any of it, which led to the
| company shipping a broken product to some very annoyed
| customers.
|
| If you are ever managing someone who is "your best employee"
| and they want to do something you think is useless, please
| ask them why and what the expected results will be. Also when
| you get that data, listen a little? If priorities clash do a
| balancing act: "If you can do X in Y time, do it. But if day
| D comes along, switch over to my priority".
| musicale wrote:
| TL;DR: using Docker is an anti-pattern
|
| > Big images
|
| > Requiring rebuilds for every single change
|
| > Forcing things to run in containers
|
| > Using overly complex tools
|
| > Building with Dockerfiles
| BossingAround wrote:
| Docker containers are easy to build in a way that's pretty bad.
| I don't think that necessarily implicates Docker containers as
| a bad tech. In the same way, you can write an _insert language
| here_ application that 's horribly unmaintainable, and that
| tells you nothing of the language.
|
| I don't like Docker as a company, but I can't imagine living my
| tech life without containers + Kubernetes, and Dockerfiles are
| really easy to write and read.
| justin_oaks wrote:
| While I won't address each item in your list. I'll say that I
| love Dockerfiles.
|
| They make it clear what steps occurred to arrive at the
| resulting image. For any image that is built from an open-
| source repository, I can find the Dockerfile, look at it and
| say "Oh, that's how they did it!"
|
| In the article it says that Dockerfiles might not be the best
| way to build your image. I think some of that may have come
| from before the days of multi-stage builds. With multi-stage
| builds you can use a lot of tools (like the Go compiler) at
| build time without including those tools in your final build.
|
| Can we do better than Dockerfiles as they are today? I'm sure
| we can, but can I appreciate what I have while knowing it could
| be better.
| throwaway894345 wrote:
| > Or, if you use Docker, you can use docker cp to copy some
| static tools (e.g. busybox) to a running container and check
| what's going on.
|
| I tried this recently with `kubectl cp` for my scratch image and
| it turns out that command will fail if the target container lacks
| a `tar` program in its PATH. I guess under the covers it's just
| doing a `kubectl exec <pod> -- /bin/tar ...` on the file rather
| than asking the kubelet to untar the file into the target
| container's filesystem.
| [deleted]
___________________________________________________________________
(page generated 2021-12-02 23:03 UTC)