[HN Gopher] BuildKit in depth: Docker's build engine explained
       ___________________________________________________________________
        
       BuildKit in depth: Docker's build engine explained
        
       Author : kylegalbraith
       Score  : 79 points
       Date   : 2024-02-06 17:31 UTC (5 hours ago)
        
 (HTM) web link (depot.dev)
 (TXT) w3m dump (depot.dev)
        
       | inetknght wrote:
       | Buildx also has a completely different build output and it's
       | completely useless for anything. Use `--output=plain`.
       | 
       | Also, buildx won't commit the intermediate layers during the
       | build. So if something fails, you can't just grab the previous
       | intermediate layer and do `docker run --entrypoint /bin/bash` on
       | that layer to poke around.
       | 
       | I've had cases where BuildKit will get stuck and `docker buildx
       | prune -a` will leave things around. The way to fix it is with
       | `systemctl restart containerd.io`. This seems to happen _a lot_
       | when I interrupt the build (eg, I know it will fail and need to
       | restart it with a fix), but it 's also happened on successful
       | builds.
       | 
       | All of that is on top of docker itself having extremely poor
       | error messages.
       | 
       | I really do not like docker or buildx.
        
         | adamgordonbell wrote:
         | If you build your dockerfile with earthly, we have a -i flag
         | that will pop you into a shell on a failure, and we are using
         | buildkit, so it's totally possible.
        
         | 4oo4 wrote:
         | Well said, these are the reasons I use podman/buildah now.
        
           | EdwardDiego wrote:
           | Was it a drop in replacement or did you need to give things a
           | jiggle to get it going?
        
             | Narushia wrote:
             | I've had a good experience with Podman as a Docker
             | alternative. The Podman CLI is designed to be API-
             | compatible with Docker's CLI, so most things should just
             | work. The only thing I've had to change with my Dockerfiles
             | (or Containerfiles) is the `SHELL` instruction, since
             | Podman by default builds OCI images, and that instruction
             | is not supported by the OCI standard. But you can still
             | specify the shell individually for the `RUN` instructions,
             | or even just use `--format=docker` if you don't want OCI
             | images.
        
         | cpuguy83 wrote:
         | "BUILDX_EXPERIMENTAL=1 docker buildx build <args> --invoke
         | /bin/sh"
         | 
         | There's actually a whole debugging client available.
        
         | mdaniel wrote:
         | and its "oh, you want multi-arch, do you?" friend. While
         | prosecuting this <https://gitlab.com/gitlab-
         | org/gitlab/-/issues/339567> I learned that they bundle binfmt
         | into the buildkit docker image <https://github.com/moby/buildki
         | t/blob/v0.12.5/Dockerfile#L11...> which itself is compiled
         | against a static copy of qemu <https://github.com/tonistiigi/bi
         | nfmt/blob/deploy/v6.2.0-26/D...> with various
         | /proc/sys/fs/binfmt_misc wizardry <https://github.com/tonistiig
         | i/binfmt/blob/deploy/v6.2.0-26/c...>
         | 
         | We ended up switching to using multiarch/qemu-user-
         | static:7.2.0-1 <https://hub.docker.com/layers/multiarch/qemu-
         | user-static/7.2...> which also mutates binfmt_misc in buildx's
         | context in order to exec the static copy of qemu in it but I
         | find their shell script much more legible about what's going
         | on: <https://github.com/multiarch/qemu-user-
         | static/blob/v7.2.0-1/...> not to mention the fact that it more
         | clearly represents the recent qemu version versus secreting it
         | away inside the moby/buildkit:heheh-good-luck-friend
         | 
         | They have an entire GH label for that:
         | https://github.com/docker/buildx/labels/area%2Fqemu and my
         | journey into the buildx sewers started from
         | https://github.com/docker/buildx/issues/1170
        
       | adamgordonbell wrote:
       | I did not know about the CargoWharf frontend!
       | 
       | Writing your own front end to BuildKit can be pretty simple. At
       | my work, this is sort of a starter task for everyone who joins
       | the team.
       | 
       | My frontend was based on intercal and I don't recommend anyone
       | use it but it was fun to play around with[1].
       | 
       | [1]: https://github.com/adamgordonbell/compiling-
       | containers/tree/...
        
         | notso411 wrote:
         | Really? Just busy work? Senior engineers incapable of thinking
         | of good onboarding tasks?
        
           | adamgordonbell wrote:
           | Nope, i mean i can see how youd think that, but familarity
           | with BuildKit is just part of the job.
           | 
           | We are building on top of it, and sometimes adding to it.
        
       | nicornk wrote:
       | So how did you ,,further optimized [buildkit] to build Docker
       | images up to 40x faster"
        
         | maxmcd wrote:
         | hopefully depot will reply, but from my perspective it is
         | mostly laid out on their homepage. they are comparing against
         | builds in other CI products that use network-backed disks,
         | virtualized hardware, and don't keep a layer cache around.
         | Depot provides fast hardware and disks and is good at making
         | the layer cache available for subsequent builds.
         | 
         | You could likely get very similar performance by provisioning a
         | single host with good hardware and simply leverage the on-host
         | cache.
        
       ___________________________________________________________________
       (page generated 2024-02-06 23:00 UTC)