[HN Gopher] Immutable Software Deploys Using ZFS Jails on FreeBSD
___________________________________________________________________
Immutable Software Deploys Using ZFS Jails on FreeBSD
Author : vermaden
Score : 159 points
Date : 2025-11-08 00:21 UTC (22 hours ago)
(HTM) web link (conradresearch.com)
(TXT) w3m dump (conradresearch.com)
| fukka42 wrote:
| Isn't this just docker with extra steps?
| ssl-3 wrote:
| No.
|
| ZFS has been stable in FreeBSD for something like 17 years, and
| FreeBSD jails have been around for something like 25 years.
|
| By the time Docker hit 1.0 (about 11 years ago), the use of
| snapshots and jails had already been normal parts of life in
| the FreeBSD space for over half of a decade.
| IgorPartola wrote:
| I never find arguments like this compelling (but I agree with
| your sentiment). I don't much care to fire up the time
| machine to go back 12 or more years to develop software
| today. If your argument is that ZFS and jails provide the
| same functionality but are more stable than Docker. But as is
| it comes off as "get off my lawn you young whipper snappers".
|
| But at the same time, the reason Docker won was not because
| it was groundbreaking tech or because it was amazingly well
| tested or anything. Just as one example, it has a years old
| bug which actively gets more comments every week having to do
| with Docker grossly mishandling quotes in env files.
|
| No, the reason it won is because the development experience
| and the deploy experience is easy, especially when you are on
| Linux AND on macOS. I can't run FreeBSD jails or ZFS on
| macOS, can I? Definitely not with one file and one command.
|
| Jails and ZFS are amazing tech but they are not _accessible_.
| Docker made simple things very simple while being cross-
| platform enough. Do I feel gross using it? Yeah. It's a
| kludgy solution to the problem. But it gets the job done and
| is supported by every provider out there. I am excited that
| it is being ported to FreeBSD though I know it will be a very
| long process.
| stingraycharles wrote:
| Yeah I would love to use FreeBSD jails with ZFS and
| everything, it's just that the whole cloud and
| containerization thing happened based on Linux and FreeBSD
| just never made it into that ecosystem.
|
| You'll be sacrificing a _lot_ and have to hand-roll a _lot_
| if you want your organization to switch from Linux+docker
| to FreeBSD+jails
| jonhohle wrote:
| What about FreeBSD+OCI containers
| https://freebsdfoundation.org/blog/oci-containers-on-
| freebsd...
| nesarkvechnep wrote:
| In my opinion this is the path forward. I can already
| imagine Hashicorp Nomad orchestrator, with the podman
| driver, running fleets of FreeBSD containers.
| randmeerkat wrote:
| Hacker News serves +4M requests per day using nothing but
| two FreeBSD servers...
| kbenson wrote:
| It's all just history now for all I know, but there was
| work in the past to make Linux containers work on a
| Solaris fork (SmartOS, specifically) by emulating the
| Linux syscall table and presenting that to the
| containers. Joyent did work on this (alas, and there's an
| excellent and entertaining talk from Bryan Cantrill[1]
| that goes over it.
|
| I imagine FreeBSD could do something similar if they
| aren't already. IIRC FreeBSD has a Linux emulation layer
| (but I don't know how much attention it still gets), and
| it's had containerization primitives longer than linux,
| so some amount of filling in the gaps in containerization
| features and syscall tables (if needed) could possibly
| yield an OCI compatibility later (for all I know all this
| already exists).
|
| The problem, and the reason if this doesn't exist why
| people probably weren't as interested in doing the work,
| is it would always be "mostly" compatible and working and
| there would be no guarantee that the underlying software
| wouldn't exhibit bugs or weird behavior from small
| behavior differences in the platform when emulating
| something else. Why open yourself up to the headache when
| you can just run Linux with containers or build what you
| want on FreeBSD with jails and their own native
| containerization primitives.
|
| 1: https://www.youtube.com/watch?v=coFIEH3vXPw
| lukeh wrote:
| Yeah, emulating syscalls is fine until it isn't. See
| WSL1.
| baq wrote:
| Some kernels are more similar to others, some are less.
| Turns out NT is less similar to Linux than required for
| good performance. I wouldn't be surprised if Solaris was
| similar enough given that Linux tries to be Unix-like and
| Solaris is actually Unix.
| toast0 wrote:
| I think there's FreeBSD images for all the clouds now.
|
| You would need to do more work yourself to fetch and run
| jails probably, and I don't know if there's a hosted
| repository of 'jail images', but in return, you'd
| probably have a nicer system (at least, I'd like such a
| system more than running containers on google container
| optimized linux)
| righthand wrote:
| Bastille has "templates":
| https://bastillebsd.org/templates/
| badgersnake wrote:
| Not Hetzner, sadly.
| MathiasPius wrote:
| You can always upload your own, it's pretty simple doing
| so in a reproducible manner using something like Packer,
| but even without it you can just boot a VM into a rescue
| system, write your OS of choice to the VM disk and
| reboot.
| tiltowait wrote:
| Is it doable on a VPS? The documentation made it sound
| like it was for dedicated only. I went with OVH for this
| reason (which was cheaper anyway).
| BobbyTables2 wrote:
| I think it also helps that Docker started with an enormous
| amount of VC funding because their promise was to bring the
| "App Store" to Linux and enterprise servers.
|
| Who couldn't become famous with something like a $200M
| budget?
|
| Feel like they spent it on marketing instead.
|
| Podman is arguably technically superior yet people stay
| with Docker out of habit...
| baq wrote:
| Docker engine is one thing, but access to docker hub
| without rate limits is what people actually pay for if
| they're too cheap to host their own proxy registry (which
| everyone except the smallest companies should
| regardless).
| perrygeo wrote:
| Docker was the first viable containerization technology on
| Linux. Despite the 15 year late start vs FreeBSD Jails,
| it's certainly winning by the numbers.
|
| But that has nothing to do with their respective UXs. It's
| a Linux vs FreeBSD signal.
| cyphar wrote:
| Not really, OpenVZ was/is really quite good, it was just
| hampered by the fact it requires out of tree modules. LXC
| has also always been very usable (Docker even used it for
| several years) but it was IMHO too focused on the VM-like
| management scheme that Zones and Jails had.
|
| Docker's killer selling point was that it solved a very
| common and specific developer problem, not that it
| provided operational improvements over the state of the
| art on Linux. From an _operational_ perspective, Docker
| has generally been a downgrade compared to LXC. (I say
| this as a maintainer of runc, the container runtime that
| underpins Docker, and as someone who worked a lot on
| Docker back in the early days and somewhat less today.)
| hnlmorg wrote:
| > Docker was the first viable containerization technology
| on Linux.
|
| No it wasn't. Docker was late to the party even for Linux
| (and Linux was late compared to every other "UNIX").
|
| OpenVZ was around for years before docker. Its main issue
| was that it required out-of-tree kernel code. But there
| were some distributions that did still ship OpenVZ
| support. In fact it's what Proxmox originally used. And
| it worked very well.
|
| Then LXC came along. Fun fact, Docker was originally used
| LXC itself. But obviously that was a long time ago too.
|
| I've used both OpenVZ and LXC in production systems
| before Docker came along. But I've always preferred
| FreeBSD Jails + ZFS to anything Linux has offered.
| blahgeek wrote:
| > especially when you are on Linux AND on macOS. I can't
| run FreeBSD jails or ZFS on macOS, can I? Definitely not
| with one file and one command.
|
| On macOS, docker actually launches a Linux VM to run
| containers. If this counts, then yes, you can run FreeBSD
| jails or zfs on macOS, by running a FreeBSD VM.
| IgorPartola wrote:
| But it works with one command and one (Docker)file.
| That's what I mean by Docker being a kludgy solution:
| this is way less than ideal. But for developer experience
| this is very nice. And that same Dockerfile runs on
| everything from AWS, to GCP, to k8s, to Dokku, etc.
|
| I dislike the implementation but I cannot deny that the
| UX is good enough to be very popular.
| ssl-3 wrote:
| The argument was just "No [it is not the same]." The rest
| was just some supporting facts. :)
|
| You've got a good take on things, and I do not disagree
| with what you've written.
| Arch-TK wrote:
| You can't use Docker on Mac or FreeBSD. Now, I am not
| calling you a liar, you _can_ use Docker on Mac and
| FreeBSD. But you would probably only want to do so for
| development, as Docker on Mac and FreeBSD requires running
| a Linux VM which is the thing which _actually_ runs the
| containers.
|
| There is work ongoing to try to make this more native on
| FreeBSD (by using Linux jails) but that work is not
| complete yet.
|
| So, if you want to get the same kind of experience as
| Docker on FreeBSD, you are forced to use jails.
|
| The only reason Docker seems accessible is because it's
| native to the platform people seem to like for running all
| their services, but if you're dealing with FreeBSD, you
| most certainly would not just "use Docker" to deploy your
| stuff. Because you would get worse performance than if you
| had just used Linux.
|
| So the answer to "Isn't this just Docker with extra steps?"
| is truly and absolutely "No". Not because of some kind of
| old man shouting at cloud argument, but because if you are
| on FreeBSD (for whatever reason that might be) you can't
| just use Docker as an easier replacement for Jails (at
| least right now).
| atmosx wrote:
| MacOS runs containers spinning VMs. That's why it's slow
| (processing and filesystem I/O) compared to Linux.
|
| Same thing could be done in FreeBSD. Someone needs to put
| in the work...
| chuckadams wrote:
| One of Docker's big advantages is its client/server
| architecture made it easier to run on all those macOS and
| Windows boxes by just spinning up a VM for the server side,
| then making that VM fully managed to where a large percent
| of users don't even know it's there.
| soupbowl wrote:
| This is more like making an immutable linux container using
| only OS base tools. Docker is a whole stack doing the work for
| you.
| miladyincontrol wrote:
| I have to imagine systemd's nspawn with btrfs integration
| took much inspiration. Combined with systemd's service
| configuration it really makes a wonderful way of running
| distroless, immutable containers.
| arminiusreturns wrote:
| I second systemd-nspawn being a hidden gem for this
| usecase. I use git post-recieve hooks that target it for
| much of my ci/cd pipelines.
|
| I also find myself using nspawn just to isolate apps like
| firefox, etc.
| righthand wrote:
| This is like using Lxc with fewer steps and less tape.
|
| FreeBsd has jail managers aka container managers aka "Docker"
| as well.
| righthand wrote:
| Try Bastille FreeBSD!
| soupbowl wrote:
| For those unfamiliar with FreeBSD, this is using base OS tools to
| manually create this type of immutable jail/container. This can
| be done with 'less effort' by using a jail manager.
| pacija wrote:
| Jail managers come and go. Base OS tools stay and are getting
| better and better. I would definitely stay away from ezjail as
| it us quite old, active development or even maintenance has
| stopped long time ago.
|
| Author of the article seem to know what they are doing so I'm
| puzzled why they don't use `bsdinstall jail /path/to/jail` to
| implement basejail instead of manually unpacking archives.
|
| No need for separate custom rc script to start `lo1`, it can be
| done with `cloned_interfaces` directive in rc.conf.
|
| Updating and upgrading jails by passing `-b /path/to/jail` to
| `freebsd-update` works, but new recommended way has lately been
| `-j <jailname>`.
|
| Cool article overall, the beauty of FreeBSD is also in
| possibility to do things in many different ways.
| AdieuToLogic wrote:
| The ezjail[0] port is another option for achieving the article's
| stated goal: FreeBSD's native support for ZFS
| snapshots and jails provides a powerful foundation for
| immutable deployments.
|
| I have not used the article's tool(s) and am not comparing the
| functionality provided by each. I have used ezjail[0] and found
| it exceptionally useful for similar concerns.
|
| 0 - https://erdgeist.org/arts/software/ezjail/
| fsmv wrote:
| I built a reverse proxy server for this kind of usecase
| specifically. I have a chroot launcher with a simple config file
| and the reverse proxy is auto configured over RPC and
| automatically acts as a CA root so internal communication is
| HTTPS. You can also set the API key as an env var and just run
| your binary on the command line for testing and it will auto
| configure and be available on your URL with https.
|
| https://github.com/fsmv/daemon/
|
| It's a bad time for me to be mentioning it because I have a major
| update that's not quite ready to release that changes some client
| APIs and makes the whole thing much nicer with fully automatic
| lets encrypt. I haven't had the space to work on it for a while
| unfortunately.
| nesarkvechnep wrote:
| I hope OCI containers support make this boilerplate obsolete.
| soupbowl wrote:
| It already is, if you use most jail managers, this post is a
| manual approach.
| nesarkvechnep wrote:
| I really don't like jail managers because I need to learn a
| non-standard way to operate jails when I already know the
| manual approach. I just wouldn't miss the boilerplate. Also,
| OCI containers > jail managers.
| cyphar wrote:
| FYI, we just merged FreeBSD jail support into the OCI runtime
| spec v1.3[1]. There is already at least one implementation of
| it[2] as well.
|
| [1]: https://github.com/opencontainers/runtime-spec/pull/1286
| [2]: https://github.com/samuelkarp/runj
| righthand wrote:
| If you want to do this kind of thing but don't like the looks of
| this indepth tutorial, I highly reccomend installing Bastille
| FreeBSD instead and let the bastille tooling do the heavy lifting
| of managing zpool cloning and what not. Their getting started
| tutorial should get you 90% of the way there and has Docker like
| tooling. [0]
|
| Also found this self host on a Raspberry Pi helpful.[1]
|
| [0] https://bastillebsd.org/getting-started/
|
| [1] https://www.sharpwriting.net/project/bastille-jail-
| managemen...
| cullumsmith wrote:
| I highly recommend this approach. I run ~all of my digital
| footprint on FreeBSD jails. After surveying how many jail
| managers have come and go in the last decade, I decided to just
| roll my own using a single shell script called jailctl [0].
|
| Nothing fancy, just VNET jails based on ZFS templates (vanilla
| FreeBSD rootfs) and epair interfaces (which I truck to various
| VLANs on the host's egress interface).
|
| One pattern that I've found useful is to give each jail a
| persistently delegated ZFS dataset called "data." This lets me
| reprovision the OS image for the jail without having to backup
| and restore its application data (such as a Postgres DB). It also
| allows each jail to manage its own ZFS snapshots.
|
| The only thing that was a bit hairy was generating unique
| interface names and MAC addresses for each jail's VNET interface.
| My first instinct was to derive the interface name from the jail
| name, but interface names on FreeBSD are limited to 15
| characters, and occasionally I'd hit this limit.
|
| In the end I did some dark magic using md5 sums of the jail name
| / host interface MAC address. Kind of ugly but I really didn't
| want to introduce any dependencies besides /bin/sh.
|
| [0]
| https://github.com/cullumsmith/infrastructure/blob/master/fi...
| ritcgab wrote:
| Very neat!
| inopinatus wrote:
| I used to do something like this but with ZFS on an OpenSolaris /
| Illumos storage server, exporting copy-on-write clones of
| snapshots of iSCSI volumes to boot Xen VMs on neighbouring
| blades, with tagged VLANs from each host because there were
| multiple guests to launch with varying roles in the application
| cluster. We made the VLAN number match 12 bits of the IP
| addresses and numbered the clones similarly. It merely remained
| to create new snapshots every release and any dev could launch an
| bugfix/test/showcase/etc environment for that version, and
| connect to it via a VPN. I was always worried about scale if we
| hired more than 4096 developers but fortunately the company was
| acquired and its product discontinued before that happened.
|
| That was in 2007 so the control plane (scheduler and automation)
| were built from scratch and we had very few reference points for
| the overall design. If I was building that today I'd probably
| still use ZFS clones but at filesystem level instead of block
| devices, and serve jails over NFS if I can get away with it, the
| iSCSI part was always a little janky.
| evanjrowley wrote:
| Detailed writeup here: https://docs.skunkwerks.at/s/fUiAmi4pE
___________________________________________________________________
(page generated 2025-11-08 23:01 UTC)