[HN Gopher] FreeBSD Jails for Fun and Profit
___________________________________________________________________
FreeBSD Jails for Fun and Profit
Author : kettunen
Score : 228 points
Date : 2021-12-22 13:34 UTC (9 hours ago)
(HTM) web link (topikettunen.com)
(TXT) w3m dump (topikettunen.com)
| drclau wrote:
| FreeBSD Jails were so much better than everything else out there,
| for a long time. I'll just copy&paste part of a comment I wrote
| on another HN thread some time ago, since it's relevant here:
|
| [...] In fact, many years ago, when FreeBSD was my main OS
| (including on notebook) I went as far as to isolate each app that
| used internet into its own custom-setup jail [0][1]. I had
| Firefox, Thunderbird, Pidgin and a few others running in complete
| isolation from the base system, and from each other. I even had a
| separate Firefox jail that was only allowed to get out via a Tor
| socks proxy to avoid leaks (more of an experiment than a
| necessity, to be fair). Communication between jails was done via
| commonly mounted nullfs. I have also setup QoS via PF for each of
| them. They were all running on the host's Xorg, which was
| probably also the weakness of this setup. It was a pretty sweet
| setup, but required quite a bit of effort to maintain, even tho I
| automated most of the stuff. [...]
|
| The original comment is here:
| https://news.ycombinator.com/item?id=27709256
| vegai_ wrote:
| Is there a reason why FreeBSD doesn't default to running all
| applications in jails? Seems like this would be a pretty huge
| advantage compared to the typical Unix system's almost complete
| lack of sandboxing.
| kazen44 wrote:
| what should run in a jail? you need the base system to run
| jails. and that is pretty much the only thing that is
| installed.
| rsync wrote:
| "... I even had a separate Firefox jail that was only allowed
| to get out via a Tor socks proxy to avoid leaks ..."
|
| I have looked into doing this many times and it's neither
| simple nor straightforward.
|
| Specifically: jailing a _GUI app_ that you can interact with on
| your desktop.
|
| I can't remember what the most promising recipe I saw for this
| was but it wasn't quite promising enough to compel me to built
| it up ... and this discussion is always (rightly) hijacked with
| "just use Qubes" ...
| VTimofeenko wrote:
| Firejail with spawning nested Xorg works fine for me,
| including text-only copypaste between "host" and "guest" and
| automatic file synchronization through bind-like mounts. For
| some firejails I also use Linux network namespaces to control
| traffic going through taps. My introduction to this approach
| was the alternative Gentoo handbook by Sakaki[1], but the
| principles would apply on any distro.
|
| There's also a very interesting read on Qubes-like experience
| on NixOs with Wayland and XWayland[2,3].
|
| [1]: https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_
| Inst...
|
| [2]: https://roscidus.com/blog/blog/2021/03/07/qubes-lite-
| with-kv...
|
| [3]: https://roscidus.com/blog/blog/2021/10/30/xwayland/
| haroldp wrote:
| Bastille template to bootstrap Firefox:
|
| https://github.com/ddowse/jailfox
| ajross wrote:
| Yes, but in a sense that's the essence of why the technology
| got left behind. Jails were a mechanism for expert admins to
| play with container ideas.
|
| What the market actually wanted was Docker. And what Docker
| needed was Linux containers (complicated, flexible, piecewise
| technology) and not jails, which were higher level abstractions
| (but yet not high enough) with jargon and framework assumptions
| that didn't match Docker's needs 1:1.
| pjmlp wrote:
| And slowly the only thing left from Docker will be the image
| format.
| GordonS wrote:
| And the Registry API too, I expect, but yeah, the untimely
| demise of Docker as a company feels inevitable at this
| point.
| StreamBright wrote:
| The "market" did not want Docker. Docker as a product failed.
|
| There are many reasons why FreeBSD jails count not get out
| for FreeBSD land, one, very important thing is the Linux
| community's NIH attitude.
| shatteredgate wrote:
| I don't think you can explain that as NIH. From what I have
| seen, Linux namespaces are a much more powerful primitive
| than BSD jails.
| kazen44 wrote:
| namespaces in what sense? network namespaces? because
| freebsd has VNET for a while now. which seems to do the
| same thing.
| shatteredgate wrote:
| And all the other ones: https://man7.org/linux/man-
| pages/man7/namespaces.7.html
|
| BSD jails are similar but not quite the same thing.
| ori_b wrote:
| I don't get it. How are people using this flexibility to
| get things done in practice, and what uses aren't allowed
| by the jail model?
| shatteredgate wrote:
| You can just compare the APIs, namespaces are like the
| individual components of a jail. You can use them to
| build something like a jail, or something different that
| has a different security model. This was discussed a lot
| in an old HN thread:
| https://news.ycombinator.com/item?id=13982620
| ajross wrote:
| > How are people using this flexibility to get things
| done in practice
|
| Um... to loop back to the upthread point: Docker. People
| are using Docker, and docker is using this stuff.
| redis_mlc wrote:
| cnst wrote:
| Sounds pretty cool! TBH, I don't understand how the "modern"
| app security models are not deemed to be fundamentally insecure
| in 2021 where any random app on any random system can simply
| leak your whole photo library to their servers, or delete all
| your files. Why do banking apps need full access to my
| filesystem? Why don't the reviewers of the app stores prohibit
| such practices of excessive permissions?
|
| Yes, there's always a question of usability, but if you're an
| advanced user, it's still scary that you're not afforded any
| control to prevent these incidents unless you go ahead and
| redesign the whole way all these apps are working all by
| yourself.
|
| It seems terribly inefficient if every engineer has to do it on
| their own, and in their own incompatible way. Obviously most
| people simply give up after a while, since maintaining such a
| setup might itself be a whole full-time job.
| pjmlp wrote:
| I beg to differ, given that HP-UX Virtual Vaults were quite
| alright and precede Jails.
| drclau wrote:
| (noticed your reply to my other comment too, please consider
| this a reply to both)
|
| I wasn't aware of HP-UX Virtual Vaults, thanks. However, I'd
| say FreeBSD Jails still had an advantage, due to being free
| and running on various hardware platforms, and more
| importantly on commodity hardware.
| pjmlp wrote:
| Eventually they were replaced by HP-UX Containers (SRP) a
| couple of years later.
|
| Unfortunately it is hard to still find documentation, given
| the troubles HP-UX has gone through at HP (which kind of
| plays into your remark regarding FreeBSD).
|
| Still there you go, https://support.hpe.com/hpesc/public/do
| cDisplay?docLocale=en...
| hestefisk wrote:
| In the same way LPARs preceded Virtual Vaults on System z? :)
| pjmlp wrote:
| I guess so, but I was constraining myself to UNIXes. :)
| qqumut wrote:
| Are Jails really that safe & secure?
| bigodanktime wrote:
| Counts what you are afraid against. There's always some side
| channel attack that could possibly used to gain information,
| even on VM's this is true. Off the top of my head there could
| be some timing attack to gain information on which libraries
| others are using by reading in libraries and seeing if they are
| warm in the buffer cache, counts if you care about sharing the
| same kernel. I generally find them secure enough considering
| how fast they can be brought up and down.
| antihero wrote:
| Can a process in them exec on the rest of your system?
| marcosdumay wrote:
| The fundamental thing about those features (and the
| equivalent on every system except Windows) is that you can
| never get more capabilities, only less. Once you are in a
| jail, there is no API for getting out of it.
|
| You can't even see a binary from the rest of your system,
| and exec won't get you out.
| tptacek wrote:
| They're approximately as safe as modern Docker is. Upside to
| Docker: more security knobs (eBPF, kernel MAC, &c); upside to
| jails: probably easier to get right out of the box, fewer
| footguns. Both jails and containers (and Solaris Zones) share a
| fundamental security weakness, which is a kernel shared between
| tenants.
| freemint wrote:
| I never understood the appeal of BSD jails over Solaris zones
| which seem to be more hardened and seem to vitualize more of the
| OS.
| laumars wrote:
| FreeBSD is the appeal.
|
| Back in the earlier days of containerisation Linux had no
| options (Linux was pretty late to that particular game) and
| Solaris wasn't free. So FreeBSD made a lot of sense.
|
| These days the tooling around Linux is better and there are
| open source forks of Solaris so FreeBSD might seem like an odd
| choice for some. However I still think FreeBSD is a rock solid
| operating system and one that doesn't get taken as seriously
| these days as it should do.
| AndrewUnmuted wrote:
| FreeBSD's appeal for me is the easy maintenance. Everything
| related to config is in /etc/rc.conf; tunables in
| /boot/loader.conf. The ZFS implementation is rock-solid. I
| therefore only have to fiddle with my FreeBSD server, running
| 80TB ZFS storage pool, once a year or so.
|
| No other OS gives me this kind of comfort and stability.
| freemint wrote:
| I wonder if Solaris has bigger userbase it would also
| appeal to you.
| AndrewUnmuted wrote:
| Perhaps it would! But the userbase is definitely what
| finally got me into FreeBSD. The fact that hobbyists were
| contributing such excellent cli tools as `iocage` [0] and
| `vm-bhyve` [1] really is what got me over my fears.
|
| [0] https://iocage.io/
|
| [1] https://github.com/churchers/vm-bhyve
| deltarholamda wrote:
| Way back when OpenSolaris was around, I put it on a
| server to experiment with. There was a lot to like about
| it, but one thing I didn't like was there was a lot of
| obvious cruft that has built up over the years. Things
| were in places that I didn't expect, but they had been
| there since 1904 when people wrote C with a quill pen, so
| SUN couldn't just move it around, because it would break
| MasterCard or something.
|
| The base FreeBSD system is well thought out, and because
| it had the ability to, it coalesced into a very coherent
| system. Man pages are great, everything is where it
| should be.
| lordgroff wrote:
| While I'm thoroughly a Linux person, I agree. I understand
| why Docker and the Linux container won the day, but the fact
| that FreeBSD is one system helps a ton with the overall
| intelligibility of the system inside that container. Oh, this
| container is Debian based, this is Centos, that one is
| busybox... No, every jail is just a FreeBSD system you know
| what you get for its bones. That's really nice.
| tomc1985 wrote:
| Pretty sure you can run Debian or Busybox as a FreeBSD jail
| if you want
| trasz wrote:
| Shameless plug: https://wiki.freebsd.org/LinuxJails
| drclau wrote:
| Note that FreeBSD Jails were introduced in 1999, while Solaris
| Containers and Zones were introduces in 2004. At the time
| FreeBSD Jails were introduced, probably the only alternative
| that was wildly available was chroot, which is really far from
| what Jails offer. Full virtualization was too slow to be
| practical for most scenarios, back then. 1999 is the year when
| Pentium III was released.
| betaby wrote:
| > probably the only alternative that was wildly available was
| chroot
|
| Opensource Linux-vserver https://en.wikipedia.org/wiki/Linux-
| VServer was available in 2001. At the same era commercial
| Virtuozzo was ubiquitous amount hosting providers.
| pjmlp wrote:
| Bringing other UNIX into the party, HP-UX Virtual Vaults,
| which were introduced in HP-UX 11, available in 1997.
| freemint wrote:
| I've seen nothing which would suggest that Solaris zones are
| more taxing than BSD jails. Did you imply that claim?
| drclau wrote:
| > Did you imply that claim?
|
| No, I didn't. I was comparing full virtualization to Jails.
| However, I realize now that there may not even have been a
| full virtualization solution available back then anyway, at
| least not for consumer hardware. I'll have to dig a bit on
| Wikipedia.
|
| I've never used Solaris Containers / Zones, but my
| understanding is that the implementation was similar to
| FreeBSD Jails, so I have no reason to believe the
| performance was different.
| shrubble wrote:
| The issue with zones was the lack in the early days of GNU
| software and the difficulty in compiling certain things.
|
| For instance, awk can act differently and if you don't know the
| 25 year old decisions that led to the differences, it can be
| very confusing. (There are different behaviors and command line
| switches between GNU's AWK and the version from SVR4/XPG)
| eddieh wrote:
| Lack of GNU stuff would be a selling point.
| zokula wrote:
| eddieh wrote:
| Wow. Name calling, especially such wholly inappropriate
| and inflammatory name calling, is way outside the norms
| here on HN.
|
| With brand ambassadors like yourself is there any wonder
| why I and many others want nothing to do with GNU or the
| FSF?
| fsflover wrote:
| Same here, but I never understood how BSD jails or Solaris
| zones are better over normal hardware virtualization which is
| used in Qubes OS. In addition, you get a great UX in the
| latter.
| laumars wrote:
| Containerisation makes much more sense than virtualisation in
| a lot of areas:
|
| * lower runtime overhead
|
| * quicker deployment time
|
| * smaller image foot print
|
| Plus most of the advantages of VMs can still be applied.
|
| However it does massively depend on individual use cases.
| Qubes aims to be ultra secure and containers in Linux weren't
| up to that task at the time (the situation has since improved
| massively).
| StreamBright wrote:
| - lower runtime overhead
|
| - quicker deployment time
|
| - smaller image foot print
|
| Firecracker beg to differ. I think you can do
| virtualisation and have all three you listed.
| laumars wrote:
| Sure. But for every edge case where virtualisation
| bridges the gap, there will be containerisation solutions
| that further the gap too.
|
| Also I'm not really in the mood to engage in a dumb flame
| war. I'm just answering the question as to why some
| people favour containers for some workflows. If
| firecracker works for you then keep at it.
| throwawayboise wrote:
| With modern hardware support for virtualization, is runtime
| overhead still a big deal (assuming your virtual
| environments are not emulating some other CPU?)
| laumars wrote:
| Depends on the industry. For general purpose computing
| it's generally not an issue but it does matter for some
| specific workflows.
|
| I tend to look at it the other way though: given
| containerisation is so easy these days, what are the
| compelling reasons to run virtual machines.
|
| Neither is a wrong answer though.
| throwawayboise wrote:
| VMs still provide better isolation and security, or is
| that no longer true?
| gaadd33 wrote:
| I don't think Qubes OS existed around 2000, did it?
| fsflover wrote:
| I was asking about what to use now.
| freemint wrote:
| Also OS level emulation. Linux system calls can be translated
| to Solaris system calls allowing to run Linux specific
| workloads without having to emulate hardware and a full Linux
| kernels. Furthermore before Dtrace was ported Linux it
| allowed to debug Linux workloads under DTrace.
| trasz wrote:
| You can do both with FreeBSD in the exact same way.
| freemint wrote:
| I am fully aware. Parent was probably not.
| luto wrote:
| (FreeBSD) jails are amazing. I just wish there were easier ways
| to use them more "cattle"-like, so I can augment or replace
| Docker/Podman. At the moment tooling and many of the real-world
| setups remind me a lot of "pet" LXC containers or even VMs in the
| Linux world.
|
| The tooling is slowly moving in a direction I like, though :)
| heresie-dabord wrote:
| On a Debian GNU/Linux or derivative:
|
| apt-cache search jail
|
| firejail - sandbox to restrict the application environment
|
| firejail-profiles - profiles for the firejail application
| sandbox
|
| firetools - Qt frontend for the Firejail application sandbox
| stjohnswarts wrote:
| Are they superior to firejail on linux? I kind of always
| figured they were similar level of "sandboxing" but I never had
| enough interest in BSD to dig in myself.
| trasz wrote:
| They are completely different mechanism for doing different
| kind of stuff. Firejail sounds like something closer to
| Capsicum, but without the security model.
| tome wrote:
| Could you elaborate on the differences? As far as I
| understood it firejail, or rather the Linux features that
| it depends upon, is far more powerful than FreeBSD jails.
| stjohnswarts wrote:
| I was under the impression they were like jails, to sandbox
| a program and make it more siloed off and secure? Whatever
| the underlying mechanisms for obtaining that. I'll research
| some more I guess.
| milofeynman wrote:
| I use LXC on Proxmox and I do everything with Ansible scripts.
| Is there something moving towards docker-like repository in LXC
| land? Would love to just run the latest pihole or nginx or what
| have you on LXC
| kettunen wrote:
| Author here!
|
| This is an old post of mine which I happened to find useful.
| Orchestration of jails moved quite bit forward lately! For
| example, you can manage your jails quite nicely with containerd
| today! See great post from Samuel Karp about the topic:
| https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-w...
| dgellow wrote:
| Just curious, is there an equivalent (or at least similar in
| spirit) to FreeBSD jails in the Windows world?
| erkserkserks wrote:
| There's Sandboxie, which works decently, but does have some
| compatibility issues: https://sandboxie-plus.com/
| VTimofeenko wrote:
| There used to be Virtuozzo containers for Windows and looks
| like they are bringing it back as a technical preview:
|
| https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_users_g...
| hestefisk wrote:
| Windows 10 kernel supports "Sandbox", which might be what comes
| closest.
| dgellow wrote:
| Do you mean the graphical Windows Sandbox tool? Can it also
| be used headless to run a service for example? That would
| come pretty close.
| mastax wrote:
| There's Windows Containers[0], which are analogous to Docker
| containers. They can either be run shared-kernel or in a
| Hyper-V container. Also Virtualization-based security[1], which
| is supposed to be completely transparent so it's not really a
| developer or deployment tool.
|
| [0]: https://docs.microsoft.com/en-
| us/virtualization/windowsconta... [1]:
| https://docs.microsoft.com/en-us/windows-hardware/design/dev...
| [deleted]
| shrubble wrote:
| I have been playing around with the bastillebsd.org scripts for
| creating and managing jails. I think it aims to be more 'Docker
| like'.
|
| Worked well from the limited testing I have done so far
| tambourine_man wrote:
| I always hoped for macOS to borrow FreeBSD jails for itself.
|
| A Docker-like solution with a pretty UI could be really useful
| for pros. For novices, it could mean a less cumbersome security
| measure than the restrictions we've been experiencing since
| Catalina.
| jbverschoor wrote:
| I would like an auto-jail, based on the directory you're in
| (similar to asdf/rbenv). There's just too much complexity in
| software, and so many things run on your machine these days.
| You can't really trust any application anymore
| eddieh wrote:
| macOS would definitely benefit from adding support for jails
| and a whole host other features like procfs, unionfs, bindfs,
| tmpfs, etc.
| trasz wrote:
| Erm, why would anyone want procfs? It's obsoleted in FreeBSD
| for a good reason.
| eddieh wrote:
| It doesn't have to be procfs, but it would be nice to have
| a /proc like interface.
| okokwhatever wrote:
| I am amazed at how many interesting things I still have to learn
| in this life. Too many tools, too little time to see them all...
| :(
| moeris wrote:
| ars longa vita brevis
|
| https://en.wikipedia.org/wiki/Ars_longa%2C_vita_brevis
| bigodanktime wrote:
| A great wrapper UI I have used for FreeBSD Jails is iocage
| (https://iocage.readthedocs.io/en/latest/). Its a great project.
| Timothycquinn wrote:
| I was about to say the same. iocage is now the default jails
| wrapper on FreeNAS, which means that there is good
| documentation and support. The previous jails wrapper used by
| FreeNAS was not very well documented, but was written by some
| smart folks.
|
| One thing I like about iocage is how easy it is to grant the
| jail access to the host ZFS datasets.
|
| On a Jails note, I have had issues creating a jail that can do
| network inspection. I believe this is an issue with network
| restrictions of the jails subsystem itself. Eg, I could never
| run nmap or get mac addresses of remote hosts from within a
| jail.
| rsync wrote:
| One thing I find so valuable about jails is the ability to jail
| _a single command_ with no other userland than what that binary
| requires.
|
| Here's an example from my personal name server:
| /usr/sbin/jail /jails/www www 10.10.10.36 /lighttpd -f
| conf/lighttpd.conf
|
| ... and although this jail has a lot of _content_ files in it,
| the actual UNIX userland is only what is required to run
| 'lighttpd': # find /jails/www/usr | wc -l
| 43
|
| So it's an extremely lightweight environment with very little
| attack surface.
|
| You can also _share a lightweight environment with multiple
| commands_ - here are two other jail commands:
| /usr/sbin/jail /jails/dns ns1 10.10.10.30 /nsd/nsd -c
| /nsd/nsd.conf /usr/sbin/jail /jails/dns dns 10.10.10.37
| /unbound/unbound -c /unbound/unbound.conf
|
| ... see how _both_ jailings of 'nsd' and 'unbound' point to the
| same '/jails/dns' userland ? Once again, that userland is very,
| very compact: # find /jails/dns/|wc -l 97
|
| ... so, 97 files total to run both name servers.
|
| No 'make world' necessary, no building and maintaining of a full
| FreeBSD system - just the lightest skeleton required for both
| 'nsd' and 'unbound'.
| cpuguy83 wrote:
| Totally feasible on Linux, btw.
|
| Packaging an entire system is more about convenience than
| anything else. It's also pretty difficult to package _just_ the
| libs one needs when you are dependent on libc and other C libs.
|
| I suspect that if one was really ok with it, some tooling could
| be built to copy/link in system libs into the rootfs
| automatically from the host.
| chriswarbo wrote:
| > It's also pretty difficult to package just the libs one
| needs when you are dependent on libc and other C libs.
|
| It's pretty easy using Nix, e.g. this example defines a
| container for running a shell script:
| https://ryantm.github.io/nixpkgs/builders/images/ocitools
|
| That script depends on bash, bash depends on libc, etc. so
| those dependencies (and _only_ those dependencies) will be
| put in the container. (See https://nixos.org/guides/nix-
| pills/enter-environment.html#id... for an example of what
| dependencies look like in Nix).
|
| > I suspect that if one was really ok with it, some tooling
| could be built to copy/link in system libs into the rootfs
| automatically from the host.
|
| Eww, no thanks! I want my containers to be reproducible.
| [deleted]
| pricci wrote:
| Exodus?
|
| Exodus - relocation of Linux binaries-and all of their deps-
| without containers - https://github.com/intoli/exodus
| hda111 wrote:
| How does it know what is needed to run? What if its a program
| that controls another program via CLI?
| toast0 wrote:
| One needs to assemble the jail. There may be tools for this,
| although I haven't used them. It's real easy if you can build
| a static executable, if it's a normal dynamic executable, you
| can use ldd (or something) to get all the libraries and just
| remember to pull in ld.so (the runtime linker). If it's
| dynamically loading modules (like Apache HTTP and PHP often
| are configured) or running other executables like you asked,
| that's harder to automate.
___________________________________________________________________
(page generated 2021-12-22 23:00 UTC)