When would it benefit the BSDs to get on the containerization train? reddit / BSD ---------------------------------------------------------------------- cerka · 2022-05-13 22:51 Apologies if this is long. I have been waking up from my decade-long slumber and I'm realizing that everyone on Linux is going crazy about flatpaks, snaps, and containers. I understand the following are among the motivations for Linux distros to adopt this framework: 1. To save on the effort of keeping conflicting dependency chains in sync for various packages. Every package can have its own libraries bundled into the flatpak or snap. These can be maintained by the distros, like packages have been. 2. To solve the issue of having to re-package the same software for half a dozen different distros. Every developer could prepare their own flatpaks or snaps and bypass the distros completely. 3. To limit the damage from malicious software that will now be installed with non-zero probability due to the previous point. That is, bypassing the distro introduces huge security concerns. 4. To create immutable base systems that can be updated more cleanly. My understanding is that this is how OS X, iOS, etc. work, and Fedora Silverblue is doing this for Linux desktops. Of these four, the second motivation (and thus the third as well) is effectively a non-issue for BSDs. So only the first and the last seem like legitimate motivations to me. But it is not clear to me * how much effort is required by conflicting dependencies in present- day ports/pkgsrc, and * how much we gain from container-based immutable base systems. Are immutable base systems only of real benefit for managing enterprise workstations and cloud shenanigans, or are there real security benefits there relative to where we are in the BSDs today? Am I missing something about what is motivating this seeming obsession with containers? In 10 years time, will any of the major BSDs also be riding the containerization train? ---------------------------------------------------------------------- DiggyTroll · 2022-05-13 23:39 FreeBSD implemented jails over 20 years ago. Immutable base filesystems existed, but they got a nice boost with ZFS snapshots. What you call the “containerization train” is a user-friendly set of tools and protocols that many folks find more approachable than the way we’ve always done things with scripts. There’s just not enough emergent demand for new-style containers on the BSDs. If there were, we would have implemented them. ---------------------------------------------------------------------- cerka · 2022-05-13 23:54 So do you think that new-style containers are an inevitability? Do you see user-friendly jailing as the main motivation for it? ---------------------------------------------------------------------- desnudopenguino · 2022-05-14 02:11 As someone else said, freebsd has had jails for a long time. Though jails arent quite the same as containers, and there are isnt an orchestration layer out there for jails like containers (k8s, and docker swarm and such). I like jails a bit more to be honest. Check out bastillebsd. You could use that with some config shell scripts and have something pretty close to a container deployment. I have been interested in building a management/orchestration tool for a bit, but havent had the time to throw into it. But it would be cool to do some load balancing and HA stuff across nodes. The patterns are out there to follow. And with jails, you can have stupid thin/light jails akin to a serverless function environment. I'm kinda (well more than kinda) bummed something like this doesnt already exist for jails. Openbsd has their own virtualization stack, but that project isnt geared towards containerization. And they have a set of existing chrooting practices to separate services from the host. I cant speak for net- or dragonflybsd, as I'm not as familiar with those. ---------------------------------------------------------------------- DiggyTroll · 2022-05-14 02:38 Inevitable? I don't know. But with Docker falling out of favor (a container control daemon is a single point of failure) and stable container conventions, the chances seem better. If anything, I think the motivation will be pressure to accommodate DevOps and SRE practices. Edit: Yikes. I just meant process support, like CI/CD. Didn’t mean to start an argument. ---------------------------------------------------------------------- PunishedRaion · 2022-05-14 04:03 Yuck, DevOps is BS. Lawl downvote me all you want, but traditional separation of sysadmin and development roles is best. Ops shit is ruining the field and it's corporations cost cutting and removing specialization, so fuck them. ---------------------------------------------------------------------- PunishedRaion · 2022-05-14 04:13 These are no actual benefits to containerization via flatpak, docker, whatever. It's all tradeoffs. On one hand you have a stable package that runs anywhere, something Apple-jerkers love to point out. On the other the increased space waste, inability to update the libs for security issues unless the vendor bumps it, and various other issues negate a fair bit. A container or jail can be broken and can cause inflexibility. It's possible to not bother with that at all. Security is relative and a process. As BSDs lack distros the way GNU/Linux does, we have no benefits. and there's alien for binary package conversions. An immutable base is not an issue with the BSDs. You simply uninstall everything, delete /usr/local or /usr/pkg and update base. Then restore your configs. Plain and simple this trend is a macOS thing, and as I've said time and time again, macOS is the antithesis of UNIX and it's principles. It's a shitty Gen 1 ukernel, and everything GPL zealots weave stories about. Apple users are a pest to the BSD and UNIX communities and need to bugger off, like Jordan Hubbard. ---------------------------------------------------------------------- kraileth · 2022-05-14 04:15 There was also PC-BSD, a FreeBSD-based desktop spin with KDE (which turned into graphical TrueOS with the native Lumina DE only to be discontinued and succeeded by Project Trident which then switched over to Linux to die a silent death). It had PBI (aka "Push Button Installer") way before on Linux they came up with Flatpak or Snap. If you want to know more about it, I suggest you read the article by the inventor, Kris Moore, which was published in the March/April issue of the FreeBSD Journal (it's only 3 pages, just looked it up). PBI eventually fell out of favor and died. I'm not sure about the exact reasons, though. For containers / jails: Like others have pointed out, it was on FreeBSD where the whole thing started (proprietary addon in FreeBSD 3.x, officially part of 4.0 for everyone to use). What Linux did with Docker was providing a compelling tool around it that allowed things to take off in whole new dimensions. FreeBSD does have interesting tools which can do mostly the same thing. There's Christer Edwards' `bastille` for example which supports stackable templates, Luca Pizzamiglio's `pot` which even supports Hashi's *Nomad* container orchestration (Kubernetes-like) and Oleg Ginzburg's `cbsd` which is a complete *virtualization framework* for BSD (i.e. it manages jails on both FreeBSD and DragonFly BSD, Bhyve VMs on FreeBSD, NVMM on NetBSD and DragonFly BSD, etc.). Disclaimer: I'm loosely affiliated with the latter project, so I'm not neutral on the matter. However I can only recommend that if you're interested in getting on the containerization train, you take a look at the [guide](https://github.com/myb-project/guide) for the `MyBee` project (which is based on cbsd's abilities). It basically provides a REST API for running your own cloud hosting service on FreeBSD. While there's currently a focus on VMs (ready-made images for Linux systems and even the fastest possible deploying of Kubernetes is also available!), you can us it to manage jails as well. There's also `ClonOS` which aims at delivering a Proxmox-like web UI for management of VMs and jails. And yes, this is not daydreaming - all these things do already exist today. None of those tools is perfect and they will all need more work, but they are usable right now. Unfortunately they are kind of a well-kept secret even within the BSD community and have a rather small user base. But we hope that this is going to change soon. \[edit: Fix typo to make ectbot a happy bot\] ---------------------------------------------------------------------- ectbot · 2022-05-14 04:15 Hello! You have made the mistake of writing "ect" instead of "etc." "Ect" is a common misspelling of "etc," an abbreviated form of the Latin phrase "et cetera." Other abbreviated forms are **etc.**, **&c.**, **&c**, and **et cet.** The Latin translates as "et" to "and" + "cetera" to "the rest;" a literal translation to "and the rest" is the easiest way to remember how to use the phrase. [Check out the wikipedia entry if you want to learn more.](https://en.wikipedia.org/wiki/Et_cetera) ^(I am a bot, and this action was performed automatically. Comments with a score less than zero will be automatically removed. If I commented on your post and you don't like it, reply with "!delete" and I will remove the post, regardless of score. Message me for bug reports.) ---------------------------------------------------------------------- Kernigh · 2022-05-14 04:53 I don't like how containers have extra copies of libraries, so you need more RAM and disk space to run them. About item 2 (re-packaging), BSD packages must be rebuilt for each different BSD (FreeBSD, NetBSD, ...) on each different hardware platform (amd64, arm64, riscv64, ...). Containers would not simplify this. Item 4 (immutable base system) is least useful on OpenBSD. Each reboot of OpenBSD mutates the system by reordering the kernel and some libraries. ---------------------------------------------------------------------- vext01 · 2022-05-14 07:21 There are no benefits to containerisation? You can't think of one? I'm an OpenBSD guy. I've used (and developed) it for over a decade. It's great, but sometimes I wish I could sling up a disposable (but reproducible) environment like docker would allow. Often the nearest you get is a VM, which is much more time consuming and not easily reproducible. I do think the configuration files and ways the user accounts are handled in docker is questionable, but that doesn't invalidate the above points. I have a Linux box nearby for those times where I need docker. I don't evangelise any given OS. They all have their uses. ---------------------------------------------------------------------- motific · 2022-05-14 09:31 I think you need to understand what docker does and how it solves those problems before we engage with all the hype. 1) That’s basically downloadable jails. 2) Docker does not make code portable, it relies because everyone on Linux shares the same kernel (and WSL in the case of windows) and then the container brings its own userland so basically their community has settled on Alpine Linux. So the usefulness of containers to BSD is limited 3) Trades the relative security of partial sandboxing for running a completely unknown VM from an outside source. Jails aren’t security as we’re often reminded. 4) is what we have with jails. We lack the single fire & forget packaging, but look at bastille templates and that’s actually not a million miles away. pkg / port conflicts are most definitely a thing but honestly 10 mins with the bastille docs and it’s solved. The big benefit to containerisation in my book is preconfiguration and easy load balancing - the package set is immediately ready to go, minimal configuration. ---------------------------------------------------------------------- PunishedRaion · 2022-05-14 12:58 As I said, just tradeoffs. It's not a surefire way one way or the other. I've worked on IBM i and other much higher end things. See I never really need that. I used iocage on FreeBSD but as that's now a legacy environment (We will not be upgrading to 13) we moved our workflow out of containers and I found plenty of ways to do a workflow much better. Docker and such never appealed, but I'm a Solaris and Sys V guy foremost. ---------------------------------------------------------------------- cerka · 2022-05-14 14:19 Thank you, huh, I just learned about [KARL](https://undeadly.org/cgi?action=article;sid=20170613041706)! ---------------------------------------------------------------------- cerka · 2022-05-14 14:30 Thank you for this overview! And do you think that this means that eventually FreeBSD will shift to a Fedora Silverblue-like organization of the userland? Or will these remain tools that can achieve containerization in specific use cases but for the default use case the userland will remain as it is today? ---------------------------------------------------------------------- cerka · 2022-05-14 14:43 Thank you! Yeah, it's the space waste and the security issues that put me off the most. I'm trying to see if it's all just a passing fad or if there's a good reason that so many Linux distros put themselves thru this. Right now my impression is that there's a rather specialized enterprise use case for which containerization is great, and now for some reason everyone on Linux (okay, obviously not _everyone_) has to bend themselves to live with the enterprise use case on their computers. ---------------------------------------------------------------------- cerka · 2022-05-14 14:46 Thank you! So would something like BastilleBSD fit this use case already? ---------------------------------------------------------------------- cerka · 2022-05-14 14:47 Thank you! ---------------------------------------------------------------------- Chousuke · 2022-05-14 14:57 People just misunderstand the point of devops. The point of devops isn't for developers to do operations, but for operations to use the same tools developers do, ie. CI/CD and testing. The point is for ops to work together with software developers because custom software is only one part of any system that solves a problem. Of course, ops people programming is nothing new; they have always used automation and programming to solve their problems; many are competent programmers, they just aren't always interested in software development. ---------------------------------------------------------------------- kraileth · 2022-05-14 14:58 If I had to guess (I'm just an advanced user not a FreeBSD dev), I would say that it will stay as it is. FreeBSD is a general purpose OS and is not likely to completely leave classic setups behind. However: What is kind of revolutionary on Linux is no so much a new thing on BSD. In case of using jails you can not only run them read-only (which is in fact the standard option in cbsd - you need to make it mutable if you need to!). It's also possible to build jails that do not come with a complete userland but with a very minimal set of libraries and applications. What might also be interesting for you: You can have a read-only base jail and then delegate certain ZFS datasets into the jail so that e.g. /var/log is still writable but things like /etc and /usr/sbin aren't. ---------------------------------------------------------------------- PunishedRaion · 2022-05-14 16:25 I'm skilled with coding to some degree, but I'm not interested in transitioning into a job where I am forced to adhere to deadlines, or fight SALT/Puppet/Ansible automation gone wrong. I'm old fashioned and thats why I bailed when I did. If I go back to tech, it'll be mainframes where things won't change in the next 40 years but so much. I want to come to work, sit down and monitor/correct shit and handle non automation tasks. Leave that to development. I'm a sysadmin, and OPS as we know it is a modern thing in the last 5-8 years. ---------------------------------------------------------------------- motific · 2022-05-14 16:44 Many use cases would be covered but not quite all, shifting workloads would be one example, and of course it relies on there being templates for what you want to do. But after some basic setup to get bastille running and the networking how you want it then it’s basically 2 lines to deploy a jail, 1 to grab a template, 1 to create the instance using the template and start the jail… ---------------------------------------------------------------------- LinuxLeafFan · 2022-05-14 16:55 As others have mentioned, FreeBSD has been doing jails for a number of years, long before containers on Linux were a thing. The thing that has hurt the adoption of jails is the lack of “friendly” and scalable tooling. Personally, I think FreeBSD has a lot of potential to do something awesome here but it’s developer community doesn’t seem interested in such efforts. It’d be very interesting if efforts were put towards making FreeBSD an OCI compatible tooling and runtime supporting Linux native containers and FreeBSD jails. This would make FreeBSD an option as a K8s host or Linux container development host. It would also open up some interesting capabilities for running Linux-native apps. Sadly it seems most of the developer community is uninterested in this. There was a dev working on it a few years ago and presented it at a con where basically he was laughed off the stage for trying to get Linux containers working on FreeBSD with most responses to his presentation amounting to “jails tho”. SmartOS was doing this for a few years but it seems to be unmaintained/destined to become yet another memory in Unix history. Of course someone will likely reply with “BHYVE” and my response is simple. That’s not what I’m discussing, shut up. ---------------------------------------------------------------------- vext01 · 2022-05-15 05:48 So you are a zones guy ;) ---------------------------------------------------------------------- Chousuke · 2022-05-15 10:41 I'm pretty much the opposite; I'd much rather work with configuration management where changes are clearly visible in git history than try to figure out after the fact what changed, why it broke things and what the expected configuration is supposed to be. Beyond a *very* small number of systems to manage, manual change management is just too stressful for me. With properly managed automation (especially Puppet and its static typing) I can easily and confidently deploy changes to hundreds or thousands of systems in one go and not worry about surprise breakages either due to unknown configuration or some typo or mistake I make during execution. ---------------------------------------------------------------------- PunishedRaion · 2022-05-15 13:04 If I have to make a change to more than two I always make up a quick script to execute it, then deploy that. My issue with config management I guess more than anything is I don't care for any of the options. ---------------------------------------------------------------------- PunishedRaion · 2022-05-15 13:05 Everything I said applies to those or WPARs for AIX etc. ---------------------------------------------------------------------- CoolTheCold · 2022-05-20 11:29 Disclaimer: that's my theories and thoughts, I'm not having any internal statistics, not having much of CustDev/DX (developer experience) polls. Let's split topic into dealing with containers in style of Docker, where it's mostly for background services (daemons) and Flatpak/Snaps/AppImage which are mostly for end-user, what we name "applications", like some Office suite or Video editor. So, on FlatPak, Snaps, AppImages and similar things of bringing applications images to End User. I strongly believe this happening due to fact of Linux user base is growing in Desktop/Laptop area and to no surprise not all of such user base is hardcore C & kernel devs who can easily deal with coredumps, gdb and tracking things deeply in system. Moreover, as it's becoming less technical on average, at the same time on average, request for running _lastest_ OpenOffice rises vs being happy with Vim. And that leads us into the land of distribution modelling - stable vs rolling, pinned versions, myriads of WM/DE choices, X11 vs Wayland and so on. Imagine yourself in place of OpenOffice devteam receiving bug reports which they hardly can repeat due to _their_ system env is quite different. Imagine yourself in place of End User, who more or less wants to just be able to install _latest, with bug fixes_ OpenOffice from some type of store/package manager and seeing some cryptic message about `libmesagl something` or whatever similar, he/she has no any idea about. Or even breaking packages due to installing with some guide on internet which says: `you need to use --force to install it` and crashing everything. Thus, application images, are heavily toward of helping average Joe to get it in _predictable_ and _guaranteed_ way, which can be prebuild and tested by product's dev team and distributed in a way which prevents manual changes by that Joe. And across multiple distros (it's just abstracted away). Everyone happy, including support team. Between "this will probably work, probably not on your system" vs "this will 99% work on your system but will take extra 1GB of data", you may guess who chooses what. ---------------------------------------------------------------------- CoolTheCold · 2022-05-20 11:34 Coming next thing, immutable images. The best explanation here I believe would be to place your self into shoes of Chromebook team. Or PinePhone. Or similar customer oriented solution. How would your organize updates? Updates Over The Air? Testing such updates before shipping? Rolling back? Not letting users [who are again, can be just 8 years old scholar] break the system? When you start imagine how you would solve it yourself, you will better understand such decisions. To my understanding, *BSDs are far far away from that yet. ---------------------------------------------------------------------- CoolTheCold · 2022-05-20 12:01 and part 3, the last thing, on Docker (or what has it inspired after being 9 years old already, be it `podman` or `OCI` or ...) >I'm realizing that everyone on Linux is going crazy about flatpaks, snaps, and containers. This statement while correct, but not full. It's not Linux going crazy on containers, it's developers going crazy on containers. The fact containers run on Linux bothers only sysadmins/nerds [who are shrinking and being extinguished since then]. If it would be Minix inside, it will be used as well. Linux was just a good enough tool with support of features which Docker Inc. needed to create their tooling. There is at least one example in this thread on ephemeral containers usage, will try to express one more, a bit more detailed, from real life. So, situation, given: * frontend developer - guy who deals with javascript in browser and in general of nice looking pages and interactive design, doing `supersite.com` pages * using MacOS on work laptop * DevOps/sysadmin - who is helping with all that black terminal magic and WEB SERVERS, Nginx and such * uses Linux (let it be Suse for example) on laptop * QA Engineer, * using Windows 11 with WSL (and Docker Desktop) on laptop * CI/CD system with some sort of linters enabled, basic tests, so on * one of them is running https://www.sitespeed.io/ based tooling to generate reports on various site metrics (TTFB, LCP, Repeat View and so on), which is based on Chrome * dev/prod environment with k8s for hosting software/services that company produces Case: It was found that on Nginx of our lovely `supersite.com` for some big background picture (background.jpg), doesn't properly set `Expires` HTTP header to `-1`, enforcing users to redownload and redownload it on each visit, instead of caching in browser. What happens next? * DevOps (Linux on local machine) guy pulls repo of that project from their Gitlab/Github/Other VCS, create new branch * updates Nginx config to be proper to his understanding * runs quick test locally [on his laptop/desktop] with using Dockerfile which already exists in repo * once things look good - commits changes, pushes branch back to repo, creates Merge Request * QA guy ( Windows user ) - repeats the same, but does some manual tests (let's imagine). Using the same Dockerfile of course * Frontend guy ( those one on MacOS ) - after receiving OK from QA, merges request into main branch, pulls changes, rebuilds/restarts container, which is still based on the same Dockerfile and keeps developing Case is solved. During the flow, everyone had the _SAME_ environment, has spend 0 minutes on dealing with reading & setting up Nginx locally in proper way and _THE SAME_ will at the end will be shipped to dev/prod. EDITS: formatting ---------------------------------------------------------------------- lazy-xo · 2022-06-18 13:26 Ohm - you are aware bsd was “containerized” 20 years ago right? You realize FreeBSD has literally jailed zfs datasets jailed the entire network stack (ENTIRE STACK) hell jailed Linux (JAILED LINUX) - run your Linux containers in a jail ---------------------------------------------------------------------- lazy-xo · 2022-06-18 13:30 Everyone talking about containers are this and jails are that please watch this - and follow with the same speaker’s talk on how docker is SUPPOSED to be deployed as originally INTENDED.. https://paperswelove.org/2016/video/bryan-cantrill-jails-and-solaris- zones/ ^^ he co wrote dtrace / created smartOS / had a hand in zones and zfs - deep into docker hosting - solid source is my point.