[HN Gopher] From Proxmox to FreeBSD and Sylve in our office lab
___________________________________________________________________
From Proxmox to FreeBSD and Sylve in our office lab
Author : arch1e
Score : 66 points
Date : 2026-03-28 12:20 UTC (2 days ago)
(HTM) web link (www.iptechnics.com)
(TXT) w3m dump (www.iptechnics.com)
| gcifuentes wrote:
| Bhyve doesn't feature nested virt though.
| liendolucas wrote:
| Honestly asking, in which cases nested virtualization is
| useful?
| zenoprax wrote:
| One example: when learning Proxmox itself. I was able to set
| up a multi-node cluster with more complicated networking than
| I was normally comfortable with and experiment with failures
| of all sorts (killing a node, disabling NICs, etc.) without
| needing more hardware or affecting my existing things.
|
| Outside of learning and testing I am not sure of what uses
| there might be but I'm curious to know if there are.
| belthesar wrote:
| Nested virtualization can be very handy in both the lab and
| in production. In the lab, you can try out a new hosting
| platform by running one atop the other. IE: Proxmox on
| VMWare, Hyper-V on KVM. This lets you try things out without
| needing fresh bare metal hardware.
|
| In prod, let's say you run workloads in Firecracker VMs. You
| have plenty of headroom on your existing hardware. Nested
| virtualization would allow you to set up Firecracker hosts on
| your existing hardware.
| liendolucas wrote:
| Perhaps I'm misunderstanding, but wouldn't that case be
| covered by simply putting some vms under a vnet and others
| on another vnet and make them talk to each other? I can't
| also understand what you mean by "fresh bare metal
| hardware". In either case you don't need bare metal, being
| a top level vm or a nested one.
| mbreese wrote:
| If you're evaluating VM hosts (proxmox, hyper-V, vmware,
| etc...) You need to have support for nested
| virtualization all the way down. Otherwise, if you want
| to evaluate a VM infrastructure, you need to start with
| bare-metal. Really, you just need to make sure that your
| top level support nested virtualization, but I understand
| their point.
|
| However, the point about firecracker VMs in place of
| containers I think is really a good use-case. Firecracker
| can provide a better isolation environment, so it would
| be great to be able to run Firecracker VMs for workloads,
| which would require that the host (and the VM host above)
| support nested virtualization.
| wingmanjd wrote:
| We run Proxmox VMs that are running Hashicorp's Nomad
| orchestration at $DAYJOB. The Nomad clients are then turning
| around and running the docker containers (Proxmox -> Nomad VM
| -> Docker). For us it's easier to manage and segregate duties
| on the initial metal this way.
| doubled112 wrote:
| WSL2 in a virtual desktop environment.
| arch1e wrote:
| That's true, bhyve doesn't support nested virtualization right
| now.
|
| In practice though, most setups don't actually need it if
| you're running workloads directly on the host.
|
| Also, if your goal is testing or simulating clusters, you can
| already run Sylve inside jails. That gives you multiple
| isolated "nodes" on a single machine without needing nested
| virt. We have a guide for it here:
| https://sylve.io/guides/advanced-topics/jailing-sylve/
|
| So you can still experiment with things like clustering,
| networking, failure scenarios, etc., just using jails instead
| of spinning up hypervisors inside VMs.
|
| Nested virt is still useful for specific cases like testing
| other hypervisors or running Firecracker inside VMs, but for
| most Sylve-style setups it hasn't really been a blocker.
| sidkshatriya wrote:
| I love FreeBSD but Linux just provides every feature under the
| sun when it comes to virtualization. Do you find any missing
| features on bhyve ? Is bhyve reliable ? I can't imagine its been
| tested as thoroughly as KVM ...
| gcifuentes wrote:
| Bhyve is quite cool but no nested virt which means you cannot
| nest vm_enter/exit calls with EPT pages so you cannot
| virtualize within those guests. I found this crucial. For
| instance Qubes OS won't run in Bhyve by any means.
| rwmj wrote:
| Or Windows VBS, if you need to run full-featured Windows VMs.
| seanw444 wrote:
| Do they have any near-future plans to resolve that?
| MisterTea wrote:
| If you are unsure of bhyve's abilities then why not test
| yourself? Speculation and guessing about stability or testing
| is useless without seeing if it works in your application.
| sidkshatriya wrote:
| > If you are unsure of bhyve's abilities then why not test
| yourself?
|
| It is not possible to come to a conclusion about everything
| in the world yourself "from scratch". No one has the time to
| try out everything themselves. Some filteration process needs
| to be applied to prevent wasting your finite time.
|
| That is why you ask for recommendations of hotels,
| restaurants, travel destinations, good computer brands,
| software and so on from friends, relatives or other trusted
| parties/groups. This does not mean your don't form your
| opinions. You use the opinions of others as a sort of
| bootstrap or prior which you can always refine.
|
| HN is actually the perfect place to ask for opinions. Someone
| just said bhyve does not support nested virtualization
| (useful input !). Someone else might chime in and say they
| have run bhyve for a long time and they trust it (and so
| on...)
|
| So I can't agree with your viewpoint.
| MisterTea wrote:
| The phrasing of your questions is the problem. They are
| uninformed, too general, and assuming. The last sentence
| reads as if you outright dismiss bhyve because YOU can't
| imagine it was tested thoroughly.
|
| > It is not possible to come to a conclusion about
| everything in the world yourself "from scratch". No one has
| the time to try out everything themselves. Some filteration
| process needs to be applied to prevent wasting your finite
| time.
|
| It's totally possible when you know what your application
| requires but you didn't state anything.
|
| > Someone just said bhyve does not support nested
| virtualization (useful input !).
|
| What nested applications are you planning to run?
| sidkshatriya wrote:
| Ok you have a problem with the way I framed my questions
| and my (unintentional) tonality. Fair enough. Let's move
| from critique of the way I asked my questions to what
| your experience with bhyve has been, if you're willing to
| share that.
|
| Have you used bhyve ? What has your experience been with
| it ? Have you used KVM+QEMU -- can you compare your
| experience between both of them ?
| LeFantome wrote:
| I agree with you and do not understand the "I read every
| manual" and "I test all software" crowd. I play around with
| A LOT of software but I cannot test it all.
|
| Speculation is not useless if you are saying "the answer I
| got makes it 99% likely that this solution will not work
| for me". Curation has immense value in the world today. I
| investigate only the options most likely to be useful. And
| that still takes all my time.
| evanjrowley wrote:
| Anecdotally, Bhyve has worked in FreeBSD for a decade now.
| Eventually it got ported to Illumos because it was better than
| their implementation of QEMU.
| KaiserPro wrote:
| What does Sylve provide that proxmox doesn't?
|
| Or better, _how_ does it do it better than proxmox?
|
| This isn't to say that proxmox is the best thing since sliced
| bread, I'm curious as to what makes sylve better, is it the API?
| evanjrowley wrote:
| Without looking at the Sylve docs, I'll conjecture that it has
| deeper integration with ZFS. With a foundation on FreeBSD,
| there is a likelihood Sylve can support ZFS-on-root rollbacks
| better than hacking it into Proxmox. A rollback capability is
| why I'm looking for Proxmox alternatives. In the Linux world,
| Talos Linux and IncusOS provide A/B updates which achieve a
| similar rollback capability. With something based on FreeBSD,
| your "immutable" OS and all of it's data can be treated equally
| as ZFS datasets. There's also a higher risk that a Linux kernel
| update will break ZFS.
| justsomehnguy wrote:
| > Sylve can support ZFS-on-root rollbacks better than hacking
| it into Proxmo
|
| Can you explain your use case when you absolutely can't
| provide a separate M.2 drive solely for the OS?
| evanjrowley wrote:
| Regardless of the number of drives available, you gain an
| advantage when your file system can leverage snapshots to
| roll backwards or forwards. There are other Linux-native
| filesystems that can provide this capability too, but many
| admins prefer ZFS because the full range of capabilities is
| unparelleled.
| TacticalCoder wrote:
| > What does Sylve provide that proxmox doesn't?
|
| A Un*x system that doesn't use systemd as an init system.
| BeepInABox wrote:
| Right, so nothing?
| hedora wrote:
| In the same way as not releasing angry bees into your cube
| farm is doing nothing, yes!!
| vablings wrote:
| Great to see we are back to counter jerking systemd after
| years of great work...
| arch1e wrote:
| OP here. It's less about Sylve doing something Proxmox _can't_
| do, and more about a bunch of QoL improvements that come from
| us being heavy Proxmox users and building what we felt was
| missing.
|
| A few concrete things:
|
| ZFS-first UX: Not just "ZFS as storage", but everything built
| around it. Snapshots, clones, ZVOLs, replication, all cleanly
| exposed in the UI without dropping to CLI.
|
| Simple backups without extra infra: Any remote box with SSH +
| ZFS works. No need to deploy something like PBS just to get
| decent backups.
|
| Built-in Samba shares: You can spin up and manage shares
| directly from the UI without having to manually configure
| services.
|
| Magnet / torrent downloader baked in: Sounds small, but for
| homelab use it removes a whole extra container/VM people
| usually end up running.
|
| Clustering: but not all-or-nothing, You can cluster nodes when
| you need it, and also disable/unwind it later. Proxmox clusters
| are much more rigid once set up.
|
| Templates done right: Create a base VM/jail once and spin up N
| instances from it in one go, straight from the UI.
|
| FreeBSD base: It's not really a benefit of Sylve, but rather
| the ecosystem that FreeBSD provides.. Tighter system
| integration, smaller surface area, no systemd, etc. (depending
| on what you care about)
|
| None of this is to say Proxmox is bad, it's great. This is more
| "we used it a lot, hit some friction points, and built
| something that feels smoother for our workflows."
| alwillis wrote:
| Sounds awesome! Looking forward to trying it out.
| KaiserPro wrote:
| Excellent, this makes a lot of sense. I'll try it out
| hedora wrote:
| I get the impression bhyve does all that stuff too. Is sylve
| basically just a thin GUI wrapper on top?
|
| (That'd be amazing if it's possible to do stuff like dump
| configs + check them into git from the cli, then stand them
| up on any bhve/sylve box later...)
| j45 wrote:
| Sylve appears to be a FreeBSD/BSD exclusive implementation of
| managing vms, etc.
|
| Proxmox is Debian/Ubuntu based.
|
| Both will have their advantages. It might not be about better
| or worse, the particular things you use may in some cases run
| better on BSD, or the security management could more fit what
| you are after.
|
| I wonder why not run both :).
|
| Proxmox is due for it's viral moment though.
| dizhn wrote:
| Sylve looks like a decent project with a promising future but
| this article really doesn't explain why they picked it over
| Proxmox at all. They explain a lot of things but I can't see the
| advantage over prox other than they wanted to use it.
| Cyph0n wrote:
| Sometimes unification can be an advantage.
|
| I run Proxmox at home, but now that I have been drinking the
| NixOS koolaid over the past 2 years, all of my homelab problems
| suddenly look like Nix-shaped nails.
| dizhn wrote:
| I have the same thing with proxmox especially after I
| realized how well it integrates with proxmox backup server.
| And I haven't even gotten into clustering yet. It really is a
| very solid product.
| Cyph0n wrote:
| Indeed, Proxmox VE is an amazing product.
| EnigmaCurry wrote:
| Same. Here's how I scratch the NixOS itch on Proxmox and/or
| libvirt[1]. One interface for both targets.
|
| [1] https://github.com/EnigmaCurry/nixos-vm-template
| dizhn wrote:
| That feature list looks really good. It would actually be
| really nice to standardize the guest operating systems in
| such a way.
|
| I actually have a few hosts that only run docker. I might
| be able to test with those.
| craftkiller wrote:
| Well it looks like we might soon be able to have the benefits
| of NixOS while also having bhyve (and presumably Sylve):
| https://github.com/nixos-bsd/nixbsd
| dizhn wrote:
| https://github.com/SaumonNet/proxmox-nixos
|
| Looks like Nix will eat the world soon. :)
| Cyph0n wrote:
| Damn this is crazy!
| TacticalCoder wrote:
| > They explain a lot of things but I can't see the advantage
| over prox other than they wanted to use it.
|
| A huge, totally obvious, advantage is that FreeBSD isn't using
| systemd. I'm now nearly systemd-free, if not for Proxmox. But
| my VMs are systemd free. And, by definition, my containers too
| (where basically the entire point is that there's a PID 1 for
| the service and that PID 1, in a container is _not_ systemd).
|
| So the last piece missing for me is getting rid of Proxmox
| because Proxmox is using systemd.
|
| I was thinking about going straight to FreeBSD+bhyve (the
| hypervisor) but that felt a bit raw. FreeBSD+Sylve (using bhyve
| under the hood) seems to be, at long last, my way out of
| systemd.
|
| I've got several servers at home with Proxmox but I never, on
| purpose, relied too much on Proxmox: I kept it to the bare
| minimum. I create VMs and use cloudinit and tried to have most
| of it automated and always made it with the idea of getting rid
| of Promox.
|
| I've got nothing against Proxmox but _fuck_ systemd. Just fuck
| that system.
| redserk wrote:
| Whether an appliance OS uses SystemD or not is as silly of a
| concern as "does the lead developer prefer cheddar or brie"
|
| What about performance characteristics? Recoverability of
| workloads?
|
| I'm interested in a FreeBSD base OS because it seems ZFS is
| better integrated and ZFS has a lot of incredibly useful
| tools that come with it. If Bhyve is at least nearly as
| performant as KVM, I'd be hard pressed not to give it a
| whirl.
| hedora wrote:
| It's not silly at all.
|
| I've been repeatedly burned by systemd, both on machines
| I've administered and on appliances. In every situation,
| the right fix was either "switch distros" or "burn
| developer-months of work in a fire drill".
|
| In fact, I just decided to go with FreeBSD instead of
| proxmox specifically because proxmox requires systemd. The
| last N systemd machines I've had the misfortune to touch
| were broken due to various systemd related issues. (For
| large values of N.)
|
| I assume that means anything built on top of it is flaky +
| not stable enough for production use.
| redserk wrote:
| It absolutely is silly. I've been responsible for
| managing low-thousands of Linux servers with systemd and
| it's standardized a lot of things that otherwise would've
| been a lot of bespoke scripts.
| arch1e wrote:
| OP here. One thing we mentioned in the blog but probably didn't
| emphasize enough is how deeply ZFS is integrated into the UI.
|
| With Sylve, you rarely need to touch the CLI. Snapshots,
| datasets, ZVOLs, even flashing images directly to ZVOLs, it's
| all handled from the UI in a straightforward way.
|
| That tight ZFS integration also lets us build more flexible
| backup workflows. You can back up VMs, jails, or entire
| datasets to any remote machine that supports SSH + ZFS. This is
| powered by Zelta (https://zelta.space) (which is embedded
| directly into the Go backend), so it's built-in rather than
| something you bolt on.
|
| In Proxmox, you can achieve similar things, but it's less
| intuitive and usually involves setting up additional components
| like Proxmox Backup Server.
| xoa wrote:
| Do you have any opinions on how this works vs doing iSCSI to
| some other storage system using ZFS? That's how I've been
| handling Proxmox on the backend, and have mixed feelings. The
| GUI leaves a very great deal to be desired in honestly
| curious ways, have to touch the CLI a lot even for super
| basic networking or auth stuff, and of course neither side
| has the same insight to the data structures in question.
| Either you've got to do ZVOL instances and thus manual effort
| or scripting, or you give Proxmox a single big blob then let
| it manage that with LVM but that means the storage side can't
| give any granular help on snapshots and the like. It still
| can deal with data integrity and backups and storage
| redundancy and all that but no further, and some increased
| overhead. But on the other hand, I do feel like a really firm
| separation of concerns isn't without value. Having native
| support though is an interesting alternative I hadn't really
| considered.
| dizhn wrote:
| I did actually notice the ZFS gui which is indeed something
| lacking in proxmox which doesn't default to ZFS in the
| installer. However once you do install it using ZFS it
| actually makes use of it pretty well and the user does not
| need to mess with the zfs cli tools much. Obviously it would
| be nice to have a GUI for all zfs operations too. Then again
| even TrueNAS refers you back to the cli for SOME operations.
|
| On proxmox ZFS syncs do not require proxmox backup server,
| which actually has its own format which is very efficient in
| speed and disk space, but you do either need something like
| sanoid/syncoid or use of the shell.
| whalesalad wrote:
| Wow, TIL! https://github.com/AlchemillaHQ/Sylve
|
| Also: https://www.youtube.com/watch?v=wo4oD5UON30
| SoftTalker wrote:
| Thanks, I hadn't heard of it either.
| gnabgib wrote:
| There was a bit of discussion over the weekend (70 points, 21
| comments) https://news.ycombinator.com/item?id=47557392
|
| Not sure why this copy made the SCP
| wolvoleo wrote:
| This is really interesting. I've played with bhyve before but I
| didn't realise anyone actually used it in anger. And that people
| had written such great tooling around it.
|
| My home lab still uses ESXi 8. But it needs something new and I
| was looking at proxmox. However I may give this a try first.
| ggm wrote:
| I'd love a simple explanation of the virt/bridge interface
| choices, and also why people pick NAT vs true address for their
| jails and virtuals.
|
| Likewise for disk i/o -some people swear by 9P as a backing
| mechanism, some by ZVOL.
___________________________________________________________________
(page generated 2026-03-30 23:01 UTC)