Bare metal vs Virtualization reddit / selfhosted ---------------------------------------------------------------------- tangerinine · 2023-10-24 20:49 Hey all, I know this is a bit off-topic but this sub is one of the few where one can talk about technical stuff and not about tech careers lol. I have been thinking about that a lot lately. Are there use cases where bare metal is better than virtualization? Have you ever encountered an use case that the virtualization overhead is an issue? I would love if you share your experiences. Thanks =) ---------------------------------------------------------------------- ElevenNotes · 2023-10-24 21:02 Yes. Licensing. I run all my container nodes bare metal. I run all my blockchain nodes bare metal (due to IO issues). There are plenty more but nothing that fits into this sub to be honest. Most people who selfhost don’t have applications that are that demanding, and even then, hypervisors like ESXi can be configured to give near bare metal performance to a few VMs for mission critical applications. The licensing issue remains though. ---------------------------------------------------------------------- Reverent · 2023-10-24 21:19 Containerisation for worker nodes is typically baremetal for licensing and performance reasons. Control planes are typically virtualised to take advantage of backup and availability mechanisms. For VMs, two situations where you may prefer baremetal is if you're doing hardware passthrough (which negates many VM advantages), or high performance compute. ---------------------------------------------------------------------- Firestarter321 · 2023-10-24 21:25 I prefer to have my NASes run on bare metal as I don't want to pass either drives or HBA's through to a VM. I used to think the same thing about my router, however, at least with Proxmox and Untangle it's been working very well having it virtualized. I also use Docker for some things and LXC's for others. Emby and Nextcloud are 2 examples of my moving from Docker to individual LXC's as I wanted to use different backup strategies for them as well as reduce the chances of them going down due to Docker issues. ---------------------------------------------------------------------- ElevenNotes · 2023-10-24 21:32 Two node vSAN? ---------------------------------------------------------------------- Firestarter321 · 2023-10-24 22:48 Yeah I thought about that as I’ve seen it mentioned but that’d actually require me to add 2 more machines running 24/7 since I run Proxmox (and don’t want to change). I still may look into it though and what it’d take to move from Proxmox. Id also need 2 VMUG licenses which are $250ish/yr I think? I have a couple of spare machines so I may install it and test at some point. ---------------------------------------------------------------------- epsiblivion · 2023-10-24 23:25 for selfhosting stuff, I don't see any requirements to be bare metal. the exception is the router/firewall. if you want to do software for that, then at least have a dumb all in one router as a spare when you inevitably break something and have family waiting on you to get the network back up. ---------------------------------------------------------------------- retrodaredevil · 2023-10-25 00:30 Can you elaborate on what you mean by licensing issue? ---------------------------------------------------------------------- DarkKnyt · 2023-10-25 01:10 Esxi has a free tier but is very limited. The paid licensed version is much more expensive. I use the free version of proxmox. It doesn't have the paid for repo or official support but it has almost every feature you might want except advanced networking (well at least not via the GUI, it's Debian underneath something you can do with Debian you can likely do with proxmox) ---------------------------------------------------------------------- retrodaredevil · 2023-10-25 01:14 I also use the free version of Proxmox. It's top tier software even without whatever advanced features their paid tier has. I don't really see the licensing as a reason not to virtualize, though. I guess if you're in enterprise it might be a turn off, though. ---------------------------------------------------------------------- DarkKnyt · 2023-10-25 01:15 Oh I agree about virtualizing. I mean, even if you went with Ubuntu, you can easily virtualized in lxd or docker. ---------------------------------------------------------------------- HumbleOraclea · 2023-10-25 02:18 Sounds complicated, but worth considering! ---------------------------------------------------------------------- befitting_semicolon · 2023-10-25 02:23 Worth virtualizing for sure! ---------------------------------------------------------------------- wooden_fixing · 2023-10-25 02:28 Totally agree, worth considering! ---------------------------------------------------------------------- desnudopenguino · 2023-10-25 03:05 I'd say it depends. I run BSD's in my setup, with a freebsd "development" server running a bunch of jails for projects. I find that easier to maintain than a vm host lately. I just have one more vm to port over to a jail, then I'm retiring my 12 year old server, and looking for a new one. I have another freebsd box running as a file server for the house, where I drop all our media and a copy of important backups. I kind of use bsd jails like containers. Once I get one working, I can package it and send it to a server, jnpackage it, and run it with minimal effort/reconfiguration. ---------------------------------------------------------------------- DIYSRE · 2023-10-25 04:02 99% of the time, you won't see performance benefits from running things on bare metal compared to a hypervisor. QEMU VMs can be NUMA aware, so there is theoretically next to no difference between running services in QEMU VMs with the correct configuration and running them on bare metal. Additionally, a good hypervisor will support IOMMU. There will be a performance impact for doing this however I highly doubt it would be noticeable unless you know that you need to notice it. Generally speaking, the only time that you care about the performance impact is when you're using every bit of performance out of your hardware and you're trying to min-max that load. There are better solutions to moving to bare metal in these circumstances. The only time that I prefer bare metal to virtualization is for storage as it's one less layer of complexity to account for. My home storage nodes have direct access to their disks (ZFS stuff) and I just run K3S ontop to handle services like NFS. I also *prefer* to run my firewalls as physical applications. This is purely just a mental siloing of hardware doing a job, and to keep the physical connections simplistic for L1/L2 techs. I have run firewalls "on a stick" or as VMs and done tricky bridging/vneting, and they work perfectly fine servicing load beyond what home servers would experience. Enterprise orgs can run Samba servers (whatever MS calls it) on VMDKs. This is fine and it works for them and their VMWare based setups. From what I am told, they get some benefits to doing this however they also don't mind overspending on gear to keep these kinds of deployments redundant. In previous setups, we've purchased units like Synology or EMC and just let the hardware do what the hardware was designed to do. Compute nodes would be purchased and the SAN/NAS accessed via something like iSCSI or NFS. You do see some performance impact doing this, but it can be mitigated "cheaply" (in the enterprise world at least) with networking that surpasses your read/write speeds and using SPT to keep latency low. Beyond min-maxing extreme use-case scenarios, bare metal deployments will also be vendor specific. If that vendor has an implementation plan that calls for bare metal - for whatever reason they want - then it will be bare metal. In a business, you want to adhere to the specification of your vendor for support reasons. At home, do whatever you want. ---------------------------------------------------------------------- usrdef · 2023-10-25 04:36 Another note. I've noticed some people like to use Cpanel for some reason. Cpanel sells licensing depending on if it's a bare metal server or virtual / VPS. Bare metal licenses cost more. ---------------------------------------------------------------------- ElevenNotes · 2023-10-25 04:52 The license cost per server. As bare metal there is no license cost. ---------------------------------------------------------------------- ElevenNotes · 2023-10-25 04:54 There is, another 🏴‍☠️ ---------------------------------------------------------------------- PaulEngineer-89 · 2023-10-25 10:51 One big advantage of VMs is better resource allocation. If you have multiple different server types load leveling is better (fewer idle cores) with VMs. You also have the security implications. These days I tend to run Docker though and eliminate VM overhead even more. Another major benefit is security. Say someone hacks your web server. Ok so from there they are stuck inside the VM. Assuming you’ve practiced zero tier compromising that one server is useless against the others. Plus there is not just backups but maintenance and recovery. If a server has a hardware failure (say one of a couple PSUs or a fan) with a VM environment you can just move the servers over or set them to auto boot on a second hardware server. And recovery is simply copying over the VM and booting it in seconds. I ran performance tests years ago on a couple Dells with VMWare vs Xen vs bare metal. What I found is that VMWare has better advertising than Xen but basically uses the same software on a buried RHEL host. Performance wise on any load it was something like 99.7% throughput/CPU vs bare metal. So there is a difference but it literally comes down to roughly the overhead of running the VM if it was a process on the VM instead of the host. As far as pass through hardware this has gotten to be less and less of a thing. Only a few annoying products “require” bare metal (TrueNAS). Not so much licensing as just stupid implementations. You aren’t “losing benefits of VMs” though except perhaps storage allocation flexibility or sharing a GPU. ---------------------------------------------------------------------- Fluffer_Wuffer · 2023-10-25 12:52 Bare metal for storage, such as a Synology NAS. Everything else gets virtualised, even the firewall. Why? Several reasons, it means your getting better utilisation of the hardware, it's much easier (and quicker) to backup and recover VMs.. you can migrate your VMs between machines, for example if you want to upgrade one, but keep all servers/services online.. The hit you take from virtualising is negligible, at least when it comes to 99% of services that people here run. Few people need to use bare metal, only in extenuating circumstances, such as performance.. I.e. crypto, or data.analysis ---------------------------------------------------------------------- nyrosis · 2023-10-25 14:33 This... It's good to have the router/firewall as it's own device. An argument could be made for a NAS too simply doing NAS functions. For servers I see no point because in the home environment you can squeeze more out of the system using a hypervisor. Even in an enterprise environment you are likely thinking about clustering/HA which still will be utilizing a hypervisor. ---------------------------------------------------------------------- antaresuk · 2023-10-27 14:45 Latency sensitive operations like gaming and downloading linux torrents. Makes me think unraid's claim of "hard core gamers" is a joke :p ---------------------------------------------------------------------- Pvt-Snafu · 2023-10-27 15:04 Yup, the biggest limitation for me with ESXi free is that there is no vStorage API so no easy way to backup VMs for example with Veeam CE or something else. ---------------------------------------------------------------------- DarkKnyt · 2023-10-27 16:59 Fwiw I recently learned there is a vmug version which is $200 a year. It has all the features but is limited to 6 sockets. ---------------------------------------------------------------------- Pvt-Snafu · 2023-11-09 18:13 Right, forgot to mention VMUG Advantage. It's basically Enterprise Plus. Just need to renew once a year.