Which setup do you recommend for a router + server on the same computer? reddit / BSD ---------------------------------------------------------------------- _inf3rno · 2021-01-05 05:00 Currently I don't have the hardware to have a dedicated router, so I am trying to figure out how to do it with my microserver. What is clear that I need to give one core to the router and some dedicated memory, so when there is load on the server, then it won't affect network performance. Some people suggested to use virtualization, while others to use jails, but even if I do virtualization some wrote that better to do it with ESXi, because network bandwidth was low for him with all Linux hypervisors, though I am not sure if the guy tried FreeBSD + bhyve too. Yet another thing here, that I read somewhere that OpenBSD+PF is more secure and a lot faster than FreeBSD+PF, while on the other hand I just read that OpenBSD on a virtual machine can be really slow in networking compared to FreeBSD. So what I have here is a complete mess and I rather ask than experimenting for months. What do you recommend for this problem, jails or virtual machines and if the latter, then which OS to use for the hypervisor (ESXi/FreeBSD) and for the router (OpenBSD/FreeBSD) part? ---------------------------------------------------------------------- DiggyTroll · 2021-01-05 06:27 First, I would let the intended applications drive the choice. What OS is recommended/required? If you’re just needing standard services (DNS, SMTP, etc), any BSD would be fine. Don’t forget to look at pfSense and its packages as an easy router/firewall management choice. Jails are fine, but defer installing a hypervisor until you can justify a need for it under load. You easily can add one later. In any case, You won’t even need to worry about network load on a box with multiple 1GbE ports; modern *BSD handles them efficiently, even under high load. You don’t even need to worry about 10GbE as long as you use the NIC’s offloading features (unless using pfSense - leave them off then). ---------------------------------------------------------------------- northrupthebandgeek · 2021-01-05 08:30 Worth noting that [OpenBSD's included hypervisor can run OpenBSD (and Linux)](https://www.openbsd.org/faq/faq16.html). So you could run OpenBSD as the hypervisor, with the host doing all routing via PF (thus avoiding any potential hypervisor-related bottlenecks) and with one or more OpenBSD or Linux VMs running server applications. One downside of this approach is that OpenBSD's vmm does not (currently) support guest SMP, so you'd be limited to one virtual CPU per VM. Whether this'll jive with your expected "server" roles depends on exactly what the server part would be running; if it's something like HTTP that can be readily load-balanced across VMs, or if it's something like DNS/DHCP/etc. that would have a hard time saturating a single core anyway, then this is likely barely an inconvenience, but if it's something beefy like a (heavily-used) SQL database you'll need to either get real comfortable with sharding / read replicas or consider a different approach. ---------------------------------------------------------------------- johnklos · 2021-01-05 16:18 CPU load won't affect network performance unless you're seriously resource starved and are actively pushing tons of traffic. Virtualization will add more overhead than it can save unless you're doing something very wrong in the native OS. I'd recommend you forget about virtualizing and just run FreeBSD (if you care so much about performance) for both serving and routing and be done with it. For reference, I run NetBSD on a 2 GHz quad core AMD Athlon (2014 AM1 type with DDR3) and I can get the full 500 Mbps of the Internet connection on clients behind it while running BIND, BitTorrent, ffmpeg and more while doing NAT for IPv4 and routing IPv6. It also bridges over a tinc VPN at the same time. I've never had any slowdowns related to CPU load. ---------------------------------------------------------------------- _inf3rno · 2021-01-05 16:27 Thanks! Currently I need only 1Gbps routing and 10Gbps on the local network. Well my only real problem from application perspective that I need to do machine learning on an nvidia card. I researched the topic today and FreeBSD barely supports machine learning. Maybe OpenCL is available, but CUDA+Tensorflow would be a lot better for me currently. So I think I will need Linux too and I read that bhyve does not support PCI passthrough, so I'll need something like ESXi. I'll check the other hypervisors too, but so far it looks like I'll have ESXi as hypervisor, some Linux for machine learning, FreeBSD for webserver and maybe pfsense or OpenBSD+PF for router+firewall. ---------------------------------------------------------------------- _inf3rno · 2021-01-05 16:33 Sadly I need CUDA and PCI passthrough does not seem to be something stable atm. neither on FreeBSD+bhyve nor on OpenBSD+VMM. [https://www. reddit.com/r/openbsd/comments/ih4vns/pci\_passthrough\_on\_vmm/](https ://www.reddit.com/r/openbsd/comments/ih4vns/pci_passthrough_on_vmm/) Maybe in the future I can use these as hypervisor too, but I'll need to do some machine learning early this year so I can't wait. ---------------------------------------------------------------------- northrupthebandgeek · 2021-01-05 18:17 Ah, then yeah, that kinda narrows it to KVM, Xen, and ESXi as your hypervisors. It also rules out OpenBSD for the "server" part, since I'm assuming you'd be using an Nvidia GPU (given that CUDA's an Nvidia thing) and OpenBSD's Nvidia support is pretty lacking last I checked. My recommendation would be to try either Xen (with whatever dom0 you want; FreeBSD, NetBSD, and Linux all work) or ESXi for the hypervisor, FreeBSD for the "server" stuff, and OpenBSD for the "router" stuff. Both FreeBSD and OpenBSD support paravirtualized disk and network IO under both Xen and ESXi, so I would expect both to be reasonably performant in either case. And both Xen and ESXi (ostensibly) support Nvidia GPU passthrough (though I'm assuming you've already verified your GPU itself supports this, since not all do). ---------------------------------------------------------------------- _inf3rno · 2021-01-05 18:39 It is a GTX 750 Ti, I am not sure if it is possible to reach CUDA from the guests. Afaik. the Windows drivers actively block this kind of usage for gaming cards and some people hack around it. Not sure what is possible with the Linux drivers or if the guest OS can reach the card directly on any hypervisor without going through additional layers. Usually if they add a virtual video card in the middle, then CUDA support is not implemented for it. Worst case I'll run the code on the host OS or buy an AMD card with ROCm and forget about Nvidia GPUs and CUDA entirely. Even with ROCm I could not use a BSD OS, so please upvote this, maybe they will do it sooner if they see it is needed by many ppl: [https://github.com/RadeonOpenCompute/ROCm/issues/ 138](https://github.com/RadeonOpenCompute/ROCm/issues/138) Or just open a new issue. ---------------------------------------------------------------------- northrupthebandgeek · 2021-01-05 19:20 > GTX 750 Ti If that's the case, then that narrows you down to ESXi; [this guide](https://tysonz.net/nvidia-gpu-passthrough-esxi-6-7/) looks to be a solid bet, with the important thing being to set `Hypervisor.CPUID.v0 = "FALSE"` (which prevents ESXi from telling the guest that it's a VM, and therefore prevents consumer Nvidia GPUs from knowing they're in a VM and freaking out). FreeBSD should still work for the CUDA stuff (at least as well as it'd work on bare metal), and OpenBSD should still work for the router stuff. (There were some whispers of Xen supporting a similar method of CPUID masking, but I can't find solid info on whether those patches actually landed) ---------------------------------------------------------------------- _inf3rno · 2021-01-05 19:35 Wow, sounds promising! I found another solution, Tensorflow has a Tensorflow.js project which uses WebGL, but I guess performance would be less than 10% compared to CUDA. Yours is a lot better. ---------------------------------------------------------------------- northrupthebandgeek · 2021-01-05 19:40 I should probably disclaim the above by stating that I haven't touched FreeBSD, ESXi, or Nvidia GPUs in at least 5 years, so *caveat emptor*. But best of luck and let us know how it goes :) ---------------------------------------------------------------------- ZarK-eh · 2021-01-05 21:42 I used esxi on a perimer and vm router... Kinda felt like hanging yer fruit for all to see. ***shrugs*** I now use OPNsense on a mini/micro i3 with sata ssd (Web proxy)! Maybe even look and see if your consumer router can run the likes of OpenWrt/LeDe or dd-wrt or other FOSS firmware? If I was doing it again, I would try truenas/freenas iocage bhyve for ease of setup (and keep it bsd-ish related) ---------------------------------------------------------------------- _inf3rno · 2021-01-06 05:26 Okay, I will! I'll try out both CUDA and Tensorflow.js and I have an RX 580 in my PC, so I'll try ROCm too. ---------------------------------------------------------------------- _inf3rno · 2021-01-06 05:29 The current router is a modem+router owned by the ISP. I could buy a SOHO router and use OpenWRT, but I'd like to learn PF. I think even pfsense would hide too much from me. If this project will be a success, then I'll build a dedicated router, but for now I don't want to spend money on it, if I can try it out on my microserver.