Optimizations to get best network performance on Linux VMs in bhyve? reddit / freebsd ---------------------------------------------------------------------- alexsm_ · 2023-06-15 21:23 I’m looking for advice and/or tips to have an optimized network performance in bhyve VMs running Ubuntu, Alma, Rocky and other Linux flavors. Testing different virtualization solutions - both FreeBSD+bhyve and Linux+KVM - on an old HP server, I am seeing iperf3 in a Ubuntu 20.04 VM over bhyve with network throughput around ~600Mbps on a 1GbE NIC, while the same hardware running Proxmox7.2 running an Ubuntu 20.04 VM gave better throughput. As there’s a need to run different networking applications mostly from open-source R&D projects either in CLI or web-based apps developed mostly in / for Linux environments. Note: There’s no manpower and/or time to port the apps to run natively on FreeBSD or under Linuxulator, so that’s why the need to support multiple Linux flavors, kernels and package systems. There’s no need for desktop or graphical environments, as they are mostly server apps. So the idea is to have a flexible and stable hypervisor to run any other Linux OS required by the network apps for R&D and network performance measurements tests, as well as network automation and orchestration softwares. Current setup: OS: FreeBSD 13.2 + Bhyve Hardware: HP ProLiant DL380 G7 - 2x Xeon X5660 2.80GHz - 96GB RAM - 5x 300GB SAS HDDs Current VMs have different specs and run: - Ubuntu 18.04, 20.04 and 22.04 - Alma Linux (Latest) - Rocky Linux (Latest) There are also some containerd and Docker containers in the mix. I’ve checked that iperf3 running in a Ubuntu 20.04 VM on the HW platform above is achieving around ~600Mbps on a TCP throughput test with a 1GbE NIC testing to another baremetal Linux node with 1GbE NIC. iperf3 throughput was higher - around ~900Mbps on a Ubuntu 20.04 VM created by Proxmox 7.2 on baremetal, when testing against a dedicated baremetal server node at 1Gbps running CentOS 7. Having a great past experience managing FreeBSD on production internet servers from 3.x-RELEASE to 8.x-RELEASE & CURRENT using ZFS, I rather run FreeBSD servers anytime, but as I haven’t worked with bhyve before, I don’t have reference numbers to compare it’s performance with the multiple Linux virtualization combinations to find out which performs best overall in terms of network performance. As there are many layers of hardware, software, network and protocol stacks that can impact end to end performance, including the network itself and remote nodes, I’m hoping someone have tried some combinations before and can share some insights. ---------------------------------------------------------------------- ethanappgenius · 2023-06-15 23:42 Sure, here are some tips for optimizing network performance on Linux VMs in bhyve: * **Use a recent version of FreeBSD.** FreeBSD 13.2 is a good choice, as it includes a number of performance improvements for bhyve. * **Use a high-performance NIC.** A 1GbE NIC is the minimum requirement, but a 10GbE NIC will give you better performance. * **Enable jumbo frames.** Jumbo frames are larger than standard Ethernet frames, and they can improve performance by reducing the number of network packets that need to be sent and received. * **Configure the VM's network interface with a high transmit queue depth.** This will allow the VM to send more packets in parallel, which can improve throughput. * **Use a fast storage device.** The VM's storage device can have a significant impact on network performance. A fast SSD will give you better performance than a traditional hard drive. * **Reduce the VM's memory overhead.** The more memory that a VM uses, the more CPU time it will need to access that memory. Reducing the VM's memory overhead can improve network performance. * **Disable unnecessary services in the VM.** Running unnecessary services in a VM can consume CPU time and memory, which can impact network performance. * **Update the VM's kernel and drivers.** Outdated kernels and drivers can cause performance problems. Make sure to update the VM's kernel and drivers to the latest versions. By following these tips, you can improve network performance on Linux VMs in bhyve. Here are some additional tips that can help improve network performance in general: * **Use a wired connection instead of a wireless connection.** Wired connections are generally more reliable and faster than wireless connections. * **Keep your network devices up to date.** Network devices, such as routers and switches, can become outdated over time. Keeping your network devices up to date can help improve performance. * **Avoid using too many network devices.** Too many network devices can add latency and reduce throughput. * **Use a good quality network cable.** A good quality network cable can help improve performance. * **Place your network devices in a central location.** This will help reduce the distance that network packets have to travel, which can improve performance. * **Use a network monitoring tool.** A network monitoring tool can help you identify and troubleshoot performance problems. By following these tips, you can improve network performance in your environment. ---------------------------------------------------------------------- _arthur_ · 2023-06-16 17:54 You don't specify how you're enabling networking in the VMs, but if at all possible to PCI passthrough of VF devices rather than tap interfaces. tap performance will make everyone sad. ---------------------------------------------------------------------- edthesmokebeard · 2023-06-17 00:25 " I’m looking for advice and/or tips " yet you like to some random github page - astroturf much? ---------------------------------------------------------------------- edthesmokebeard · 2023-06-17 00:25 So you're saying to use faster hardware, so that your machine runs faster? genius. ---------------------------------------------------------------------- ethanappgenius · 2023-06-17 00:32 obviously ---------------------------------------------------------------------- alexsm_ · 2023-06-21 11:15 Thanks for the heads up. Yes, the VMs were created in bhyve using BVCP - which is how they are being managed - and they are using tap interfaces. I could not find PCI passthrough config option in BVCP. I’ll enable it manually. ---------------------------------------------------------------------- AveryFreeman · 2023-09-03 07:16 These are all good points, but why is the Linux solution providing better throughput OTB? Want to see the forest through the trees...