[HN Gopher] Linux Hypervisor Setup (Libvirt/QEMU/KVM)
___________________________________________________________________
Linux Hypervisor Setup (Libvirt/QEMU/KVM)
Author : willdaly
Score : 154 points
Date : 2021-05-01 13:58 UTC (9 hours ago)
(HTM) web link (octetz.com)
(TXT) w3m dump (octetz.com)
| sneak wrote:
| What do you all use for quick one-command (non gui) new vm
| bringup?
|
| I haven't had much luck with terraform with libvirt, and I miss
| Vagrant but don't want to lock myself into its syntax. Maybe I
| should just start using it again but I feel like there has got to
| be a good libvirt-based way.
|
| Provisioning new test vms with userdata embedded into an
| kickstart-or-whatever iso that is provided to libvirt also seems
| clunky. docker-machine is no longer maintained so I can't use it
| as a vagrant replacement to just bring up sshable hosts.
| Multipass?
|
| Curious as to how others have solved this.
| bonzini wrote:
| virt-install or virt-builder.
| sneak wrote:
| I have some extremely clunky 50 line shell scripts around
| virt-install presently but was hoping to find something a
| little more streamlined (like the usual Vagrant or docker-
| machine workflow).
|
| Do you just manually run a huge virt-install command each
| time?
| bonzini wrote:
| I usually use virt-builder actually. But it's like 3-4
| command line options for virt-install, not 50.
| ohyeshedid wrote:
| Have you looked into Weaveworks Ignite?
|
| https://github.com/weaveworks/ignite
| madjam002 wrote:
| You mention Terraform, I'm using
| https://github.com/dmacvicar/terraform-provider-libvirt quite
| heavily and it works a treat.
| sneak wrote:
| That's the one I've had suboptimal experience with. Maybe I
| need to learn it better. What kind of storage are you using
| it with? I ran into issues using file-backed trying to keep
| everything in tf.
| vegardx wrote:
| I was playing around with this a while ago, using cloud-init
| and virt-install. But I found it to be quite cumbersome if you
| don't have some kind of metadata service that can provide
| cloud-init with the user-data, ie. not using the nocloud
| provider.
|
| I started looking at this mock EC2 metadata service, but never
| did anything with it: https://github.com/sjjf/md_server
|
| Ninja-edit: It seems like virt-install rencently abstracted the
| whole NoCloud-provider, this changes a lot of things:
| https://blog.wikichoon.com/2020/09/virt-install-cloud-init.h...
| SteveNuts wrote:
| We implemented our own mock ec2 metadata service for KVM
| cloud-init, probably the most fun project I've worked on.
|
| The tricky part is intercepting the requests and redirecting
| them to your service
| mazatta wrote:
| I've had luck with Proxmox and this Terraform provider:
| https://registry.terraform.io/providers/Telmate/proxmox/late...
|
| Other than a working Proxmox install, you'll need to create a
| cloud-init template, which is documented here:
| https://pve.proxmox.com/wiki/Cloud-Init_Support
| hyperpl wrote:
| I use qemu and never really understood the need for any further
| abstraction such as libvirt over it's cli usage. Can anyone tell
| me what there is to be gained other than needing further setup,
| packages and systemd services?
| dijit wrote:
| libvirt is mainly a consistency wrapper. So you can use kvm or
| lxc.
|
| As for what it "gives", I suppose I've never migrated a VM
| outside of libvirt, though I'm not 100% sure if that's not
| possible with plain old qemu+kvm
| kbumsik wrote:
| I am not an expert but libvirt is not only for qemu but it
| supports various virtualization technologies such as Xen, LXC,
| VMware. So if you decide to stick with QEMU then I guess you
| don't need to use libvirt.
| bonzini wrote:
| This is a blog post I wrote about the KVM userspace
| architecture, it goes into why Red Hat uses Libvirt in its
| virtualization products.
|
| https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u...
| (fixed now)
| dooglius wrote:
| Getting a 404
| curiousgal wrote:
| https://www.redhat.com/en/blog/all-you-need-know-about-
| kvm-u...
| teilo wrote:
| It's not just about abstraction. VirtIO is significantly more
| efficient than SATA emulation, and as it is built into the
| Linux kernel it just works. There is also a Windows driver
| package that adds VirtIO support, but it's a bit tricky to get
| it to work when porting in an existing Windows VM. VirtIO also
| makes it possible to do USB relay.
| benlwalker wrote:
| There's more cool stuff coming in this area too. For a long
| time there's been the virtio family of protocols for
| shuttling IO to something outside QEMU to handle. Originally
| that was always KVM and the implementation is called vhost.
| Then later it became clear that these same messages could be
| sent to another user space process to handle instead (called
| vhost-user). These work great for creating virtio devices in
| the guest. But operating systems like Windows don't have
| virtio device drivers in-box, so it's a little annoying.
|
| Recently, a new protocol to replace virtio has been defined.
| It is modeled on vfio ioctls and currently only can forward
| to another user space process, so we're calling it vfio-user.
| With this protocol, it's possible to emulate any PCI device
| rather than only virtio devices. Projects like SPDK (what I
| work on) can now use this to present fully emulated NVMe
| devices into guests and back them with whatever actual
| storage is available (a file, something over the network, a
| real NVMe SSD, etc). This allows an OS, including Windows, to
| boot from the virtual disk using it's in-box NVMe driver.
| This hasn't quite made it into a QEMU release yet, but it's
| close!
| alschwalm wrote:
| VirtIO is very usable via QEMU, without libvirt (naturally,
| because in the configuration described in the article,
| libvirt is just calling QEMU). It is usually as simple as
| `qemu-system-x86_64 -drive file=/path/to/my/disk,if=virtio`.
| mackal wrote:
| When I started using virt-manager I started there because I was
| specifically looking to replace Virtual box due to having
| kernel modules for VB sometimes block kernel upgrades. So I
| guess I was looking for a GUI since I was on a desktop anyways.
| (Just have some dev VMs so I don't have to clutter my desktop
| with running MySQL etc for some projects and so I can match the
| distro of the production server)
|
| So I guess, it worked, didn't look into raw qemu. I also use it
| infrequently enough that I would have to relearn the CLI all
| the time.
| linsomniac wrote:
| It's been probably a decade since I've used libvirt, but I can
| say why I like using ganeti:
|
| - I can get a list of the host machines in my cluster and how
| much memory and storage they have available.
|
| - I can easily move VMs between hosts if I want to evacuate a
| host for hardware/software/firmware maintenance.
|
| - It has the ability to set up DRBD backed VMs and live migrate
| between the host nodes.
|
| - List what machines are running and on what hosts.
|
| - Start and stop commmands don't require me to remember the
| settings on individual VMs, the qemu commands that get run are
| something like 700 characters long.
| MegaDeKay wrote:
| I'm like you and use the qemu command line. But libvirt does
| make it easier to define cpu groups, thread affinities and the
| like and without needing to be root to do so. Something like
| this. There are other ways to accomplish this I think without
| libvirt, but it gets a little hairy.
|
| https://www.reddit.com/r/VFIO/comments/9iaj7x/smt_not_suppor...
| psanford wrote:
| I've been quite happy with LXD's support for VMs. Its really easy
| to pull an image an have a running VM with a single command.
| inshadows wrote:
| LXD does virtual machines now? I though it does only
| "containers" (running processes namespaces, like Docker).
| HumblyTossed wrote:
| Proxmox exists.
| zaat wrote:
| One annoyance I had with the defaults are the short dhcp lease
| time configured for dnsmasq, causing repeated dhcp lease logging
| in the logs. Turns out you can configure the lease time using
| virsh net-edit. The libvrit docs contain additional useful
| configuration options.
|
| Another small thing but big quality of life improvement is
| exporting LIBVIRT_DEFAULT_URI='qemu:///system', so that
| everything you execute uses the correct connection.
| guerby wrote:
| Cockpit is a nice web UI for libvirt:
|
| https://cockpit-project.org/
|
| On debian add backports then it's as simple as "apt-get install
| cockpit-machines" then point your browser to
| http://localhost:9090
| 7kay wrote:
| Do I guess correctly that you have cockpit running on the host
| and manage guests via the cockpit-machines plugin? If so, do
| you have any hints on where to find documentation about that? A
| cursory glance at the repo didn't produce much information.
| invokestatic wrote:
| Probably my favorite feature of libvirt is the security and
| isolation features provided by sVirt. It applies a security
| policy via SELinux (or AppArmor) that ensures that in the event
| of a VM breakout exploit, the attacker can only access resources
| allocated to that VM. So it isolates VMs from the host and from
| each other. Really cool!
| egberts1 wrote:
| I've been using Proxmox community edition for my home datacenter.
|
| It's good to see what this article details into it and how some
| can pull all this together ... for free (or for a small sum).
|
| URL: https://proxmox.com/en/proxmox-ve/features
| linsomniac wrote:
| I was recently going to set up Proxmox as well, since I had
| used it ~7 years ago and it was working well. But I ran into an
| expectation that I have a drive dedicated to putting ISOs on,
| in addition to the OS install and the virtual machine storage.
| I was trying to set it up on a SFF machine and was limited in
| what I could put into it and I just kind of noped out of it. I
| couldn't install any VMs until I did this.
|
| I don't recall this being an issue before.
|
| I ended up re-installing TrueNAS Scale beta, which I had
| installed and tried, but ran into a problem. Ended up finding
| the solution on the forums, something related to bridged or
| VLAN interfaces not coming up.
|
| In the end I probably will just install Ubuntu and put Ganeti
| on these boxes, since that's been really reliable for me at
| work. I was actually hoping to try out Proxmox to possibly move
| to at work.
| ylk wrote:
| What made you think that you need a separate disk just for
| ISOs? They can be uploaded to and stored on the default
| directory storage.
| synergy20 wrote:
| One thing is missing comparing to virtualbox, is the bridge-mode
| especially when host uses wireless which is very common these
| days.
|
| virtualbox hacked bridge-mode and made it working well all the
| time, with libvirt/qemu/kvm you will need write a lot of scripts
| and click around to set this up.
|
| This is the sole reason I had to stick with virtualbox these
| years.
| MuffinFlavored wrote:
| I'm a big VirtualBox fan as well and I'm sad to see that Apple
| M1 (ARM) support isn't really a priority.
| numbsafari wrote:
| I get the impression Oracle is killing VirtualBox through
| neglect.
| 2ion wrote:
| Then you need to compare to the (for non-commercial use free )
| vmware workstation (player) as well.
|
| - Simple OpenGL accelerated guest graphics (alternatives: intel
| gvt-g, card passthrough, virgl in qemu (not generally
| available), SPICE+qxl in qemu (not nearly as performant))
|
| - bridged networking "just works"
|
| - open-vm-tools enable quite an interesting array of
| comfortable guest integration
|
| - disk image format can be handled by qemu-img for conversion
| purposes, so it's easy to migrate vmware <> virtualbox <>
| qemu+kvm.
|
| The graphics part is why for desktop VMs, I prefer the vmware
| solution at the moment.
| neurostimulant wrote:
| This is the main virtualbox feature I sorely missed running
| kvm. Is there any easy way to configure bridge mode on kvm? I
| didn't dig too much because I also run docker and kubernetes on
| the same machine and I don't want to break their network config
| accidentally.
| VMtest wrote:
| If anyone knows how to setup the bridge with wireless in
| libvirt/qemu/kvm following the steps with/without ebtables
| listed in Debian and Arch wiki, please do tell
|
| 1.
| https://wiki.archlinux.org/index.php/QEMU#Network_sharing_be...
|
| 2.
| https://wiki.archlinux.org/index.php/Network_bridge#Wireless...
|
| 3.
| https://wiki.debian.org/BridgeNetworkConnections#Bridging_wi...
| qwertox wrote:
| How annoying that this page uses a non-standard port for the
| WebSocket "livereload" feature (1313).
|
| It makes my firewall pop up constantly.
| [deleted]
| cyberlab wrote:
| Browse with JS disabled by default. Then in uBlock Origin you
| can temporarily whitelist the page and browse it with JS
| enabled if it really requires it.
| ffeiek wrote:
| Qubes OS (Libvirt/QEMU/Xen) would also be good
| sascha_sl wrote:
| libvirt has too many gotchas to be as easy to use as virtualbox,
| vmware or proxmox. Proxmox also uses qemu+kvm, but is opinionated
| enough to pick the most appropriate option by default.
|
| Example? For Networking, you probably want the virtio driver for
| maximum throughput if it is supported on the guest (even Windows
| has drivers now), using the emulated realtek device/e1000 is very
| slow. That means you should probably also use the virtio block
| devices, right? Nope, wrong, work on that was abandoned years
| ago, use SATA!
| nzmsv wrote:
| The point about storage doesn't seem to be true:
| https://www.qemu.org/2021/01/19/virtio-blk-scsi-configuratio...
|
| However, I am not an expert and would love to learn more.
| Intuitively though I wouldn't expect emulated SATA to be better
| than paravirt of any kind.
___________________________________________________________________
(page generated 2021-05-01 23:01 UTC)