If you are running an Ubuntu host, you have multiple choices for a virtualization hypervisor. I have written up several articles on using VirtualBox, but now letâÂÂs consider a bare metal hypervisor like KVM.
KVM is a type 1 hypervisor implemented as a Linux kernel module that utilizes a modern processorâÂÂs virtualization extensions, making it capable of direct CPU execution without translation. Each virtual machine is a regular Linux process, scheduled by the standard Linux scheduler.
An an example of something that KVM can do that VirtualBox cannot, KVM has the ability to pass on virtualization capability to its guest OS, which would allow nested virtualization.
Installation
First, install KVM and assorted tools:
sudo apt-get install qemu-system-x86 qemu-kvm qemu libvirt-dev libvirt-clients virt-manager virtinst bridge-utils cpu-checker virt-viewer -y # less than Ubuntu 20 sudo apt-get install libvirt-bin
Then validate that that KVM was installed and that the CPU has VT-x virtualization enabled with kvm-ok.
$ sudo kvm-ok INFO: /dev/kvm exists KVM acceleration can be used
If you instead get a message that looks like below, then go in at the BIOS level and enable VT-x.
INFO: /dev/kvm does not exist HINT: sudo modprobe kvm_intel INFO: Your CPU supports KVM extensions INFO: KVM (vmx) is disabled by your BIOS HINT: Enter your BIOS setup and enable Virtualization Technology (VT), and then hard poweroff/poweron your system KVM acceleration can NOT be used
Then run the virt-host-validate utility to run a whole set of checks against your virtualization ability and KVM readiness.
# if this fails, you may have an older version still installed virt-host-validate --version # the newest version comes from /usr/bin (not /usr/local/bin) which virt-host-validate # this utility comes from the libvirt-clients package sudo virt-host-validate
Relax AppArmor enforcement
Ubuntu comes with AppArmor enabled for libvirt. Disable the security_driver, set to an empty set of namespaces, and restart libvirt or you may not be able to create a guest VM.
echo 'security_driver = "none"' | sudo tee -a /etc/libvirt/qemu.conf echo 'namespaces =| sudo tee -a /etc/libvirt/qemu.conf sudo systemctl restart libvirtd sudo systemctl status libvirtd
You should not have to modify any of the AppArmor profiles to complain. But if you must, there are notes at the bottom of this article.
Add user to libvirt groups
So that we can manage the guest VM as a regular user, we can add ourselves to all of the libvirt groups (e.g. libvirt, libvirt-qemu) and the kvm group.
cat /etc/group | grep libvirt | awk -F {'print $1'} | xargs -n1 sudo adduser $USER # add user to kvm group also sudo adduser $USER kvm # relogin, then show group membership exec su -l $USER id | grep libvirt
Group membership requires a user to log back in, so if the âÂÂidâ command does not show your libvirt* group membership, logout and log back in, or try âÂÂexec su -l $USERâÂÂ.
QEMU connection to system
If not explicitly set, the userspace QEMU connection will be to âÂÂqemusessionâÂÂ, and not to âÂÂqemusystemâÂÂ. This will cause you to see different domains, networks, and disk pool when executing virsh as your regular user versus sudo.
Modify your profile so that the environment variable below is exported to your login sessions.
# use same connection and objects as sudo LIBVIRT_DEFAULT_URI=qemusystem
Default network
By default, KVM creates a virtual switch that shows up as a host interface named âÂÂvirbr0â using 192.168.122.0/24. The image below is courtesy of libvirt.org.
httpsfabianlee.org/wp-content/uploads/2018/08/libvirt_network_default_network_overview.jpg
This interface should be visible from the Host using the âÂÂipâ command below.
$ ip addr show virbr0 3: virbr0: mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever
âÂÂvirbr0â operates in NAT mode, which allows the guest OS to communicate out, but only allowing the Host(and those VMs in its subnet) to make incoming connections.
Bridged network
To enable guest VMs on the same network as the Host, you should create a bridged network to your physical interface (e.g. eth0, ens4, epn1s0).
Read my article here for how to use NetPlan on Ubuntu to bridge your physical network interface to âÂÂbr0â at the OS level. And then use that to create a libvirt network named âÂÂhost-bridgeâ that uses br0.
# bridge to physical network $ virsh net-dumpxml host-bridge host-bridge 44d2c3f5-6301-4fc6-be81-5ae2be4a47d8
This âÂÂhost-bridgeâ will be required in later articles.
Enable IPv4 forwarding on KVM host
In order to handle NAT and routed networks for KVM, enable IPv4 forwarding on this host.
# this needs to be "1" cat /proc/sys/net/ipv4/ip_forward # if not, then add it echo net.ipv4.ip_forward=1 | sudo tee -a /etc/sysctl.conf # make permanent sudo sysctl -p /etc/sysctl.conf
Default storage pool
The âÂÂdefaultâ storage pool for guest disks is âÂÂ/var/lib/libvirt/imagesâÂÂ. This is fine for test purposes, but if you have another mount that you want to use for guest OS disks, then you should create a custom storage pool.
Below are the commands to create a âÂÂkvmpoolâ on an SSD mounted at âÂÂ/data/kvm/poolâÂÂ.
$ virsh pool-list --all Name State Autostartdefault active yes $ virsh pool-define-as kvmpool --type dir --target /data/kvm/pool Pool kvmpool defined $ virsh pool-list --all $ virsh pool-start kvmpool $ virsh pool-autostart kvmpool $ virsh pool-list --all Name State Autostartdefault active yes kvmpool active yes
Create basic VM using virt-install
In order to test you need an OS boot image. Since we are on an Ubuntu host, letâÂÂs download the ISO for the network installer of Ubuntu 20.04 Focal. This file is only 74Mb, so it is perfect for testing. When complete, you should have a local file named Downloads/mini.isoâÂÂ
wget httparchive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/current/legacy-images/netboot/mini.iso
First list what virtual machines are running on our host:
# chown is only necessary if virsh was run previously as sudo ls -l virtinst sudo chown -R $USER:$USER virtinst # list VMs virsh list --all
This should return an empty list of VMs, because no guest OS have been deployed. Create your first guest VM with 1 vcpu/1G RAM using the default virbr0 NAT network and default pool storage.
virt-install --virt-type=kvm --name=ukvm-focal --ram 1024 --vcpus=1 --virt-type=kvm --hvm --cdrom ~/Downloads/mini.iso --network network=default --graphics vnc --disk pool=default,size=20,bus=virtio,format=qcow2 --noautoconsole # open console to VM virt-viewer ukvm-focal
âÂÂvirt-viewerâ will popup a window for the Guest OS, when you click the mouse in the window and then press you will see the initial Ubuntu network install screen.
If you want to delete this guest OS completely, close the GUI window opened with virt-viewer, then use the following commands:
virshukvm-focal virsh undefine ukvm-focal
Test from GUI
The virt-viewer utility will open a basic window to the guest OS, but notice it does not give any control beside sending keys. If you want a full GUI for managing KVM, I would suggest using âÂÂvirt-managerâÂÂ.
httpsfabianlee.org/wp-content/uploads/2018/08/kvm-virt-manager.png
To install and start virt-manager:
sudo apt-get install qemu-system virt-manager virt-manager
virt-manager provides a convenient interface for creating or managing a guest OS, and any guest OS you create from the CLI using virt-install will show up in this list also.
REFERENCES
thegeekway, kvm vs qemu vs libvirt
httpswww.cyberciti.biz/faq/installing-kvm-on-ubuntu-16-04-lts-server/ (configure bridged networking manually)
httpslinuxconfig.org/simple-virtualization-with-ubuntu-16-04-and-kvm (libvertd group, adding bridged networking using Ubuntu Network Manager GUI or console)
httpshelp.ubuntu.com/community/KVM/Networking (networking modes for KVM, by default private to host, then bridged; troubleshooting)
httpswww.ostechnix.com/setup-headless-virtualization-server-using-kvm-ubuntu/ (install, use console commands for mgmt of KVM, VNC/TigerVNC access)
httpswww.linuxjournal.com/article/9764 (nice description of kernel/user/guest space and virtualization with KVM)
httpswww.ostechnix.com/how-to-find-if-a-cpu-supports-virtualization-technology-vt/ (multiple ways to determine if virtualization supported by CPU)
httpslinuxconfig.org/virtualization-solutions-on-linux-systems-kvm-and-virtualbox (KVM versus Virtualbox features)
httpswww.virtualbox.org/ticket/4032 (VirtualBox does not pass through VT-x to guests)
httpsserverfault.com/questions/208693/difference-between-kvm-and-qemu (Qemu vs KVM)
httpswww.linux-kvm.org/page/FAQ (using as non-privilege user,checking hw accel,time sync)
httpswww.altaro.com/vmware/how-to-set-up-a-nested-vsphere-6-environment-part-1/
httpswww.virtualbox.org/manual/ch10.html#gimproviders (VirtualBox docs describing paravirtualization)
httpswiki.ubuntu.com/KvmWithBridge (manual instructions)
httpswiki.libvirt.org/page/VirtualNetworking (virtual networking diagrams and full descriptions)
httpslinux..net/man/1/virt-install (man page)
httpcdimage.ubuntu.com/netboot/16.04/ (network install ISO for Ubuntu 16.04)
httpswww.jethrocarr.com/2012/08/04/virt-viewer-remote-access-tricks/ (virt-viewer over ssh tunnel)
httpsblog.programster.org/kvm-cheatsheet (resizing memory using âÂÂvirsh editâÂÂ, cpu sched params,cpu affinity,guest net list,manual bridge,snapshot)
httpswww.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests (kvm-intel nested,ept)
httpredsymbol.net/linux-kernel-boot-parameters/ (kernel boot parameters for kvm)
httpswiki.openstack.org/wiki/LibvirtXMLCPUModel (cpu.type=host-passthrough|host-model)
httpsdocs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/ (enabling nested virt)
httpswiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU (IOMMU steps)
httpsvirtuallyfun.com/wordpress/2017/03/07/running-vmware-esxi-6-5-linuxkvm/ (qemu-system-x86.conf)
Enable VT-x on HP workstations
KVM, network bridge on Ubuntu bionic
Enabling VT-x in UEFI, disable Microsoft Hyper-V
IBM, virsh-pool summary of commands
NOTES
**SPECIFYING OS FLAVOR IF AUTOSENSE DOES NOT WORK**
Use virt-install to list the known flavors of Ubuntu.
On Ubuntu 16.04+ use:
sudo apt-get install libosinfo-bin osinfo-query os | grep ubuntu | awk {'print $1'}
On Ubuntu 14.04 use
virt-install --os-variant list | grep -i ubuntu
Then specify âÂÂâÂÂos-variant=ubuntutrustyâ (on trusty) or âÂÂâÂÂos-variant=ubuntu14.04â (on xenial) as parameter in virt-install.
virsh edit ukvm1404 (will show variant at /domain/os/type@machine)
**Checking true size of sparse disk**
sparse files consist of runs of empty âÂÂ0âÂÂ, ls will report inflated apparent size, du will report true size. For details on disk use qemu-img
sudo qemu-img info .qcow2
**making sure VT-x is enabled on host**
==add to /etc/modprobe.d/qemu-system-x86.conf
options kvm_intel nested=1 enable_apicv=n
options kvm ignore_msrs=1
check with
cat /sys/module/kvm/parameters/ignore_msrs (want Y)
cat /sys/module/kvm_intel/parameters/enable_apicv (want N)
cat /sys/module/kvm_intel/parameters/nested (want Y)
$ sudo kvm-ok
$ virt-host-validate
$ egrep -c âÂÂ(vmx|svm)â /proc/cpuinfo (want non-zero value for # cpus)
==Then ssh into the esxi host, /etc/vmware/config
vmx.allowNested=TRUE
**Checking VT-d virtualization for IO, IOMMU (different from cpu VT-x)
**
dmesg | grep -iE "dmar|iommu|aspm" cat /var/log/kern.log | grep IOMMU
==add to /etc/default/grub (for VT-d) bug in 18.04
GRUB_CMDLINE_LINUX_DEFAULT=âÂÂintel_iommu=onâÂÂ
grub-install --version # equiv to grub2-mkconfig sudo update-grub
**kvm settings worked for esxi6.7 installation and running**
cpus > copy host CPU configuration; disk1=IDE (not sata|not scsi); nic=e1000; cd=IDE; video=QXL 16Mb (cirrus caused cycling at initial boot screen); display spice|vnc
**Verbose logs for libvirt-bin**
$ sudo vi /etc/libvirt/libvirtd.conf log_level = 1 log_outputs="1:syslog:libvirtd" $ sudo systemctl restart libvirt-bin $ journalctl -f $ journalctl -u libvirt-bin
**Removing virsh storage pool, link**
virsh pool-autostart --disable virsh pool-destroy # pool-delete is optional, just to remove local dir virsh pool-delete virsh pool-undefine
**Pre-create image for use with virt-install**
# 1200Gb but created sparse qemu-img create -f qcow2 esxi1.qcow2 1200G qemu-img info esxi1.qcow2 # when using virt-install, refer to path of disk virt-install --disk /path/to/imported/esxi1.qcow2
**Determining version of QEMU, KVM, virsh**
# below from Ubuntu 20 Focal $ sudo apt show qemu-system-x86 Package: qemu-system-x86 Version: 1:4.2-3ubuntu6.17 $ kvm --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers $ qemu-system-x86_64 --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) $ virsh --version 6.0.0
If upgrading from older versions watch for âÂÂ/local/sbin/libvirtd: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_PRIVATE_4.8.0â² not found (required by ./local/sbin/libvirtd)âÂÂ
# This could come from 'virt-host-validate' utility or 'libvirtd' service not starting. $ virt-host-validate $ sudo systemctl status libvirtd $ sudo journalctl -u libvirtd --no-pager # this gives the error because it is the older version $ /usr/local/sbin/libvirtd -V # whereas this one print 6.0.0 as expected $ /usr/sbin/libvirtd -V # so we must get rid of references to this old libvirtd # look at libvirt systemd service file $ sudo grep ExecStart /lib/systemd/system/libvirtd.service ExecStart=/usr/sbin/libvirtd $libvirtd_opts # here is the problem, change it $ sudo grep ExecStart /usr/local/lib/systemd/system/libvirtd.service ExecStart=/usr/local/sbin/libvirtd $LIBVIRTD_ARGS # get these old files out of PATH, clear path cache $ cd /usr/local/bin $ sudo mkdir bak $ sudo mv *virt* bak/. $ hash -r $ sudo systemctl daemon-reload $ sudo systemctl restart libvirtd $ sudo systemctl status libvirtd $ sudo journalctl -u libvirtd --no-pager # ok, good now
**Checking AppArmor status, setting to complain instead of enforce**
sudo aa-status sudo aa-status | grep virt # set to 'complain' so it writes to log, but doesn't stop process aa-complain /usr/sbin/libvirtd aa-complain /usr/sbin/virtlogd aa-complain /usr/lib/libvirt/virt-aa-helper sudo systemctl restart libvirtd sudo systemctl status libvirtd
**KSM for sharing memory among virtualized hosts [1,2
KSM does share memory pages among common guest VMs, but I have seen more nested ESXi errors when this is enabled. For example, k8s clusters and vcenter interfaces will error and become unavailable, requiring a restart of the nested esxi host.
# KSM enabled if set to 1 cat sys/kernel/mm/ksm/run # monitor sharing watch cat /sys/kernel/mm/ksm/pages_shared # enable KSM echo 1 > /sys/kernel/mm/ksm/run # enable KSM ever after reboot echo 'w /sys/kernel/mm/ksm/run 1' > /etc/tmpfiles.d/ksm.conf # verify enabled grep -H '' /sys/kernel/mm/ksm/*
**LIBVIRT_DEFAULT_URI environment variable [1,2,3]
**
has value in /etc/profile.d/libvirt-uri.and /etc/libvirt/libvirt.conf. Needs to be set to qemusystem or user will get qemusession and see a completely different set of domains, networks, disk pool.
**Changes to qemu.conf that can help with disk permissions**
echo 'group = "libvirt"' | sudo tee -a /etc/libvirt/qemu.conf echo 'dynamic_ownership = 1' | sudo tee -a /etc/libvirt/qemu.conf # restart service sudo systemctl restart libvirtd