Si está ejecutando un host Ubuntu, tiene múltiples opciones para un hipervisor de virtualización. He escrito varios artículos sobre el uso de VirtualBox, pero ahora consideremos un hipervisor bare metal como KVM.

KVM es un hipervisor de tipo 1 implementado como un módulo del kernel de Linux que utiliza las extensiones de virtualización de un procesador moderno, lo que lo hace capaz de ejecutar directamente la CPU sin traducción. Cada máquina virtual es un proceso regular de Linux, programado por el programador estándar de Linux.

Como ejemplo de algo que KVM puede hacer que VirtualBox no puede, KVM tiene la capacidad de transmitir la capacidad de virtualización a su sistema operativo invitado, lo que permitiría la virtualización anidada.

 Instalación

Primero, instale KVM y una variedad de herramientas:

sudo apt-get install qemu-system-x86 qemu-kvm qemu libvirt-dev libvirt-clients virt-manager virtinst bridge-utils cpu-checker virt-viewer -y # menos que Ubuntu 20 sudo apt-get install libvirt-bin

Luego valide que se instaló ese KVM y que la CPU tiene la virtualización VT-x habilitada con kvm-ok.

$ sudo kvm-ok INFO: /dev/kvm existe Se puede usar la aceleración KVM

Si, en cambio, recibe un mensaje como el que se muestra a continuación, acceda al nivel del BIOS y habilite VT-x.

INFORMACIÓN: /dev/kvm no existe SUGERENCIA: sudo modprobe kvm_intel INFORMACIÓN: su CPU admite extensiones KVM INFORMACIÓN: KVM (vmx) está deshabilitado por su BIOS SUGERENCIA: ingrese la configuración de su BIOS y habilite la tecnología de virtualización (VT), y luego apague por completo /poweron your system La aceleración KVM NO se puede usar

Luego ejecute la utilidad virt-host-validate para ejecutar un conjunto completo de comprobaciones contra su capacidad de virtualización y preparación para KVM.

# si esto falla, es posible que aún tenga instalada una versión anterior virt-host-validate --version # la versión más reciente proviene de /usr/bin (no /usr/local/bin) de la cual virt-host-validate # proviene esta utilidad del paquete libvirt-clients sudo virt-host-validate

 Relaje la aplicación de AppArmor

Ubuntu viene con AppArmor habilitado para libvirt. Deshabilite security_driver, establezca un conjunto vacío de espacios de nombres y reinicie libvirt o es posible que no pueda crear una máquina virtual invitada.

echo 'controlador_seguridad = "ninguno"'| sudo tee -a /etc/libvirt/qemu.conf echo 'espacios de nombres =| sudo tee -a /etc/libvirt/qemu.conf sudo systemctl reiniciar libvirtd sudo systemctl status libvirtd

No debería tener que modificar ninguno de los perfiles de AppArmor para presentar una queja. Pero si es necesario, hay notas al final de este artículo.

 Agregar usuario a grupos de libvirt
Para que podamos administrar la máquina virtual invitada como un usuario normal, podemos agregarnos a todos los grupos libvirt (por ejemplo, libvirt, libvirt-qemu) y al grupo kvm.

gato /etc/grupo | grep biblioteca virtual | awk -F {'imprimir $1'} | xargs -n1 sudo adduser $USER # agregar usuario al grupo kvm también sudo adduser $USER kvm # volver a iniciar sesión, luego mostrar la membresía del grupo exec su -l $USER id | grep biblioteca virtual

La pertenencia a un grupo requiere que el usuario vuelva a iniciar sesión, por lo que si el comando âÂÂidâ no muestra su pertenencia al grupo libvirt*, cierre la sesión y vuelva a iniciar sesión, o intente con âÂÂexec su -l $USUARIOâÂÂ.

 Conexión QEMU al sistema

Si no se establece explícitamente, la conexión QEMU del espacio de usuario será a "qemusession"y no a "qemusystem". Esto hará que vea diferentes dominios, redes y grupos de discos cuando ejecute virsh como su usuario normal versus sudo.

Modifique su perfil para que la variable de entorno a continuación se exporte a sus sesiones de inicio de sesión.

# usar la misma conexión y objetos que sudo LIBVIRT_DEFAULT_URI=qemusystem

 Red predeterminada

De forma predeterminada, KVM crea un conmutador virtual que aparece como una interfaz de host denominada âÂÂvirbr0â mediante 192.168.122.0/24. La imagen a continuación es cortesía de libvirt.org.

httpsfabianlee.org/wp-content/uploads/2018/08/libvirt_network_default_network_overview.jpg

Esta interfaz debe ser visible desde el host usando el comando âÂÂipâ a continuación.

$ ip addr show virbr0 3: virbr0:,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever

“virbr0” operates in NAT mode, which allows the guest OS to communicate out, but only allowing the Host(and those VMs in its subnet) to make incoming connections.

 Bridged network 

To enable guest VMs on the same network as the Host, you should create a bridged network to your physical interface (e.g. eth0, ens4, epn1s0).

Read my article here for how to use NetPlan on Ubuntu to bridge your physical network interface to ‘br0’ at the OS level. And then use that to create a libvirt network named ‘host-bridge’ that uses br0.

# bridge to physical network $ virsh net-dumpxml host-bridge  host-bridge 44d2c3f5-6301-4fc6-be81-5ae2be4a47d8 
In order to handle NAT and routed networks for KVM, enable IPv4 forwarding on this host.

# this needs to be "1" cat /proc/sys/net/ipv4/ip_forward # if not, then add it echo net.ipv4.ip_forward=1 | sudo tee -a /etc/sysctl.conf # make permanent sudo sysctl -p /etc/sysctl.conf

 Default storage pool 

The “default” storage pool for guest disks is “/var/lib/libvirt/images”. This is fine for test purposes, but if you have another mount that you want to use for guest OS disks, then you should create a custom storage pool.

Below are the commands to create a “kvmpool” on an SSD mounted at “/data/kvm/pool”.

$ virsh pool-list --all Name State Autostartdefault active yes $ virsh pool-define-as kvmpool --type dir --target /data/kvm/pool Pool kvmpool defined $ virsh pool-list --all $ virsh pool-start kvmpool $ virsh pool-autostart kvmpool $ virsh pool-list --all Name State Autostartdefault active yes kvmpool active yes

 Create basic VM using virt-install 

In order to test you need an OS boot image. Since we are on an Ubuntu host, let’s download the ISO for the network installer of Ubuntu 20.04 Focal. This file is only 74Mb, so it is perfect for testing. When complete, you should have a local file named Downloads/mini.iso”

wget httparchive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/current/legacy-images/netboot/mini.iso

First list what virtual machines are running on our host:

# chown is only necessary if virsh was run previously as sudo ls -l virtinst sudo chown -R $USER:$USER virtinst # list VMs virsh list --all

This should return an empty list of VMs, because no guest OS have been deployed. Create your first guest VM with 1 vcpu/1G RAM using the default virbr0 NAT network and default pool storage.

virt-install --virt-type=kvm --name=ukvm-focal --ram 1024 --vcpus=1 --virt-type=kvm --hvm --cdrom ~/Downloads/mini.iso --network network=default --graphics vnc --disk pool=default,size=20,bus=virtio,format=qcow2 --noautoconsole # open console to VM virt-viewer ukvm-focal

“virt-viewer” will popup a window for the Guest OS, when you click the mouse in the window and then press  you will see the initial Ubuntu network install screen.

If you want to delete this guest OS completely, close the GUI window opened with virt-viewer, then use the following commands:

virshukvm-focal virsh undefine ukvm-focal

 Test from GUI 


The virt-viewer utility will open a basic window to the guest OS, but notice it does not give any control beside sending keys. If you want a full GUI for managing KVM, I would suggest using “virt-manager“.

httpsfabianlee.org/wp-content/uploads/2018/08/kvm-virt-manager.png

To install and start virt-manager:

sudo apt-get install qemu-system virt-manager virt-manager

virt-manager provides a convenient interface for creating or managing a guest OS, and any guest OS you create from the CLI using virt-install will show up in this list also.

REFERENCES

thegeekway, kvm vs qemu vs libvirt

httpswww.cyberciti.biz/faq/installing-kvm-on-ubuntu-16-04-lts-server/ (configure bridged networking manually)

httpslinuxconfig.org/simple-virtualization-with-ubuntu-16-04-and-kvm (libvertd group, adding bridged networking using Ubuntu Network Manager GUI or console)

httpshelp.ubuntu.com/community/KVM/Networking (networking modes for KVM, by default private to host, then bridged; troubleshooting)

httpswww.ostechnix.com/setup-headless-virtualization-server-using-kvm-ubuntu/ (install, use console commands for mgmt of KVM, VNC/TigerVNC access)

httpswww.linuxjournal.com/article/9764 (nice description of kernel/user/guest space and virtualization with KVM)

httpswww.ostechnix.com/how-to-find-if-a-cpu-supports-virtualization-technology-vt/ (multiple ways to determine if virtualization supported by CPU)

httpslinuxconfig.org/virtualization-solutions-on-linux-systems-kvm-and-virtualbox (KVM versus Virtualbox features)

httpswww.virtualbox.org/ticket/4032 (VirtualBox does not pass through VT-x to guests)

httpsserverfault.com/questions/208693/difference-between-kvm-and-qemu (Qemu vs KVM)

httpswww.linux-kvm.org/page/FAQ (using as non-privilege user,checking hw accel,time sync)

httpswww.altaro.com/vmware/how-to-set-up-a-nested-vsphere-6-environment-part-1/

httpswww.virtualbox.org/manual/ch10.html#gimproviders (VirtualBox docs describing paravirtualization)

httpswiki.ubuntu.com/KvmWithBridge (manual instructions)

httpswiki.libvirt.org/page/VirtualNetworking (virtual networking diagrams and full descriptions)

httpslinux..net/man/1/virt-install (man page)

httpcdimage.ubuntu.com/netboot/16.04/ (network install ISO for Ubuntu 16.04)

httpswww.jethrocarr.com/2012/08/04/virt-viewer-remote-access-tricks/ (virt-viewer over ssh tunnel)

httpsblog.programster.org/kvm-cheatsheet (resizing memory using ‘virsh edit’, cpu sched params,cpu affinity,guest net list,manual bridge,snapshot)

httpswww.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests (kvm-intel nested,ept)

httpredsymbol.net/linux-kernel-boot-parameters/ (kernel boot parameters for kvm)

httpswiki.openstack.org/wiki/LibvirtXMLCPUModel (cpu.type=host-passthrough|host-model)

httpsdocs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/ (enabling nested virt)

httpswiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU (IOMMU steps)

httpsvirtuallyfun.com/wordpress/2017/03/07/running-vmware-esxi-6-5-linuxkvm/ (qemu-system-x86.conf)

Enable VT-x on HP workstations

KVM, network bridge on Ubuntu bionic

Enabling VT-x in UEFI, disable Microsoft Hyper-V

IBM, virsh-pool summary of commands

NOTES

**SPECIFYING OS FLAVOR IF AUTOSENSE DOES NOT WORK**

Use virt-install to list the known flavors of Ubuntu.

On Ubuntu 16.04+ use:

sudo apt-get install libosinfo-bin osinfo-query os | grep ubuntu | awk {'print $1'}

On Ubuntu 14.04 use

virt-install --os-variant list | grep -i ubuntu

Then specify “–os-variant=ubuntutrusty” (on trusty) or “–os-variant=ubuntu14.04” (on xenial) as parameter in virt-install.

virsh edit ukvm1404 (will show variant at /domain/os/type@machine)

**Checking true size of sparse disk**

sparse files consist of runs of empty “0”, ls will report inflated apparent size, du will report true size. For details on disk use qemu-img

sudo qemu-img info .qcow2

**making sure VT-x is enabled on host**

==add to /etc/modprobe.d/qemu-system-x86.conf

options kvm_intel nested=1 enable_apicv=n

options kvm ignore_msrs=1

check with

cat /sys/module/kvm/parameters/ignore_msrs (want Y)

cat /sys/module/kvm_intel/parameters/enable_apicv (want N)

cat /sys/module/kvm_intel/parameters/nested (want Y)



$ sudo kvm-ok

$ virt-host-validate

$ egrep -c ‘(vmx|svm)’ /proc/cpuinfo (want non-zero value for # cpus)

==Then ssh into the esxi host, /etc/vmware/config

vmx.allowNested=TRUE

**Checking VT-d virtualization for IO, IOMMU (different from cpu VT-x)

**

dmesg | grep -iE "dmar|iommu|aspm" cat /var/log/kern.log | grep IOMMU

==add to /etc/default/grub (for VT-d) bug in 18.04

GRUB_CMDLINE_LINUX_DEFAULT=”intel_iommu=on”

grub-install --version # equiv to grub2-mkconfig sudo update-grub

**kvm settings worked for esxi6.7 installation and running**

cpus > copy host CPU configuration; disk1=IDE (not sata|not scsi); nic=e1000; cd=IDE; video=QXL 16Mb (cirrus caused cycling at initial boot screen); display spice|vnc

**Verbose logs for libvirt-bin**

$ sudo vi /etc/libvirt/libvirtd.conf log_level = 1 log_outputs="1:syslog:libvirtd" $ sudo systemctl restart libvirt-bin $ journalctl -f $ journalctl -u libvirt-bin


**Removing virsh storage pool, link**

virsh pool-autostart  --disable virsh pool-destroy  # pool-delete is optional, just to remove local dir virsh pool-delete  virsh pool-undefine 

**Pre-create image for use with virt-install**

# 1200Gb but created sparse qemu-img create -f qcow2 esxi1.qcow2 1200G qemu-img info esxi1.qcow2 # when using virt-install, refer to path of disk virt-install --disk /path/to/imported/esxi1.qcow2

**Determining version of QEMU, KVM, virsh**

# below from Ubuntu 20 Focal $ sudo apt show qemu-system-x86 Package: qemu-system-x86 Version: 1:4.2-3ubuntu6.17 $ kvm --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers $ qemu-system-x86_64 --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) $ virsh --version 6.0.0

If upgrading from older versions watch for “/local/sbin/libvirtd: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_PRIVATE_4.8.0′ not found (required by ./local/sbin/libvirtd)”

# This could come from 'virt-host-validate' utility or 'libvirtd' service not starting. $ virt-host-validate $ sudo systemctl status libvirtd $ sudo journalctl -u libvirtd --no-pager # this gives the error because it is the older version $ /usr/local/sbin/libvirtd -V # whereas this one print 6.0.0 as expected $ /usr/sbin/libvirtd -V # so we must get rid of references to this old libvirtd # look at libvirt systemd service file $ sudo grep ExecStart /lib/systemd/system/libvirtd.service ExecStart=/usr/sbin/libvirtd $libvirtd_opts # here is the problem, change it $ sudo grep ExecStart /usr/local/lib/systemd/system/libvirtd.service ExecStart=/usr/local/sbin/libvirtd $LIBVIRTD_ARGS # get these old files out of PATH, clear path cache $ cd /usr/local/bin $ sudo mkdir bak $ sudo mv *virt* bak/. $ hash -r $ sudo systemctl daemon-reload $ sudo systemctl restart libvirtd $ sudo systemctl status libvirtd $ sudo journalctl -u libvirtd --no-pager # ok, good now

**Checking AppArmor status, setting to complain instead of enforce**

sudo aa-status sudo aa-status | grep virt # set to 'complain' so it writes to log, but doesn't stop process aa-complain /usr/sbin/libvirtd aa-complain /usr/sbin/virtlogd aa-complain /usr/lib/libvirt/virt-aa-helper sudo systemctl restart libvirtd sudo systemctl status libvirtd

**KSM for sharing memory among virtualized hosts [1,2


KSM does share memory pages among common guest VMs, but I have seen more nested ESXi errors when this is enabled. For example, k8s clusters and vcenter interfaces will error and become unavailable, requiring a restart of the nested esxi host.

# KSM enabled if set to 1 cat sys/kernel/mm/ksm/run # monitor sharing watch cat /sys/kernel/mm/ksm/pages_shared # enable KSM echo 1 > /sys/kernel/mm/ksm/run # enable KSM ever after reboot echo 'w /sys/kernel/mm/ksm/run 1' > /etc/tmpfiles.d/ksm.conf # verify enabled grep -H '' /sys/kernel/mm/ksm/*

**LIBVIRT_DEFAULT_URI environment variable [1,2,3]

**

has value in /etc/profile.d/libvirt-uri.and /etc/libvirt/libvirt.conf. Needs to be set to qemusystem or user will get qemusession and see a completely different set of domains, networks, disk pool.

**Changes to qemu.conf that can help with disk permissions**

echo 'group = "libvirt"' | sudo tee -a /etc/libvirt/qemu.conf echo 'dynamic_ownership = 1' | sudo tee -a /etc/libvirt/qemu.conf # restart service sudo systemctl restart libvirtd