Se stai eseguendo un host Ubuntu, hai più scelte per un hypervisor di virtualizzazione. Ho scritto diversi articoli sull'utilizzo di VirtualBox, ma ora consideriamo un hypervisor bare metal come KVM.

KVM è un hypervisor di tipo 1 implementato come modulo del kernel Linux che utilizza estensioni di virtualizzazione di un processore moderno, rendendolo in grado di eseguire direttamente la CPU senza traduzione. Ogni macchina virtuale è un normale processo Linux, pianificato dallo scheduler Linux standard.

Un esempio di qualcosa che KVM può fare che VirtualBox non può, KVM ha la capacità di trasmettere la capacità di virtualizzazione al suo sistema operativo guest, che consentirebbe la virtualizzazione nidificata.

 Installazione

Innanzitutto, installa KVM e strumenti assortiti:

sudo apt-get install qemu-system-x86 qemu-kvm qemu libvirt-dev libvirt-clients virt-manager virtinst bridge-utils cpu-checker virt-viewer -y # meno di Ubuntu 20 sudo apt-get install libvirt-bin

Quindi convalidare che quel KVM sia stato installato e che la CPU abbia la virtualizzazione VT-x abilitata con kvm-ok.

$ sudo kvm-ok INFO: /dev/kvm esiste È possibile utilizzare l'accelerazione KVM

Se invece ricevi un messaggio simile al seguente, vai a livello di BIOS e abilita VT-x.

INFO: /dev/kvm non esiste SUGGERIMENTO: sudo modprobe kvm_intel INFO: la tua CPU supporta le estensioni KVM INFO: KVM (vmx) è disabilitato dal tuo BIOS SUGGERIMENTO: accedi alla configurazione del BIOS e abilita la tecnologia di virtualizzazione (VT), quindi spegni completamente /poweron l'accelerazione KVM del sistema NON può essere utilizzata

Quindi esegui l'utility virt-host-validate per eseguire un'intera serie di controlli rispetto alla tua capacità di virtualizzazione e alla disponibilità di KVM.

# se fallisce, potresti avere una versione precedente ancora installata virt-host-validate --version # la versione più recente proviene da /usr/bin (non /usr/local/bin) che virt-host-validate # viene fornita da questa utility dal pacchetto libvirt-clients sudo virt-host-validate

 Rilassa l'applicazione di AppArmor

Ubuntu viene fornito con AppArmor abilitato per libvirt. Disabilita security_driver, imposta su un set vuoto di spazi dei nomi e riavvia libvirt o potresti non essere in grado di creare una VM guest.

echo 'security_driver = "none"'| sudo tee -a /etc/libvirt/qemu.conf echo 'namespaces =| sudo tee -a /etc/libvirt/qemu.conf sudo systemctl restart libvirtd sudo systemctl status libvirtd

Non dovresti dover modificare nessuno dei profili AppArmor per presentare un reclamo. Ma se devi, ci sono note in fondo a questo articolo.

 Aggiungi utente ai gruppi libvirt
Per poter gestire la VM ospite come utente normale, possiamo aggiungerci a tutti i gruppi libvirt (ad esempio libvirt, libvirt-qemu) e al gruppo kvm.

gatto /etc/gruppo | grep libvirt | awk -F {'print $1'} | xargs -n1 sudo adduser $USER # aggiunge l'utente al gruppo kvm anche sudo adduser $USER kvm # effettua nuovamente il login, quindi mostra l'appartenenza al gruppo exec su -l $USER id | grep libvirt

L'appartenenza al gruppo richiede che un utente effettui nuovamente l'accesso, quindi se il comando âÂÂidâ non mostra l'appartenenza al gruppo libvirt*, disconnettersi e riconnettersi o provare âÂÂexec su -l $USERâÂÂ.

 Connessione QEMU al sistema

Se non impostato esplicitamente, la connessione QEMU in spazio utente sarà a âÂÂqemusessionâÂÂ, e non a âÂÂqemusystemâÂÂ. Ciò ti farà vedere domini, reti e pool di dischi diversi quando esegui virsh come utente normale rispetto a sudo.

Modifica il tuo profilo in modo che la variabile di ambiente sottostante venga esportata nelle tue sessioni di accesso.

# utilizza la stessa connessione e gli stessi oggetti di sudo LIBVIRT_DEFAULT_URI=qemusystem

 Rete predefinita

Per impostazione predefinita, KVM crea uno switch virtuale che si presenta come un'interfaccia host denominata âÂÂvirbr0â utilizzando 192.168.122.0/24. L'immagine qui sotto è per gentile concessione di libvirt.org.

httpsfabianlee.org/wp-content/uploads/2018/08/libvirt_network_default_network_overview.jpg

Questa interfaccia dovrebbe essere visibile dall'Host utilizzando il comando âÂÂipâ di seguito.

$ ip addr show virbr0 3: virbr0:,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever

“virbr0” operates in NAT mode, which allows the guest OS to communicate out, but only allowing the Host(and those VMs in its subnet) to make incoming connections.

 Bridged network 

To enable guest VMs on the same network as the Host, you should create a bridged network to your physical interface (e.g. eth0, ens4, epn1s0).

Read my article here for how to use NetPlan on Ubuntu to bridge your physical network interface to ‘br0’ at the OS level. And then use that to create a libvirt network named ‘host-bridge’ that uses br0.

# bridge to physical network $ virsh net-dumpxml host-bridge  host-bridge 44d2c3f5-6301-4fc6-be81-5ae2be4a47d8 
In order to handle NAT and routed networks for KVM, enable IPv4 forwarding on this host.

# this needs to be "1" cat /proc/sys/net/ipv4/ip_forward # if not, then add it echo net.ipv4.ip_forward=1 | sudo tee -a /etc/sysctl.conf # make permanent sudo sysctl -p /etc/sysctl.conf

 Default storage pool 

The “default” storage pool for guest disks is “/var/lib/libvirt/images”. This is fine for test purposes, but if you have another mount that you want to use for guest OS disks, then you should create a custom storage pool.

Below are the commands to create a “kvmpool” on an SSD mounted at “/data/kvm/pool”.

$ virsh pool-list --all Name State Autostartdefault active yes $ virsh pool-define-as kvmpool --type dir --target /data/kvm/pool Pool kvmpool defined $ virsh pool-list --all $ virsh pool-start kvmpool $ virsh pool-autostart kvmpool $ virsh pool-list --all Name State Autostartdefault active yes kvmpool active yes

 Create basic VM using virt-install 

In order to test you need an OS boot image. Since we are on an Ubuntu host, let’s download the ISO for the network installer of Ubuntu 20.04 Focal. This file is only 74Mb, so it is perfect for testing. When complete, you should have a local file named Downloads/mini.iso”

wget httparchive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/current/legacy-images/netboot/mini.iso

First list what virtual machines are running on our host:

# chown is only necessary if virsh was run previously as sudo ls -l virtinst sudo chown -R $USER:$USER virtinst # list VMs virsh list --all

This should return an empty list of VMs, because no guest OS have been deployed. Create your first guest VM with 1 vcpu/1G RAM using the default virbr0 NAT network and default pool storage.

virt-install --virt-type=kvm --name=ukvm-focal --ram 1024 --vcpus=1 --virt-type=kvm --hvm --cdrom ~/Downloads/mini.iso --network network=default --graphics vnc --disk pool=default,size=20,bus=virtio,format=qcow2 --noautoconsole # open console to VM virt-viewer ukvm-focal

“virt-viewer” will popup a window for the Guest OS, when you click the mouse in the window and then press  you will see the initial Ubuntu network install screen.

If you want to delete this guest OS completely, close the GUI window opened with virt-viewer, then use the following commands:

virshukvm-focal virsh undefine ukvm-focal

 Test from GUI 


The virt-viewer utility will open a basic window to the guest OS, but notice it does not give any control beside sending keys. If you want a full GUI for managing KVM, I would suggest using “virt-manager“.

httpsfabianlee.org/wp-content/uploads/2018/08/kvm-virt-manager.png

To install and start virt-manager:

sudo apt-get install qemu-system virt-manager virt-manager

virt-manager provides a convenient interface for creating or managing a guest OS, and any guest OS you create from the CLI using virt-install will show up in this list also.

REFERENCES

thegeekway, kvm vs qemu vs libvirt

httpswww.cyberciti.biz/faq/installing-kvm-on-ubuntu-16-04-lts-server/ (configure bridged networking manually)

httpslinuxconfig.org/simple-virtualization-with-ubuntu-16-04-and-kvm (libvertd group, adding bridged networking using Ubuntu Network Manager GUI or console)

httpshelp.ubuntu.com/community/KVM/Networking (networking modes for KVM, by default private to host, then bridged; troubleshooting)

httpswww.ostechnix.com/setup-headless-virtualization-server-using-kvm-ubuntu/ (install, use console commands for mgmt of KVM, VNC/TigerVNC access)

httpswww.linuxjournal.com/article/9764 (nice description of kernel/user/guest space and virtualization with KVM)

httpswww.ostechnix.com/how-to-find-if-a-cpu-supports-virtualization-technology-vt/ (multiple ways to determine if virtualization supported by CPU)

httpslinuxconfig.org/virtualization-solutions-on-linux-systems-kvm-and-virtualbox (KVM versus Virtualbox features)

httpswww.virtualbox.org/ticket/4032 (VirtualBox does not pass through VT-x to guests)

httpsserverfault.com/questions/208693/difference-between-kvm-and-qemu (Qemu vs KVM)

httpswww.linux-kvm.org/page/FAQ (using as non-privilege user,checking hw accel,time sync)

httpswww.altaro.com/vmware/how-to-set-up-a-nested-vsphere-6-environment-part-1/

httpswww.virtualbox.org/manual/ch10.html#gimproviders (VirtualBox docs describing paravirtualization)

httpswiki.ubuntu.com/KvmWithBridge (manual instructions)

httpswiki.libvirt.org/page/VirtualNetworking (virtual networking diagrams and full descriptions)

httpslinux..net/man/1/virt-install (man page)

httpcdimage.ubuntu.com/netboot/16.04/ (network install ISO for Ubuntu 16.04)

httpswww.jethrocarr.com/2012/08/04/virt-viewer-remote-access-tricks/ (virt-viewer over ssh tunnel)

httpsblog.programster.org/kvm-cheatsheet (resizing memory using ‘virsh edit’, cpu sched params,cpu affinity,guest net list,manual bridge,snapshot)

httpswww.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests (kvm-intel nested,ept)

httpredsymbol.net/linux-kernel-boot-parameters/ (kernel boot parameters for kvm)

httpswiki.openstack.org/wiki/LibvirtXMLCPUModel (cpu.type=host-passthrough|host-model)

httpsdocs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/ (enabling nested virt)

httpswiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU (IOMMU steps)

httpsvirtuallyfun.com/wordpress/2017/03/07/running-vmware-esxi-6-5-linuxkvm/ (qemu-system-x86.conf)

Enable VT-x on HP workstations

KVM, network bridge on Ubuntu bionic

Enabling VT-x in UEFI, disable Microsoft Hyper-V

IBM, virsh-pool summary of commands

NOTES

**SPECIFYING OS FLAVOR IF AUTOSENSE DOES NOT WORK**

Use virt-install to list the known flavors of Ubuntu.

On Ubuntu 16.04+ use:

sudo apt-get install libosinfo-bin osinfo-query os | grep ubuntu | awk {'print $1'}

On Ubuntu 14.04 use

virt-install --os-variant list | grep -i ubuntu

Then specify “–os-variant=ubuntutrusty” (on trusty) or “–os-variant=ubuntu14.04” (on xenial) as parameter in virt-install.

virsh edit ukvm1404 (will show variant at /domain/os/type@machine)

**Checking true size of sparse disk**

sparse files consist of runs of empty “0”, ls will report inflated apparent size, du will report true size. For details on disk use qemu-img

sudo qemu-img info .qcow2

**making sure VT-x is enabled on host**

==add to /etc/modprobe.d/qemu-system-x86.conf

options kvm_intel nested=1 enable_apicv=n

options kvm ignore_msrs=1

check with

cat /sys/module/kvm/parameters/ignore_msrs (want Y)

cat /sys/module/kvm_intel/parameters/enable_apicv (want N)

cat /sys/module/kvm_intel/parameters/nested (want Y)



$ sudo kvm-ok

$ virt-host-validate

$ egrep -c ‘(vmx|svm)’ /proc/cpuinfo (want non-zero value for # cpus)

==Then ssh into the esxi host, /etc/vmware/config

vmx.allowNested=TRUE

**Checking VT-d virtualization for IO, IOMMU (different from cpu VT-x)

**

dmesg | grep -iE "dmar|iommu|aspm" cat /var/log/kern.log | grep IOMMU

==add to /etc/default/grub (for VT-d) bug in 18.04

GRUB_CMDLINE_LINUX_DEFAULT=”intel_iommu=on”

grub-install --version # equiv to grub2-mkconfig sudo update-grub

**kvm settings worked for esxi6.7 installation and running**

cpus > copy host CPU configuration; disk1=IDE (not sata|not scsi); nic=e1000; cd=IDE; video=QXL 16Mb (cirrus caused cycling at initial boot screen); display spice|vnc

**Verbose logs for libvirt-bin**

$ sudo vi /etc/libvirt/libvirtd.conf log_level = 1 log_outputs="1:syslog:libvirtd" $ sudo systemctl restart libvirt-bin $ journalctl -f $ journalctl -u libvirt-bin


**Removing virsh storage pool, link**

virsh pool-autostart  --disable virsh pool-destroy  # pool-delete is optional, just to remove local dir virsh pool-delete  virsh pool-undefine 

**Pre-create image for use with virt-install**

# 1200Gb but created sparse qemu-img create -f qcow2 esxi1.qcow2 1200G qemu-img info esxi1.qcow2 # when using virt-install, refer to path of disk virt-install --disk /path/to/imported/esxi1.qcow2

**Determining version of QEMU, KVM, virsh**

# below from Ubuntu 20 Focal $ sudo apt show qemu-system-x86 Package: qemu-system-x86 Version: 1:4.2-3ubuntu6.17 $ kvm --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers $ qemu-system-x86_64 --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) $ virsh --version 6.0.0

If upgrading from older versions watch for “/local/sbin/libvirtd: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_PRIVATE_4.8.0′ not found (required by ./local/sbin/libvirtd)”

# This could come from 'virt-host-validate' utility or 'libvirtd' service not starting. $ virt-host-validate $ sudo systemctl status libvirtd $ sudo journalctl -u libvirtd --no-pager # this gives the error because it is the older version $ /usr/local/sbin/libvirtd -V # whereas this one print 6.0.0 as expected $ /usr/sbin/libvirtd -V # so we must get rid of references to this old libvirtd # look at libvirt systemd service file $ sudo grep ExecStart /lib/systemd/system/libvirtd.service ExecStart=/usr/sbin/libvirtd $libvirtd_opts # here is the problem, change it $ sudo grep ExecStart /usr/local/lib/systemd/system/libvirtd.service ExecStart=/usr/local/sbin/libvirtd $LIBVIRTD_ARGS # get these old files out of PATH, clear path cache $ cd /usr/local/bin $ sudo mkdir bak $ sudo mv *virt* bak/. $ hash -r $ sudo systemctl daemon-reload $ sudo systemctl restart libvirtd $ sudo systemctl status libvirtd $ sudo journalctl -u libvirtd --no-pager # ok, good now

**Checking AppArmor status, setting to complain instead of enforce**

sudo aa-status sudo aa-status | grep virt # set to 'complain' so it writes to log, but doesn't stop process aa-complain /usr/sbin/libvirtd aa-complain /usr/sbin/virtlogd aa-complain /usr/lib/libvirt/virt-aa-helper sudo systemctl restart libvirtd sudo systemctl status libvirtd

**KSM for sharing memory among virtualized hosts [1,2


KSM does share memory pages among common guest VMs, but I have seen more nested ESXi errors when this is enabled. For example, k8s clusters and vcenter interfaces will error and become unavailable, requiring a restart of the nested esxi host.

# KSM enabled if set to 1 cat sys/kernel/mm/ksm/run # monitor sharing watch cat /sys/kernel/mm/ksm/pages_shared # enable KSM echo 1 > /sys/kernel/mm/ksm/run # enable KSM ever after reboot echo 'w /sys/kernel/mm/ksm/run 1' > /etc/tmpfiles.d/ksm.conf # verify enabled grep -H '' /sys/kernel/mm/ksm/*

**LIBVIRT_DEFAULT_URI environment variable [1,2,3]

**

has value in /etc/profile.d/libvirt-uri.and /etc/libvirt/libvirt.conf. Needs to be set to qemusystem or user will get qemusession and see a completely different set of domains, networks, disk pool.

**Changes to qemu.conf that can help with disk permissions**

echo 'group = "libvirt"' | sudo tee -a /etc/libvirt/qemu.conf echo 'dynamic_ownership = 1' | sudo tee -a /etc/libvirt/qemu.conf # restart service sudo systemctl restart libvirtd