KVM Implementation with Proxmox¶

KVM Server implementation with Proxmox repositories and setup iommu , pci passtrouhg settings

MainBoard Information¶

Proxmox Repositories¶

Enterprise or Not Enterprise

Enterprise Repositories¶

The Enterprise repository for The Proxmox subscription model is based on the number of physical servers and their occupied CPU sockets. A CPU socket (or slot) is a specific place on the motherboard where your CPU plugs in. Most motherboards have one, two, or four CPU sockets. The CPU socket count is the number of CPU sockets on your motherboard that are actually occupied with a physical CPU.

1

grep "physical id" /proc/cpuinfo | sort -u | wc -l

Install Proxmox VE¶

Add an /etc/hosts entry for your IP address The hostname of your machine must be resolvable via /etc/hosts.

This means that in /etc/hosts you need one of the following entries for your hostname:

• 1 IPv4 or

• 1 IPv6 or

• 1 IPv4 and 1 IPv6

Note: This also means removing the address 127.0.1.1 that might be present as default.

For instance, if your IP address is 192.168.15.77, and your hostname prox4m1, then your /etc/hosts file could look like:

127.0.0.1       localhost
192.168.15.77   prox4m1.proxmox.com prox4m1

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

You can test if your setup is ok using the hostname command:

hostname --ip-address
192.168.15.77 # should return your IP address here

Add the Proxmox VE repository key as root (or use sudo):

1

echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list

1

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

1

sha512sum /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg 7fb03ec8a1675723d2853b84aa4fdb49a46a3bb72b9951361488bfd19b29aab0a789a4f8c7406e71a69aabbc727c936d3549731c4659ffa1a08f44db8fdcebfa /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

1

echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bullseye pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list

1

wget https://enterprise.proxmox.com/debian/proxmox-release-bullseye.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg

1

sha512sum /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg 7fb03ec8a1675723d2853b84aa4fdb49a46a3bb72b9951361488bfd19b29aab0a789a4f8c7406e71a69aabbc727c936d3549731c4659ffa1a08f44db8fdcebfa /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg

Update your repository and system by running:

apt update && apt full-upgrade

Install the Proxmox VE Kernel¶

First you need to install and boot the Proxmox VE kernel, as some packages depend on specific kernel compile flags to be set or feature extensions (e.g., for apparmor) to be available.

1

apt install pve-kernel-6.2

apt install pve-kernel-5.15

1

systemctl reboot

Note that while the pve-kernel-5.15 is the default stable kernel for the Proxmox VE 7.x series, there’s also a newer opt-in kernel available with pve-kernel-6.2, possibly helping to resolve some hardware related woes on modern systems.

Install the Proxmox VE packages¶

apt install proxmox-ve postfix open-iscsi

Configure packages which require user input on installation according to your needs.

If you have a mail server in your network, you should configure postfix as a satellite system. Your existing mail server will then be the relay host which will route the emails sent by Proxmox VE to their final recipient. If you don’t know what to enter here, choose local only and leave the system name as is.

Remove the Debian Kernel¶

Proxmox VE ships its own kernel and keeping the Debian default kernel can lead to trouble on upgrades, for example, with Debian point releases. Therefore, you must remove the default Debian kernel:

1

apt remove linux-image-amd64 'linux-image-6.1*'

1

apt remove linux-image-amd64 'linux-image-5.10*'

Update and check grub2 config by running:

update-grub2

Recommended: Remove the os-prober Package¶

The os-prober package scans all the partitions of your host to create dual-boot GRUB entries. But the scanned partitions can also include those assigned to virtual machines, which one doesn’t want to add as boot entry.

If you didn’t install Proxmox VE as dual boot beside another OS, you can safely remove the os-prober package:

apt remove os-prober

Connect to the Proxmox VE web interface¶

Connect to the admin web interface of the local hypervisor link If you have a fresh install and have not added any users yet, you should select PAM authentication realm and login with root user account.

Network Configuration¶

Bridged networking aka: “shared physical device”¶

Install bridge utils on debian

sudo apt install bridge-utils

Config an interface as bridge to guest machines

How to view the current configuration of the interfaces

With ip command

1	ip -c a show dev enp5s0f0

With ifconfig

1	ifconfig enp5s0f0

Add bridge configuration to interfaces file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug enp5s0f0
auto enp5s0f0
iface enp5s0f0 inet static
        address 192.168.20.9
        netmask 255.255.255.0
        gateway 192.168.20.255
dns-search invalid
dns-nameservers 1.1.1.1 8.8.8.8

1
2
3
4
5
6
7

auto vmbr0
iface vmbr0 inet static
        address 10.10.8.1
        bridge_ports none
        bridge_stp off
        bridge_fd 0
up ip route add 10.10.8.0/24 via 10.10.8.1 dev vmbr0

1
2

# Enable ip forwarding
post-up echo 1 > /proc/sys/net/ipv4/ip_forward

1
2
3

# Add table routes
post-up iptables -t nat -A POSTROUTING -s '10.10.8.0/24' -o enp5s0f0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.8.0/24' -o enp5s0f0 -j MASQUERADE

Enabling IOMMU¶

Do I have iommu ?

1

sudo dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

How to enable IOMMU on Intel and AMD product?

Go to BIOS screen and find the VT-d or IOMMU from the NorthBridge Configuration and change that setting to “Enabled”, and save it. In case of CentOS6.x and RedHat6.x, you have to add intel_iommu=on or amd_iommu=on to kernel option too.

Manually enable IOMMU support by setting the correct kernel parameter depending on the type of CPU in use:

You should also append the iommu=pt parameter. This will prevent Linux from touching devices which cannot be passed through.

Check Grub Parameter¶

For check if iommu is enable in grub kernel option

1
2
3

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-5.15.108-1-pve root=UUID=9b2564e3-5b97-4d48-8295-3ca5f0669908 ro quiet

Add the parameter in /etc/default/grub

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#  For amd set -> amd_iommu=on
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt splash"
GRUB_CMDLINE_LINUX=""

After that update grub:

sudo update-grub2

Modules Enable¶

1
2
3
4

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

After changing anything modules related, you need to refresh your initramfs. On {pve} this can be done by executing:

# update-initramfs -u -k all

Finish Configuration¶

Finally reboot to bring the changes into effect and check that it is indeed enabled.

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

Verify IOMMU interrupt remapping is enabled¶

It is not possible to use PCI passthrough without interrupt remapping. Device assignment will fail with ‘Failed to assign device “[device name]”: Operation not permitted’ or ‘Interrupt Remapping hardware not found, passing devices to unprivileged domains is insecure.’.

All systems using an Intel processor and chipset that have support for Intel Virtualization Technology for Directed I/O (VT-d), but do not have support for interrupt remapping will see such an error. Interrupt remapping support is provided in newer processors and chipsets (both AMD and Intel).

To identify if your system has support for interrupt remapping:

dmesg | grep 'remapping'

[    0.179058] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.179350] DMAR-IR: Enabled IRQ remapping in x2apic mode

If you see one of the following lines:

• AMD-Vi: Interrupt remapping enabled

• DMAR-IR: Enabled IRQ remapping in x2apic mode (‘x2apic’ can be different on old CPUs, but should still work)

then remapping is supported.

Testing the Configuration under Linux¶

The following instructions will explain how to test if the CPU supports VT-x and, if that is the case, how to test if the BIOS settings permit this feature to be used.

Does the CPU support VT-x?¶

On a Linux system, you can test if the CPUs will support VT-x with the help of the /proc/cpuinfo utility. If the vmx setting is listed among the flags, the CPU will support VT-x, in principle:[2]

root@grml~# cat /proc/cpuinfo
processor: 0
vendor_id: GenuineIntel
cpu family: 6
model 15:
model name: Intel(R) Xeon(R) CPU            5110  @ 1.60GHz
stepping: 6
cpu MHz: 1595.896
cache size: 4096 KB
physical id: 0
siblings: 2
core id: 0
cpu cores: 2
apicid: 0
initial apicid: 0
fpu: yes
fpu_exception: yes
cpuid level: 10
wp: yes
flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall
nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl
vmx tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow
bogomips: 3193.78
clflush size: 64
cache_alignment: 64
address sizes: 36 bits physical, 48 bits virtual
power management:
[...]

Do the BIOS Settings permit VT-x usage?¶

If the BIOS will permit the usage of VT-x can be determined with the help of the rdmsr (read msr) utility from the msr-tools tool package[3] MSR stands for machine-specific register, although sometimes model-specific register is used.[4] MSRs are used to set specific values for the hardware in use or to transfer values between the BIOS and the kernel.

The VT-x lock function is stored in the IA32_FEATURE_CONTROL machine-specific register (MSR address 0x3a).[5] This MSR contains the following three bits:[6]

• Bit 0: lock bit

• Bit 1: activate VMXON in SMX mode

• Bit 2: activate VMXON outside of SMX mode

The BIOS must set Bits 1 and 2, or all three bits (including Bit 0) so that VT-x support will be enabled.

The Linux Live CD from grml 64 for May 2009 will be used in this example. The msr-tools will be installed by means of the apt-get update and subsequent apt-get install msr-tools commands. So that rdmsr can access the MSRs, the msr kernel module will have to be installed.

Example 1: Deactivating Intel VT-x in the BIOS¶

If Intel VT-x has been deactivated in the BIOS (or the server was not turned off and back on after the setting was activated), the rdmsr 0x3a command will return 1 (meaning that only Bit 0 has been set).

root@grml ~ # modinfo msr
filename:       /lib/modules/2.6.28-grml64/kernel/arch/x86/kernel/msr.ko
license:        GPL
description:    x86 generic MSR driver
author:         H. Peter Anvin <hpa@zytor.com>
srcversion:     645F15590C9CD893B156AD6
depends:
vermagic:       2.6.28-grml64 SMP preempt mod_unload
root@grml ~ # modprobe msr
root@grml ~ # rdmsr 0x3a
1
root@grml ~ #

Example 2: Activating Intel VT-x in the BIOS¶

With the example server (an Intel SR2500), the rdmsr 0x3a command returns 5, if Intel VT-x has been activated in the BIOS (Bits 0 and 2 have therefore been set).

root@grml ~ # modinfo msr
filename:       /lib/modules/2.6.28-grml64/kernel/arch/x86/kernel/msr.ko
license:        GPL
description:    x86 generic MSR driver
author:         H. Peter Anvin <hpa@zytor.com>
srcversion:     645F15590C9CD893B156AD6
depends:
vermagic:       2.6.28-grml64 SMP preempt mod_unload
root@grml ~ # modprobe msr
root@grml ~ # rdmsr 0x3a
5
root@grml ~ #

IOMMU supported but deactivated¶

IOMMU Full Activated¶

High Availability Cluster¶

Overall Progress¶