Blog/OpenGL acceleration with KVM and VirGL

OpenGL acceleration with QEMU, KVM and VirGL

SuperTuxKart running in a virtual machine with OpenGL acceleration via VirGL

Hardware acceleration for 3D rendering within virtual machines has historically been challenging, often requiring GPU pass-through configurations with KVM. However, the introduction of VirGL support in QEMU has provided a solution. With VirGL, it's possible to enable hardware acceleration for OpenGL graphics within QEMU/KVM guests without the need for GPU pass-through setups. This allowes a single host GPU to be utilised by several guests at the same time.

Many Linux desktop environments use OpenGL for desktop composition, and enabling VirGL can offer improved graphics performance for these guests.

The spice protocol allows for better video performance over congested networks. It also allows for remotely connecting USB devices to the guest machine as well as audio playback and recording. More information on spice is available at https://www.spice-space.org/

aSPICE Android remote desktop client with Firefox playing a video.

VirGL

VirGL is a virtual 3D GPU for use inside QEMU virtual machines, that allows the guest operating system to use the capabilities of the host GPU to accelerate 3D rendering. The goal of VirGL is to have a guest GPU that is fully independent of the host GPU.

The software stack that makes up VirGL support is:

Guest side:

• MESA OpenGL driver
• Linux kernel VirtIO GPU driver

Host Side:

• Linux kernel with KVM enabled
• QEMU
• The GPU hardware and drivers

Libvirt

Libvirt is a set of tools to make virtualisation and administration of virtual machines easier. While it is possible to run QEMU and VirGL without Libvirt, it is much easier to get going using libvirt.

The tools needed on a host are:

• libvirt
• virt-manager
• QEMU

On the guest side:

• qemu-guest-agent
• spice-vdagentd (optional, for better spice client intergration)

Enabling VirGL

I find it easiest to create a new VM using virt-manager and make sure it is running as intended and afterwards add support for VirGL. This is because virt-manager does not yet have direct support for adding the required VirGL configuration.

Once a VM has been created with virt-manager, you need to edit the XML configuration for the VM using virsh edit from a shell. It should look like this:

<domain type='kvm'>
  <name>alpinelinux3.19</name>
  <uuid>afb30432-7a93-4282-b9fb-5577016130b8</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://alpinelinux.org/alpinelinux/3.19"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/media/vm/libvirt/images/alpinelinux3.19.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:19:b1:2b'/>
      <source bridge='lan'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <audio id='1' type='spice'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
</domain>

Now we need to replace the graphics and video sections with the following. You need to keep <address ..> line in the video section preserved from the original XML definition.

    <graphics type='spice' autoport='yes' listen='192.168.0.1'>
      <listen type='address' address='192.168.0.1'/>
      <image compression='auto_glz'/>
      <jpeg compression='auto'/>
      <zlib compression='auto'/>
      <playback compression='on'/>
      <streaming mode='filter'/>
      <clipboard copypaste='yes'/>
      <filetransfer enable='yes'/>
    </graphics>
    <graphics type='egl-headless'>
      <gl rendernode='/dev/dri/renderD128'/>
    </graphics>
     <video>
      <driver iommu='on' ats='on' packed='on'/>
      <model type='virtio' heads='1' primary='yes'>
        <acceleration accel3d='yes'/>
      </model>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>

Make sure <gl rendernode='/dev/dri/renderD128'/> points to your hosts GPU.

Now when you boot your VM, the Virtio GPU kernel driver should be automatically loaded. This can be verified using dmesg:

# dmesg |grep drm

[    1.406522][    T1] ACPI: bus type drm_connector registered
[    1.406820][    T1] [drm] pci: virtio-vga detected at 0000:00:01.0
[    1.407134][    T1] [drm] features: +virgl +edid -resource_blob -host_visible
[    1.407135][    T1] [drm] features: -context_init
[    1.408178][    T1] [drm] number of scanouts: 1
[    1.408344][    T1] [drm] number of cap sets: 2
[    1.421567][    T1] [drm] cap set 0: id 1, max-version 1, max-size 308
[    1.422196][    T1] [drm] cap set 1: id 2, max-version 2, max-size 1376
[    1.422578][    T1] [drm] Initialized virtio_gpu 0.1.0 0 for 0000:00:01.0 on minor 0
[    1.458307][    T1] virtio-pci 0000:00:01.0: [drm] fb0: virtio_gpudrmfb frame buffer device

You can also check if the hosts GPU is available via virgl with the glinfo tool

# glinfo

GL_VERSION: 4.3 (Compatibility Profile) Mesa 23.3.6
GL_RENDERER: virgl (AMD Radeon Graphics (radeonsi, renoir, LLVM 17.0.6, ...)
GL_VENDOR: Mesa

glmark2 running with hardware acceleration over VirGL.

References

• https://docs.mesa3d.org/drivers/virgl.html
• https://libvirt.org/
• https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine