Three processor flaws have been made public in three weeks. On 24 July came Zenbleed (CVE-2023-20593), affecting AMD Zen 2 processors; on 8 August came Downfall (CVE-2022-40982), affecting several generations of Intel processors, and on the same day Inception (CVE-2023-20569), affecting AMD Zen 3 and Zen 4 processors. All three belong to the same family as Spectre and Meltdown in 2018: by exploiting the processor’s internal workings, a program can read data it has no access to – even from another virtual machine. That is why they are most urgent for anyone running a virtualisation host or a multi-tenant server.
Downfall (Intel, CVE-2022-40982)
The flaw was found by Daniel Moghimi, a researcher at Google; Intel refers to it as “Gather Data Sampling” (GDS). During speculative execution, the “gather” instructions of the AVX2 and AVX-512 instruction sets can leak the contents of the processor’s internal vector registers that previously belonged to another process or virtual machine on the same physical core. According to the researcher’s demonstration, this can be used to steal encryption keys and other sensitive data, in a cloud environment even from another customer.
According to Intel, the affected processors are:
- the Skylake family (from 6th-generation Core through the Kaby Lake, Coffee Lake and Comet Lake processors, and on the server side the Skylake-SP, Cascade Lake and Cooper Lake Xeons),
- the Tiger Lake family (11th-generation Core),
- the Ice Lake family (Ice Lake, including the 3rd-generation Xeon Scalable processors, and Rocket Lake).
Alder Lake, Raptor Lake and Sapphire Rapids (4th-generation Xeon Scalable) are not affected. The fix is a microcode update, which is enabled by default.
A reminder: the July AMD flaw
Zenbleed, published on 24 July, affects Zen 2 processors (including the 2nd-generation EPYC “Rome” server processors). For EPYC Rome, fixed microcode was already available at the time of the announcement; for the other models, the Linux kernel disables the faulty feature in the meantime. You can read the details on the blog of our hosting brand, aWh; the checks below apply to this flaw as well.
Inception (AMD Zen 3 and Zen 4, CVE-2023-20569)
Inception (also known as Speculative Return Stack Overflow, SRSO), found by researchers at ETH Zürich, hijacks the processor’s return address prediction and can thereby leak arbitrary data from kernel memory. AMD is providing microcode or BIOS updates for Zen 3 and Zen 4 processors (for example EPYC 7003 “Milan” and 9004 “Genoa”, Ryzen 5000 and 7000); for Zen 1 and Zen 2 models, according to the vendor, no microcode is needed, and the kernel provides the protection there. According to AMD, the flaw can only be exploited locally, with code running on the machine, and it is not aware of any exploitation outside research environments so far.
What does this mean on a virtualisation host?
The essence of all three flaws is the same: code running on the same physical processor core as the victim can see the victim’s data. This sets the order of priority:
- Multi-tenant hosts (web hosting, VPSs, rented virtual machines): anyone can run their own code here, so this is the most urgent.
- Shared build and CI servers, where code from different projects runs on the same machine.
- Internal Proxmox hosts, where only your own virtual machines run: the risk is lower, because an attacker first has to get into one of the machines, but a compromised web server could then expose the data of the other virtual machines as well.
Importantly, the protection has to be on the host. Updating the guest system does not replace the host’s microcode and kernel; in the guest, the status of gather_data_sampling in such cases is often “Unknown: Dependent on hypervisor status”.
How to check your servers
The kernel shows in the /sys/devices/system/cpu/vulnerabilities/ directory which protection is active against which flaw, and /proc/cpuinfo shows the version of the loaded microcode:
grep -r . /sys/devices/system/cpu/vulnerabilities/
grep -m1 microcode /proc/cpuinfo
journalctl -k -b | grep -iE 'microcode|zenbleed'
What should you look for in the output?
| File | Good state | Action needed |
|---|---|---|
gather_data_sampling (Downfall) |
Not affected or Mitigation: Microcode |
Vulnerable: No microcode – microcode needed |
spec_rstack_overflow (Inception) |
Not affected or Mitigation: Safe RET |
Vulnerable… – kernel and/or microcode update needed |
If the two files don’t exist at all, the kernel is too old and doesn’t know about these flaws. Zenbleed has no file of its own: if the kernel log contains the message “Zenbleed: please update your microcode for the most optimal fix”, the kernel is protecting the system with the fallback solution, and the definitive fix is the microcode.
Updating on Debian and Proxmox VE
This morning Debian released the advisories DSA-5474-1 (intel-microcode, 3.20230808.1) and DSA-5475-1 (kernel, 6.1.38-4 on Debian 12, 5.10.179-5 on Debian 11) against Downfall and Inception. For Zenbleed, amd64-microcode (DSA-5459-1) and the kernel fix already arrived in July.
The microcode packages are not in the main repository. On Proxmox VE 8 and Debian 12 you need to enable the non-free-firmware component, and on Proxmox VE 7 and Debian 11 the non-free component:
# /etc/apt/sources.list – Debian 12 / Proxmox VE 8
deb http://deb.debian.org/debian bookworm main contrib non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free-firmware
deb http://security.debian.org/debian-security bookworm-security main contrib non-free-firmware
apt update
apt install intel-microcode # Intel processors
apt install amd64-microcode # AMD processors
reboot
The package puts the microcode into the initramfs, and – if it is newer than the one in the BIOS – it is loaded early at every boot. So the update only takes effect after a reboot.
Proxmox VE uses its own kernel. The fix for Zenbleed arrived in the pve-no-subscription repository at the end of July (on Proxmox VE 8 in version 6.2.16-6 of the pve-kernel-6.2.16-5-pve package, on 7 in version 5.15.108-2 of pve-kernel-5.15.108-1-pve). At the time of writing, the kernel update against Downfall and Inception has not yet been released; until then, the developers recommend installing the microcode. In a cluster, move the virtual machines with live migration and reboot node by node – this is when you find out what high availability is really worth, which we also covered in our article on the advantages of Proxmox VE. If you are moving to version 8 now, our upgrade guide will help.
What does it cost in performance?
- Downfall: according to the researcher’s site, Intel itself mentions an overhead of up to 50% for some workloads. Mainly programs that make heavy use of the AVX2/AVX-512 gather instructions are affected: scientific computing and some machine learning and media processing workloads. For an average web or database server, a much smaller impact is expected.
- Inception: according to AMD, the impact of the microcode is minimal, and the cost of the kernel’s “Safe RET” protection depends on the workload.
- Zenbleed: according to its discoverer, the kernel’s fallback solution (chicken bit) may have some performance cost, so here too the microcode is the better solution.
The protections can be disabled with kernel parameters (gather_data_sampling=off, spec_rstack_overflow=off), but only consider this on a machine where only trusted code of your own runs. Don’t disable them on a multi-tenant host. It’s worth measuring your own workload before and after the update, because general figures say little.
In summary
Install the microcode now, and the kernel as soon as your distribution or Proxmox releases it, then reboot the hosts. On old systems that no longer receive updates, this kind of flaw stays unfixed – we wrote about this in our article on the risks of unsupported software.
If you need help updating Proxmox hosts or multi-tenant servers, get in touch: you can read about our operations for hosting providers and our Infrastructure and operations service on our site.