[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria
Mauricio Faria de Oliveira
1960758 at bugs.launchpad.net
Wed Sep 20 21:51:59 UTC 2023
Hi Corey,
While verifying (bionic) ussuri-proposed, I noticed bionic doesn't
necessarily need the fix, since its ovmf package does not ship
OVMF_CODE.secboot.fd (introduced later, present in focal).
Keeping the fix is safe, though, as it is a no-op by default (no functional change),
and may be desired to get nova in bionic-ussuri back in sync with focal, after the regression handling for CVE-2023-2088.patch, but I'm not sure of all the details involved.
I'll not mark this as verification-done-ussuri yet, in order to wait for
your confirmation.
Thanks,
Mauricio
...
Verification on bionic:
With ussuri-updates, an uefi server _does_ boot (differently than
focal):
$ juju ssh nova-compute/0 'dpkg -s nova-compute | grep ^Version:' 2>/dev/null
Version: 2:21.2.4-0ubuntu2.2~cloud0
$ openstack image set jammy --property hw_firmware_type=uefi
$ openstack server create --image jammy --flavor m1.small --network private test
$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000001 2>&1 | sed -n '/<os>/,/<\/os>/p'
...
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
...
$ openstack console log show test | grep -o 'test login:'
test login:
This works because the problematic loader (OVMF_CODE.secboot.fd) is not
yet available on bionic, only in focal:
$ ls -1 /usr/share/OVMF/
OVMF_CODE.fd
OVMF_VARS.fd
$ dpkg -S /usr/share/OVMF/OVMF_CODE.fd
ovmf: /usr/share/OVMF/OVMF_CODE.fd
$ dpkg -L ovmf | grep -F .fd
/usr/share/OVMF/OVMF_CODE.fd
/usr/share/OVMF/OVMF_VARS.fd
/usr/share/ovmf/OVMF.fd
/usr/share/qemu/OVMF.fd
$ lsb_release -cs
bionic
$ pull-lp-debs ovmf focal
$ dpkg-deb -c ovmf_*.deb
...
-rw-r--r-- root/root 1966080 2022-11-08 05:40 ./usr/share/OVMF/OVMF_CODE.secboot.fd
...
With ussuri-proposed and new option disabled (default), an uefi server
still boots (no regression by default):
$ juju ssh nova-compute/0 'sudo add-apt-repository --yes "deb http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri main"'
$ juju ssh nova-compute/0 'sudo apt install --yes --option=Dpkg::Options::=--force-confnew --option=Dpkg::Options::=--force-confdef nova-compute'
$ juju ssh nova-compute/0 'sudo systemctl restart nova-compute.service' 2>/dev/null
$ juju ssh nova-compute/0 'dpkg -s nova-compute | grep ^Version:' 2>/dev/null
Version: 2:21.2.4-0ubuntu2.6~cloud0
$ openstack server stop test
$ openstack server start test
$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000001 2>&1 | sed -n '/<os>/,/<\/os>/p'
...
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
...
$ openstack console log show test | grep -o 'test login:'
test login:
With ussuri-proposed and the new option enabled (changed), an uefi
server still boots (no regression)
$ juju config nova-compute config-flags='ubuntu_libvirt_uefi_loader_path=True'
$ juju ssh nova-compute/0 sudo grep ubuntu_libvirt_uefi_loader_path /etc/nova/nova.conf 2>/dev/null
ubuntu_libvirt_uefi_loader_path = True
$ openstack server stop test
$ openstack server start test
$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000001 2>&1 | sed -n '/<os>/,/<\/os>/p'
...
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
...
$ openstack console log show test | grep -o 'test login:'
test login:
** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2023-2088
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1960758
Title:
UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
Ussuri/Victoria
Status in Ubuntu Cloud Archive:
Invalid
Status in Ubuntu Cloud Archive ussuri series:
Fix Committed
Status in Ubuntu Cloud Archive victoria series:
Fix Released
Status in OpenStack Compute (nova):
Invalid
Status in OpenStack Compute (nova) ussuri series:
Invalid
Status in OpenStack Compute (nova) victoria series:
Invalid
Status in nova package in Ubuntu:
Invalid
Status in nova source package in Focal:
Fix Released
Bug description:
Impact:
===
Currently, setting `hw_firwmare_type=uefi` may create
_unbootable_ servers on 20.04 hypervisors with Ussuri
and Victoria (Wallaby and later are OK).
We should not use the Secure Boot firmware on the 'pc'
machine type, as 'q35' is _required_ by OVMF firmware
if SMM feature is built (usually the case, to actually
secure the SB feature).
[See comment #6 for research and #7 for test evidence.]
We should not use the Secure Boot firmware on the 'q35'
machine type _either_, as it might not work regardless,
since other libvirt XML options such as SMM and S3/S4
disable may be needed for Secure Boot to work, but are
_not_ configured by Openstack Ussuri (no SB support).
Approach:
===
Considering how long Focal/Ussuri have been out there
(and maybe worked with UEFI enabled for some cases?)
add a config option to _opt-in_ to actually supported
UEFI loaders for nova/libvirt.
This seems to benefit downstream/Ubuntu more (although
other distros might be affected) add the config option
"ubuntu_libvirt_uefi_loader_path" (disabled by default)
in the DEFAULT libvirt config section (so it can be set
in nova-compute charm's 'config-flags' option).
Test Plan:
===
$ openstack image set --property hw_firmware_type=uefi $IMAGE
$ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server
(with patched packages:)
Set `ubuntu_libvirt_uefi_loader_path = true` in `[DEFAULT]` in /etc/nova/nova.conf
(eg `juju config nova-compute config-flags='ubuntu_libvirt_uefi_loader_path=true'`)
$ openstack server stop uefi-server
$ openstack server start uefi-server
- Expected Result:
The server's libvirt XML uses UEFI _without_ Secure Boot.
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
The guest boots, and console log confirms UEFI mode:
$ openstack console log show srv | grep -i -e efi -e bios
...
Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
...
[ 0.000000] efi: EFI v2.70 by EDK II
[ 0.000000] efi: SMBIOS=0x7fbcd000 ACPI=0x7fbfa000 ACPI
2.0=0x7fbfa014 MEMATTR=0x7eb30018
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
...
- Actual Result:
The server's libvirt XML uses UEFI _with_ Secure Boot.
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
The guest doesn't boot; empty console log; qemu-kvm looping at 100%
CPU.
$ openstack console log show srv | grep -i -e efi -e bios
$ openstack console log show srv | wc -l
0
$ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
67205 libvirt+ ... 100.0 1.4 1:18.35 qemu-sy+
67205 libvirt+ ... 100.0 1.4 1:19.36 qemu-sy+
67205 libvirt+ ... 99.0 1.4 1:20.36 qemu-sy+
67205 libvirt+ ... 101.0 1.4 1:21.37 qemu-sy+
67205 libvirt+ ... 100.0 1.4 1:22.38 qemu-sy+
Where problems could occur:
===
The changes are opt-in with `ubuntu_libvirt_uefi_loader_path=true`,
so users are not affected by default.
Theoretically, regressions would more likely manifest and be contained
in nova's libvirt driver, when `hw_firwmare_type=uefi` (not by default).
The expected symptoms of regressions are boot failures (server starts
from openstack perspective, but doesn't boot to the operating system).
Other Info:
===
- Hypervisor running Ubuntu 20.04 LTS (Focal)
- Nova packages from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1960758/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list