[Bug 2133230] [NEW] Recent OVMF updates break UEFI PXE network boot on Ubuntu Noble and Jammy

[Riccardo Pittau]

Public bug reported:

Recent OVMF package updates introduced a regression that prevents UEFI
PXE network boot from working correctly on both Ubuntu Noble (24.04) and
Ubuntu Jammy (22.04). VMs drop into the UEFI Shell instead of booting
from the network.

## Package Information

### Ubuntu Noble (24.04)
- **Package:** ovmf
- **Working version:** 2024.02-2ubuntu0.5 (released ~Nov 26, 2024)
- **Broken version:** 2024.02-2ubuntu0.6 (released ~Nov 27, 2024)

### Ubuntu Jammy (22.04)
- **Package:** ovmf
- **Working version:** 2022.02-3ubuntu0.22.04.3 (released ~Nov 24, 2024)
- **Broken version:** 2022.02-3ubuntu0.22.04.4 (released ~Nov 27-28, 2024)

## Impact
This regression affects:
- OpenStack Ironic and Bifrost CI jobs on Ubuntu Noble and Jammy
- Any bare metal provisioning workflows using UEFI PXE boot
- QEMU/KVM virtual machines configured for UEFI network boot
- Both Ubuntu 24.04 (Noble) and 22.04 (Jammy) releases

## Description
After recent ovmf package updates on November 27-28, 2024, UEFI PXE network boot stopped working on both Ubuntu Noble and Jammy. Virtual machines configured to boot from the network via PXE now drop into the UEFI Shell instead of successfully performing network boot.

This was discovered when OpenStack Ironic/Bifrost CI jobs on both Ubuntu
releases started timing out simultaneously. The jobs had been working
consistently until the ovmf package was updated on both releases.

## Steps to Reproduce

### On Ubuntu Noble (24.04):
1. Set up a Ubuntu Noble host with QEMU/KVM and libvirt
2. Ensure ovmf package version 2024.02-2ubuntu0.6 is installed
3. Create a VM configured for:

### On Ubuntu Jammy (22.04):
1. Set up a Ubuntu Jammy host with QEMU/KVM and libvirt
2. Ensure ovmf package version 2022.02-3ubuntu0.22.04.4 is installed
3. Create a VM configured for:
   - UEFI firmware (using OVMF)
   - Network boot (PXE) as boot device
   - Connected to a network with a working PXE/DHCP server
4. Start the VM
5. Observe that the VM drops into UEFI Shell instead of PXE booting

## Expected Behavior
The VM should:
1. Initialize UEFI firmware
2. Attempt network boot via PXE
3. Contact DHCP server and receive network configuration
4. Download and boot the network boot image (e.g., iPXE, GRUB, or kernel/ramdisk)

## Actual Behavior
The VM:
1. Initializes UEFI firmware
2. Fails to perform network boot
3. Drops into the UEFI Shell
4. Hangs indefinitely in the shell

## Environment
```
OS: Ubuntu Noble (24.04) or Ubuntu Jammy (22.04)
Virtualization: QEMU/KVM with libvirt
Firmware: OVMF (UEFI)
Boot mode: UEFI with PXE network boot
Network: Virtual network with dnsmasq-based PXE server
```

## Log Evidence
>From failing CI job logs, the following behavior is observed:

### Console output shows VM dropping into UEFI Shell:
```
UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):HD0a1:;BLK1:
          PciRoot(0x0)/Pci(0x1,0x1)/Ata(0x0)/HD(1,MBR,0xBE1AFDFA,0x3F,0xFBFC1)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x1)/Ata(0x0)
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell>
```

### Ironic logs show node stuck in "clean wait" state:
```
TASK [bifrost-ironic-install : Pull ironic facts until provision state available] ***
FAILED - RETRYING: [localhost]: Pull ironic facts until provision state available (45 retries left).
FAILED - RETRYING: [localhost]: Pull ironic facts until provision state available (44 retries left).
[... continues for all 45 retries ...]
```

The node never successfully boots the Ironic Python Agent (IPA) ramdisk
because the PXE boot fails.

### Timeline

**Ubuntu Noble (24.04):**
- **Nov 26, 2024:** CI jobs passing with ovmf 2024.02-2ubuntu0.5
- **Nov 27, 2024:** CI jobs failing after ovmf updated to 2024.02-2ubuntu0.6

**Ubuntu Jammy (22.04):**
- **Nov 24, 2024:** CI jobs passing with ovmf 2022.02-3ubuntu0.22.04.3
- **Nov 27-28, 2024:** CI jobs failing after ovmf updated to 2022.02-3ubuntu0.22.04.4

## Workaround
Downgrading to the previous version resolves the issue on both releases

After applying this workaround, UEFI PXE network boot works correctly
again on both releases.

## Verification
This issue has been confirmed across multiple OpenStack CI runs on both Ubuntu releases:

**Ubuntu Noble (24.04):**
- Bifrost integration tests
- Ironic tempest tests
- All tests pass with ovmf 2024.02-2ubuntu0.5, fail consistently with 2024.02-2ubuntu0.6

**Ubuntu Jammy (22.04):**
- Bifrost integration tests
- Bifrost upgrade tests
- All tests pass with ovmf 2022.02-3ubuntu0.22.04.3, fail consistently with 2022.02-3ubuntu0.22.04.4

The identical timing and symptoms across both releases strongly suggests
a common root cause in the OVMF updates.

## Additional Information
The regression appears to be specific to the network boot functionality in OVMF. Other UEFI boot modes (e.g., booting from virtual media/ISO) may or may not be affected.

## Related Projects
This issue affects the following OpenStack projects:
- OpenStack Ironic (bare metal provisioning)
- OpenStack Bifrost (standalone Ironic)
- Any deployment using UEFI PXE boot with Ubuntu Noble (24.04) or Jammy (22.04)

## Request
Please investigate what changed in the recent OVMF updates that could cause UEFI PXE network boot to fail on both Ubuntu releases:

**For Ubuntu Noble (24.04):**
- Investigate changes between ovmf 2024.02-2ubuntu0.5 and 2024.02-2ubuntu0.6
- Consider reverting the problematic changes or releasing a fix as 2024.02-2ubuntu0.7

**For Ubuntu Jammy (22.04):**
- Investigate changes between ovmf 2022.02-3ubuntu0.22.04.3 and 2022.02-3ubuntu0.22.04.4
- Consider reverting the problematic changes or releasing a fix as 2022.02-3ubuntu0.22.04.5

The fact that both releases were affected simultaneously by updates on
the same date suggests this may be a coordinated security update or
backport that introduced the regression across both versions.

** Affects: edk2 (Ubuntu)
     Importance: Undecided
         Status: New

** Summary changed:

- OVMF 2024.02-2ubuntu0.6 breaks UEFI PXE network boot
+ Recent OVMF updates break UEFI PXE network boot on Ubuntu Noble and Jammy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to edk2 in Ubuntu.
https://bugs.launchpad.net/bugs/2133230

Title:
  Recent OVMF updates break UEFI PXE network boot on Ubuntu Noble and
  Jammy

Status in edk2 package in Ubuntu:
  New

Bug description:
  Recent OVMF package updates introduced a regression that prevents UEFI
  PXE network boot from working correctly on both Ubuntu Noble (24.04)
  and Ubuntu Jammy (22.04). VMs drop into the UEFI Shell instead of
  booting from the network.

  ## Package Information

  ### Ubuntu Noble (24.04)
  - **Package:** ovmf
  - **Working version:** 2024.02-2ubuntu0.5 (released ~Nov 26, 2024)
  - **Broken version:** 2024.02-2ubuntu0.6 (released ~Nov 27, 2024)

  ### Ubuntu Jammy (22.04)
  - **Package:** ovmf
  - **Working version:** 2022.02-3ubuntu0.22.04.3 (released ~Nov 24, 2024)
  - **Broken version:** 2022.02-3ubuntu0.22.04.4 (released ~Nov 27-28, 2024)

  ## Impact
  This regression affects:
  - OpenStack Ironic and Bifrost CI jobs on Ubuntu Noble and Jammy
  - Any bare metal provisioning workflows using UEFI PXE boot
  - QEMU/KVM virtual machines configured for UEFI network boot
  - Both Ubuntu 24.04 (Noble) and 22.04 (Jammy) releases

  ## Description
  After recent ovmf package updates on November 27-28, 2024, UEFI PXE network boot stopped working on both Ubuntu Noble and Jammy. Virtual machines configured to boot from the network via PXE now drop into the UEFI Shell instead of successfully performing network boot.

  This was discovered when OpenStack Ironic/Bifrost CI jobs on both
  Ubuntu releases started timing out simultaneously. The jobs had been
  working consistently until the ovmf package was updated on both
  releases.

  ## Steps to Reproduce

  ### On Ubuntu Noble (24.04):
  1. Set up a Ubuntu Noble host with QEMU/KVM and libvirt
  2. Ensure ovmf package version 2024.02-2ubuntu0.6 is installed
  3. Create a VM configured for:

  ### On Ubuntu Jammy (22.04):
  1. Set up a Ubuntu Jammy host with QEMU/KVM and libvirt
  2. Ensure ovmf package version 2022.02-3ubuntu0.22.04.4 is installed
  3. Create a VM configured for:
     - UEFI firmware (using OVMF)
     - Network boot (PXE) as boot device
     - Connected to a network with a working PXE/DHCP server
  4. Start the VM
  5. Observe that the VM drops into UEFI Shell instead of PXE booting

  ## Expected Behavior
  The VM should:
  1. Initialize UEFI firmware
  2. Attempt network boot via PXE
  3. Contact DHCP server and receive network configuration
  4. Download and boot the network boot image (e.g., iPXE, GRUB, or kernel/ramdisk)

  ## Actual Behavior
  The VM:
  1. Initializes UEFI firmware
  2. Fails to perform network boot
  3. Drops into the UEFI Shell
  4. Hangs indefinitely in the shell

  ## Environment
  ```
  OS: Ubuntu Noble (24.04) or Ubuntu Jammy (22.04)
  Virtualization: QEMU/KVM with libvirt
  Firmware: OVMF (UEFI)
  Boot mode: UEFI with PXE network boot
  Network: Virtual network with dnsmasq-based PXE server
  ```

  ## Log Evidence
  From failing CI job logs, the following behavior is observed:

  ### Console output shows VM dropping into UEFI Shell:
  ```
  UEFI Interactive Shell v2.2
  EDK II
  UEFI v2.70 (EDK II, 0x00010000)
  Mapping table
        FS0: Alias(s):HD0a1:;BLK1:
            PciRoot(0x0)/Pci(0x1,0x1)/Ata(0x0)/HD(1,MBR,0xBE1AFDFA,0x3F,0xFBFC1)
       BLK0: Alias(s):
            PciRoot(0x0)/Pci(0x1,0x1)/Ata(0x0)
  Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
  Shell>
  ```

  ### Ironic logs show node stuck in "clean wait" state:
  ```
  TASK [bifrost-ironic-install : Pull ironic facts until provision state available] ***
  FAILED - RETRYING: [localhost]: Pull ironic facts until provision state available (45 retries left).
  FAILED - RETRYING: [localhost]: Pull ironic facts until provision state available (44 retries left).
  [... continues for all 45 retries ...]
  ```

  The node never successfully boots the Ironic Python Agent (IPA)
  ramdisk because the PXE boot fails.

  ### Timeline

  **Ubuntu Noble (24.04):**
  - **Nov 26, 2024:** CI jobs passing with ovmf 2024.02-2ubuntu0.5
  - **Nov 27, 2024:** CI jobs failing after ovmf updated to 2024.02-2ubuntu0.6

  **Ubuntu Jammy (22.04):**
  - **Nov 24, 2024:** CI jobs passing with ovmf 2022.02-3ubuntu0.22.04.3
  - **Nov 27-28, 2024:** CI jobs failing after ovmf updated to 2022.02-3ubuntu0.22.04.4

  ## Workaround
  Downgrading to the previous version resolves the issue on both releases

  After applying this workaround, UEFI PXE network boot works correctly
  again on both releases.

  ## Verification
  This issue has been confirmed across multiple OpenStack CI runs on both Ubuntu releases:

  **Ubuntu Noble (24.04):**
  - Bifrost integration tests
  - Ironic tempest tests
  - All tests pass with ovmf 2024.02-2ubuntu0.5, fail consistently with 2024.02-2ubuntu0.6

  **Ubuntu Jammy (22.04):**
  - Bifrost integration tests
  - Bifrost upgrade tests
  - All tests pass with ovmf 2022.02-3ubuntu0.22.04.3, fail consistently with 2022.02-3ubuntu0.22.04.4

  The identical timing and symptoms across both releases strongly
  suggests a common root cause in the OVMF updates.

  ## Additional Information
  The regression appears to be specific to the network boot functionality in OVMF. Other UEFI boot modes (e.g., booting from virtual media/ISO) may or may not be affected.

  ## Related Projects
  This issue affects the following OpenStack projects:
  - OpenStack Ironic (bare metal provisioning)
  - OpenStack Bifrost (standalone Ironic)
  - Any deployment using UEFI PXE boot with Ubuntu Noble (24.04) or Jammy (22.04)

  ## Request
  Please investigate what changed in the recent OVMF updates that could cause UEFI PXE network boot to fail on both Ubuntu releases:

  **For Ubuntu Noble (24.04):**
  - Investigate changes between ovmf 2024.02-2ubuntu0.5 and 2024.02-2ubuntu0.6
  - Consider reverting the problematic changes or releasing a fix as 2024.02-2ubuntu0.7

  **For Ubuntu Jammy (22.04):**
  - Investigate changes between ovmf 2022.02-3ubuntu0.22.04.3 and 2022.02-3ubuntu0.22.04.4
  - Consider reverting the problematic changes or releasing a fix as 2022.02-3ubuntu0.22.04.5

  The fact that both releases were affected simultaneously by updates on
  the same date suggests this may be a coordinated security update or
  backport that introduced the regression across both versions.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/2133230/+subscriptions