[Bug 1889556] Re: grub-install failure does not fail package upgrade (and does not roll back to matching modules)

[halfgaar]

I had an amazon EC2 Ubuntu 18.04 instance fail to boot. Grub was updated
to the fixed version on aug 3 (2.02-2ubuntu8.17). A fix for this bug
that prevents this in the future may have been released, but the broken
state the machine is in, is not recovered.

Amazon AWS is especially tricky, because you don't have a VGA console to
see the error in. Even tier 1 support only has virtual console access,
which is not there yet.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/1889556

Title:
  grub-install failure does not fail package upgrade (and does not roll
  back to matching modules)

Status in grub2 package in Ubuntu:
  Fix Released
Status in grub2 source package in Xenial:
  Fix Released
Status in grub2 source package in Bionic:
  Fix Released
Status in grub2 source package in Focal:
  Fix Released
Status in grub2 source package in Groovy:
  Fix Released

Bug description:
  [SRU Justification]
  SRU regression triggered by latent misconfiguration of grub on the host system, leading to a failure to boot due to ABI mismatch between /boot/grub/i386-pc which is updated, and the grub image in the MBR which is not

  [Test case]
  1. Boot a system in BIOS mode.
  2. Run sudo sed -i -e'/Name: grub-pc\/install_devices/,/Name/ s,Value: .*,Value: /dev/non-existent,' /var/cache/debconf/config.dat
  3. Enable -proposed in apt sources.list.
  4. Run sudo env DEBIAN_FRONTEND=noninteractive apt install grub-pc
  5. Reboot
  6. Confirm that the system boots successfully
  7. Boot a system in UEFI mode.
  8. Enable -proposed in apt sources.list.
  9. Run sudo env DEBIAN_FRONTEND=noninteractive apt install grub-efi-amd64-signed
  10. Confirm that the contents of /boot/efi/EFI/ubuntu have been updated with the current timestamp.

  [Original description]
  Failure to install new grub core to the specified device does not correctly prevent upgrade to incompatible modules (LP: #1889509)

  $ sudo debconf-get-selections |grep sda
  grub-pc	grub-pc/install_devices_disks_changed	multiselect	/dev/sda
  grub-pc	grub-pc/install_devices	multiselect	/dev/sda

  $ mount|grep nvme
  /dev/nvme0n1p1 on / type ext4 (rw,relatime,discard,data=ordered)

  $ ls /dev/sda
  ls: cannot access '/dev/sda': No such file or directory

  $ sudo env DEBIAN_FRONTEND=noninteractive apt full-upgrade -y
  ...
  Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 grub-pc-bin amd64 2.02~beta2-36ubuntu3.26 [891 kB]
  ...
  Installing for i386-pc platform.
  grub-install: error: cannot find a GRUB drive for /dev/sda.  Check your device.map.
  Generating grub configuration file ...
  Found linux image: /boot/vmlinuz-4.4.0-1111-aws
  Found linux image: /boot/vmlinuz-4.4.0-1109-aws
  Found initrd image: /boot/initrd.img-4.4.0-1109-aws
  done
  ...

  # update-grub failed during the install but the return code is 0
  $ echo $?
  0

  # The package is installed without apparent error, but the instance will fail to reboot (LP: #1889509)
  $ dpkg -l|grep grub-pc
  ii  grub-pc                          2.02~beta2-36ubuntu3.26                    amd64        GRand Unified Bootloader, version 2 (PC/BIOS version)
  ii  grub-pc-bin                      2.02~beta2-36ubuntu3.26                    amd64        GRand Unified Bootloader, version 2 (PC/BIOS binaries)

  # If I reboot it will fail to boot:
  Booting from Hard Disk 0...
  error: symbol `grub_calloc' not found.
  Entering rescue mode...
  grub rescue> _

  ---

  Xenial in AWS (us-west-2 ami-060d1be0dd4526759 built on 20200611)
  The  debconf for grub was not set to the correct device when cloud-init first ran (LP: #1877491) or when the fix for that was applied (LP: #1889555)
  The fact that grub-install fails during the upgrade but does not fail the package install (and cause a rollback) means that how we have a mismatch between grub core and modules which breaks boot (LP: #1889509).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889556/+subscriptions