[Bug 2041739] Re: update-grub giving errors and apparently not locating /boot on correct zfs pool after upgrade to Ubuntu Mantic

[Mathias Aerts]

Had the same issue after upgrading to Ubuntu 23.10. Tried repairing grub
multiple times in chroot after booting from a live USB. Even tried to
roll back to a zfs snapshot from before the upgrade, which also did not
work.

Was finally able to recover from this after finding this bug report with
the mention of recreating the bpool.

I created a script to prepare the chroot to be able to speed up the
tries to fix grub. Since I'm running zfs on root with encryption, it was
taking too much effort to do this manually every time. I've included the
script which might be useful for anyone else encountering this issue.
You will have to be aware of your partition layout and adjust the script
accordingly.

** Attachment added: "grub-repair-in-chroot.sh"
   https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+attachment/5729852/+files/grub-repair-in-chroot.sh

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/2041739

Title:
  update-grub giving errors and apparently not locating /boot on correct
  zfs pool after upgrade to Ubuntu Mantic

Status in grub2 package in Ubuntu:
  Confirmed

Bug description:
  After upgrading to Ubuntu Mantic, grub failed to load vmlinuz from my
  zfs bpool giving me the following message:

    error: file `/BOOT/ubuntu_07flvq@/vmlinuz-6.5.0-0-9-generic' not found.
    error: you need to load the kernel first.

    Press any key to continue..._

  After this I rebooted from Ubuntu Live USB, imported zfs bpool and
  rpool's, entered a chroot (with bind mounts), and performed an update-
  grub.

  update-grub log showed it found multiple linux and initrd images from
  rpool and snapshots but not bpool. Log also contained the following
  error:

    /usr/sbin/grub-probe: error: compression algorithm inherit not
  supported

  I deleted all old snapshots of /boot from zfs bpool and rpool's. I am not sure how there ended up being a /boot on rpool as well as bpool. After re-running update-grub I was still getting the error about the zfs 'inherit' compression flag, so I explicitly set the zfs compression flag to "on" on the bpool, bpool/BOOT and bpool/BOOT/ubuntu_07flvq datasets. After re-running update-grub the warning was still there. I installed the grub-common v2.12~rc1-10ubuntu4 src package and searched for the warning, and it seems it is reading the zfs compression flag from the block. (./grub-core/fs/zfs/zfs.c zio_read() function line 1853) Chatting with the AI (Bing, Phind.com or perpexity.ai - can't remember) it said the flag may be set per block and changing the flag on the dataset doesn't necessarily mean the block flag will be updated. I don't have time to dig further to verify this, and not sure if I'm on the right track. 
  I tried rebooting again but still got the same error, so I rebooted from Ubuntu Live USB again and re-entered the chroot. This time I backed up the bpool/BOOT/ubuntu_07flvq dataset to /tmp/boot_backup, destroyed the bpool pool on /dev/sda3 and re-created it with the proper structure and flags, and copied the boot backup back to /boot and ran update-grub again. From memory the compression flag warning went away - but my memory is hazy and I wasn't taking notes. After rebooting, grub booted successfully. However several days later I noticed during an apt upgrade that update-grub was giving the same warnings and had found some more linux/initrd images in snapshots on rpool again. I verified the zfs "com.sun:auto-snapshot" flag on all the bpool datasets and they were set to false, so zfs-auto-snapshot had not created auto-snapshots on bpool. (as expected) However, there was a /boot directory under the "rpool/ROOT/ubuntu_07flvq" dataset, which was being included in it's zfs-auto-snapshot. Even after unmounting bpool and removing this /boot directory and remounting bpool, update-grub still claims it is finding it's linux and initrd images from rpool/ROOT/ubuntu_07flvq. And when I browse the rpool/ROOT/ubuntu_07flvq snapshots under /.zfs I can't see any linux or initrd images under /boot. So the update-grub messages have me mystified. Are the messages incorrect? Is grub-probe just hopelessly confused? Does it only think the images are on rpool because bpool/BOOT/ubuntu_07flvq is mounted at /boot? 

  I hope someone can understand all this because it's a big mystery to
  me. Is this the right place to post it?

  (base) danny at envy:/boot/grub$ lsb_release -rd
  No LSB modules are available.
  Description:	Ubuntu 23.10
  Release:	23.10

  (base) danny at envy:/boot/grub$ apt-cache policy grub-common
  grub-common:
    Installed: 2.12~rc1-10ubuntu4
    Candidate: 2.12~rc1-10ubuntu4
    Version table:
   *** 2.12~rc1-10ubuntu4 990
          990 mirror+file:/etc/apt/mirrorlist-ubuntu.txt mantic/main amd64 Packages
          100 /var/lib/dpkg/status

  ProblemType: Bug
  DistroRelease: Ubuntu 23.10
  Package: grub-common 2.12~rc1-10ubuntu4
  ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
  Uname: Linux 6.5.0-9-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.27.0-0ubuntu5
  Architecture: amd64
  CasperMD5CheckResult: pass
  CurrentDesktop: GNOME
  Date: Sun Oct 29 16:37:56 2023
  InstallationDate: Installed on 2022-06-05 (511 days ago)
  InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220602)
  SourcePackage: grub2
  UpgradeStatus: Upgraded to mantic on 2023-04-21 (191 days ago)
  modified.conffile..etc.grub.d.05_debian_theme: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+subscriptions