[Bug 2041739] Re: update-grub giving errors and apparently not locating /boot on correct zfs pool after upgrade to Ubuntu Mantic

Tue Dec 19 01:55:27 UTC 2023

I think this is related to several of the other reported bugs related to
zfs:

https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1982897 - 26/07/2022
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1632694 - 12/10/2016
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1635115 - 20/10/2016
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1688424 - 05/05/2017
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1867542 - 15/03/2020

What I don't understand is how update-grub worked with zfs on prior
versions of Ubuntu. And why does the problem temporarily resolve itself
if I recreate bpool from scratch? And why hasn't it been fixed already
given the problems been causing issues for up to 7 years? I thought ZFS
was supposed to be quite well supported by Linux distro's or are people
moving away from it now? It has great features.

The comments on bug 1867542 imply grub doesn't like zsys snapshots. Or
maybe zsys changes some flag on the pool that grub doesn't like? Or
maybe grub doesn't like some zfs feature flag enabled on one of my
zpools? I have attached a file showing the currently enabled flags.

I have been without my personal laptop for over 7 weeks now as I took a
break from looking at this problem as it was doing my head in. After
several weeks break I have now pulled all the latest Ubuntu Mantic
v23.10 updates and will reboot and see if it made any difference.
Otherwise I try and go through all my zpool flags and see if there is
anything that doesn't look compatible.

** Attachment added: "bpool and rpool flags"
   https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+attachment/5730545/+files/zpool-get-all-flags.txt

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/2041739

Title:
  update-grub giving errors and apparently not locating /boot on correct
  zfs pool after upgrade to Ubuntu Mantic

Status in grub2 package in Ubuntu:
  Confirmed

Bug description:
  After upgrading to Ubuntu Mantic, grub failed to load vmlinuz from my
  zfs bpool giving me the following message:

    error: file `/BOOT/ubuntu_07flvq@/vmlinuz-6.5.0-0-9-generic' not found.
    error: you need to load the kernel first.

    Press any key to continue..._

  After this I rebooted from Ubuntu Live USB, imported zfs bpool and
  rpool's, entered a chroot (with bind mounts), and performed an update-
  grub.

  update-grub log showed it found multiple linux and initrd images from
  rpool and snapshots but not bpool. Log also contained the following
  error:

    /usr/sbin/grub-probe: error: compression algorithm inherit not
  supported

  I deleted all old snapshots of /boot from zfs bpool and rpool's. I am not sure how there ended up being a /boot on rpool as well as bpool. After re-running update-grub I was still getting the error about the zfs 'inherit' compression flag, so I explicitly set the zfs compression flag to "on" on the bpool, bpool/BOOT and bpool/BOOT/ubuntu_07flvq datasets. After re-running update-grub the warning was still there. I installed the grub-common v2.12~rc1-10ubuntu4 src package and searched for the warning, and it seems it is reading the zfs compression flag from the block. (./grub-core/fs/zfs/zfs.c zio_read() function line 1853) Chatting with the AI (Bing, Phind.com or perpexity.ai - can't remember) it said the flag may be set per block and changing the flag on the dataset doesn't necessarily mean the block flag will be updated. I don't have time to dig further to verify this, and not sure if I'm on the right track. 
  I tried rebooting again but still got the same error, so I rebooted from Ubuntu Live USB again and re-entered the chroot. This time I backed up the bpool/BOOT/ubuntu_07flvq dataset to /tmp/boot_backup, destroyed the bpool pool on /dev/sda3 and re-created it with the proper structure and flags, and copied the boot backup back to /boot and ran update-grub again. From memory the compression flag warning went away - but my memory is hazy and I wasn't taking notes. After rebooting, grub booted successfully. However several days later I noticed during an apt upgrade that update-grub was giving the same warnings and had found some more linux/initrd images in snapshots on rpool again. I verified the zfs "com.sun:auto-snapshot" flag on all the bpool datasets and they were set to false, so zfs-auto-snapshot had not created auto-snapshots on bpool. (as expected) However, there was a /boot directory under the "rpool/ROOT/ubuntu_07flvq" dataset, which was being included in it's zfs-auto-snapshot. Even after unmounting bpool and removing this /boot directory and remounting bpool, update-grub still claims it is finding it's linux and initrd images from rpool/ROOT/ubuntu_07flvq. And when I browse the rpool/ROOT/ubuntu_07flvq snapshots under /.zfs I can't see any linux or initrd images under /boot. So the update-grub messages have me mystified. Are the messages incorrect? Is grub-probe just hopelessly confused? Does it only think the images are on rpool because bpool/BOOT/ubuntu_07flvq is mounted at /boot? 

  I hope someone can understand all this because it's a big mystery to
  me. Is this the right place to post it?

  (base) danny at envy:/boot/grub$ lsb_release -rd
  No LSB modules are available.
  Description:	Ubuntu 23.10
  Release:	23.10

  (base) danny at envy:/boot/grub$ apt-cache policy grub-common
  grub-common:
    Installed: 2.12~rc1-10ubuntu4
    Candidate: 2.12~rc1-10ubuntu4
    Version table:
   *** 2.12~rc1-10ubuntu4 990
          990 mirror+file:/etc/apt/mirrorlist-ubuntu.txt mantic/main amd64 Packages
          100 /var/lib/dpkg/status

  ProblemType: Bug
  DistroRelease: Ubuntu 23.10
  Package: grub-common 2.12~rc1-10ubuntu4
  ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
  Uname: Linux 6.5.0-9-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.27.0-0ubuntu5
  Architecture: amd64
  CasperMD5CheckResult: pass
  CurrentDesktop: GNOME
  Date: Sun Oct 29 16:37:56 2023
  InstallationDate: Installed on 2022-06-05 (511 days ago)
  InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220602)
  SourcePackage: grub2
  UpgradeStatus: Upgraded to mantic on 2023-04-21 (191 days ago)
  modified.conffile..etc.grub.d.05_debian_theme: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+subscriptions