[Bug 2041739] Re: update-grub giving errors and apparently not locating /boot on correct zfs pool after upgrade to Ubuntu Mantic

Tue Dec 19 19:54:32 UTC 2023

I've been testing this some more and it seems that any enabled zpool
feature that requires or also enables the extensible_dataset feature
will cause this issue. 'extensible_dataset' is listed in grub2
compatibility mode (/usr/share/zfs/compatibility.d/grub2) but it doesn't
seem that this is correct, or at least this bug is causing it to not be
fully compatible.

When creating the bpool with any feature disabled that would also enable
extensible_dataset, I can create a snapshot on bpool with grub-probe
still recognizing it as zfs (test script attached):

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -o compatibility=grub2 \
    -o feature at extensible_dataset=disabled \
    -o feature at bookmarks=disabled \
    -o feature at filesystem_limits=disabled \
    -o feature at large_blocks=disabled \
    -o feature at large_dnode=disabled \
    -o feature at sha512=disabled \
    -o feature at skein=disabled \
    -o feature at edonr=disabled \
    -o feature at userobj_accounting=disabled \
    -o feature at encryption=disabled \
    -o feature at project_quota=disabled \
    -o feature at obsolete_counts=disabled \
    -o feature at bookmark_v2=disabled \
    -o feature at redaction_bookmarks=disabled \
    -o feature at redacted_datasets=disabled \
    -o feature at bookmark_written=disabled \
    -o feature at livelist=disabled \
    -o feature at zstd_compress=disabled \
    -o feature at zilsaxattr=disabled \
    -o feature at head_errlog=disabled \
    -o feature at blake3=disabled \
    -o feature at vdev_zaps_v2=disabled \
    -O devices=off \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off \
    bpool /dev/nvme0n1p3

Enabling any of the features in the command above will cause grub not to
recognize /boot as zfs again when a snapshot is created on bpool.

** Attachment added: "grub-zpool-test.sh"
   https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+attachment/5730777/+files/grub-zpool-test.sh

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/2041739

Title:
  update-grub giving errors and apparently not locating /boot on correct
  zfs pool after upgrade to Ubuntu Mantic

Status in grub2 package in Ubuntu:
  Confirmed

Bug description:
  After upgrading to Ubuntu Mantic, grub failed to load vmlinuz from my
  zfs bpool giving me the following message:

    error: file `/BOOT/ubuntu_07flvq@/vmlinuz-6.5.0-0-9-generic' not found.
    error: you need to load the kernel first.

    Press any key to continue..._

  After this I rebooted from Ubuntu Live USB, imported zfs bpool and
  rpool's, entered a chroot (with bind mounts), and performed an update-
  grub.

  update-grub log showed it found multiple linux and initrd images from
  rpool and snapshots but not bpool. Log also contained the following
  error:

    /usr/sbin/grub-probe: error: compression algorithm inherit not
  supported

  I deleted all old snapshots of /boot from zfs bpool and rpool's. I am not sure how there ended up being a /boot on rpool as well as bpool. After re-running update-grub I was still getting the error about the zfs 'inherit' compression flag, so I explicitly set the zfs compression flag to "on" on the bpool, bpool/BOOT and bpool/BOOT/ubuntu_07flvq datasets. After re-running update-grub the warning was still there. I installed the grub-common v2.12~rc1-10ubuntu4 src package and searched for the warning, and it seems it is reading the zfs compression flag from the block. (./grub-core/fs/zfs/zfs.c zio_read() function line 1853) Chatting with the AI (Bing, Phind.com or perpexity.ai - can't remember) it said the flag may be set per block and changing the flag on the dataset doesn't necessarily mean the block flag will be updated. I don't have time to dig further to verify this, and not sure if I'm on the right track. 
  I tried rebooting again but still got the same error, so I rebooted from Ubuntu Live USB again and re-entered the chroot. This time I backed up the bpool/BOOT/ubuntu_07flvq dataset to /tmp/boot_backup, destroyed the bpool pool on /dev/sda3 and re-created it with the proper structure and flags, and copied the boot backup back to /boot and ran update-grub again. From memory the compression flag warning went away - but my memory is hazy and I wasn't taking notes. After rebooting, grub booted successfully. However several days later I noticed during an apt upgrade that update-grub was giving the same warnings and had found some more linux/initrd images in snapshots on rpool again. I verified the zfs "com.sun:auto-snapshot" flag on all the bpool datasets and they were set to false, so zfs-auto-snapshot had not created auto-snapshots on bpool. (as expected) However, there was a /boot directory under the "rpool/ROOT/ubuntu_07flvq" dataset, which was being included in it's zfs-auto-snapshot. Even after unmounting bpool and removing this /boot directory and remounting bpool, update-grub still claims it is finding it's linux and initrd images from rpool/ROOT/ubuntu_07flvq. And when I browse the rpool/ROOT/ubuntu_07flvq snapshots under /.zfs I can't see any linux or initrd images under /boot. So the update-grub messages have me mystified. Are the messages incorrect? Is grub-probe just hopelessly confused? Does it only think the images are on rpool because bpool/BOOT/ubuntu_07flvq is mounted at /boot? 

  I hope someone can understand all this because it's a big mystery to
  me. Is this the right place to post it?

  (base) danny at envy:/boot/grub$ lsb_release -rd
  No LSB modules are available.
  Description:	Ubuntu 23.10
  Release:	23.10

  (base) danny at envy:/boot/grub$ apt-cache policy grub-common
  grub-common:
    Installed: 2.12~rc1-10ubuntu4
    Candidate: 2.12~rc1-10ubuntu4
    Version table:
   *** 2.12~rc1-10ubuntu4 990
          990 mirror+file:/etc/apt/mirrorlist-ubuntu.txt mantic/main amd64 Packages
          100 /var/lib/dpkg/status

  ProblemType: Bug
  DistroRelease: Ubuntu 23.10
  Package: grub-common 2.12~rc1-10ubuntu4
  ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
  Uname: Linux 6.5.0-9-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.27.0-0ubuntu5
  Architecture: amd64
  CasperMD5CheckResult: pass
  CurrentDesktop: GNOME
  Date: Sun Oct 29 16:37:56 2023
  InstallationDate: Installed on 2022-06-05 (511 days ago)
  InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220602)
  SourcePackage: grub2
  UpgradeStatus: Upgraded to mantic on 2023-04-21 (191 days ago)
  modified.conffile..etc.grub.d.05_debian_theme: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+subscriptions