[Bug 1973669] [NEW] 10_linuxzfs fails if a non-root filesystem has a mountpoint of /

Tue May 17 02:36:26 UTC 2022

Public bug reported:

For ~4 years, this server ran Ubuntu-18.04 with the OS on mirrored SSDs
and all persistent data in a zpool on separate disks. Last week I
upgraded it.  Installation of Ubuntu-22.04 ZFS-root on a new SSD ran
flawlessly. But after importing the persistent data pool, update-grub
(specifically 10_linux_zfs) found the valid kernels but wouldn't create
grub entries for them so /boot/grub/grub.cfg only contained memtest86.
10_linux_zfs's section of grub-mkconfig output is below.

The problem turned out to be a FreeBSD backup I'd zfs-send/rec'd to the
Ubuntu system 2 years ago.  At the time, I added a prefix ("zmule") to
all of the FreeBSD filesystems' mountpoints and turned off "canmount".
Somehow I missed the *actual* root filesystem so it was still set to
mount on '/' with "canmount" enabled - though Ubuntu22.04 wouldn't mount
it on top of /, so I never noticed it.  However, that's what was
confusing /etc/grub.d/10_linux_zfs even though 18.04 was fine with it.
Once I changed FreeBSD's root mountpoint to not '/', update-grub ran as
expected.

I know this is a strange, unusual situation that shouldn't happen, but a
missing /etc/os-release file in one root(ish) filesystem shouldn't block
it from creating entries for valid ones.  If you'd like me to break it
again and run 10_linux_zfs with 'set -x' or similar, I'll be happy to do
that.  (I'm a Linux SRE professionally so breaking stuff is all in a
days work.)

One more note: the bug submission wouldn't let me use "grub-common" as
the package, so I changed that to "grub2" even though, as you can see
below, that's not what's installed.

===================================================================
More detail:

$ lsb_release -rd 
Description:    Ubuntu 22.04 LTS
Release:        22.04
$ apt-cache policy grub-common
grub-common:
  Installed: 2.06-2ubuntu7
  Candidate: 2.06-2ubuntu7
  Version table:
 *** 2.06-2ubuntu7 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
$ apt-cache policy grub2
grub2:
  Installed: (none)
  Candidate: 2.06-2ubuntu7
  Version table:
     2.06-2ubuntu7 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

-----------------

### BEGIN /etc/grub.d/10_linux_zfs ###
Found linux image: vmlinuz-5.15.0-27-generic in rpool/ROOT/ubuntu_lva06t
Found initrd image: initrd.img-5.15.0-27-generic in rpool/ROOT/ubuntu_lva06t
Found linux image: vmlinuz-5.15.0-25-generic in rpool/ROOT/ubuntu_lva06t
Found initrd image: initrd.img-5.15.0-25-generic in rpool/ROOT/ubuntu_lva06t
/etc/grub.d/10_linux_zfs: 404: .: cannot open /tmp/zfsmnt.Z7nWCz/etc/os-release: No such file
-----------------

$ zfs get mountpoint persist/backup/mule-zfs/ROOT/default 
NAME                                    PROPERTY    VALUE       SOURCE
persist/backup/mule-zfs/ROOT/default    mountpoint  /           local
-----------------

# oops!  This fixed it:
$ sudo zfs set mountpoint=zmule persist/backup/mule-zfs/ROOT/default 
$ zfs get mountpoint persist/backup/mule-zfs/ROOT/default 
NAME                                    PROPERTY    VALUE       SOURCE
persist/backup/mule-zfs/ROOT/default    mountpoint  /zmule      local

** Affects: grub2 (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/1973669

Title:
  10_linuxzfs fails if a non-root filesystem has a mountpoint of /

Status in grub2 package in Ubuntu:
  New

Bug description:
  For ~4 years, this server ran Ubuntu-18.04 with the OS on mirrored
  SSDs and all persistent data in a zpool on separate disks. Last week I
  upgraded it.  Installation of Ubuntu-22.04 ZFS-root on a new SSD ran
  flawlessly. But after importing the persistent data pool, update-grub
  (specifically 10_linux_zfs) found the valid kernels but wouldn't
  create grub entries for them so /boot/grub/grub.cfg only contained
  memtest86.  10_linux_zfs's section of grub-mkconfig output is below.

  The problem turned out to be a FreeBSD backup I'd zfs-send/rec'd to
  the Ubuntu system 2 years ago.  At the time, I added a prefix
  ("zmule") to all of the FreeBSD filesystems' mountpoints and turned
  off "canmount".  Somehow I missed the *actual* root filesystem so it
  was still set to mount on '/' with "canmount" enabled - though
  Ubuntu22.04 wouldn't mount it on top of /, so I never noticed it.
  However, that's what was confusing /etc/grub.d/10_linux_zfs even
  though 18.04 was fine with it.  Once I changed FreeBSD's root
  mountpoint to not '/', update-grub ran as expected.

  I know this is a strange, unusual situation that shouldn't happen, but
  a missing /etc/os-release file in one root(ish) filesystem shouldn't
  block it from creating entries for valid ones.  If you'd like me to
  break it again and run 10_linux_zfs with 'set -x' or similar, I'll be
  happy to do that.  (I'm a Linux SRE professionally so breaking stuff
  is all in a days work.)

  One more note: the bug submission wouldn't let me use "grub-common" as
  the package, so I changed that to "grub2" even though, as you can see
  below, that's not what's installed.

  ===================================================================
  More detail:

  $ lsb_release -rd 
  Description:    Ubuntu 22.04 LTS
  Release:        22.04
  $ apt-cache policy grub-common
  grub-common:
    Installed: 2.06-2ubuntu7
    Candidate: 2.06-2ubuntu7
    Version table:
   *** 2.06-2ubuntu7 500
          500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
          100 /var/lib/dpkg/status
  $ apt-cache policy grub2
  grub2:
    Installed: (none)
    Candidate: 2.06-2ubuntu7
    Version table:
       2.06-2ubuntu7 500
          500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

  -----------------

  ### BEGIN /etc/grub.d/10_linux_zfs ###
  Found linux image: vmlinuz-5.15.0-27-generic in rpool/ROOT/ubuntu_lva06t
  Found initrd image: initrd.img-5.15.0-27-generic in rpool/ROOT/ubuntu_lva06t
  Found linux image: vmlinuz-5.15.0-25-generic in rpool/ROOT/ubuntu_lva06t
  Found initrd image: initrd.img-5.15.0-25-generic in rpool/ROOT/ubuntu_lva06t
  /etc/grub.d/10_linux_zfs: 404: .: cannot open /tmp/zfsmnt.Z7nWCz/etc/os-release: No such file
  -----------------

  $ zfs get mountpoint persist/backup/mule-zfs/ROOT/default 
  NAME                                    PROPERTY    VALUE       SOURCE
  persist/backup/mule-zfs/ROOT/default    mountpoint  /           local
  -----------------

  # oops!  This fixed it:
  $ sudo zfs set mountpoint=zmule persist/backup/mule-zfs/ROOT/default 
  $ zfs get mountpoint persist/backup/mule-zfs/ROOT/default 
  NAME                                    PROPERTY    VALUE       SOURCE
  persist/backup/mule-zfs/ROOT/default    mountpoint  /zmule      local

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1973669/+subscriptions