[Bug 2110891] Re: System freeze on release upgrade 24.10 oneiric to 25.04 plucky with root fs on ZFS and snapshots

[Benjamin Schmid]

I've created a really ugly hack solution which allowed me to upgrade
without errors on deadlocks. Sharing, so that others can upgrade their
Ubuntu 24.10 installation, in case they use root on ZFS and they are
still on 24.10 or were able to restore their 24.10 installation after a
failed upgrade attempt

With root permissions:

1. Update your system:
    apt update ; apt upgrade --with-new-pkg

2. Download 00_zfs_oneiric_upgradehack.cfg to /etc/default/grub.d/
  (not /etc/grub.d/. Also keep the filename incl. .cfg suffix)

3. Prepare for the worst:
    cp -ra /boot /boot.oracular
    zfs snapshot -r rpool at oracular

4. I had issues with virtualbox DKMS; maybe temporarily uninstall if you
have it

5. Run the upgrade – it now should pass through successfully
    do-release-upgrade

6. Reboot into Ubuntu 25.04 after upgrade completion and remove the hack:
    rm /etc/default/grub.d/00_zfs_oneiric_upgradehack.cfg*

7. Run update-grub twice:
    update-grub ; update-grub

You should be set.


** Attachment added: "00_zfs_oneiric_upgradehack.cfg"
   https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/2110891/+attachment/5879003/+files/00_zfs_oneiric_upgradehack.cfg

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/2110891

Title:
  System freeze on release upgrade 24.10 oneiric to 25.04 plucky with
  root fs on ZFS and snapshots

Status in ubuntu-release-upgrader package in Ubuntu:
  Confirmed

Bug description:
  SUMMARY:
  I'm running Ubuntu 24.10 on a ZFS root fs on LUKS which also has snapshots. As part of the release upgrade process ubuntu-release-upgrader is triggering a bug in ZFS triggered by the update-grub steps which executes ls /.zfs/snapshot/… via 10_linux_zfs and eventually causes a kernel deadlock on mount.zfs.

  As the `ls` statement in question works flawlessly before the upgrade
  process, I assume the hang is caused due to ZoL been replaced by apt
  previously? That's why I feel this is a ubuntu-release-upgrader
  package bug.

  I've reported this bug also upstream at ZoL:
  https://github.com/openzfs/zfs/issues/17337

  
  DETAILED DESCRIPTION:
  I've failed now 7 time to upgrade my Ubuntu 24.10 to 25.04 (Beta, RC, First Release, Today). In all cases the upgrade runs into a complete system freeze deadlock. `zfs rollback`  for the rescue. Today, only using text console with screen & running `dmesg -Hxw`, `htop -d 5` and `do-release-upgrade` in parallel I finally was able to pinpoint the problem:

    1. `do-release-upgrade` downloads & install all updated .deb
  packages

    2. Eventually, the upgrade tries to run `upgrade-grub`

    3. The grub script executes the hook `10_linux_zfs` to identify the
  available kernel versions for the grub boot menu

    4. As part of this discovery an `ls /.zfs/snapshot/[snapshot]/etc`
  is executed which causes a system freeze

    5. Eventually, after a long pause the kernel reports `task ls:…
  blocked for more than 122 seconds.`

    6. Only option left for me is to shutdown server and revert to ZFS
  snapshot I did before the upgrade

  
  REMARKS:
  * Executing "ls /.zfs/snapshot/[snapshot]/etc" and "update-grub" works successfully BEFORE upgrade
  * Rebooting into the partial upgraded system, running "strace update-grub" reproduces the problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/2110891/+subscriptions