[Bug 2110891] [NEW] System freeze on release upgrade 24.10 oneiric to 25.04 plucky with root fs on ZFS and snapshots

[Benjamin Schmid]

Public bug reported:

SUMMARY:
I'm running Ubuntu 24.10 on a ZFS root fs on LUKS which also has snapshots. As part of the release upgrade process ubuntu-release-upgrader is triggering a bug in ZFS triggered by the update-grub steps which executes ls /.zfs/snapshot/… via 10_linux_zfs and eventually causes a kernel deadlock on mount.zfs.

As the `ls` statement in question works flawlessly before the upgrade
process, I assume the hang is caused due to ZoL been replaced by apt
previously? That's why I feel this is a ubuntu-release-upgrader package
bug.

I've reported this bug also upstream at ZoL:
https://github.com/openzfs/zfs/issues/17337


DETAILED DESCRIPTION:
I've failed now 7 time to upgrade my Ubuntu 24.10 to 25.04 (Beta, RC, First Release, Today). In all cases the upgrade runs into a complete system freeze deadlock. `zfs rollback`  for the rescue. Today, only using text console with screen & running `dmesg -Hxw`, `htop -d 5` and `do-release-upgrade` in parallel I finally was able to pinpoint the problem:

  1. `do-release-upgrade` downloads & install all updated .deb packages

  2. Eventually, the upgrade tries to run `upgrade-grub`

  3. The grub script executes the hook `10_linux_zfs` to identify the
available kernel versions for the grub boot menu

  4. As part of this discovery an `ls /.zfs/snapshot/[snapshot]/etc` is
executed which causes a system freeze

  5. Eventually, after a long pause the kernel reports `task ls:…
blocked for more than 122 seconds.`

  6. Only option left for me is to shutdown server and revert to ZFS
snapshot I did before the upgrade


REMARKS:
* Executing "ls /.zfs/snapshot/[snapshot]/etc" and "update-grub" works successfully BEFORE upgrade
* Rebooting into the partial upgraded system, running "strace update-grub" reproduces the problem.

** Affects: ubuntu-release-upgrader (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/2110891

Title:
  System freeze on release upgrade 24.10 oneiric to 25.04 plucky with
  root fs on ZFS and snapshots

Status in ubuntu-release-upgrader package in Ubuntu:
  New

Bug description:
  SUMMARY:
  I'm running Ubuntu 24.10 on a ZFS root fs on LUKS which also has snapshots. As part of the release upgrade process ubuntu-release-upgrader is triggering a bug in ZFS triggered by the update-grub steps which executes ls /.zfs/snapshot/… via 10_linux_zfs and eventually causes a kernel deadlock on mount.zfs.

  As the `ls` statement in question works flawlessly before the upgrade
  process, I assume the hang is caused due to ZoL been replaced by apt
  previously? That's why I feel this is a ubuntu-release-upgrader
  package bug.

  I've reported this bug also upstream at ZoL:
  https://github.com/openzfs/zfs/issues/17337

  
  DETAILED DESCRIPTION:
  I've failed now 7 time to upgrade my Ubuntu 24.10 to 25.04 (Beta, RC, First Release, Today). In all cases the upgrade runs into a complete system freeze deadlock. `zfs rollback`  for the rescue. Today, only using text console with screen & running `dmesg -Hxw`, `htop -d 5` and `do-release-upgrade` in parallel I finally was able to pinpoint the problem:

    1. `do-release-upgrade` downloads & install all updated .deb
  packages

    2. Eventually, the upgrade tries to run `upgrade-grub`

    3. The grub script executes the hook `10_linux_zfs` to identify the
  available kernel versions for the grub boot menu

    4. As part of this discovery an `ls /.zfs/snapshot/[snapshot]/etc`
  is executed which causes a system freeze

    5. Eventually, after a long pause the kernel reports `task ls:…
  blocked for more than 122 seconds.`

    6. Only option left for me is to shutdown server and revert to ZFS
  snapshot I did before the upgrade

  
  REMARKS:
  * Executing "ls /.zfs/snapshot/[snapshot]/etc" and "update-grub" works successfully BEFORE upgrade
  * Rebooting into the partial upgraded system, running "strace update-grub" reproduces the problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/2110891/+subscriptions