[Bug 2007827] Re: flash-kernel failure when upgrading f-k anad kernel in the same cycle

[Dave Jones]

Okay, I've now confirmed that this is indeed the rather more wide-
ranging bug that I suspected it was (it also affects the Pi images, but
in a different way). I'll try and explain the problem with reference to
the log provided in comment:3. First, the scenario:

* the device currently has kernel 5.15.0-1015-xilinx-zynqmp
* kernel 5.15.0-1018-xilinx-zynqmp is available for upgrade
* the device currently has flash-kernel 3.104ubuntu14 installed
* flash-kernel 3.106ubuntu12test1 is available for upgrade

This in turn means that the following package states are currently true:

* linux-image-5.15.0-1015-xilinx-zynqmp is "installed"
* linux-image-5.15.0-1018-xilinx-zynqmp is "not installed"
* flash-kernel is "installed"

Now selected entries from the log, in chronological order:

  Preparing to unpack .../0-flash-kernel_3.106ubuntu12test1_arm64.deb ...
  Unpacking flash-kernel (3.106ubuntu12test1) over (3.104ubuntu14) ...
  ...
  Selecting previously unselected package linux-image-5.15.0-1018-xilinx-zynqmp.
  Preparing to unpack .../4-linux-image-5.15.0-1018-xilinx-zynqmp_5.15.0-1018.20_arm64.deb ...
  ...
  Setting up flash-kernel (3.106ubuntu12test1) ...

At this point:

* flash-kernel has now been upgraded but is now in "triggers-pending" as its upgrade triggers itself (this is the issue fixed in LP: #1667742)
* linux-image-5.15.0-1015-xilinx-zynqmp is still "installed"
* linux-image-5.15.0-1018-xilinx-zynqmp is now "unpacked"

Further entries:

  I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-1018-xilinx-zynqmp
  I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-1018-xilinx-zynqmp
  Setting up linux-image-xilinx-zynqmp (5.15.0.1018.19) ...
  Setting up linux-modules-5.15.0-1018-xilinx-zynqmp (5.15.0-1018.20) ...
  Setting up linux-xilinx-zynqmp (5.15.0.1018.19) ...

At this point, linux-image-5.15.0-1018-xilinx-zynqmp is now "triggers-
awaiting". The kernel postinst has run, updating the /boot symlinks to
point to the 1018 kernel. However, the /boot/initrd.img symlink is
"dead" because the initrd for this version has not yet been generated.
This doesn't matter though as the /boot kernel isn't the one actually
used at boot time; it's just the one that flash-kernel copies to
<wherever the platform needs>, and flash-kernel hasn't been called yet.

Carrying on:

  Processing triggers for initramfs-tools (0.140ubuntu13) ...
  initramfs-tools: update-initramfs -u
  update-initramfs: Generating /boot/initrd.img-5.15.0-1015-xilinx-zynqmp

The initramfs-tools hook is called for the *old* kernel, regenerating
its initrd. This appears largely pointless (and in this case, actually
is) but is necessary to cover the case of linux-modules-extra packages
being added / removed, requiring a regeneration of the initrd (and
possibly other scenarios I'm unaware of).

Carrying on:

  Processing triggers for flash-kernel (3.106ubuntu12test1) ...
  flash-kernel: installing version 5.15.0-1018-xilinx-zynqmp

Here's the root of the error. flash-kernel finally runs "properly" (no
deferral), and uses "linux-version list" to determine the set of
installed kernel versions. This simply looks for /boot/vmlinu[xz]-$ver
files and reports the $ver suffixes. At this point
/boot/vmlinuz-5.15.0-1018-xilinx-zynqmp exists but the corresponding
initrd doesn't. More importantly, the linux-image-5.15.0-1018-xilinx-
zynqmp which owns the kernel is still in "triggers-awaiting" state.
flash-kernel should *not* be considering this a valid candidate for
flashing at this stage, but it currently does.

The suggested fix comes next:

  flash-kernel: Initrd required for FIT method missing; expecting future
run

In this case, there will be a future run but this is not guaranteed by
the fix (it doesn't re-fire the trigger), but further this also misses
the case that the earlier regeneration of the initrd for the old kernel
was actually necessary. I contend that what *should* happen here is that
flash-kernel filters the output of "linux-version list" to determine
only what packages are fully "installed" and proceeds with the highest
of *those*. This would lead to a redundant run in this case, but as I'll
explain a bit further on it turns out to be quite important in the Pi
case too.

More log entries:

  Processing triggers for linux-image-5.15.0-1018-xilinx-zynqmp (5.15.0-1018.20) ...
  /etc/kernel/postinst.d/initramfs-tools:
  W: initramfs-tools: linux-image-5.15.0-1018-xilinx-zynqmp passes a version number: 5.15.0-1018-xilinx-zynqmp
  initramfs-tools: update-initramfs -c -k 5.15.0-1018-xilinx-zynqmp -b /boot
  update-initramfs: Generating /boot/initrd.img-5.15.0-1018-xilinx-zynqmp

At this point the initramfs-tools triggers for the *new* kernel have
run, and the initrd for the 1018 version has been generated. The
/boot/initrd.img symlink is now "valid", and the linux-
image-5.15.0-1018-xilinx-zynqmp package is now in "installed" state.

Proceeding onwards:

  flash-kernel: deferring update (trigger activated)
  /etc/kernel/postinst.d/zz-flash-kernel:
  flash-kernel: deferring update (trigger activated)
  Processing triggers for flash-kernel (3.106ubuntu12test1) ...
  flash-kernel: installing version 5.15.0-1018-xilinx-zynqmp

A few more deferrals of the flash-kernel trigger and eventually we reach
its final run. This time everything is in place, and the kernel is
flashed successfully (this would also be the case if we were filtering
for "installed" state kernel packages only).

Why does this matter in the Pi case? We haven't noticed this issue in
the Pi images because it plays out a bit differently there, but with
quite serious side effects. The "pi" method in flash-kernel permits
flashing of a kernel with no initrd (this isn't valid on Ubuntu itself,
but is in RaspiOS and some derivatives including some from Ubuntu). In
the first run of flash-kernel above, instead of crashing out complaining
of no initrd.img, flash-kernel simply *deletes initrd.img* from the boot
partition (mounted at /boot/firmware in Ubuntu's case). On a standard
Ubuntu system this would render it unbootable. I suspect one reason we
haven't noticed is that the second flash-kernel run then "fixes" this
after the new initrd has been generated.

However, this still leaves the system in an unbootable state for a
period in the middle of the upgrade, which is obviously not desirable.
Again, the remedy of filtering linux versions for only those in fully
"installed" state would fix this.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to flash-kernel in Ubuntu.
https://bugs.launchpad.net/bugs/2007827

Title:
  flash-kernel failure when upgrading f-k anad kernel in the same cycle

Status in flash-kernel package in Ubuntu:
  New

Bug description:
  [Impact]
  In version 3.104ubuntu15 of flash-kernel, when both f-k and the kernel are upgraded in the same cycle, depending on the ordering of dpkg trigger execution, f-k may find the content of /boot "inconsistent" causing it to fail and return error exit status 1.

  Erorr message:
  Processing triggers for man-db (2.10.2-1) ...
  Processing triggers for flash-kernel (3.104ubuntu15) ...
  flash-kernel: installing version 5.15.0-1018-xilinx-zynqmp
  Initrd required for FIT method
  dpkg: error processing package flash-kernel (--configure):
   installed flash-kernel package post-installation script subprocess returned error exit status 1
  Processing triggers for linux-image-5.15.0-1018-xilinx-zynqmp (5.15.0-1018.20) ...
  /etc/kernel/postinst.d/initramfs-tools:
  update-initramfs: Generating /boot/initrd.img-5.15.0-1018-xilinx-zynqmp

  flash-kernel gets the latest kernel version by "linux-version list".
  When flash-kernel was triggered to generate fitimage, the kernel version is "5.15.0-1018" and the initrd for it wasn't ready. So, flash-kernel failed to generate the fitimage.

  A subsequent run of "apt install -f" fixed things because, by that
  point, the kernel's own trigger had executed, ensuring that update-
  initramfs had been run. In the case that f-k is run "prematurely" and
  finds itself in this situation (/boot/kernel-$[ver} exists, but
  /boot/initrd-${ver}) doesn't), it should probably bail out silently
  under the assumption that whatever is responsible for it will rectify
  the situation and trigger f-k again (as happens in the kernel postinst
  hooks).

  [Test Case]
  1. Flash an old image (with an out of date kernel and flash-kernel)
  2. sudo apt-get update
  3. sudo apt install flash-kernel with the fix and linux packages
  4. Upgrade should proceed without issue

  [Regression Potential]
  As with the previous flash-kernel uploads, it is possible that a breakage in the changed code can lead to issues with upgrading kernels (due to f-k being executed via a trigger at the end) or with Xilinx devices in the field not upgrading correctly. I will test all the changes extensively though.

  Related issues:
  LP: #1861292 flash-kernel failure during kernel upgrade

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/flash-kernel/+bug/2007827/+subscriptions