[Bug 2028862] Re: Mantic hangs during install with continuous 'Job snapd.seeded.service/start running' messages

[Dan Bungert]

** Description changed:

+ [ Impact ]
+ 
+  * The subiquity systemd units and cloud-init 23.3 have an
+    incompatibility that results in a stuck boot, when cloud-init
+    23.3 is present on an unpatched install ISO.
+ 
+  * This is a necessary prerequisite to allow for Jammy dailies /
+    eventually 22.04.4 with updated cloud-init.
+ 
+ [ Test Plan ]
+ 
+  * Live-server Daily ISOs constructed with the fixed version
+    should boot correctly to the Subqiuity TUI, regardless of the
+    cloud-init version present in that ISO.
+ 
+  * If desired, a test plan with ~16 steps can be provided with
+    detailed steps to modify ISOs to simulate the bug and the fix.
+ 
+ [ Where problems could occur ]
+ 
+  * Changes to livecd-rootfs cause frequent regressions to other
+    build projects.  The risk here is relatively localized to
+    live-server builds as we are only modifying drop-in systemd
+    files in use by live-server.
+ 
+  * If this change is done incorrectly, when cloud-init 23.3 is
+    SRUed and allowed to migrate, the boot of the live-server
+    dailies may show the same problem as the original bug.
+ 
+  * "The livecd-rootfs package is a frequent target of SRUs as part
+    of development of changes to image builds for the target
+    series, and is not intended for general installation on
+    end-user systems. The risk of user-affecting regression is
+    lower as a result, because the impact of changes to this
+    package to end users is mediated by way of image builds."
+ https://wiki.ubuntu.com/StableReleaseUpdates?action=show&redirect=StableReleaseUpdate#livecd-rootfs
+ 
+ [ Other Info ]
+ 
+  * Using kernel command line "systemd.mask=snapd.seeded.service"
+    allows affected ISOs to boot successfully
+ 
+ 
+ ---
+ previous description:
+ 
  While trying to install the latest mantic ISO image (tried current and
  pending) on s390x (but it's probably not limited to a certain
  architecture) the installation hangs with never-ending messages like
  this:
  
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 23s / no limit)
  M[K[[0m[0;31m*     [0m] Job snapd.seeded.service/start running (16min 23s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 24s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 24s / no limit)
  M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] Job snapd.seeded.service/start running (16min 25s / no limit)
  M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] Job snapd.seeded.service/start running (16min 25s / no limit)
  M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] Job snapd.seeded.service/start running (16min 26s / no limit)
  M[K[    [0;31m*[0;1;31m*[0m] Job snapd.seeded.service/start running (16min 26s / no limit)
  M[K[     [0;31m*[0m] Job snapd.seeded.service/start running (16min 27s / no limit)
  M[K[    [0;31m*[0;1;31m*[0m] Job snapd.seeded.service/start running (16min 27s / no limit)
  M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] Job snapd.seeded.service/start running (16min 28s / no limit)
  M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] Job snapd.seeded.service/start running (16min 28s / no limit)
  M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] Job snapd.seeded.service/start running (16min 29s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 29s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 30s / no limit)
  M[K[[0m[0;31m*     [0m] Job snapd.seeded.service/start running (16min 30s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 31s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 31s / no
  
  (sorry for the 'special' characters, but it's because I copied the
  content from the HMC console)
  
  I was told that using the kernel arg "systemd.mask=snapd.seeded.service"
  should help to workaround this, still need to try this out ...

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/2028862

Title:
  Mantic hangs during install with continuous 'Job
  snapd.seeded.service/start running' messages

Status in snapd:
  Invalid
Status in subiquity:
  Invalid
Status in Ubuntu on IBM z Systems:
  Fix Released
Status in livecd-rootfs package in Ubuntu:
  Fix Released
Status in livecd-rootfs source package in Jammy:
  New

Bug description:
  [ Impact ]

   * The subiquity systemd units and cloud-init 23.3 have an
     incompatibility that results in a stuck boot, when cloud-init
     23.3 is present on an unpatched install ISO.
     The symptom looks like the following:
     "A start job is waiting for Wait until snapd is fully seeded"
     This will never complete.

   * This is a necessary prerequisite to allow for Jammy dailies /
     eventually 22.04.4 with updated cloud-init.

  [ Test Plan ]

   * Live-server Daily ISOs constructed with the fixed version
     should boot correctly to the Subqiuity TUI, regardless of the
     cloud-init version present in that ISO.

   * If desired, a test plan with ~16 steps can be provided with
     detailed steps to modify ISOs to simulate the bug and the fix.

  [ Where problems could occur ]

   * Changes to livecd-rootfs cause frequent regressions to other
     build projects.  The risk here is relatively localized to
     live-server builds as we are only modifying drop-in systemd
     files in use by live-server.

   * If this change is done incorrectly, when cloud-init 23.3 is
     SRUed and allowed to migrate, the boot of the live-server
     dailies may show the same problem as the original bug.

   * "The livecd-rootfs package is a frequent target of SRUs as part
     of development of changes to image builds for the target
     series, and is not intended for general installation on
     end-user systems. The risk of user-affecting regression is
     lower as a result, because the impact of changes to this
     package to end users is mediated by way of image builds."
  https://wiki.ubuntu.com/StableReleaseUpdates?action=show&redirect=StableReleaseUpdate#livecd-rootfs

  [ Other Info ]

   * Using kernel command line "systemd.mask=snapd.seeded.service"
     allows affected ISOs to boot successfully

  ---
  previous description:

  While trying to install the latest mantic ISO image (tried current and
  pending) on s390x (but it's probably not limited to a certain
  architecture) the installation hangs with never-ending messages like
  this:

  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 23s / no limit)
  M[K[[0m[0;31m*     [0m] Job snapd.seeded.service/start running (16min 23s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 24s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 24s / no limit)
  M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] Job snapd.seeded.service/start running (16min 25s / no limit)
  M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] Job snapd.seeded.service/start running (16min 25s / no limit)
  M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] Job snapd.seeded.service/start running (16min 26s / no limit)
  M[K[    [0;31m*[0;1;31m*[0m] Job snapd.seeded.service/start running (16min 26s / no limit)
  M[K[     [0;31m*[0m] Job snapd.seeded.service/start running (16min 27s / no limit)
  M[K[    [0;31m*[0;1;31m*[0m] Job snapd.seeded.service/start running (16min 27s / no limit)
  M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] Job snapd.seeded.service/start running (16min 28s / no limit)
  M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] Job snapd.seeded.service/start running (16min 28s / no limit)
  M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] Job snapd.seeded.service/start running (16min 29s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 29s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 30s / no limit)
  M[K[[0m[0;31m*     [0m] Job snapd.seeded.service/start running (16min 30s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 31s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 31s / no

  (sorry for the 'special' characters, but it's because I copied the
  content from the HMC console)

  I was told that using the kernel arg
  "systemd.mask=snapd.seeded.service" should help to workaround this,
  still need to try this out ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/snapd/+bug/2028862/+subscriptions