[Bug 1986781] Re: [Ubuntu 22.04]cloud-init failed to complete after 10 minutes of waiting was shown during Installation via iDRAC Virtual Console

[Dan Bungert]

Verification done for jammy-proposed.

I have purposefully done a different test procedure, based on injecting
the proposed casper version  and using systemd-analyze.

Above, in comments such as https://bugs.launchpad.net/cloud-
init/+bug/1986781/comments/25, we can see that casper-md5check needs to
complete before hitting multi-user.target.

With casper 1.470.2 in place, it looks like:

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

casper-md5check.service +13.837s
└─multi-user.target @18.459s
  └─ssh.service @18.336s +30ms
    └─basic.target @7.833s
      └─sockets.target @7.831s
        └─snap.lxd.user-daemon.unix.socket @15.746s
          └─sysinit.target @7.778s
            └─cloud-init.service @6.080s +1.695s
              └─systemd-networkd-wait-online.service @6.061s +10ms
                └─systemd-networkd.service @5.950s +109ms
                  └─network-pre.target @5.948s
                    └─cloud-init-local.service @1.615s +4.332s
                      └─systemd-remount-fs.service @884ms +135ms
                        └─systemd-journald.socket @705ms
                          └─system.slice @624ms
                            └─-.slice @624ms

This is the expected behavior, so the SRU LGTM.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to casper in Ubuntu.
https://bugs.launchpad.net/bugs/1986781

Title:
  [Ubuntu 22.04]cloud-init failed to complete after 10 minutes of
  waiting was shown during Installation via iDRAC Virtual Console

Status in cloud-init:
  Invalid
Status in subiquity:
  Invalid
Status in casper package in Ubuntu:
  Fix Released
Status in casper source package in Focal:
  Triaged
Status in casper source package in Jammy:
  Fix Committed
Status in casper source package in Kinetic:
  Won't Fix

Bug description:
  [Impact]

   * Users that install Ubuntu Server through slow
     media (eg, virtual optical drive over network,
     which may be common on enterprise deployments)
     might hit the following subiquity startup error:

    'cloud-init failed to complete after 10 minutes of waiting'

   * (That in addition to 10 minutes of waiting themselves.)

   * This happens because casper-md5check.service is
     (slowly) verifying the integrity of install media,
     which blocks `multi-user.target`,
     which blocks `cloud-final.service`,
     which blocks `cloud-init status --wait`
     which is used in subiquity / `waiting on cloud-init`).

  [Fix]

   * The adopted solution (merged on lunar) is simply
     not to block `multi-user.target`, but rather run
     _after_ it.

  [Test Steps]

   For a synthetic reproducer of slowness of casper-md5check:

   * boot with `break=init` to break into initramfs-tools
     before exec() systemd.
   * chroot /root /bin/bash
   * edit /usr/lib/systemd/system/casper-md5check.service
   * prepend `strace --inject read:delay_enter=5s` to the
     command in `ExecStart`, to introduce a 5 secs delay
     to every read() syscall performed by casper-md5check.
   * exit twice (chroot, initramfs shell) to resume boot.

    See comment 37 for examples.

  [Regression Potential]

   * Functionality related to install media integrity check.

   * Users with corrupted install media might not realize
     this until later on; but this is rarely the case and
     even w/out the fix, there's a lot that runs _before_
     we even get to casper-md5check, so they may (still)
     see errors early anyway.

   * There's a cosmetic glitch in the proposed solution:
     the systemd line when casper-md5check finishes
     shows up on top of subiquity's menu (screenshot):

     "[ OK ] Finished casper-md5check Verify Live ISO checksums."

     Dan Bungert mentioned this is known and should be
     addressed in a future change to subiquity, and is
     not supposed to block the SRU for Jammy / 22.04.2.

  [Original Description]

  Description:

  On Dell EMC PowerEdge system when Install Ubuntu 22.04 via iDRAC
  Virtual Console, cloud-init failed to complete after 10 minutes of
  waiting.

  Steps to Reproduce:

  1. Login to iDRAC and Launch Virtual Console.
  2. Connect to Virtual Media and Map ubuntu 22.04 iso file using Map CD/DVD option.
  3. Try Installing Ubuntu server.
  4. "cloud-init" failed to complete after 10 minutes of waiting was shown during Installation.

  Expected Results :-

  Installation should be successful.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1986781/+subscriptions