[Bug 1986781] Re: [Ubuntu 22.04]cloud-init failed to complete after 10 minutes of waiting was shown during Installation via iDRAC Virtual Console

[Mauricio Faria de Oliveira]

Verification done for jammy-proposed.

...

Booting jammy-live-server-amd64.iso with `break=init`
(more details in comment #37).

(initramfs) chroot /root /bin/bash
root at ubuntu-server:/# dhclient enp1s0

root at ubuntu-server:/# add-apt-repository -y -p proposed

root at ubuntu-server:/# apt install -y casper/jammy-proposed

root at ubuntu-server:/# apt policy casper
casper:
  Installed: 1.470.2
  Candidate: 1.470.2
  Version table:
 *** 1.470.2 500
        500 http://security.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
...
root at ubuntu-server:/# add-apt-repository -y -p proposed -r

...

root at ubuntu-server:/# sed -i '/^ExecStart=/ s,=,=/usr/bin/strace
--inject read:delay_enter=60s ,' /usr/lib/systemd/system/casper-
md5check.service

root at ubuntu-server:/# cat /usr/lib/systemd/system/casper-md5check.service
[Unit]
Description=casper-md5check Verify Live ISO checksums
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/bin/strace --inject read:delay_enter=60s /usr/lib/casper/casper-md5check /cdrom /cdrom/md5sum.txt
...

root at ubuntu-server:/# exit
exit
(initramfs) exit
...


...
[  OK  ] Reached target Multi-User System.
         Starting casper-md5check Verify Live ISO checksums...
         Starting Execute cloud user/final scripts...
...
[  OK  ] Finished Execute cloud user/final scripts.
[  OK  ] Reached target Cloud-init target.

Ubuntu 22.04.1 LTS ubuntu-server ttyS0


connecting...
<a few seconds later>

================================================================================
  Serial                                                              [ Help ]
================================================================================

  As the installer is running on a serial console, it has started in basic
  mode, using only the ASCII character set and black and white colours.

  If you are connecting from a terminal emulator such as gnome-terminal that
  supports unicode and rich colours you can switch to "rich mode" which uses
  unicode, colours and supports many languages.

  You can also connect to the installer over the network via SSH, which will
  allow use of rich mode.




                          [ Continue in rich mode  > ]
                          [ Continue in basic mode > ]
                          [ View SSH instructions    ]


** Tags removed: verification-needed verification-needed-jammy
** Tags added: verification-done verification-done-jammy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to casper in Ubuntu.
https://bugs.launchpad.net/bugs/1986781

Title:
  [Ubuntu 22.04]cloud-init failed to complete after 10 minutes of
  waiting was shown during Installation via iDRAC Virtual Console

Status in cloud-init:
  Invalid
Status in subiquity:
  Invalid
Status in casper package in Ubuntu:
  Fix Released
Status in casper source package in Focal:
  Triaged
Status in casper source package in Jammy:
  Fix Committed
Status in casper source package in Kinetic:
  Won't Fix

Bug description:
  [Impact]

   * Users that install Ubuntu Server through slow
     media (eg, virtual optical drive over network,
     which may be common on enterprise deployments)
     might hit the following subiquity startup error:

    'cloud-init failed to complete after 10 minutes of waiting'

   * (That in addition to 10 minutes of waiting themselves.)

   * This happens because casper-md5check.service is
     (slowly) verifying the integrity of install media,
     which blocks `multi-user.target`,
     which blocks `cloud-final.service`,
     which blocks `cloud-init status --wait`
     which is used in subiquity / `waiting on cloud-init`).

  [Fix]

   * The adopted solution (merged on lunar) is simply
     not to block `multi-user.target`, but rather run
     _after_ it.

  [Test Steps]

   For a synthetic reproducer of slowness of casper-md5check:

   * boot with `break=init` to break into initramfs-tools
     before exec() systemd.
   * chroot /root /bin/bash
   * edit /usr/lib/systemd/system/casper-md5check.service
   * prepend `strace --inject read:delay_enter=5s` to the
     command in `ExecStart`, to introduce a 5 secs delay
     to every read() syscall performed by casper-md5check.
   * exit twice (chroot, initramfs shell) to resume boot.

    See comment 37 for examples.

  [Regression Potential]

   * Functionality related to install media integrity check.

   * Users with corrupted install media might not realize
     this until later on; but this is rarely the case and
     even w/out the fix, there's a lot that runs _before_
     we even get to casper-md5check, so they may (still)
     see errors early anyway.

   * There's a cosmetic glitch in the proposed solution:
     the systemd line when casper-md5check finishes
     shows up on top of subiquity's menu (screenshot):

     "[ OK ] Finished casper-md5check Verify Live ISO checksums."

     Dan Bungert mentioned this is known and should be
     addressed in a future change to subiquity, and is
     not supposed to block the SRU for Jammy / 22.04.2.

  [Original Description]

  Description:

  On Dell EMC PowerEdge system when Install Ubuntu 22.04 via iDRAC
  Virtual Console, cloud-init failed to complete after 10 minutes of
  waiting.

  Steps to Reproduce:

  1. Login to iDRAC and Launch Virtual Console.
  2. Connect to Virtual Media and Map ubuntu 22.04 iso file using Map CD/DVD option.
  3. Try Installing Ubuntu server.
  4. "cloud-init" failed to complete after 10 minutes of waiting was shown during Installation.

  Expected Results :-

  Installation should be successful.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1986781/+subscriptions