[Bug 2039441] Re: "System is booting up. Unprivileged users are not permitted to log in yet." causes wait subcommand to fail

[James Page]

Updates for mantic, jammy and focal sponsored into the UNAPPROVED queue
ready for SRU team review.

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/2039441

Title:
  "System is booting up. Unprivileged users are not permitted to log in
  yet." causes wait subcommand to fail

Status in uvtool:
  Fix Committed
Status in cloud-init package in Ubuntu:
  Fix Committed
Status in uvtool package in Ubuntu:
  Fix Released
Status in cloud-init source package in Focal:
  Fix Committed
Status in uvtool source package in Focal:
  Triaged
Status in cloud-init source package in Jammy:
  Fix Committed
Status in uvtool source package in Jammy:
  Triaged
Status in cloud-init source package in Lunar:
  Fix Committed
Status in uvtool source package in Lunar:
  Won't Fix
Status in cloud-init source package in Mantic:
  Fix Committed
Status in uvtool source package in Mantic:
  Triaged

Bug description:
  [ Impact ]

  The bug breaks `uvt-kvm wait` promise that the virtualized system is up and running. `uvt-kvm` naively waited for the ssh server to be up, but if the
  ssh server is up before authentication (pam notably) is ready, the system is still not ready to use.
  For that reason, all automation relying on it breaks and report failures blaming the underlying image instead of waiting for it to be really ready.
  This fix will make `uvt-kvm wait` retry in case the login fails.

  [ Test Plan ]

  Use the patched uvt-kvm to start then wait for this image to come up:
  http://cloud-images.ubuntu.com/releases/focal/release-20231011/ubuntu-20.04-server-cloudimg-amd64.img
  This image was know to trigger this error, so verify ten times we can: start it, wait then ssh without any failure.
  Verify also this with a previously working image such as http://cloud-images.ubuntu.com/daily/server/focal/20231003/focal-server-cloudimg-amd64.img

  [ Where problems could occur ]

  The retry mechanism could be triggered for other reasons than login
  error and slow down the whole process.

  [ Original Bug Report ]
  After doing `uvt-kvm wait`, we can expect to be able to ssh into the VMs.
  That's not always the case as the ssh port can be up before PAM is setup:
  `System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).`
  This means that subsequent programs can't rely on `uvt-kvm wait` to know if the system is up, which defeats the purpose of this function and drives the complexity up in highly automated environment.

  Personally, I see two ways to fix the wait to handle this case:
  - Change the behavior of the created VM to avoid this edge case.
  - Makes `uvt-kvm wait` smarter by actually establishing a communication to check if we really can login.

  The last option seems less intrusive but will make the library more complex.
  I'm not convinced that would be a reasonable default or would be better as an option to `uvt-kvm wait`.

To manage notifications about this bug go to:
https://bugs.launchpad.net/uvtool/+bug/2039441/+subscriptions