[Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online

[Philip Roche]

I have also successfully verified that -proposed amd64 kernel
`6.5.0-7-generic` results in successful network configuration when
tested using qemu on an amd64 host with older hardware (ThinkPad T460
with 6th gen intel i5 which is the same hardware which we were able to
reproduce the issue on previously). See
https://people.canonical.com/~philroche/20231003-mantic-minimal-
proposed-kernel/amd64/ for cloud-init logs, some debug output and test
image.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2036968

Title:
  Mantic minimized/minimal cloud images do not receive IP address during
  provisioning; systemd regression with wait-online

Status in cloud-images:
  New
Status in linux package in Ubuntu:
  Fix Committed
Status in systemd package in Ubuntu:
  Triaged

Bug description:
  Following a recent change from linux-kvm kernel to linux-generic
  kernel in the mantic minimized images, there is a reproducable bug
  where a guest VM does not have an IP address assigned as part of
  cloud-init provisioning.

  This is easiest to reproduce when emulating arm64 on amd64 host. The
  bug is a race condition, so there could exist fast enough
  virtualisation on fast enough hardware where this bug is not present
  but in all my testing I have been able to reproduce.

  The latest mantic minimized images from http://cloud-
  images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and
  no initrd to fallback to.

  This but is not present in the non minimized/base images @
  http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with
  the required drivers present for virtio-net.

  Reproducer

  ```
  wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh

  chmod +x ./launch-qcow2-image-qemu-arm64.sh
  wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img
  ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img
  ```

  You will then be able to log in with user `ubuntu` and password
  `passw0rd`.

  You can run `ip a` and see that there is a network interface present
  (separate to `lo`) but no IP address has been assigned.

  ```
  ubuntu at cloudimg:~$ ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
      link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff

  ```

  This is because when cloud-init is trying to configure network
  interfaces it doesn't find any so it doesn't configure any. But by the
  time boot is complete the network interface is present but cloud-init
  provisioning has already completed.

  You can verify this by running `sudo cloud-init clean && sudo cloud-
  init init`

  You can then see a successfully configured network interface

  ```
  ubuntu at cloudimg:~$ ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
      inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1
         valid_lft 86391sec preferred_lft 86391sec
      inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute
         valid_lft 86393sec preferred_lft 14393sec
      inet6 fe80::5054:ff:fe12:3456/64 scope link
         valid_lft forever preferred_lft forever

  ```

  The bug is also reproducible with amd64 guest on adm64 host on
  older/slower hardware.

  The suggested fixes while debugging this issue are:

  * to include `virtio-net` as a built-in in the mantic generic kernel
  * understand what needs to change in cloud-init so that it can react to late additions of network interfaces

  I will file a separate bug against cloud-init to address the race
  condition on emulated guest/older hardware.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2036968/+subscriptions