[Bug 2036358] Re: systemd wait-online now times out after jammy and lunar upgrade

Fri Sep 22 15:41:54 UTC 2023

I have verified the fix using systemd 249.11-0ubuntu3.11 from jammy-
proposed:

Creating a fresh container:

nr at six:~$ lxc launch ubuntu-daily:jammy jammy
Creating jammy
Starting jammy                              
nr at six:~$ lxc exec jammy bash

Within the container, after enabling -proposed:

root at jammy:~# vi /etc/apt/sources.list
root at jammy:~# apt update && apt install systemd -y 
[ ... ]
root at jammy:~# apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.11
  Candidate: 249.11-0ubuntu3.11
  Version table:
 *** 249.11-0ubuntu3.11 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root at jammy:~# vi /etc/netplan/50-cloud-init.yaml 
root at jammy:~# cat /etc/netplan/50-cloud-init.yaml 
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true
            optional: true
root at jammy:~# netplan generate
root at jammy:~# netplan apply
root at jammy:~# time SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=5
Found link 11
Found link 1
eth0: link is ignored
lo: link is ignored

real	0m0.004s
user	0m0.003s
sys	0m0.000s
root at jammy:~# echo $?
0

I also ran the test from bug 1982218 again to ensure that does not
regress:

On the host, I have clean-jammy-amd64, and detached it's default network
interface:

nr at six:~$ virsh detach-interface clean-jammy-amd64 network
Interface detached successfully

Then, in the VM, configure systemd-networkd to use DHCP on all en*
interfaces:

nr at clean-jammy-amd64:~$ apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.11
  Candidate: 249.11-0ubuntu3.11
  Version table:
 *** 249.11-0ubuntu3.11 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
nr at clean-jammy-amd64:~$ sudo su
root at clean-jammy-amd64:/home/nr# cat > /etc/systemd/network/10-dhcp.network << EOF
[Match]
Name=en*

[Network]
DHCP=yes
EOF
root at clean-jammy-amd64:/home/nr# systemctl restart systemd-networkd

Now, back in the host, I attach a network interface without DHCP:

nr at six:~$ virsh attach-interface clean-jammy-amd64 network no-dhcp
Interface attached successfully

In the VM, check that wait-online times out since the interface does not
get configured:

root at clean-jammy-amd64:/home/nr# /lib/systemd/systemd-networkd-wait-online --any --timeout=5
Timeout occurred while waiting for network connectivity.
root at clean-jammy-amd64:/home/nr# networkctl 
IDX LINK TYPE     OPERATIONAL SETUP      
  1 lo   loopback carrier     unmanaged
  3 ens3 ether    degraded    configuring

2 links listed.

>From the host, attach another interface that will get DHCP:

nr at six:~$ virsh attach-interface clean-jammy-amd64 network default
Interface attached successfully

And finally, in the VM, run wait-online again and see that it will exit
successfully when the second interface is configured:

root at clean-jammy-amd64:/home/nr# /lib/systemd/systemd-networkd-wait-online --any --timeout=0
root at clean-jammy-amd64:/home/nr# networkctl 
IDX LINK TYPE     OPERATIONAL SETUP      
  1 lo   loopback carrier     unmanaged
  3 ens3 ether    degraded    configuring
  4 ens9 ether    routable    configured 

3 links listed.

** Tags removed: verification-needed verification-needed-lunar
** Tags added: verification-done verification-done-lunar

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2036358

Title:
  systemd wait-online now times out after jammy and lunar upgrade

Status in systemd package in Ubuntu:
  Invalid
Status in systemd source package in Jammy:
  Fix Committed
Status in systemd source package in Lunar:
  Fix Committed

Bug description:
  [NOTE]

  If you are running a desktop system and you see this issue, you should
  run:

  $ systemctl disable --now systemd-networkd.service

  This will disable systemd-networkd and associated units, including
  systemd-networkd-wait-online.service. NetworkManager and systemd-
  networkd should not be running at the same time. On desktop,
  NetworkManager is the default network stack.

  [Impact]

  When all interfaces are "not required for online", e.g. when they are
  marked "optional: true" in netplan, systemd-networkd-wait-online will
  timeout. Or, in other words, systemd-networkd-wait-online will timeout
  even though all interfaces are ignored, hence none of them will ever
  be marked as "ready." Depending on what units depend on network-
  online.target, this can delay boot by 120 seconds (the default timeout
  for systemd-networkd-wait-online).

  [Test Plan]

  1. Create a new LXD container. These instructions assume jammy is the
  release, but the same can be done for lunar.

  $ lxc launch ubuntu-daily:jammy jammy
  $ lxc exec jammy bash

  2. Once in the container, modify the default /etc/netplan/10-lxc.yaml
  so that eth0 is configured with "optional: true":

  $ vi /etc/netplan/10-lxc.yaml # Use whatever editor you like
  $ cat /etc/netplan/10-lxc.yaml
  network:
    version: 2
    ethernets:
      eth0:
        dhcp4: true
        dhcp-identifier: mac
        optional: true

  3. Re-generate and apply the netplan configuration.

  $ netplan generate
  $ netplan apply

  4. Manually run systemd-networkd-wait-online, and observe that all
  links are ignored, and the command times out:

  $ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
  Found link lo(1)
  Found link eth0(19)
  lo: link is ignored
  eth0: link is ignored
  Timeout occurred while waiting for network connectivity.

  [Where problems could occur]

  This patch partially re-instates a patch remove in bug 1982218.
  However, instead of exiting if all links are unmanaged, we exit if all
  links are ignored in manager_configured(). If the patch was wrong, we
  may re-introduce bug 1982218, so as part of this SRU verification,
  that bug should be tested too. Any other regressions would also be
  related to systemd-networkd-wait-online behavior.

  [Original Description]

  On Ubuntu 22.04 desktop system using network-manager and upgrading to
  systemd 249.11-0ubuntu3.10, wait-online now times out which prevents
  logins (GDM, ssh, console) until it does time out. This seems to be
  introduced by the change for
  https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982218.

  https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982218/comments/21
  also mentioned the problem on Lunar.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2036358/+subscriptions