[Bug 2036358] Re: systemd wait-online now times out after jammy and lunar upgrade

Wed Feb 26 10:46:28 UTC 2025

Here the "same?" problem. My environment it's a juju model with a jammy
lxc deployed on a jammy host:

ceph-mon/27           error     idle   14/lxd/0  10.93.6.24
hook failed: "install"

Inside the lxc we see this:

[  653.060669] cloud-init[312]: 2025-02-26 10:06:20,530 - ubuntu.py[ERROR]: Failed to wait for network: Unexpected error while running command.
[  653.060858] cloud-init[312]: Command: ['systemctl', 'start', 'systemd-networkd-wait-online.service']
[  653.061003] cloud-init[312]: Exit code: 1
[  653.061100] cloud-init[312]: Reason: -
[  653.061195] cloud-init[312]: Stdout:
[  653.061290] cloud-init[312]: Stderr: Job for systemd-networkd-wait-online.service failed because the control process exited with error code.
[  653.061393] cloud-init[312]:         See "systemctl status systemd-networkd-wait-online.service" and "journalctl -xeu systemd-networkd-wait-online.service" for details.

root at juju-841b16-14-lxd-0:~# apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.12
  Candidate: 249.11-0ubuntu3.12
  Version table:
 *** 249.11-0ubuntu3.12 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

root at juju-841b16-14-lxd-0:~# SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 10
Found link 1
eth0: link is not managed by networkd (yet?).
lo: link is ignored

root at juju-841b16-14-lxd-0:~# sudo netplan get
network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "00:16:3e:d5:e2:d4"
      addresses:
      - "10.93.6.24/23"
      nameservers:
        addresses:
        - 127.0.0.53
      gateway4: 10.93.6.1

root at juju-841b16-14-lxd-0:~# sudo netplan apply
root at juju-841b16-14-lxd-0:~# cat /run/systemd/network/10-netplan-eth0.network
[Match]
PermanentMACAddress=00:16:3e:d5:e2:d4

[Network]
LinkLocalAddressing=ipv6
Address=10.93.6.24/23
Gateway=10.93.6.1
DNS=127.0.0.53

root at juju-841b16-14-lxd-0:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
12: eth0 at if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:d5:e2:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fed5:e2d4/64 scope link
       valid_lft forever preferred_lft forever

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2036358

Title:
  systemd wait-online now times out after jammy and lunar upgrade

Status in systemd package in Ubuntu:
  Invalid
Status in systemd source package in Jammy:
  Fix Released
Status in systemd source package in Lunar:
  Fix Released

Bug description:
  [NOTE]

  If you are running a desktop system and you see this issue, you should
  run:

  $ systemctl disable --now systemd-networkd.service

  This will disable systemd-networkd and associated units, including
  systemd-networkd-wait-online.service. NetworkManager and systemd-
  networkd should not be running at the same time. On desktop,
  NetworkManager is the default network stack.

  [Impact]

  When all interfaces are "not required for online", e.g. when they are
  marked "optional: true" in netplan, systemd-networkd-wait-online will
  timeout. Or, in other words, systemd-networkd-wait-online will timeout
  even though all interfaces are ignored, hence none of them will ever
  be marked as "ready." Depending on what units depend on network-
  online.target, this can delay boot by 120 seconds (the default timeout
  for systemd-networkd-wait-online).

  [Test Plan]

  1. Create a new LXD container. These instructions assume jammy is the
  release, but the same can be done for lunar.

  $ lxc launch ubuntu-daily:jammy jammy
  $ lxc exec jammy bash

  2. Once in the container, modify the default /etc/netplan/10-lxc.yaml
  so that eth0 is configured with "optional: true":

  $ vi /etc/netplan/50-cloud-init.yaml # Use whatever editor you like
  $ cat /etc/netplan/50-cloud-init.yaml
  network:
    version: 2
    ethernets:
      eth0:
        dhcp4: true
        dhcp-identifier: mac
        optional: true

  3. Re-generate and apply the netplan configuration.

  $ netplan generate
  $ netplan apply

  4. Manually run systemd-networkd-wait-online, and observe that all
  links are ignored, and the command times out:

  $ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
  Found link lo(1)
  Found link eth0(19)
  lo: link is ignored
  eth0: link is ignored
  Timeout occurred while waiting for network connectivity.

  [Where problems could occur]

  This patch partially re-instates a patch remove in bug 1982218.
  However, instead of exiting if all links are unmanaged, we exit if all
  links are ignored in manager_configured(). If the patch was wrong, we
  may re-introduce bug 1982218, so as part of this SRU verification,
  that bug should be tested too. Any other regressions would also be
  related to systemd-networkd-wait-online behavior.

  [Original Description]

  On Ubuntu 22.04 desktop system using network-manager and upgrading to
  systemd 249.11-0ubuntu3.10, wait-online now times out which prevents
  logins (GDM, ssh, console) until it does time out. This seems to be
  introduced by the change for
  https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982218.

  https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982218/comments/21
  also mentioned the problem on Lunar.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2036358/+subscriptions