[Bug 1636912] Re: systemd-networkd runs too late for cloud-init.service (net)

[Martin Pitt]

@Steve:

> There's feedback in the end of that bug that cloud-init should *not*
be using Before=basic.target / Before=dbus.socket, but instead use
Before=sysinit.target.

Correct, that would avoid starting it in between sysinit.target and
basic.target when the  sockets start, and avoid the deadlock in a
simpler way.

> I thought that once resolved supported DNS, we would use that /instead
of/ the NSS module. Is there really a benefit to using both?

Both "resolve" and "dns" are NSS modules, and you can't not use NSS for
name resolution (as that's what glibc's gethostbyname() and friends
use). Maybe you meant "instead of the dns module"? This is a required
fallback for either (1) foreign architecture programs which don't have
the corresponding nss-resolve:arch installed, or (2) early boot when
D-Bus and hence resolved are not yet running.

> because Ubuntu Core 16 does contain libnss-resolve,

"contains" yes, but we don't use nss-resolve in Ubuntu 16.04 and hence
snappy. This got introduced step-wise in 16.10, isn't completly finished
yet, and IMHO not yet ready for backporting (if we ever actually want
that). If snappy's /etc/nsswitch.conf contains "resolve", this is NOT
intended.

@Ryan:

> I don't know enough to say if resolved and NSS are 100% swappable;

Again, I figure you mean s/NSS/dns/; dns does not do DNSSEC, caching and
the like, but none of this should be relevant at early boot (if you need
network that early, you are basically on your own and not guaranteed to
succeed anyway -- "dns" is by far good enough for that).

Anyway, all of this is not the primary focus for this bug -- this bug is
about the ordering of cloud-init vs. networkd, resolved is not in this
picture.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1636912

Title:
  systemd-networkd runs too late for cloud-init.service (net)

Status in cloud-init package in Ubuntu:
  Triaged
Status in systemd package in Ubuntu:
  Triaged

Bug description:
  Ubuntu Core 16 images using cloud-init fail to function when the
  DataSource is over the network (Like OpenStack) as networking is not
  yet available when cloud-init.service runs.

  cloud-init service unit deps look like this:

  [Unit]
  Description=Initial cloud-init job (metadata service crawler)
  DefaultDependencies=no
  Wants=cloud-init-local.service
  Wants=local-fs.target
  Wants=sshd-keygen.service
  Wants=sshd.service
  After=cloud-init-local.service
  After=networking.service
  Requires=networking.service
  Before=basic.target
  Before=dbus.socket
  Before=network-online.target
  Before=sshd-keygen.service
  Before=sshd.service
  Before=systemd-user-sessions.service
  Conflicts=shutdown.target

  Here's networkd unit deps:

  [Unit]
  Description=Network Service
  Documentation=man:systemd-networkd.service(8)
  ConditionCapability=CAP_NET_ADMIN
  DefaultDependencies=no
  # dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
  # dropped once tuntap is moved to netlink
  After=systemd-udevd.service dbus.service network-pre.target systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  # On kdbus systems we pull in the busname explicitly, because it
  # carries policy that allows the daemon to acquire its name.
  Wants=org.freedesktop.network1.busname
  After=org.freedesktop.network1.busname

  
  And a critical-chain output:

  root at snap-test7:~# systemd-analyze critical-chain systemd-networkd
  Failed to get ID: Unit name systemd-networkd is not valid.
  The time after the unit is active or started is printed after the "@" character.
  The time the unit takes to start is printed after the "+" character.

  root at snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
  The time after the unit is active or started is printed after the "@" character.
  The time the unit takes to start is printed after the "+" character.

  systemd-networkd.service +440ms
  └─dbus.service @11.461s
    └─basic.target @11.403s
      └─sockets.target @11.401s
        └─dbus.socket @11.398s
          └─cloud-init.service @10.127s +1.266s
            └─networking.service @9.305s +799ms
              └─network-pre.target @9.295s
                └─cloud-init-local.service @3.822s +5.469s
                  └─local-fs.target @3.813s
                    └─run-cgmanager-fs.mount @12.687s
                      └─local-fs-pre.target @1.393s
                        └─systemd-tmpfiles-setup-dev.service @1.116s +195ms
                          └─kmod-static-nodes.service @887ms +193ms
                            └─system.slice @783ms
                              └─-.slice @721ms

  
  cloud-init would need networkd to run at or before 'networking.service' so it can raise networking to then find and use network-based datasources.

  # grep systemd /usr/share/snappy/dpkg.list 
  ii  libnss-resolve:amd64          229-4ubuntu11                                            amd64        nss module to resolve names via systemd-resolved
  ii  libpam-systemd:amd64          229-4ubuntu11                                            amd64        system and service manager - PAM module
  ii  libsystemd0:amd64             229-4ubuntu11                                            amd64        systemd utility library
  ii  systemd                       229-4ubuntu11                                            amd64        system and service manager
  ii  systemd-sysv                  229-4ubuntu11                                            amd64        system and service manager - SysV links

  # grep cloud-init /usr/share/snappy/dpkg.list 
  ii  cloud-init                    0.7.8-201610260005-gf7a5756-0ubuntu1~trunk~ubuntu16.04.1 all          Init scripts for cloud instances

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1636912/+subscriptions