[Bug 1789627] Re: systemd-resolved of systemd 239 is failing in cosmic containers

Wed Aug 29 11:27:22 UTC 2018

Some more back and forth,

Interim summary:
x86-18.10->Container (liblxc1 from git): ok
x86-18.10->Container (liblxc1 from deb): ok
ppc-18.10->Container (lxc from deb) : broken
x86-18.04->Container (lxc from snap): broken
x86->KVM-18.10->Container (lxc from deb): ok
x86->KVM-18.10->Container (lxc from snap): ok

I do not recognize the pattern yet :-/

But the PPC based test is on diamond which many of you have access to,
please give that a try there.

This blocks all sort of work, so I'd appreciate if we could find what the root cause is.
For the potential that more than just me are affected to have non-usable containers this is crit I think - we can rate it down once we know why it only affects some systems.

** Changed in: systemd (Ubuntu)
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1789627

Title:
  systemd-resolved of systemd 239 is failing in cosmic containers

Status in systemd package in Ubuntu:
  New

Bug description:
  Hi,
  a few hours ago I realized that some of my containers have no working dns resolution anymore.
  Usually I'd think I broke something in my host network, but I was suspicious s it hit me on my laptop and on a server at about the same time.

  After a while I found that in those containers I have:
  systemd-resolve --status
  Failed to get global data: Failed to activate service 'org.freedesktop.resolve1': timed out (service_start_timeout=25000ms)

  Later I found two more things leading me to some assumptions:
  1. I had no resolv.conf so the service seems to have issues
  root at c:~# ll /etc/resolv.conf 
  lrwxrwxrwx 1 root root 39 Aug 28 22:18 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
  root at c:~# ll /run/systemd/resolve/stub-resolv.conf
  ls: cannot access '/run/systemd/resolve/stub-resolv.conf': No such file or directory

  2. I realized this only affects cosmic container
  Bionic container on the same machine is ok (so Host network should be ok I think).
  I didn't realize at first as other cosmic's were ok, but those were the containers not updated yet and tonight there was a publish of https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu4

  Knowing that I checked logs and found:
  Aug 29 10:23:25 c systemd[158]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[158]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: Dependency failed for Wait for Network to be Configured.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed with result 'dependency'.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[161]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[1]: cloud-init.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[161]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 2.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[165]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[165]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 3.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[168]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[168]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[171]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[171]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Start request repeated too quickly.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
  Aug 29 10:23:25 c systemd[174]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[174]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 3.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
  Aug 29 10:23:25 c systemd[183]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[183]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 4.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
  Aug 29 10:23:25 c systemd[186]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[186]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Start request repeated too quickly.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: Reached target Host and Network Name Lookups.
  Aug 29 10:23:25 c systemd[1]: Reached target Network.

  # systemctl status systemd-resolved
  ● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2018-08-29 10:39:04 UTC; 10min ago
       Docs: man:systemd-resolved.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/resolved
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
    Process: 328 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=217/USER)
   Main PID: 328 (code=exited, status=217/USER)

  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
  Aug 29 10:39:04 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Start request repeated too quickly.
  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:39:04 c systemd[1]: Failed to start Network Name Resolution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1789627/+subscriptions