[Bug 1725685] [NEW] Systemd long reboot on OpenVZ

[roma1390]

Public bug reported:

Ubuntu 17.04
systemd: 232-21ubuntu2

short: Systemd on reboot waits extra 90sec for no reason

Other references:
  https://forum.openvz.org/index.php?t=tree&goto=52177&S=0012de5a5151a907e13af2b5267cf59d

Env: 
  OpenVZ 2.6.32-openvz-042stab120.19-amd64
  Ubuntu 17.04
  
Packages:
  # dpkg -l | grep systemd
  ii  libsystemd0:amd64                232-21ubuntu2                     amd64        systemd utility library
  ii  systemd                          232-21ubuntu2                     amd64        system and service manager
  ii  systemd-sysv                     232-21ubuntu2                     amd64        system and service manager - SysV links


console log during "problem"
=================
Sending SIGTERM to remaining processes...^M
Sending SIGKILL to remaining processes...^M
Sending SIGKILL to PID 338 (systemd-cgroups).^M
Sending SIGKILL to PID 339 (systemd-cgroups).^M
system-shutdown succeeded.^M
Rebooting.^M
=================

The "extra" wait is occurring before line:
  system-shutdown succeeded

Problem analysis, steps:
1. PID 1 does exec to /lib/systemd/systemd-shutdown
2. src/core/shutdown.c:main calls broadcast_signal(SIGKILL, true, false);
3. somehow /lib/systemd/systemd-cgroups-agent is started by /sys/fs/cgroup/systemd/release_agent
4. src/core/killall.c:broadcast_signal calls killall(sig, pids, send_sighup);
writes message: Sending SIGKILL to PID 338 (systemd-cgroups).
adds pid 338 to pids sed

5. broadcast_signal calls wait_for_children(pids, &mask);
6. in paralel process systemd-cgroups finishes
7. we still in wait_for_children and stuck here for 90 sec

Tried workaround approaches:
1. mount /dev/null -o bind /lib/systemd/systemd-cgroups-agent
will not work, before shutdown mounts are umounted, so we lose that one
2. change /sys/fs/cgroup/systemd/release_agent to /bin/false
still caches running false process and waits 90 sec for finish of it
3. write 0 to /sys/fs/cgroup/systemd/notify_on_release from script in /lib/systemd/system-shutdown
this is to late, those scripts runs after long delay

Simple proposed workaround:

disable notify before systemd-shutdown by using wrapper script

  dpkg-divert --add /lib/systemd/systemd-shutdown
  mv /lib/systemd/systemd-shutdown /lib/systemd/systemd-shutdown.distrib
  cat > /lib/systemd/systemd-shutdown <<EOF
  #!/bin/sh
  
  echo 0 > /sys/fs/cgroup/systemd/notify_on_release 
  exec /lib/systemd/systemd-shutdown.distrib "$@"
  EOF
  chmod +x /lib/systemd/systemd-shutdown

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1725685

Title:
  Systemd long reboot on OpenVZ

Status in systemd package in Ubuntu:
  New

Bug description:
  Ubuntu 17.04
  systemd: 232-21ubuntu2

  short: Systemd on reboot waits extra 90sec for no reason

  Other references:
    https://forum.openvz.org/index.php?t=tree&goto=52177&S=0012de5a5151a907e13af2b5267cf59d

  Env: 
    OpenVZ 2.6.32-openvz-042stab120.19-amd64
    Ubuntu 17.04
    
  Packages:
    # dpkg -l | grep systemd
    ii  libsystemd0:amd64                232-21ubuntu2                     amd64        systemd utility library
    ii  systemd                          232-21ubuntu2                     amd64        system and service manager
    ii  systemd-sysv                     232-21ubuntu2                     amd64        system and service manager - SysV links

  
  console log during "problem"
  =================
  Sending SIGTERM to remaining processes...^M
  Sending SIGKILL to remaining processes...^M
  Sending SIGKILL to PID 338 (systemd-cgroups).^M
  Sending SIGKILL to PID 339 (systemd-cgroups).^M
  system-shutdown succeeded.^M
  Rebooting.^M
  =================

  The "extra" wait is occurring before line:
    system-shutdown succeeded

  Problem analysis, steps:
  1. PID 1 does exec to /lib/systemd/systemd-shutdown
  2. src/core/shutdown.c:main calls broadcast_signal(SIGKILL, true, false);
  3. somehow /lib/systemd/systemd-cgroups-agent is started by /sys/fs/cgroup/systemd/release_agent
  4. src/core/killall.c:broadcast_signal calls killall(sig, pids, send_sighup);
  writes message: Sending SIGKILL to PID 338 (systemd-cgroups).
  adds pid 338 to pids sed

  5. broadcast_signal calls wait_for_children(pids, &mask);
  6. in paralel process systemd-cgroups finishes
  7. we still in wait_for_children and stuck here for 90 sec

  Tried workaround approaches:
  1. mount /dev/null -o bind /lib/systemd/systemd-cgroups-agent
  will not work, before shutdown mounts are umounted, so we lose that one
  2. change /sys/fs/cgroup/systemd/release_agent to /bin/false
  still caches running false process and waits 90 sec for finish of it
  3. write 0 to /sys/fs/cgroup/systemd/notify_on_release from script in /lib/systemd/system-shutdown
  this is to late, those scripts runs after long delay

  Simple proposed workaround:

  disable notify before systemd-shutdown by using wrapper script

    dpkg-divert --add /lib/systemd/systemd-shutdown
    mv /lib/systemd/systemd-shutdown /lib/systemd/systemd-shutdown.distrib
    cat > /lib/systemd/systemd-shutdown <<EOF
    #!/bin/sh
    
    echo 0 > /sys/fs/cgroup/systemd/notify_on_release 
    exec /lib/systemd/systemd-shutdown.distrib "$@"
    EOF
    chmod +x /lib/systemd/systemd-shutdown

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1725685/+subscriptions