[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem

Louis Bouchard louis.bouchard at canonical.com
Tue Mar 14 13:25:40 UTC 2017 

Here is a recap of my work and current status :

#1) when a system has a separated /var, the unit will hang since it is looking
for /var/run to be present and it has been unmounted.

#2) when using Unattended-Upgrade::InstallOnShutdown "true"; the upgrade never
completes as the query to the online archive fails since the network is no
longer available.

#3) it is impossible to enable the unattended-upgrades.service unit. Here
is an example :

> $ systemctl status unattended-upgrades.service 
> ● unattended-upgrades.service - Unattended Upgrades Shutdown
>    Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
>    Active: inactive (dead)
>      Docs: man:unattended-upgrade(8)

According to the doc[2], during shutdown, even if Before= or After= is used, the
unit being started will only start after the Shutdown of its dependencies. Prior
to the recent addition of local-fs.target network.target, the unit ran quickly
enough for the /var to be still mounted. But even without these dependencies,
InstallOnShutdown would fail with the following :

> 2017-03-10 13:40:42,803 INFO Starting unattended upgrades script
> 2017-03-10 13:40:42,803 INFO Allowed origins are: ['o=Ubuntu,a=zesty', 'o=Ubuntu,a=zesty-security']
> 2017-03-10 13:41:40,554 ERROR An error occurred: 'Cannot initiate the connection to 192.168.200.3:8000 (192.168.200.3). - connect (101: Network is unreachable)'
> 2017-03-10 13:41:40,555 ERROR The URI 'http://fr.archive.ubuntu.com/ubuntu/pool/main/i/init-system-helpers/init-system-helpers_1.47_all.deb' failed to download, aborting

When trying to switch the unit to an ExecStop=, we find that the Stop never
runs. This is caused by the fact that the unit is disabled (#3). Trying to
enable the unit leads to :

> # systemctl enable unattended-upgrades
> Synchronizing state of unattended-upgrades.service with SysV service script with /lib/systemd/systemd-sysv-install.
> Executing: /lib/systemd/systemd-sysv-install enable unattended-upgrades
> update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting.

Adding runlevels 2 3 4 5 fixes this then the unit can be enabled. So we get 
to a unit that looks like this :

> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> After=network.target local-fs.target
> RequiresMountsFor=/var/run /var/log
> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
> TimeoutStopSec=900
>
> [Install]
> WantedBy=multi-user.target

Before= is replaced by After= as, during shutdown otherwise the unit does not
run. RequiresMountsFor= is added as both /var/log and /var/run are needed in
order to run correctly.

RemainAfterExit=yes is added so the unit appears as started. There is no longer
a requirement to have an ExecStart present.

WantedBy is switched to multi-user.target as on the way up, we do nothing and we
are no longer depending on anything related to shutdown.

Now this only works IF /var is a separate FS. The reason for that is the
presence of DefaultDependencies=no. I don't think that it is required but was
there in the initial unit. Removing it in the final unit fixes the only
remaining issue. The unit is now :

> [Unit]
> Description=Unattended Upgrades Shutdown
> After=network.target local-fs.target
> RequiresMountsFor=/var/run /var/log
> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
> TimeoutStopSec=900
>
> [Install]
> WantedBy=multi-user.target

This works correctly and has been tested on :
- Xenial with and without /var as a separate FS
- Zesty with and without /var as a separate FS

InstallOnShutdown now also works as advertized.

I am now getting confirmation on that change since it is a rather
sensible modification.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1654600

Title:
  unattended-upgrade-shutdown hangs when /var is a separate filesystem

Status in unattended-upgrades package in Ubuntu:
  In Progress
Status in unattended-upgrades source package in Xenial:
  Triaged
Status in unattended-upgrades source package in Yakkety:
  Triaged
Status in unattended-upgrades package in Debian:
  New

Bug description:
  [SRU justification]
  This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system

  [Impact]
  System can hang up to 10 minutes if not fixed.

  [Fix]
  Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced.

  [Test Case]
  In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes

  [Regression]
  None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously.

  [Original description of the problem]
  The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
  process during shutdown. This unit file is running together with all filesystem
  unmount services.

  The unattended-upgrades service checks if the lockfile for unattended-upgrade
  (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
  and the service will wait until it finishes (and therefore automatically wait at
  shutdown).

  However, if /var is a separate filesystem, it will get unmounted even though /var/run
  is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
  The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
  tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
  exit and the system will continue shutdown.

  The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
  where it tries to lock itself:

      while True:
          res = apt_pkg.get_lock(options.lock_file)
          logging.debug("get_lock returned %i" % res)
          # exit here if there is no lock
          if res > 0:
              logging.debug("lock not taken")
              break
          lock_was_taken = True

  The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
  File descriptors are just C file descriptors, so they are always positive integers.
  The code should check the result to be negative, not positive. I have attached a patch
  to reverse the logic.

  Additional information:

  1)
  Description:	Ubuntu 16.04.1 LTS
  Release:	16.04

  2)
  unattended-upgrades:
    Installed: 0.90ubuntu0.3
    Candidate: 0.90ubuntu0.3
    Version table:
   *** 0.90ubuntu0.3 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
          100 /var/lib/dpkg/status
       0.90 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
  3)
  Fast reboot
  4)
  Very slow reboot (after a 10 minutes timeout)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions

More information about the foundations-bugs mailing list