[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem

[SergeiFranco]

Hello,

Yes, I am fully aware that my /var/run->/run replacement in that script
is a hack.

To be clear I am not testing in Zesty but Ubuntu 16.04.2 LTS Xenial.

The setup is following:
/dev/sda1 on / type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc on /var type xfs (rw,relatime,attr2,inode64,noquota)

Every single machine we (my and my colleague) tested with the updated
unit still hangs.

As var as I can tell they never reboot.

I have posted a question regarding this issue to systemd mailing list,
and one of the responses was the following:

"""
> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> Before=shutdown.target reboot.target halt.target network.target
> local-fs.target

Well, you order if before local-fs.target, which means it is stopped
after local-fs.target. You get exactly what you ask for. If this is
wrong, fix unit definition, but we do not really know what is
appropriate for this service.

> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
> TimeoutStopSec=900
>
> [Install]
> WantedBy=shutdown.target
>
> It appears no one who is involved in this fix understands systemd (which is
> very worrying!).
>
> How does one would fix this unit so it is ran before the file systems get
> unmounted?
>

Order it

After=local-fs.target
"""
"Obviously" according to systemd mailing list if I want to things to run Before I need to use After!

So I believe the confusion is with what exactly Before and After mean in
systemd units.

Today I will be testing various combinations with After/Before to see if
I can make sense of it.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1654600

Title:
  unattended-upgrade-shutdown hangs when /var is a separate filesystem

Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in unattended-upgrades source package in Xenial:
  Fix Committed
Status in unattended-upgrades source package in Yakkety:
  Fix Committed
Status in unattended-upgrades package in Debian:
  New

Bug description:
  [SRU justification]
  This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system

  [Impact]
  System can hang up to 10 minutes if not fixed.

  [Fix]
  Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced.

  [Test Case]
  In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes

  [Regression]
  None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously.

  [Original description of the problem]
  The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
  process during shutdown. This unit file is running together with all filesystem
  unmount services.

  The unattended-upgrades service checks if the lockfile for unattended-upgrade
  (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
  and the service will wait until it finishes (and therefore automatically wait at
  shutdown).

  However, if /var is a separate filesystem, it will get unmounted even though /var/run
  is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
  The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
  tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
  exit and the system will continue shutdown.

  The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
  where it tries to lock itself:

      while True:
          res = apt_pkg.get_lock(options.lock_file)
          logging.debug("get_lock returned %i" % res)
          # exit here if there is no lock
          if res > 0:
              logging.debug("lock not taken")
              break
          lock_was_taken = True

  The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
  File descriptors are just C file descriptors, so they are always positive integers.
  The code should check the result to be negative, not positive. I have attached a patch
  to reverse the logic.

  Additional information:

  1)
  Description:	Ubuntu 16.04.1 LTS
  Release:	16.04

  2)
  unattended-upgrades:
    Installed: 0.90ubuntu0.3
    Candidate: 0.90ubuntu0.3
    Version table:
   *** 0.90ubuntu0.3 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
          100 /var/lib/dpkg/status
       0.90 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
  3)
  Fast reboot
  4)
  Very slow reboot (after a 10 minutes timeout)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions