[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem

Fri Jan 27 00:41:18 UTC 2017

You're right, I thought it was the lockfile for the shutdown helper
only.

Somewhere else in this thread I already mentioned that a dependency on
the local_fs and remote_fs targets fixes the systemd order:

Before=remote-fs.target local-fs.target

That makes sure the service is run before all file systems are
unmounted. If you add the shutdown.target to that rule, systemd somehow
thinks it should be run just before shutdown which is obviously too
late. I'm not sure that is a systemd bug or a systemd feature, but
either way it doesn't do what we want.

Regards,

Erik

-- 
Erik Mouw

> Op 26 jan. 2017 om 18:00 heeft Louis Bouchard <louis.bouchard at canonical.com> het volgende geschreven:
> 
> Hello,
> 
> This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to
> acquire a lockfile (/var/run/unattended-upgrades.lock) which is only
> possible if no unattended-upgrade is already running. If it can acquire
> the lock, it means that NO unattended-upgrade is running, hence shutdown
> may proceed.
> 
> while True:                                                                                                    
>        res = apt_pkg.get_lock(options.lock_file)                                                                  
>        logging.debug("get_lock returned %i" % res)                                                                
>        # exit here if there is no lock                                                                            
>        if res > 0:                                                                                                
>            logging.debug("lock not taken")                                                                        
>            break                                                                                                  
>        lock_was_taken = True
> 
> The problem is that apt_pkg.get_lock(options.lock_file) will also return
> -1 if the path to the lock file doesn't exist, which is the case when
> /var is unmounted, hence /var/run is no longer present (/var/run is a
> symlink to /run).
> 
> I'm working on identifying the proper systemd order to make sure that
> /var/run is still accessible.
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1654600
> 
> Title:
>  unattended-upgrade-shutdown hangs when /var is a separate filesystem
> 
> Status in unattended-upgrades package in Ubuntu:
>  Confirmed
> Status in unattended-upgrades source package in Xenial:
>  Confirmed
> Status in unattended-upgrades source package in Yakkety:
>  New
> 
> Bug description:
>  The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
>  process during shutdown. This unit file is running together with all filesystem
>  unmount services.
> 
>  The unattended-upgrades service checks if the lockfile for unattended-upgrade
>  (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
>  and the service will wait until it finishes (and therefore automatically wait at
>  shutdown).
> 
>  However, if /var is a separate filesystem, it will get unmounted even though /var/run
>  is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
>  The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
>  tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
>  exit and the system will continue shutdown.
> 
>  The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
>  where it tries to lock itself:
> 
>      while True:
>          res = apt_pkg.get_lock(options.lock_file)
>          logging.debug("get_lock returned %i" % res)
>          # exit here if there is no lock
>          if res > 0:
>              logging.debug("lock not taken")
>              break
>          lock_was_taken = True
> 
>  The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
>  File descriptors are just C file descriptors, so they are always positive integers.
>  The code should check the result to be negative, not positive. I have attached a patch
>  to reverse the logic.
> 
>  Additional information:
> 
>  1)
>  Description:    Ubuntu 16.04.1 LTS
>  Release:    16.04
> 
>  2)
>  unattended-upgrades:
>    Installed: 0.90ubuntu0.3
>    Candidate: 0.90ubuntu0.3
>    Version table:
>   *** 0.90ubuntu0.3 500
>          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
>          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
>          100 /var/lib/dpkg/status
>       0.90 500
>          500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
>          500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
>  3)
>  Fast reboot
>  4)
>  Very slow reboot (after a 10 minutes timeout)
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1654600

Title:
  unattended-upgrade-shutdown hangs when /var is a separate filesystem

Status in unattended-upgrades package in Ubuntu:
  Confirmed
Status in unattended-upgrades source package in Xenial:
  Confirmed
Status in unattended-upgrades source package in Yakkety:
  New

Bug description:
  The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
  process during shutdown. This unit file is running together with all filesystem
  unmount services.

  The unattended-upgrades service checks if the lockfile for unattended-upgrade
  (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
  and the service will wait until it finishes (and therefore automatically wait at
  shutdown).

  However, if /var is a separate filesystem, it will get unmounted even though /var/run
  is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
  The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
  tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
  exit and the system will continue shutdown.

  The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
  where it tries to lock itself:

      while True:
          res = apt_pkg.get_lock(options.lock_file)
          logging.debug("get_lock returned %i" % res)
          # exit here if there is no lock
          if res > 0:
              logging.debug("lock not taken")
              break
          lock_was_taken = True

  The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
  File descriptors are just C file descriptors, so they are always positive integers.
  The code should check the result to be negative, not positive. I have attached a patch
  to reverse the logic.

  Additional information:

  1)
  Description:	Ubuntu 16.04.1 LTS
  Release:	16.04

  2)
  unattended-upgrades:
    Installed: 0.90ubuntu0.3
    Candidate: 0.90ubuntu0.3
    Version table:
   *** 0.90ubuntu0.3 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
          100 /var/lib/dpkg/status
       0.90 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
  3)
  Fast reboot
  4)
  Very slow reboot (after a 10 minutes timeout)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions