[Bug 1803749] Re: apt.Cache.clear() may set apt.Package.is_auto_removable to False

Balint Reczey balint.reczey at canonical.com
Mon Nov 26 12:20:24 UTC 2018 

** Description changed:

+ [Impact]
+ 
+  * Originally autoremovable packages can be removed as newly autoremovable ones by unattended-upgrades
+  * This can surprise users potentially removing packages which are needed for the system's operation.
+ 
+ [Test Case]
+ 
+  * WIP, see Original Bug Test
+ 
+ [Regression Potential]
+ 
+  * Unattended-upgrades may use more CPU-time for operation but I did not observe a significant increase. Autopkgtest measures u-u's performance thus if this regression occurs, we can observe it easily.
+  * Due to the code changes u-u may still remove already autoremovable packages or fail to remove newly autoremovable ones in default configuration, but since the code became simpler with the change by eliminating an optimization this regression is unlikely to take place.
+ 
+ [Original Bug Text]
+ 
  While triaging LP: #1803587 I found that originally autoremovable
  packages were handled as newly autoremovable ones by unattended-upgrades
  due to not finding all of them at the beginning of u-u's run.
  
  The root cause seems to be cache.clear() resetting pkg.is_auto_removable
  to False under some circumstances I haven't fully narrowed down.
  
- The reproduction is as follows:
- Set up a Bionic (or later) system with autoremovable packages, packages upgradable from -security and blacklist at least one upgradable package and apply the following patch to u-u:
- 
+ Set up a Bionic (or later) system with autoremovable packages, packages
+ upgradable from -security and blacklist at least one upgradable package
+ and apply the following patch to u-u:
  
  root at bb-1803587:~# diff -Naur /usr/bin/unattended-upgrade.orig /usr/bin/unattended-upgrade
  --- /usr/bin/unattended-upgrade.orig	2018-11-16 16:17:58.522583254 +0000
  +++ /usr/bin/unattended-upgrade	2018-11-16 16:36:12.226675870 +0000
  @@ -948,7 +948,9 @@
-  def rewind_cache(cache, pkgs_to_upgrade):
-      # type: (apt.Cache, List[apt.Package]) -> None
-      """ set the cache back to the state with packages_to_upgrade """
+  def rewind_cache(cache, pkgs_to_upgrade):
+      # type: (apt.Cache, List[apt.Package]) -> None
+      """ set the cache back to the state with packages_to_upgrade """
  +    print([pkg.name  for pkg in cache if pkg.is_auto_removable])
-      cache.clear()
+      cache.clear()
  +    print([pkg.name for pkg in cache if pkg.is_auto_removable])
-      for pkg2 in pkgs_to_upgrade:
-          pkg2.mark_install(from_user=not pkg2.is_auto_installed)
-      if cache.broken_count > 0:
+      for pkg2 in pkgs_to_upgrade:
+          pkg2.mark_install(from_user=not pkg2.is_auto_installed)
+      if cache.broken_count > 0:
  
+ Run u-u to observe cache.clear() resetting the list of autoremovable
+ packages:
  
- Run u-u to observe cache.clear() resetting the list of autoremovable packages:
- 
-  ~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
+  ~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
  Initial blacklisted packages: systemd
- Initial whitelisted packages: 
+ Initial whitelisted packages:
  Starting unattended upgrades script
  Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
  Using (^linux-image|^linux-headers|^linux-image-extra|^linux-modules|^linux-modules-extra|^linux-signed-image|^kfreebsd-image|^kfreebsd-headers|^gnumach-image|^.*-modules|^.*-kernel|^linux-backports-modules-.*|^linux-modules-.*|^linux-tools|^linux-cloud-tools) regexp to find kernel packages
  Using (^linux-image.*4.15.0-38-generic|^linux-headers.*4.15.0-38-generic|^linux-image-extra.*4.15.0-38-generic|^linux-modules.*4.15.0-38-generic|^linux-modules-extra.*4.15.0-38-generic|^linux-signed-image.*4.15.0-38-generic|^kfreebsd-image.*4.15.0-38-generic|^kfreebsd-headers.*4.15.0-38-generic|^gnumach-image.*4.15.0-38-generic|4.15.0-38-generic.*-modules|4.15.0-38-generic.*-kernel|^linux-backports-modules-.*.*4.15.0-38-generic|^linux-modules-.*.*4.15.0-38-generic|^linux-tools.*4.15.0-38-generic|^linux-cloud-tools.*4.15.0-38-generic) regexp to find running kernel packages
  Checking: apport ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>])
  adjusting candidate version: apport=2.20.9-0ubuntu7.1
  ...
  Checking: libnss-systemd ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
  skipping blacklisted package systemd
  pkg systemd package has been blacklisted
  sanity check failed
  ['libfreetype6']
  []
  ...

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1803749

Title:
  apt.Cache.clear() may set apt.Package.is_auto_removable to False

Status in unattended-upgrades package in Ubuntu:
  In Progress

Bug description:
  [Impact]

   * Originally autoremovable packages can be removed as newly autoremovable ones by unattended-upgrades
   * This can surprise users potentially removing packages which are needed for the system's operation.

  [Test Case]

   * WIP, see Original Bug Test

  [Regression Potential]

   * Unattended-upgrades may use more CPU-time for operation but I did not observe a significant increase. Autopkgtest measures u-u's performance thus if this regression occurs, we can observe it easily.
   * Due to the code changes u-u may still remove already autoremovable packages or fail to remove newly autoremovable ones in default configuration, but since the code became simpler with the change by eliminating an optimization this regression is unlikely to take place.

  [Original Bug Text]

  While triaging LP: #1803587 I found that originally autoremovable
  packages were handled as newly autoremovable ones by unattended-
  upgrades due to not finding all of them at the beginning of u-u's run.

  The root cause seems to be cache.clear() resetting
  pkg.is_auto_removable to False under some circumstances I haven't
  fully narrowed down.

  Set up a Bionic (or later) system with autoremovable packages,
  packages upgradable from -security and blacklist at least one
  upgradable package and apply the following patch to u-u:

  root at bb-1803587:~# diff -Naur /usr/bin/unattended-upgrade.orig /usr/bin/unattended-upgrade
  --- /usr/bin/unattended-upgrade.orig	2018-11-16 16:17:58.522583254 +0000
  +++ /usr/bin/unattended-upgrade	2018-11-16 16:36:12.226675870 +0000
  @@ -948,7 +948,9 @@
   def rewind_cache(cache, pkgs_to_upgrade):
       # type: (apt.Cache, List[apt.Package]) -> None
       """ set the cache back to the state with packages_to_upgrade """
  +    print([pkg.name  for pkg in cache if pkg.is_auto_removable])
       cache.clear()
  +    print([pkg.name for pkg in cache if pkg.is_auto_removable])
       for pkg2 in pkgs_to_upgrade:
           pkg2.mark_install(from_user=not pkg2.is_auto_installed)
       if cache.broken_count > 0:

  Run u-u to observe cache.clear() resetting the list of autoremovable
  packages:

   ~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
  Initial blacklisted packages: systemd
  Initial whitelisted packages:
  Starting unattended upgrades script
  Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
  Using (^linux-image|^linux-headers|^linux-image-extra|^linux-modules|^linux-modules-extra|^linux-signed-image|^kfreebsd-image|^kfreebsd-headers|^gnumach-image|^.*-modules|^.*-kernel|^linux-backports-modules-.*|^linux-modules-.*|^linux-tools|^linux-cloud-tools) regexp to find kernel packages
  Using (^linux-image.*4.15.0-38-generic|^linux-headers.*4.15.0-38-generic|^linux-image-extra.*4.15.0-38-generic|^linux-modules.*4.15.0-38-generic|^linux-modules-extra.*4.15.0-38-generic|^linux-signed-image.*4.15.0-38-generic|^kfreebsd-image.*4.15.0-38-generic|^kfreebsd-headers.*4.15.0-38-generic|^gnumach-image.*4.15.0-38-generic|4.15.0-38-generic.*-modules|4.15.0-38-generic.*-kernel|^linux-backports-modules-.*.*4.15.0-38-generic|^linux-modules-.*.*4.15.0-38-generic|^linux-tools.*4.15.0-38-generic|^linux-cloud-tools.*4.15.0-38-generic) regexp to find running kernel packages
  Checking: apport ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>])
  adjusting candidate version: apport=2.20.9-0ubuntu7.1
  ...
  Checking: libnss-systemd ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
  skipping blacklisted package systemd
  pkg systemd package has been blacklisted
  sanity check failed
  ['libfreetype6']
  []
  ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1803749/+subscriptions

More information about the foundations-bugs mailing list