[Bug 1803749] Re: apt.Cache.clear() may set apt.Package.is_auto_removable to False

Fri Feb 8 14:49:52 UTC 2019

Tested with 1.1ubuntu1.18.04.7~16.04.1:

$ sudo chroot x-test-1803749/
root at MacBookAir-test:/# apt install unattended-upgrades zsh
Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
  iso-codes python-apt-common python3-apt xz-utils zsh-common
Suggested packages:
  isoquery python3-apt-dbg python-apt-doc bsd-mailx mail-transport-agent zsh-doc
The following NEW packages will be installed:
  iso-codes python-apt-common python3-apt unattended-upgrades xz-utils zsh zsh-common
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 6354 kB of archives.
After this operation, 32.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 zsh-common all 5.1.1-1ubuntu2 [3170 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 zsh amd64 5.1.1-1ubuntu2 [651 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 iso-codes all 3.65-1 [2268 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 python-apt-common all 1.1.0~beta1build1 [17.3 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-apt amd64 1.1.0~beta1build1 [137 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 xz-utils amd64 5.1.1alpha+20120614-2ubuntu2 [78.8 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main amd64 unattended-upgrades all 0.90 [31.6 kB]
Fetched 6354 kB in 6s (1024 kB/s)                                                                                 
Preconfiguring packages ...
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
Selecting previously unselected package zsh-common.
(Reading database ... 10444 files and directories currently installed.)
Preparing to unpack .../zsh-common_5.1.1-1ubuntu2_all.deb ...
Unpacking zsh-common (5.1.1-1ubuntu2) ...
Selecting previously unselected package zsh.
Preparing to unpack .../zsh_5.1.1-1ubuntu2_amd64.deb ...
Unpacking zsh (5.1.1-1ubuntu2) ...
Selecting previously unselected package iso-codes.
Preparing to unpack .../iso-codes_3.65-1_all.deb ...
Unpacking iso-codes (3.65-1) ...
Selecting previously unselected package python-apt-common.
Preparing to unpack .../python-apt-common_1.1.0~beta1build1_all.deb ...
Unpacking python-apt-common (1.1.0~beta1build1) ...
Selecting previously unselected package python3-apt.
Preparing to unpack .../python3-apt_1.1.0~beta1build1_amd64.deb ...
Unpacking python3-apt (1.1.0~beta1build1) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../xz-utils_5.1.1alpha+20120614-2ubuntu2_amd64.deb ...
Unpacking xz-utils (5.1.1alpha+20120614-2ubuntu2) ...
Selecting previously unselected package unattended-upgrades.
Preparing to unpack .../unattended-upgrades_0.90_all.deb ...
Unpacking unattended-upgrades (0.90) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu4) ...
Setting up zsh-common (5.1.1-1ubuntu2) ...
Setting up zsh (5.1.1-1ubuntu2) ...
update-alternatives: using /bin/zsh5 to provide /bin/zsh (zsh) in auto mode
update-alternatives: using /bin/zsh5 to provide /bin/rzsh (rzsh) in auto mode
Setting up iso-codes (3.65-1) ...
Setting up python-apt-common (1.1.0~beta1build1) ...
Setting up python3-apt (1.1.0~beta1build1) ...
Setting up xz-utils (5.1.1alpha+20120614-2ubuntu2) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up unattended-upgrades (0.90) ...

Creating config file /etc/apt/apt.conf.d/20auto-upgrades with new
version

Creating config file /etc/apt/apt.conf.d/50unattended-upgrades with new version
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu4) ...
root at MacBookAir-test:/# apt-mark auto zsh
zsh set to automatically installed.
root at MacBookAir-test:/# echo "Unattended-Upgrade::Package-Blacklist {"libs";}" > /etc/apt/apt.conf.d/51unattended-upgrades
root at MacBookAir-test:/# echo "deb http://archive.ubuntu.com/ubuntu xenial-security main" >> /etc/apt/sources.list
root at MacBookAir-test:/#  echo "deb http://archive.ubuntu.com/ubuntu xenial-updates main" >> /etc/apt/sources.list
root at MacBookAir-test:/# apt update -qq
105 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at MacBookAir-test:/#  unattended-upgrades --dry-run --verbose --debug
Initial blacklisted packages: libs
Initial whitelisted packages: 
Starting unattended upgrades script
...
adjusting candidate version: 'libkmod2=22-1ubuntu5'
adjusting candidate version: 'libstdc++6=5.4.0-6ubuntu1~16.04.10'
adjusting candidate version: 'sudo=1.8.16-0ubuntu1.4'
marking zsh-common for remove
marking zsh for remove
Packages that are auto removed: 'zsh zsh-common'
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
/usr/bin/dpkg --status-fd 197 --force-depends --force-remove-essential --remove zsh:amd64 zsh-common:all 
/usr/bin/dpkg --status-fd 199 --configure --pending 
Packages were successfully auto-removed
InstCount=0 DelCount=2 BrokenCount=0
root at MacBookAir-test:/# echo "deb http://archive.ubuntu.com/ubuntu xenial-proposed main" >> /etc/apt/sources.list
root at MacBookAir-test:/# apt update -qq

105 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at MacBookAir-test:/# 
root at MacBookAir-test:/# apt install unattended-upgrades
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  dbus gir1.2-glib-2.0 libcap-ng0 libdbus-1-3 libdbus-glib-1-2 libgirepository-1.0-1 libglib2.0-0 libglib2.0-data
  libicu55 libxml2 powermgmt-base python3-dbus python3-gi sgml-base shared-mime-info xdg-user-dirs xml-core
Suggested packages:
  dbus-user-session | dbus-x11 python-dbus-doc python3-dbus-dbg sgml-base-doc bsd-mailx default-mta
  | mail-transport-agent needrestart debhelper
The following NEW packages will be installed:
  dbus gir1.2-glib-2.0 libcap-ng0 libdbus-1-3 libdbus-glib-1-2 libgirepository-1.0-1 libglib2.0-0 libglib2.0-data
  libicu55 libxml2 powermgmt-base python3-dbus python3-gi sgml-base shared-mime-info xdg-user-dirs xml-core
The following packages will be upgraded:
  unattended-upgrades
1 upgraded, 17 newly installed, 0 to remove and 104 not upgraded.
Need to get 11.0 MB of archives.
After this operation, 43.5 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 libglib2.0-0 amd64 2.48.2-0ubuntu4.1 [1120 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 sgml-base all 1.26+nmu4ubuntu1 [12.5 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 libcap-ng0 amd64 0.7.7-1 [10.9 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libdbus-1-3 amd64 1.10.6-1ubuntu3.3 [161 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 dbus amd64 1.10.6-1ubuntu3.3 [142 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 libgirepository-1.0-1 amd64 1.46.0-3ubuntu1 [88.3 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main amd64 gir1.2-glib-2.0 amd64 1.46.0-3ubuntu1 [127 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 libdbus-glib-1-2 amd64 0.106-1 [67.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 libglib2.0-data all 2.48.2-0ubuntu4.1 [132 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 libicu55 amd64 55.1-7ubuntu0.4 [7646 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 libxml2 amd64 2.9.3+dfsg1-1ubuntu0.6 [697 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/main amd64 powermgmt-base all 1.31+nmu1 [7178 B]
Get:13 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-dbus amd64 1.2.0-3 [83.1 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-gi amd64 3.20.0-0ubuntu1 [153 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 shared-mime-info amd64 1.5-2ubuntu0.2 [405 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 xdg-user-dirs amd64 0.15-2ubuntu6.16.04.1 [61.8 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial/main amd64 xml-core all 0.13+nmu2 [23.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 unattended-upgrades all 1.1ubuntu1.18.04.7~16.04.1 [40.2 kB]
Fetched 11.0 MB in 6s (1827 kB/s)                                                                                 
...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
root at MacBookAir-test:/#  unattended-upgrades --dry-run --verbose --debug
Initial blacklisted packages: libs
...
applying set ['python3.5', 'python3.5-minimal', 'libpython3.5-stdlib', 'libpython3.5-minimal']
apt_pkg.pkgsystem_unlock() failed due to not holding the lock but trying to continue
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
/usr/bin/dpkg --status-fd 139 --unpack --auto-deconfigure /var/cache/apt/archives/python3.5_3.5.2-2ubuntu0~16.04.5_amd64.deb /var/cache/apt/archives/libpython3.5-stdlib_3.5.2-2ubuntu0~16.04.5_amd64.deb /var/cache/apt/archives/python3.5-minimal_3.5.2-2ubuntu0~16.04.5_amd64.deb /var/cache/apt/archives/libpython3.5-minimal_3.5.2-2ubuntu0~16.04.5_amd64.deb 
/usr/bin/dpkg --status-fd 141 --configure libpython3.5-minimal:amd64 python3.5-minimal:amd64 libpython3.5-stdlib:amd64 python3.5:amd64 
/usr/bin/dpkg --status-fd 143 --configure --pending 
left to upgrade set()
All upgrades installed
InstCount=0 DelCount=0 BrokenCount=0
root at MacBookAir-test:/# 

** Tags removed: verification-needed verification-needed-xenial
** Tags added: verification-done verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1803749

Title:
  apt.Cache.clear() may set apt.Package.is_auto_removable to False

Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in unattended-upgrades source package in Xenial:
  Fix Committed
Status in unattended-upgrades source package in Bionic:
  Fix Released
Status in unattended-upgrades source package in Cosmic:
  Fix Released

Bug description:
  [Impact]

   * Originally autoremovable packages can be removed as newly autoremovable ones by unattended-upgrades
   * This can surprise users potentially removing packages which are needed for the system's operation.

  [Test Case]

  * The buggy u-u version will remove the previously autoremovable zsh
  at the end, the fixed one does not.

  $ sudo debootstrap bionic uu-test-bionic-1803749
  ...
  $ sudo chroot uu-test-bionic-1803749
  # apt install unattended-upgrades zsh
  # apt-mark auto zsh
  ...
  # echo "Unattended-Upgrade::Package-Blacklist {"libs";}"  > /etc/apt/apt.conf.d/51unattended-upgrades
  # echo "deb http://archive.ubuntu.com/ubuntu bionic-security main" >> /etc/apt/sources.list
  # echo "deb http://archive.ubuntu.com/ubuntu bionic-updates main" >> /etc/apt/sources.list
  # apt update -qq
  # unattended-upgrades --dry-run --verbose --debug
  ...
  All upgrades installed
  marking zsh for removal
  marking zsh-common for removal
  Packages that were successfully auto-removed:
  Packages that are kept back:
  InstCount=0 DelCount=0 BrokenCount=0
  #

  [Regression Potential]

   * Unattended-upgrades may use more CPU-time for operation but I did not observe a significant increase. Autopkgtest measures u-u's performance thus if this regression occurs, we can observe it easily.
   * Due to the code changes u-u may still remove already autoremovable packages or fail to remove newly autoremovable ones in default configuration, but since the code became simpler with the change by eliminating an optimization this regression is unlikely to take place.

  [Original Bug Text]

  While triaging LP: #1803587 I found that originally autoremovable
  packages were handled as newly autoremovable ones by unattended-
  upgrades due to not finding all of them at the beginning of u-u's run.

  The root cause seems to be cache.clear() resetting
  pkg.is_auto_removable to False under some circumstances I haven't
  fully narrowed down.

  Set up a Bionic (or later) system with autoremovable packages,
  packages upgradable from -security and blacklist at least one
  upgradable package and apply the following patch to u-u:

  root at bb-1803587:~# diff -Naur /usr/bin/unattended-upgrade.orig /usr/bin/unattended-upgrade
  --- /usr/bin/unattended-upgrade.orig	2018-11-16 16:17:58.522583254 +0000
  +++ /usr/bin/unattended-upgrade	2018-11-16 16:36:12.226675870 +0000
  @@ -948,7 +948,9 @@
   def rewind_cache(cache, pkgs_to_upgrade):
       # type: (apt.Cache, List[apt.Package]) -> None
       """ set the cache back to the state with packages_to_upgrade """
  +    print([pkg.name  for pkg in cache if pkg.is_auto_removable])
       cache.clear()
  +    print([pkg.name for pkg in cache if pkg.is_auto_removable])
       for pkg2 in pkgs_to_upgrade:
           pkg2.mark_install(from_user=not pkg2.is_auto_installed)
       if cache.broken_count > 0:

  Run u-u to observe cache.clear() resetting the list of autoremovable
  packages:

   ~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
  Initial blacklisted packages: systemd
  Initial whitelisted packages:
  Starting unattended upgrades script
  Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
  Using (^linux-image|^linux-headers|^linux-image-extra|^linux-modules|^linux-modules-extra|^linux-signed-image|^kfreebsd-image|^kfreebsd-headers|^gnumach-image|^.*-modules|^.*-kernel|^linux-backports-modules-.*|^linux-modules-.*|^linux-tools|^linux-cloud-tools) regexp to find kernel packages
  Using (^linux-image.*4.15.0-38-generic|^linux-headers.*4.15.0-38-generic|^linux-image-extra.*4.15.0-38-generic|^linux-modules.*4.15.0-38-generic|^linux-modules-extra.*4.15.0-38-generic|^linux-signed-image.*4.15.0-38-generic|^kfreebsd-image.*4.15.0-38-generic|^kfreebsd-headers.*4.15.0-38-generic|^gnumach-image.*4.15.0-38-generic|4.15.0-38-generic.*-modules|4.15.0-38-generic.*-kernel|^linux-backports-modules-.*.*4.15.0-38-generic|^linux-modules-.*.*4.15.0-38-generic|^linux-tools.*4.15.0-38-generic|^linux-cloud-tools.*4.15.0-38-generic) regexp to find running kernel packages
  Checking: apport ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>])
  adjusting candidate version: apport=2.20.9-0ubuntu7.1
  ...
  Checking: libnss-systemd ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
  skipping blacklisted package systemd
  pkg systemd package has been blacklisted
  sanity check failed
  ['libfreetype6']
  []
  ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1803749/+subscriptions