[Bug 1396787] Re: checking trust of archives eats a lot of cpu

Thu Feb 7 14:20:37 UTC 2019

Tested with 1.1ubuntu1.18.04.7~16.04.1:

On the autopkgtest infrastructure u-u runs for 20s when all packages are
installed from xenial-security but none from xenial-updates:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/amd64/u/unattended-upgrades/20181213_182038_2962e@/log.gz
...
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial
Packages that will be upgraded: 
19.18user 1.17system 0:20.53elapsed 99%CPU (0avgtext+0avgdata 77720maxresident)k
0inputs+123512outputs (0major+38986minor)pagefaults 0swaps
...

On a 2012 MacBook Air inside a KVM qemu vm autopkgtest runner it is ~8s:
...
adt-2.log-Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial
adt-2.log-Packages that will be upgraded: 
adt-2.log:7.64user 0.31system 0:08.00elapsed 99%CPU (0avgtext+0avgdata 76516maxresident)k
..

This is basically the same speed as with 0.90ubuntu0.10.
There is a 12% speed regression when testing in qemu with kvm on a 19.04 host:

...
adt-1549534420.log-Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial
adt-1549534420.log-Packages that will be upgraded: 
adt-1549534420.log:6.72user 0.30system 0:07.07elapsed 99%CPU (0avgtext+0avgdata 77812maxresident)k
...
vs.
..
adt-2.log-Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial
adt-2.log-Packages that will be upgraded: 
adt-2.log:7.64user 0.31system 0:08.00elapsed 99%CPU (0avgtext+0avgdata 76516maxresident)k
...

Or a 24% speedup with a smaller set of upgradable packages measured in
two 16.04 lxc containers running on the same 19.04 development system:

ii  unattended-upgrades 0.90ubuntu0.10 all          automatic
installation of security upgrades

# for i in $(seq 5); do time unattended-upgrade --dry-run; done

real	0m4.326s
user	0m4.245s
sys	0m0.043s

real	0m4.309s
user	0m4.239s
sys	0m0.070s
...
# apt list --upgradable 
Listing... Done
cloud-init/xenial-proposed 18.5-21-g8ee294d5-0ubuntu1~16.04.1 all [upgradable from: 18.4-0ubuntu1~16.04.2]
kmod/xenial-proposed 22-1ubuntu5.2 amd64 [upgradable from: 22-1ubuntu5.1]
libc-bin/xenial-proposed 2.23-0ubuntu11 amd64 [upgradable from: 2.23-0ubuntu10]
libc6/xenial-proposed 2.23-0ubuntu11 amd64 [upgradable from: 2.23-0ubuntu10]
libkmod2/xenial-proposed 22-1ubuntu5.2 amd64 [upgradable from: 22-1ubuntu5.1]
locales/xenial-proposed 2.23-0ubuntu11 all [upgradable from: 2.23-0ubuntu10]
multiarch-support/xenial-proposed 2.23-0ubuntu11 amd64 [upgradable from: 2.23-0ubuntu10]
python-apt-common/xenial-proposed 1.1.0~beta1ubuntu0.16.04.3 all [upgradable from: 1.1.0~beta1ubuntu0.16.04.2]
python3-apt/xenial-proposed 1.1.0~beta1ubuntu0.16.04.3 amd64 [upgradable from: 1.1.0~beta1ubuntu0.16.04.2]
snapd/xenial-proposed 2.37.1 amd64 [upgradable from: 2.34.2]
ubuntu-core-launcher/xenial-proposed 2.37.1 amd64 [upgradable from: 2.34.2]
unattended-upgrades/xenial-proposed 1.1ubuntu1.18.04.7~16.04.1 all [upgradable from: 0.90ubuntu0.10]
#

vs.

ii  unattended-upgrades 1.1ubuntu1.18.04.7~16.04.1 all
automatic installation of security upgrades

# for i in $(seq 5); do time unattended-upgrade --dry-run; done

real	0m3.269s
user	0m3.194s
sys	0m0.076s

real	0m3.277s
user	0m3.135s
sys	0m0.115s
...
root at x-uu-ref:~# apt list --upgradable 
Listing... Done
cloud-init/xenial-proposed 18.5-21-g8ee294d5-0ubuntu1~16.04.1 all [upgradable from: 18.4-0ubuntu1~16.04.2]
kmod/xenial-proposed 22-1ubuntu5.2 amd64 [upgradable from: 22-1ubuntu5.1]
libc-bin/xenial-proposed 2.23-0ubuntu11 amd64 [upgradable from: 2.23-0ubuntu10]
libc6/xenial-proposed 2.23-0ubuntu11 amd64 [upgradable from: 2.23-0ubuntu10]
libkmod2/xenial-proposed 22-1ubuntu5.2 amd64 [upgradable from: 22-1ubuntu5.1]
locales/xenial-proposed 2.23-0ubuntu11 all [upgradable from: 2.23-0ubuntu10]
multiarch-support/xenial-proposed 2.23-0ubuntu11 amd64 [upgradable from: 2.23-0ubuntu10]
python-apt-common/xenial-proposed 1.1.0~beta1ubuntu0.16.04.3 all [upgradable from: 1.1.0~beta1ubuntu0.16.04.2]
snapd/xenial-proposed 2.37.1 amd64 [upgradable from: 2.34.2]
ubuntu-core-launcher/xenial-proposed 2.37.1 amd64 [upgradable from: 2.34.2]
root at x-uu-ref:~# 

** Tags removed: verification-needed verification-needed-xenial
** Tags added: verification-done verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1396787

Title:
  checking trust of archives eats a lot of cpu

Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in unattended-upgrades source package in Xenial:
  Fix Committed
Status in unattended-upgrades source package in Bionic:
  Fix Released

Bug description:
  [Impact]

   * Unattended-upgrades consumes tens of seconds or even minutes of CPU
  time to verify the origin of the packages

   * Using excessive amount of CPU is unpleasant for desktop/laptop
  users and also wastes computation time on servers/cloud instances.

   * Unattended-upgrades' algorithm for checking and adjusting package
  origins is redesigned to visit and adjust less packages.

  [Test Case]

   * The added upgrade-all-security autopkgtest measure the time u-u needs for upgrading security updates on the tested release starting with no security updates applied to the point where all security updates are applied but all packages are left upgradable from <release>-updates. The test also measures the time needed for --dry-run to find no updates to be installed unattended.
  * Please run autopkgtests and look for the to time results:
  ...
  All upgrades installed
  44.41user 3.06system 0:48.35elapsed 98%CPU (0avgtext+0avgdata 164872maxresident)k
  208inputs+192376outputs (0major+642657minor)pagefaults 0swaps
  ...
  No packages found that can be upgraded unattended and no pending auto-removals
  2.83user 0.11system 0:02.98elapsed 98%CPU (0avgtext+0avgdata 79308maxresident)k

  [Regression Potential] 

   * Due to algorithm redesign there is a risk that packages from
  allowed origins are not upgraded. There were unit tests for testing
  the selection of the right packages to upgrade already, but a new
  autopkgtest is also introduce to verify u-u's behavior on current
  real-life security-updates.

  [Original bug text]

  (System: Ubuntu 14.04, up to date packages)

  I noticed that unattended-upgrades spends a significant amount of time
  in phases where it runs at 100% cpu. On a slower machine (core 2 t7200
  2GHz) this goes on for minutes rather than seconds. This interferes
  with using the machine for other tasks.

  Using the --debug option to unattended-upgrades shows that the program
  outputs a lot of lines like the following during these 100% cpu
  phases:

  matching 'a'='trusty-updates' against '<Origin component:'universe'
  archive:'trusty-updates' origin:'Ubuntu' label:'Ubuntu'
  site:'de.archive.ubuntu.com' isTrusted:True>

  From this output I guess the operation executed is not so complicated
  that it should require so much cpu power. ??

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: unattended-upgrades 0.82.1ubuntu2
  ProcVersionSignature: Ubuntu 3.13.0-40.69-generic 3.13.11.10
  Uname: Linux 3.13.0-40-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.5
  Architecture: amd64
  Date: Wed Nov 26 21:53:57 2014
  InstallationDate: Installed on 2014-08-28 (90 days ago)
  InstallationMedia: Kubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.1)
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=de_DE.UTF-8
   SHELL=/bin/bash
  SourcePackage: unattended-upgrades
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1396787/+subscriptions