[Bug 1123272] Re: high cpu usage on download (not optimised SHA256, SHA512, SHA1)
Julian Andres Klode
julian.klode at gmail.com
Tue Jul 18 19:48:25 UTC 2017
Of course you don't have a CPU bottleneck with curl, it does not run 3
or 4 hashes over everything it downloads. APT needs to hash its
downloads to ensure they are secure, and it uses all available hashes to
do so, so if one hash's security is compromised, we can still hopefully
still rely on the others.
And the topic of this bug report is indeed hash CPU usage, and I
compared our CPU usage on a 1 GB test file in a tmpfs to the CPU usage
of OpenSSL and Nettle. So this test operates on a data size 5 times
higher than the usual packages and under optimal conditions to evaluate
how fast we can hash.
The test shows that on a 1 Gbit/s connection you'll likely be throttled
slightly at a comparable CPU (assuming the connection reaches about 800
Mbit/s, that is, 80% efficiency). If you have a data rate of about 500
Mbit/s, you will likely be fine (not counting parallel downloads).
If there are other problems reducing the download speeds, these are
separate bugs. This one covers the overhead of hashing, nothing else.
BTW: The original bug report talks about 20% CPU usage on a 5 year old
CPU, that seems entirely reasonable and not really an issue. If your
bandwidth is high, you'll have higher CPU usage for a shorter time (like
100% for 5 seconds or so).
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1123272
Title:
high cpu usage on download (not optimised SHA256, SHA512, SHA1)
Status in apt package in Ubuntu:
Confirmed
Bug description:
http process takes about 20% on intel i5-3317. Here is output of ps aux:
root 4309 21.7 0.0 37108 2344 pts/2 S+ 18:17 0:01 /usr/lib/apt/methods/http
it can be easy reproduces with "apt-get download firefox-dbg" or any
other upgrade realted command.
The reason of this overhead are this functions:
22,34% http libapt-pkg.so.4.12.0 [.] SHA256_Transform(_SHA256_CTX*, unsigned int const*)
14,81% http libapt-pkg.so.4.12.0 [.] SHA512_Transform(_SHA512_CTX*, unsigned long const*)
8,11% http libapt-pkg.so.4.12.0 [.] SHA1Transform(unsigned int*, unsigned char const*)
5,62% http libapt-pkg.so.4.12.0 [.] MD5Transform(unsigned int*, unsigned int const*)
libapt uses own implementation of this hasches without any optimisation. Are there any reason why libapt do not use
openssl? Beside current version of openssl has AVX optimised SHA* implementations.
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: apt 0.9.7.7ubuntu1
ProcVersionSignature: Ubuntu 3.8.0-4.9-generic 3.8.0-rc6
Uname: Linux 3.8.0-4-generic x86_64
ApportVersion: 2.8-0ubuntu4
Architecture: amd64
Date: Tue Feb 12 18:12:33 2013
InstallationDate: Installed on 2012-09-13 (152 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Alpha amd64 (20120913.1)
MarkForUpload: True
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=de_DE.UTF-8
SHELL=/bin/bash
SourcePackage: apt
UpgradeStatus: Upgraded to raring on 2013-02-06 (5 days ago)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1123272/+subscriptions
More information about the foundations-bugs
mailing list