[Bug 1638695] Re: Python 2.7.12 performance regression

[Elvis Pranskevichus]

After much testing I found what is causing the regression in 16.04 and
later.  There are several distinct causes which are attributed to the
choices made in debian/rules and the changes in GCC.

Cause #1: the decision to compile `Modules/_math.c` with `-fPIC` *and*
link it statically into the python executable [1].  This causes the
majority of the slowdown.  This may be a bug in GCC or simply a
constraint, I didn't find anything specific on this topic, although
there are a lot of old bug reports regarding the interaction of -fPIC
with -flto.

Cause #2: the enablement of `fpectl` [2], specifically the passage of
`--with-fpectl` to `configure`.  fpectl is disabled in python.org builds
by default and its use is discouraged.  Yet, Debian builds enable it
unconditionally, and it seems to cause a significant performance
degradation.  It's much less noticeable on 14.04 with GCC 4.8.0, but on
more recent releases the performance difference seems to be larger.

Plausible Cause #3: stronger stack smashing protection in 16.04, which
uses --fstack-protector-strong, wherease 14.04 and earlier used
--fstack-protector (with lesser performance overhead).

Also, debian/rules limits the scope of PGO's PROFILE_TASK to 377 test
suites vs upstream's 397, which affects performance somewhat negatively,
but this is not definitive.  What are the reasons behind the trimming of
the tests used for PGO?

Without fpectl, and without -fPIC on _math.c, 2.7.12 built on 16.04 is
slower than stock 2.7.6 on 14.04 by about 0.9% in my pyperformance runs
[3].  This is in contrast to a whopping 7.95% slowdown when comparing
stock versions.

Finally, a vanilla Python 2.7.12 build using GCC 5.4.0, default CFLAGS,
default PROFILE_TASK and default Modules/Setup.local consistently runs
faster in benchmarks than 2.7.6 (by about 0.7%), but I was not able to
pinpoint the exact reason for that.

Note: the percentages above are the relative change in the geometric
mean of pyperformance benchmark results.


[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/python2.7/tree/debian/rules?h=ubuntu/xenial-updates#n421

[2] https://git.launchpad.net/~usd-import-
team/ubuntu/+source/python2.7/tree/debian/rules?h=ubuntu/xenial-
updates#n117

[3] https://docs.google.com/spreadsheets/d/1L3_gxe-
AOYJsXFwGZgFko8jaChB0dFPjK5oMO5T5vj4/edit?usp=sharing

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python2.7 in Ubuntu.
https://bugs.launchpad.net/bugs/1638695

Title:
  Python 2.7.12 performance regression

Status in python2.7 package in Ubuntu:
  Confirmed

Bug description:
  I work on the OpenStack-Ansible project and we've noticed that testing
  jobs on 16.04 take quite a bit longer to complete than on 14.04.  They
  complete within an hour on 14.04 but they normally take 90 minutes or
  more on 16.04.  We use the same version of Ansible with both versions
  of Ubuntu.

  After more digging, I tested python performance (using the
  'performance' module) on 14.04 (2.7.6) and on 16.04 (2.7.12).  There
  is a significant performance difference between each version of
  python.  That is detailed in a spreadsheet[0].

  I began using perf to dig into the differences when running the python
  performance module and when using Ansible playbooks.  CPU migrations
  (as measured by perf) are doubled in Ubuntu 16.04 when running the
  same python workloads.

  I tried changing some of the kerne.sched sysctl configurables but they
  had very little effect on the results.

  I compiled python 2.7.12 from source on 14.04 and found the
  performance to be unchanged there.  I'm not entirely sure where the
  problem might be now.

  We also have a bug open in OpenStack-Ansible[1] that provides
  additional detail. Thanks in advance for any help you can provide!

  [0] https://docs.google.com/spreadsheets/d/18MmptS_DAd1YP3OhHWQqLYVA9spC3xLt4PS3STI6tds/edit?usp=sharing
  [1] https://bugs.launchpad.net/openstack-ansible/+bug/1637494

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1638695/+subscriptions