[Bug 1651518] Re: systemd/logind parsing problem: HTX exercisers stopped on error: rc 11, errno 11 from main(): pthread_create

bugproxy bugproxy at us.ibm.com
Mon Jul 10 15:49:51 UTC 2017 

------- Comment From cdeadmin at us.ibm.com 2017-07-10 11:46 EDT-------
This CMVC defect is being cancelled by the CDE Bridge because the corresponding CQ Defect [FW663285] was transferred out of the bridge domain.
Here are the additional details:
New Subsystem = ppc_triage
New Release = unspecified
New Component = ubuntu_linux
New OwnerInfo = Chavez, Luciano (chavez at us.ibm.com)
To continue tracking this issue, please follow CQ defect [FW663285].

Assigning for screening...
It appears that the version of logind on this system does not support the value of "infinity", and is reverting to the default of 12288:

# cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max
12288

As a workaround until this can be resolved, specify an exact value.  You
can try using the current system thread-max value:

# cat /proc/sys/kernel/threads-max
3974272

/etc/systemd/logind.conf.d/htxlogindcustom.conf:
[Login]
UserTasksMax=3974272

Created mirror request (28237) Canonical Launchpad.

Information on this bug will potentially be exposed to the public.
Before you proceed, please make sure you read Content Guidelines for LTC
Bugzilla : Confidential vs. Non-confidential[1].

[1] - ftp://ausgsa.ibm.com/projects/l/ltc/ToolsInfrastructure/ProjectStatus/Bugzilla/Bugzilla_Content_Education_v2.pdf
The bug is ready to be mirrored to:

Distro:    Canonical Launchpad.
Project:   ubuntu
Package:   systemd

This defect will go to Linux as even after making the below 2 changes in
systemd resource limit, errors are seen:

root at yellowbee:/etc/systemd/logind.conf.d# cat htxlogindcustom.conf
[Login]
UserTasksMax=infinity
root at yellowbee:/etc/systemd/logind.conf.d# cat ../system.conf.d/htxsystemdcustom.conf
[Manager]
DefaultTasksAccounting=yes
DefaultTasksMax=infinity

root at yellowbee:/etc/systemd/logind.conf.d#

logind limit for MaxUserTask as well as systemd limit was made infinite.
Please look defect SW363655 for more details and the suggestion given earlier by Linux team.

errors are still seen. So, would ask Linux team to take a look and see if anything else is causing these
errors.

== Comment: #3 - Kevin W. Rudd <ruddk at us.ibm.com> - 2016-12-19 17:34:27 ==
It appears that the version of logind on this system does not support the value of "infinity", and is reverting to the default of 12288:

# cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max
12288

As a workaround until this can be resolved, specify an exact value.  You
can try using the current system thread-max value:

# cat /proc/sys/kernel/threads-max
3974272

/etc/systemd/logind.conf.d/htxlogindcustom.conf:
[Login]
UserTasksMax=3974272

== Comment: #4 - Kevin W. Rudd - 2016-12-20 10:25:09 ==

Canonical,

This issue appears to map to the following systemd bug and patch:

https://github.com/systemd/systemd/issues/3833

https://github.com/systemd/systemd/commit/f50582649f8eee73f59aff95fadd9a963ed4ffea

This patch appears to be included in debian/232-7, but is missing in the
xenial and yakkety versions.

Default Comment by Bridge
This error recreated on the smaller 1U Open Power system with the same smaller 1-adapter/1-4U90 drawer/90 HDD. There are 2 cables connected to the drawer (one to each ESM) that requires multipath enabled.

lpar: yellowbee

root at yellowbee:~# cat /tmp/htxerr

/dev/sdao         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdak         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdt          Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdaz         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdn          Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdv          Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdaj         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdal         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create
== Comment: #1 - Application Cdeadmin <cdeadmin at us.ibm.com> - 2016-12-19 04:15:10 ==

Configuration: IBM 8001-22C (S822LC), LSI SAS adapters, SMC 4U90 disk
drawers, HDD (180) 7.3TB

Problem: HTX exercisers stopped on error, with HTX log showing "rc 11,
errno 11 from main(): pthread_create"

htxubuntu-425

lpar: busybee.aus.stglabs.ibm.com (root/ lab passwd)

root at busybee:~# uname -a
Linux busybee 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:27:59 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

root at busybee:~# cat /tmp/htxerr

/dev/sdh          Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdh          Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
Hardware Exerciser stopped on an error

/dev/sdao         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdao         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
Hardware Exerciser stopped on an error

/dev/sddx         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sddx         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
Hardware Exerciser stopped on an error

/dev/sdcz         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sdcz         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
Hardware Exerciser stopped on an error

/dev/sddp         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
rc 11, errno 11 from main(): pthread_create

/dev/sddp         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
Hardware Exerciser stopped on an error

No errors logged in syslog after starting HTX:

Thank you for taking the time to report this bug and helping to make
Ubuntu better.  It seems that your bug report is not filed about a
specific source package though, rather it is just filed against Ubuntu
in general.  It is important that bug reports be filed about source
packages so that people interested in the package can find the bugs
about it.  You can find some hints about determining what package your
bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage.
You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit
https://bugs.launchpad.net/ubuntu/+bug/1651518/+editstatus and add the
package name in the text box next to the word Package.

[This is an automated message.  I apologize if it reached you inappropriately; please just reply to this message indicating so.]
Yet another mirror burp :-(

Fixed on the LP side.
It seems that more than just the package information was not set up properly during the mirroring.  Luciano helped set the owner of the LP bug to Taco Screen team, so this bug will hopefully soon get some traction on the Canonical side.

Thanks Luciano.
The indicated upstream patch, f50582649f8eee73f59aff95fadd9a963ed4ffea, does not apply cleanly against systemd 229.  Are there known prerequisites for this patch?

It looks like the commit referenced earlier also builds upon the
addition of the config_parse_tasks_max() function added in commit
6300502b .

The following example patch (based on systemd_229-4ubuntu14) focuses on
just the UserTasksMax "infinity" parsing.

A test build of systemd with the previous patch applied resolved the
parsing issue in my testing.

# cat /etc/systemd/logind.conf.d/usertasks.conf
[Login]
UserTasksMax=infinity

# cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max
max

cde00 (cdeadmin at us.ibm.com) added native attachment /tmp/AIXOS06698101/systemd_infinity.debdiff on 2017-01-07 20:22:29
cde00 (cdeadmin at us.ibm.com) added native attachment /tmp/AIXOS06698101/systemd_229.patch on 2017-01-07 20:22:29
Confirmed that the following fix is part of 232 present in zesty. Marking as Fix Released for devel series.

Accepted systemd into yakkety-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/systemd/231-9ubuntu4 in a few
hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

systemd-231-9ubuntu4 looks good for yakkety.  It appears that there are
other changes since user-0.slice was no longer present, but the parent
pids.max is now set to "max":

root at kwr-yakkety:~# cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max
cat: /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max: No such file or directory

root at kwr-yakkety:~# cat /sys/fs/cgroup/pids/user.slice/pids.max
max

We are now just waiting on xenial for closure.

Thanks.
This bug was fixed in the package systemd - 231-9ubuntu4

---------------
systemd (231-9ubuntu4) yakkety; urgency=medium

* debian/extra/units/systemd-resolved.service.d/resolvconf.conf: if
resolved is going to be started, make sure this blocks
network-online.target.  LP: #1673860.
* debian/patches/resolved-follow-CNAMES-for-DNS-stub-replies.patch:
Cherry-pick upstream fix for resolved failing to follow CNAMES for DNS
stub replies. LP: #1647031.
* debian/patches/logind-update-empty-and-infinity-handling-for-User-T.patch:
Cherry-pick upstream fix to handle empty and "infinity" values for
[User]TasksMax.  Closes LP: #1651518.

-- Steve Langasek <steve.langasek at ubuntu.com>  Mon, 20 Mar 2017 22:14:14
-0700

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates.  Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report.  In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
This CMVC defect is being cancelled by the CDE Bridge because the corresponding CQ Defect [FW663285] was transferred out of the bridge domain.
Here are the additional details:
New Subsystem = ppc_triage
New Release = unspecified
New Component = ubuntu_linux
New OwnerInfo = Chavez, Luciano (chavez at us.ibm.com)
To continue tracking this issue, please follow CQ defect [FW663285].

** Bug watch added: github.com/systemd/systemd/issues #3833
   https://github.com/systemd/systemd/issues/3833

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1651518

Title:
  systemd/logind parsing problem: HTX exercisers stopped on error: rc
  11, errno 11 from main(): pthread_create

Status in The Ubuntu-power-systems project:
  Fix Committed
Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Xenial:
  Fix Committed
Status in systemd source package in Yakkety:
  Fix Released

Bug description:
  [SRU justification]
  Before systemd 232, UserTasksMax=infinity is not respected in logind.conf, despite the documentation referring to systemd.resource-control(5) for the definition of this field. This limits the use of Ubuntu 16.04 and later in contexts where a user session should be permitted to allocate large numbers of processes.

  [Test case]
  1. Set UserTasksMax=infinity in /etc/systemd/logind.conf.
  2. Create a new login session.
  3. Check the ulimit for user processes with 'ulimit -u'.
  4. Verify that the limit has a numeric value.
  5. Upgrade to systemd from -proposed.
  6. Create a new login session.
  7. Check the ulimit for user processes with 'ulimit -u'.
  8. Verify that the limit is now set to 'unlimited'.

  [Regression potential]
  This is an upstream patch which is part of 232 and later without issues and clearly addresses the bug in question.  While the upstream commit includes code changes that are not strictly required in order to fix this bug, these are mostly cosmetic and should not carry significant additional risk.


  == Comment: #1 - Application Cdeadmin <cdeadmin at us.ibm.com> -
  2016-12-19 04:15:10 ==

  Configuration: IBM 8001-22C (S822LC), LSI SAS adapters, SMC 4U90 disk
  drawers, HDD (180) 7.3TB

  Problem: HTX exercisers stopped on error, with HTX log showing "rc 11,
  errno 11 from main(): pthread_create"

  htxubuntu-425

  lpar: busybee.aus.stglabs.ibm.com (root/ lab passwd)

  root at busybee:~# uname -a
  Linux busybee 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:27:59 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

  root at busybee:~# cat /tmp/htxerr

  /dev/sdh          Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdh          Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  Hardware Exerciser stopped on an error

  /dev/sdao         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdao         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  Hardware Exerciser stopped on an error

  /dev/sddx         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sddx         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  Hardware Exerciser stopped on an error

  /dev/sdcz         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdcz         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  Hardware Exerciser stopped on an error

  /dev/sddp         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sddp         Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage
  Hardware Exerciser stopped on an error

  No errors logged in syslog after starting HTX:

  ==== State: Open by: asperez on 16 December 2016 10:28:02 ====
  This error recreated on the smaller 1U Open Power system with the same smaller 1-adapter/1-4U90 drawer/90 HDD. There are 2 cables connected to the drawer (one to each ESM) that requires multipath enabled.

  lpar: yellowbee

  root at yellowbee:~# cat /tmp/htxerr

  /dev/sdao         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdak         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdt          Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdaz         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdn          Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdv          Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdaj         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  /dev/sdal         Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage
  rc 11, errno 11 from main(): pthread_create

  ==== State: Open by: cde00 on 19 December 2016 02:48:46 ====

  This defect will go to Linux as even after making the below 2 changes
  in systemd resource limit, errors are seen:

  root at yellowbee:/etc/systemd/logind.conf.d# cat htxlogindcustom.conf
  [Login]
  UserTasksMax=infinity
  root at yellowbee:/etc/systemd/logind.conf.d# cat ../system.conf.d/htxsystemdcustom.conf
  [Manager]
  DefaultTasksAccounting=yes
  DefaultTasksMax=infinity

  root at yellowbee:/etc/systemd/logind.conf.d#

  logind limit for MaxUserTask as well as systemd limit was made infinite.
  Please look defect SW363655 for more details and the suggestion given earlier by Linux team.

  errors are still seen. So, would ask Linux team to take a look and see if anything else is causing these
  errors.

  == Comment: #3 - Kevin W. Rudd <ruddk at us.ibm.com> - 2016-12-19 17:34:27 ==
  It appears that the version of logind on this system does not support the value of "infinity", and is reverting to the default of 12288:

  # cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max
  12288

  As a workaround until this can be resolved, specify an exact value.
  You can try using the current system thread-max value:

  # cat /proc/sys/kernel/threads-max
  3974272

  /etc/systemd/logind.conf.d/htxlogindcustom.conf:
  [Login]
  UserTasksMax=3974272

  == Comment: #4 - Kevin W. Rudd - 2016-12-20 10:25:09 ==

  Canonical,

  This issue appears to map to the following systemd bug and patch:

  https://github.com/systemd/systemd/issues/3833

  https://github.com/systemd/systemd/commit/f50582649f8eee73f59aff95fadd9a963ed4ffea

  This patch appears to be included in debian/232-7, but is missing in
  the xenial and yakkety versions.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1651518/+subscriptions

More information about the foundations-bugs mailing list