[Bug 2023545] Comment bridged from LTC Bugzilla
bugproxy
2023545 at bugs.launchpad.net
Tue Jan 23 08:49:44 UTC 2024
------- Comment From Grgo.Mariani at ibm.com 2024-01-23 03:44 EDT-------
The problem is still visible after installing the -proposed package:
root:~# dpkg -l | grep openssl
ii openssl 3.0.2-0ubuntu1.13 s390x Secure Sockets Layer toolkit - cryptographic utility
ii openssl-ibmca 2.2.3-0ubuntu1.1 s390x libica based hardware acceleration engine for OpenSSL
ii python3-openssl 21.0.0-1 all Python 3 wrapper around the OpenSSL library
root:~# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
By running with gdb the following BT can be observed:
(gdb) r
Starting program: /usr/bin/openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj /CN=US
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+.....+..........+............+...+...........+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.....+.+..........................+...+.+......+.....+...+.........+.+.........+......+..+.......+....................+.+...............+..............+...+...+......+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+.....+............+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
Program received signal SIGSEGV, Segmentation fault.
0x000003fffd89c738 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0x0) at pthread_rwlock_common.c:603
603 pthread_rwlock_common.c: No such file or directory.
(gdb) bt
#0 0x000003fffd89c738 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0x0) at pthread_rwlock_common.c:603
#1 ___pthread_rwlock_wrlock (rwlock=0x0) at pthread_rwlock_wrlock.c:26
#2 0x000003fffdbb7812 in CRYPTO_THREAD_write_lock () from /lib/s390x-linux-gnu/libcrypto.so.3
#3 0x000003fffdb634f2 in ENGINE_finish () from /lib/s390x-linux-gnu/libcrypto.so.3
#4 0x000003fffdb86314 in EVP_CIPHER_CTX_reset () from /lib/s390x-linux-gnu/libcrypto.so.3
#5 0x000003fffdb8635c in EVP_CIPHER_CTX_free () from /lib/s390x-linux-gnu/libcrypto.so.3
#6 0x000003fffdc791ac in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#7 0x000003fffdb8ca88 in EVP_RAND_CTX_free () from /lib/s390x-linux-gnu/libcrypto.so.3
#8 0x000003fffdbe4bc2 in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#9 0x000003fffdbb0c6c in CRYPTO_free_ex_data () from /lib/s390x-linux-gnu/libcrypto.so.3
#10 0x000003fffdba8dfa in ?? () from /lib/s390x-linux-gnu/libcrypto.so.3
#11 0x000003fffdbb36b8 in OPENSSL_cleanup () from /lib/s390x-linux-gnu/libcrypto.so.3
#12 0x000003fffd84b6ec in __run_exit_handlers (status=<optimized out>, listp=0x3fffd9ce618 <__exit_funcs>, run_list_atexit=run_list_atexit at entry=true,
run_dtors=run_dtors at entry=true) at exit.c:113
#13 0x000003fffd84b7b0 in __GI_exit (status=<optimized out>) at exit.c:143
#14 0x000002aa00047c06 in main ()
syslog shows the following:
Jan 22 15:14:03 a8314006.lnxne.boe kernel: [<000003ff89d3242c>] 0x3ff89d3242c
Jan 22 15:14:03 a8314006.lnxne.boe kernel: Last Breaking-Event-Address:
Jan 22 15:14:03 a8314006.lnxne.boe kernel: User Code: 000003ff89b1c72c: b90400b2 lgr %r11,%r2
000003ff89b1c730: 47000000 bc 0,0
#000003ff89b1c734: b24f00a0 ear %r10,%a0
>000003ff89b1c738: 58102018 l %r1,24(%r2)
000003ff89b1c73c: ebaa0020000d sllg %r10,%r10,32
000003ff89b1c742: b24f00a1 ear %r10,%a1
000003ff89b1c746: 5910a0d0 c %r1,208(%r10)
000003ff89b1c74a: a7840033 brc 8,000003ff89b1c7b0
Jan 22 15:14:03 SYSTEM kernel: 000003ff89c4dd10 000003ff8a057120 000003ff89e37812 000003ffc93fe170
Jan 22 15:14:03 SYSTEM kernel: 000003ff8a270720 000003ff8a057128 000002aa000003ff 0000000000000000
Jan 22 15:14:03 SYSTEM kernel: 000002aa272ac835 000002aa0d8d5a40 0000000000000000 000002aa0d8da370
Jan 22 15:14:03 SYSTEM kernel: User GPRS: 0000000000000007 000003ff89b1c720 0000000000000000 000002aa0d8d5a40
Jan 22 15:14:03 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jan 22 15:14:03 SYSTEM kernel: User PSW : 0705000180000000 000003ff89b1c738
Jan 22 15:14:03 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jan 22 15:14:03 SYSTEM kernel: CPU: 0 PID: 98944 Comm: openssl Not tainted 5.15.0-91-generic #101-Ubuntu
Jan 22 15:14:03 SYSTEM kernel: AS:000000009b00c1c7 R3:0000000000000024
Jan 22 15:14:03 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jan 22 15:14:03 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jan 22 15:14:03 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ff89a80000+1ca000]
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/2023545
Title:
[UBUNTU 22.04] openssl with ibmca engine configured dumps core when
creating a new certificate
Status in Ubuntu on IBM z Systems:
In Progress
Status in openssl package in Ubuntu:
In Progress
Status in openssl source package in Jammy:
Fix Committed
Status in openssl source package in Lunar:
Fix Released
Bug description:
=== SRU information ===
[Meta]
This bug is part of a series of three bugs for a single SRU.
The "central" bug with the global information and debdiff is http://pad.lv/2033422
[Impact]
Openssl using an engine dumps core upon certificate creation; other operations are probably affected too. Overall, engines are likely mostly unusable.
[Test plan]
- An openssl engine is req. to test the fix.
- A z13 / LinuxONE LPAR or z/VM guest is needed, with attached APQN.
- Check with 'lszcrypt -V' the availability (online) of the hw crypto resources.
- Install the needed package that allows to exploit the hw crypto resources:
sudo apt-get install libica-utils libica? openssl-ibmca
- And copy a working sample openssf.cnf file:
sudo cp /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample /etc/ssl/openssl.cnf
- Verify if the 'openssl engine' lists an ibmca engine,
in addition to the dynamic engine:
openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support <===
- try to create a new certificate, using this cmd-line:
openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
- The above command must not lead to a 'Segmentation fault (core dumped)',
rather than create a proper certificate file.
Also watch /var/log/syslog / journalctl for more details.
- Upgrade not only the openssl package itself,
but also libssl3, before verification.
- The issue is fixed in openssl 3.0.8 which landed in lunar.
[Where problems could occur]
I don't pretend to understand the lifecycle of providers in openssl3 but the patch is simple and has been widely tested by now, including on ubuntu. Thus, I see little chance an unexpected problem would occur with it.
[Patches]
The patches come directly from upstream and apply cleanly.
https://github.com/openssl/openssl/issues/18578
*
https://git.launchpad.net/~adrien-n/ubuntu/+source/openssl/tree/debian/patches/jammy-
sru-0001-Release-the-drbg-in-the-global-default-context-
befor.patch?h=jammy-sru&id=04ef023920ab08fba214817523fba897527dfff0
=== Original description ===
openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem
-keyout __key.pem --subj '/CN=US'
---Problem Description---
OpenSSL with ibmca engine configured dumps core when creating a new certificate.
# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
# openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
Segmentation fault (core dumped)
# journalctl
Jun 07 13:06:08 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ffae080000+1ca000]
Jun 07 13:06:08 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jun 07 13:06:08 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jun 07 13:06:08 SYSTEM kernel: AS:000000009c2941c7 R3:0000000000000024
Jun 07 13:06:08 SYSTEM kernel: CPU: 2 PID: 2344 Comm: openssl Kdump: loaded Not tainted 5.15.0-73-generic #80-Ubuntu
Jun 07 13:06:08 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jun 07 13:06:08 SYSTEM kernel: User PSW : 0705000180000000 000003ffae11c708
Jun 07 13:06:08 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jun 07 13:06:08 SYSTEM kernel: User GPRS: 0000000000000007 000003ffae11c6f0 0000000000000000 000002aa3289f9d0
Jun 07 13:06:08 SYSTEM kernel: 000002aa1825980f 000002aa3289f9d0 0000000000000000 000002aa328a4300
Jun 07 13:06:08 SYSTEM kernel: 000003ffae870720 000003ffae657128 000002aa000003ff 0000000000000000
Jun 07 13:06:08 SYSTEM kernel: 000003ffae24dd10 000003ffae657120 000003ffae437c22 000003ffec2fe000
Jun 07 13:06:08 SYSTEM kernel: User Code: 000003ffae11c6fc: b90400b2 lgr %r11,%r2
000003ffae11c700: 47000000 bc 0,0
#000003ffae11c704: b24f00a0 ear %r10,%a0
>000003ffae11c708: 58102018 l %r1,24(%r2)
000003ffae11c70c: ebaa0020000d sllg %r10,%r10,32
000003ffae11c712: b24f00a1 ear %r10,%a1
000003ffae11c716: 5910a0d0 c %r1,208(%r10)
000003ffae11c71a: a7840033 brc 8,000003ffae11c780
Jun 07 13:06:08 SYSTEM kernel: Last Breaking-Event-Address:
Jun 07 13:06:08 SYSTEM kernel: [<000003ffae33242c>] 0x3ffae33242c
Jun 07 13:06:08 SYSTEM systemd[1]: Started Process Core Dump (PID 2345/UID 0).
Jun 07 13:06:08 SYSTEM systemd-coredump[2350]: Process 2344 (openssl) of user 0 dumped core.
Found module linux-vdso64.so.1 with build-id: bcfab8ac8dbd44c758c3c5494e2952db16905d2e
Found module libica.so.4 with build-id: 0cc5ace50644dfba6d0ecf4f783477cd04a55731
Found module ibmca.so with build-id: 27daaf0ed1857fdad3761c2b3db21020999eee08
Found module ld64.so.1 with build-id: 31d4856f0ba9ea058c91a34f4d684ae0fe01964c
Found module libc.so.6 with build-id: 74250317950da91d3345f258cb2dd12d22c3f2e5
Found module libcrypto.so.3 with build-id: a27f20e6cf293f214d459530ce2c0b2b52fdbdb4
Found module libssl.so.3 with build-id: e2c031c3dac06b5ce43bdea022aee7989f78dde4
Found module openssl with build-id: ed0fe325182e99d135ee6b08e6d90a9d1c42af7f
Stack trace of thread 2344:
#0 0x000003ffae11c708 __pthread_rwlock_wrlock_full64 (libc.so.6 + 0x9c708)
#1 0x000003ffae437c22 CRYPTO_THREAD_write_lock (libcrypto.so.3 + 0x1b7c22)
#2 0x000003ffae3e3472 ENGINE_finish (libcrypto.so.3 + 0x163472)
#3 0x000003ffae406844 EVP_CIPHER_CTX_reset (libcrypto.so.3 + 0x186844)
#4 0x000003ffae40688c EVP_CIPHER_CTX_free (libcrypto.so.3 + 0x18688c)
#5 0x000003ffae4f903c n/a (libcrypto.so.3 + 0x27903c)
#6 0x000003ffae40ca98 EVP_RAND_CTX_free (libcrypto.so.3 + 0x18ca98)
#7 0x000003ffae461a92 n/a (libcrypto.so.3 + 0x1e1a92)
#8 0x000003ffae430b9c CRYPTO_free_ex_data (libcrypto.so.3 + 0x1b0b9c)
#9 0x000003ffae4293ca n/a (libcrypto.so.3 + 0x1a93ca)
#10 0x000003ffae4335e8 OPENSSL_cleanup (libcrypto.so.3 + 0x1b35e8)
#11 0x000003ffae0cb6cc __run_exit_handlers (libc.so.6 + 0x4b6cc)
#12 0x000003ffae0cb790 __GI_exit (libc.so.6 + 0x4b790)
#13 0x000002aa31847c06 main (openssl + 0x47c06)
#14 0x000003ffae0aa712 __libc_start_call_main (libc.so.6 + 0x2a712)
#15 0x000003ffae0aa7f0 __libc_start_main_impl (libc.so.6 + 0x2a7f0)
#16 0x000002aa31848070 n/a (openssl + 0x48070)
Jun 07 13:06:08 SYSTEM systemd[1]: systemd-coredump at 12-2345-0.service: Deactivated successfully.
Jun 07 13:06:31 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ffbee00000+1ca000]
Jun 07 13:06:31 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jun 07 13:06:31 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jun 07 13:06:31 SYSTEM kernel: AS:000000009c2941c7 R3:0000000000000024
Jun 07 13:06:31 SYSTEM kernel: CPU: 2 PID: 2356 Comm: openssl Kdump: loaded Not tainted 5.15.0-73-generic #80-Ubuntu
Jun 07 13:06:31 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jun 07 13:06:31 SYSTEM kernel: User PSW : 0705000180000000 000003ffbee9c708
Jun 07 13:06:31 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jun 07 13:06:31 SYSTEM kernel: User GPRS: 0000000000000007 000003ffbee9c6f0 0000000000000000 000002aa176569d0
Jun 07 13:06:31 SYSTEM kernel: 000002aa3dc356c6 000002aa176569d0 0000000000000000 000002aa1765b300
Jun 07 13:06:31 SYSTEM kernel: 000003ffbf5f0720 000003ffbf3d7128 000002aa000003ff 0000000000000000
Jun 07 13:06:31 SYSTEM kernel: 000003ffbefcdd10 000003ffbf3d7120 000003ffbf1b7c22 000003ffc4efdd40
Jun 07 13:06:31 SYSTEM kernel: User Code: 000003ffbee9c6fc: b90400b2 lgr %r11,%r2
000003ffbee9c700: 47000000 bc 0,0
#000003ffbee9c704: b24f00a0 ear %r10,%a0
>000003ffbee9c708: 58102018 l %r1,24(%r2)
000003ffbee9c70c: ebaa0020000d sllg %r10,%r10,32
000003ffbee9c712: b24f00a1 ear %r10,%a1
000003ffbee9c716: 5910a0d0 c %r1,208(%r10)
000003ffbee9c71a: a7840033 brc 8,000003ffbee9c780
Jun 07 13:06:31 SYSTEM kernel: Last Breaking-Event-Address:
Jun 07 13:06:31 SYSTEM kernel: [<000003ffbf0b242c>] 0x3ffbf0b242c
Jun 07 13:06:32 SYSTEM systemd[1]: Started Process Core Dump (PID 2357/UID 0).
Jun 07 13:06:32 SYSTEM systemd-coredump[2362]: Process 2356 (openssl) of user 0 dumped core.
Found module linux-vdso64.so.1 with build-id: bcfab8ac8dbd44c758c3c5494e2952db16905d2e
Found module libica.so.4 with build-id: 0cc5ace50644dfba6d0ecf4f783477cd04a55731
Found module ibmca.so with build-id: 27daaf0ed1857fdad3761c2b3db21020999eee08
Found module ld64.so.1 with build-id: 31d4856f0ba9ea058c91a34f4d684ae0fe01964c
Found module libc.so.6 with build-id: 74250317950da91d3345f258cb2dd12d22c3f2e5
Found module libcrypto.so.3 with build-id: a27f20e6cf293f214d459530ce2c0b2b52fdbdb4
Found module libssl.so.3 with build-id: e2c031c3dac06b5ce43bdea022aee7989f78dde4
Found module openssl with build-id: ed0fe325182e99d135ee6b08e6d90a9d1c42af7f
Stack trace of thread 2356:
#0 0x000003ffbee9c708 __pthread_rwlock_wrlock_full64 (libc.so.6 + 0x9c708)
#1 0x000003ffbf1b7c22 CRYPTO_THREAD_write_lock (libcrypto.so.3 + 0x1b7c22)
#2 0x000003ffbf163472 ENGINE_finish (libcrypto.so.3 + 0x163472)
#3 0x000003ffbf186844 EVP_CIPHER_CTX_reset (libcrypto.so.3 + 0x186844)
#4 0x000003ffbf18688c EVP_CIPHER_CTX_free (libcrypto.so.3 + 0x18688c)
#5 0x000003ffbf27903c n/a (libcrypto.so.3 + 0x27903c)
#6 0x000003ffbf18ca98 EVP_RAND_CTX_free (libcrypto.so.3 + 0x18ca98)
#7 0x000003ffbf1e1a92 n/a (libcrypto.so.3 + 0x1e1a92)
#8 0x000003ffbf1b0b9c CRYPTO_free_ex_data (libcrypto.so.3 + 0x1b0b9c)
#9 0x000003ffbf1a93ca n/a (libcrypto.so.3 + 0x1a93ca)
#10 0x000003ffbf1b35e8 OPENSSL_cleanup (libcrypto.so.3 + 0x1b35e8)
#11 0x000003ffbee4b6cc __run_exit_handlers (libc.so.6 + 0x4b6cc)
#12 0x000003ffbee4b790 __GI_exit (libc.so.6 + 0x4b790)
#13 0x000002aa161c7c06 main (openssl + 0x47c06)
#14 0x000003ffbee2a712 __libc_start_call_main (libc.so.6 + 0x2a712)
#15 0x000003ffbee2a7f0 __libc_start_main_impl (libc.so.6 + 0x2a7f0)
#16 0x000002aa161c8070 n/a (openssl + 0x48070)
Contact Information = grgo.mariani at ibm.com christian.rund at de.ibm.com
---uname output---
Linux SYSTEM 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:23:03 UTC 2023 s390x s390x s390x GNU/Linux
Machine Type = Manufacturer: IBM Type: 3931
Model: 703 A01
---Steps to Reproduce---
1. Configure openssl to be used with ibmca engine.
2. run the following command:
# openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
3. Check the syslog
Userspace tool common name: openssl
The userspace tool has the following bit modes: 64
Userspace rpm: openssl 3.0.2-0ubuntu1.10
s390x
Userspace tool obtained from project website: na
== Comment: #1 - Grgo ===============================================
Further investigations point to this open issue on openssl repository https://github.com/openssl/openssl/issues/18578
The default_algorithms declaration of RAND and others has no effect on
the behaviour of this problem.
== Comment: #2 - Ingo - 2023-06-12 06:05:20 =========================
The OpenSSL fix for the mentioned issue https://github.com/openssl/openssl/issues/18578 is commit
https://github.com/openssl/openssl/commit/a88e97fcace01ecf557b207f04328a72df5110df in the master branch.
The corresponding commit for the openssl-3.0 branch is
https://github.com/openssl/openssl/commit/d0f8056c47f7aea40a34815fe459404f14501e81
This commit is included in OpenSSL 3.0.8.
Please include this commit into the OpenSSL package shipped with 22.04
(and later releases).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/2023545/+subscriptions
More information about the foundations-bugs
mailing list