[Bug 2146760] Re: Severe performance regression in OpenSSL 3.0.13 causes ~300ms TLS handshake latency on Ubuntu 24.04 LTS

[Prabu Selvam]

Hello Matthew,

Thank you for your specific questions. I have run the checks on the
affected system, and here are the results:

1. Is the system in FIPS mode?

No. The output of sudo pro status confirms FIPS is not active on this
system.

sudo pro status --all
SERVICE          AVAILABLE  DESCRIPTION
anbox-cloud      yes        Scalable Android in the cloud
cc-eal           no         Common Criteria EAL2 Provisioning Packages
esm-apps         yes        Expanded Security Maintenance for Applications
esm-apps-legacy  no         Expanded Security Maintenance for Applications on Legacy Instances
esm-infra        yes        Expanded Security Maintenance for Infrastructure
esm-infra-legacy no         Expanded Security Maintenance for Infrastructure on Legacy Instances
fips             no         NIST-certified FIPS crypto packages
fips-preview     no         Preview of FIPS crypto packages undergoing certification with NIST
fips-updates     yes        FIPS compliant crypto packages with stable security updates
landscape        yes        Management and administration tool for Ubuntu
livepatch        yes        Canonical Livepatch service
realtime-kernel  yes        Ubuntu kernel with PREEMPT_RT patches integrated
ros              no         Security Updates for the Robot Operating System
ros-updates      no         All Updates for the Robot Operating System
usg              yes        Security compliance and audit tools

2. Are we running the latest OpenSSL package (3.0.13-0ubuntu3.7)?

Yes. The output of apt-cache policy openssl confirms we are already on
the latest version available in the noble-security and noble-updates
repositories.

apt-cache policy openssl
openssl:
  Installed: 3.0.13-0ubuntu3.7
  Candidate: 3.0.13-0ubuntu3.7
  Version table:
 *** 3.0.13-0ubuntu3.7 500
        500 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.0.13-0ubuntu3 500
        500 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 Packages

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/2146760

Title:
  Severe performance regression in OpenSSL 3.0.13 causes ~300ms TLS
  handshake latency on Ubuntu 24.04 LTS

Status in openssl package in Ubuntu:
  New

Bug description:
  Hello Ubuntu Security & OpenSSL Team,

  We have identified a critical performance regression in the openssl
  package (OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan
  2024)) provided with the official Ubuntu 24.04 LTS cloud image on AWS.

  When an application creates a new, secure (TLS) connection to the AWS
  RDS instance hosted in the same VPC & Subnets, there is a ~300ms stall
  during the initial handshake. This makes the official Ubuntu AMI
  unsuitable for any production workload that involves high-frequency,
  secure connections.

  We have confirmed this issue is resolved by manually compiling and
  installing OpenSSL 3.3.6, which points to a specific performance bug
  in the 3.0.13 version shipped with Ubuntu 24.04.

  Below are the details:

  OS: Ubuntu 24.04 LTS
  AMI: ami-01f79b1e4a5c64257
  Instance Type: m5.2xlarge
  Region: eu-central-1

  [Steps to Reproduce]

  1. Launch a standard Ubuntu 24.04 LTS instance on AWS.

  2. Install the MySQL client and mysqlslap tool: sudo apt-get install
  mysql-client.

  Run a mysqlslap benchmark against any TLS-enabled MySQL 8 server.
  time mysqlslap --host=[DB_HOST] --user=[USER] -p --ssl-mode=REQUIRED --query="SELECT 1" --iterations=10 --concurrency=10

  Actual Result (With Official OpenSSL 3.0.13):

  A severe ~300ms+ latency is observed. The benchmark shows:

  Benchmark
          Average number of seconds to run all queries: 0.340 seconds
          Minimum number of seconds to run all queries: 0.338 seconds
          Maximum number of seconds to run all queries: 0.345 seconds
          Number of clients running queries: 10
          Average number of queries per client: 1

  
  real    0m5.959s
  user    0m6.549s
  sys     0m0.220s

  Proof of Resolution (With Manual OpenSSL 3.3.6 Upgrade)

  After manually compiling and installing OpenSSL 3.3.6 on the same
  instance, the exact same benchmark yields excellent results,
  confirming the bug is within OpenSSL itself.

  Benchmark
          Average number of seconds to run all queries: 0.026 seconds
          Minimum number of seconds to run all queries: 0.024 seconds
          Maximum number of seconds to run all queries: 0.029 seconds
          Number of clients running queries: 10
          Average number of queries per client: 1

  
  real    0m3.746s
  user    0m0.263s
  sys     0m0.063s

  This performance bug severely impacts the usability of Ubuntu 24.04
  LTS for production our applications. We kindly request that this
  performance issue be investigated and that a fix be backported to the
  official OpenSSL package for Ubuntu 24.04 LTS. Ensuring a stable and
  optimized OpenSSL version is critical for maintaining the reliability
  and performance of production workloads.

  or, please let me know if there are any existing solution in place.

  Thank you.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2146760/+subscriptions