[Bug 2146760] Re: Severe performance regression in OpenSSL 3.0.13 causes ~300ms TLS handshake latency on Ubuntu 24.04 LTS

[Matthew Ruffell]

Hi Prabu,

Do you happen to be running in FIPS mode by any chance?

Or are you using the latest 3.0.13-0ubuntu3.7 package in noble
-security/-updates?

Thanks,
Matthew

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/2146760

Title:
  Severe performance regression in OpenSSL 3.0.13 causes ~300ms TLS
  handshake latency on Ubuntu 24.04 LTS

Status in openssl package in Ubuntu:
  New

Bug description:
  Hello Ubuntu Security & OpenSSL Team,

  We have identified a critical performance regression in the openssl
  package (OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan
  2024)) provided with the official Ubuntu 24.04 LTS cloud image on AWS.

  When an application creates a new, secure (TLS) connection to the AWS
  RDS instance hosted in the same VPC & Subnets, there is a ~300ms stall
  during the initial handshake. This makes the official Ubuntu AMI
  unsuitable for any production workload that involves high-frequency,
  secure connections.

  We have confirmed this issue is resolved by manually compiling and
  installing OpenSSL 3.3.6, which points to a specific performance bug
  in the 3.0.13 version shipped with Ubuntu 24.04.

  Below are the details:

  OS: Ubuntu 24.04 LTS
  AMI: ami-01f79b1e4a5c64257
  Instance Type: m5.2xlarge
  Region: eu-central-1

  [Steps to Reproduce]

  1. Launch a standard Ubuntu 24.04 LTS instance on AWS.

  2. Install the MySQL client and mysqlslap tool: sudo apt-get install
  mysql-client.

  Run a mysqlslap benchmark against any TLS-enabled MySQL 8 server.
  time mysqlslap --host=[DB_HOST] --user=[USER] -p --ssl-mode=REQUIRED --query="SELECT 1" --iterations=10 --concurrency=10

  Actual Result (With Official OpenSSL 3.0.13):

  A severe ~300ms+ latency is observed. The benchmark shows:

  Benchmark
          Average number of seconds to run all queries: 0.340 seconds
          Minimum number of seconds to run all queries: 0.338 seconds
          Maximum number of seconds to run all queries: 0.345 seconds
          Number of clients running queries: 10
          Average number of queries per client: 1

  
  real    0m5.959s
  user    0m6.549s
  sys     0m0.220s

  Proof of Resolution (With Manual OpenSSL 3.3.6 Upgrade)

  After manually compiling and installing OpenSSL 3.3.6 on the same
  instance, the exact same benchmark yields excellent results,
  confirming the bug is within OpenSSL itself.

  Benchmark
          Average number of seconds to run all queries: 0.026 seconds
          Minimum number of seconds to run all queries: 0.024 seconds
          Maximum number of seconds to run all queries: 0.029 seconds
          Number of clients running queries: 10
          Average number of queries per client: 1

  
  real    0m3.746s
  user    0m0.263s
  sys     0m0.063s

  This performance bug severely impacts the usability of Ubuntu 24.04
  LTS for production our applications. We kindly request that this
  performance issue be investigated and that a fix be backported to the
  official OpenSSL package for Ubuntu 24.04 LTS. Ensuring a stable and
  optimized OpenSSL version is critical for maintaining the reliability
  and performance of production workloads.

  or, please let me know if there are any existing solution in place.

  Thank you.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2146760/+subscriptions