[Bug 2119987] Re: haproxy reload triggers OOM-killer for TERMINATED_HTTPS loadbalancers
Launchpad Bug Tracker
2119987 at bugs.launchpad.net
Mon Feb 2 22:47:28 UTC 2026
** Merge proposal linked:
https://code.launchpad.net/~whershberger/ubuntu/+source/octavia/+git/octavia/+merge/499694
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2119987
Title:
haproxy reload triggers OOM-killer for TERMINATED_HTTPS loadbalancers
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive caracal series:
In Progress
Status in Ubuntu Cloud Archive epoxy series:
Fix Released
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in octavia:
Fix Released
Status in octavia package in Ubuntu:
Fix Released
Status in octavia source package in Noble:
In Progress
Status in octavia source package in Plucky:
Won't Fix
Status in octavia source package in Questing:
Fix Released
Bug description:
[ Impact ]
Creating a TERMINATED_HTTPS listener in an amphora with >=32GB of
memory triggers the OOM-killer during pool setup (and any subsequent
`systemctl reload` of haproxy in the amphora).
```
os loadbalancer listener create --name thttps_xxlarge --protocol TERMINATED_HTTPS --protocol-port 443 --default-tls-container-ref <URL> --wait xxlarge1
```
This was originally reported in a Caracal cloud using an Ubuntu 22.04
Amphora image.
I've been able to reproduce this reliably in my lab using the latest
devstack and an Ubuntu 24.04 Amphora image.
Workaround by setting a higher connection limit on one listener in
proportion to the 50000 default and the memory on the system. So for
an amphora with 32GiB of RAM, use --connection-limit 200000 for one
listener.
[ Test Plan ]
Deploy charmed Jammy/Caracal with Octavia and Barbican. For Noble
verification, DRU the Octavia unit to Noble before completing the rest
of the test plan.
From the machine hosting Octavia, create an ssh key and upload the public key to Openstack as `amphora-backdoor` (for shell access to the amphora):
$ os keypair create --public path/to/key amphora-backdoor
Get a compute flavor id to use for the Octavia flavorprofile:
$ openstack flavor create --ram 32768 --disk 10 --vcpus 4 --id 10 m1.xxlarge
+----------------------------+------------+
| Field | Value |
+----------------------------+------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| description | None |
| disk | 10 |
| id | 10 |
| name | m1.xxlarge |
| os-flavor-access:is_public | True |
| properties | |
| ram | 32768 |
| rxtx_factor | 1.0 |
| swap | 0 |
| vcpus | 4 |
+----------------------------+------------+
Create the flavor profile:
$ openstack loadbalancer flavorprofile create \
--name o1.xxlarge \
--provider amphora \
--flavor-data '{"compute_flavor": "10"}'
+---------------+--------------------------------------+
| Field | Value |
+---------------+--------------------------------------+
| id | f3aac848-7e77-449a-96af-bf0312c45ef9 |
| name | o1.xxlarge |
| provider_name | amphora |
| flavor_data | {"compute_flavor": "10"} |
+---------------+--------------------------------------+
And Amphora flavor:
$ openstack loadbalancer flavor create \
--name o1.xxlarge \
--flavorprofile o1.xxlarge \
--description "Extra large LB" \
--enable
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| id | 350f607c-9189-49ac-a585-84960322137a |
| name | o1.xxlarge |
| flavor_profile_id | f3aac848-7e77-449a-96af-bf0312c45ef9 |
| enabled | True |
| description | Extra large LB |
+-------------------+--------------------------------------+
For certificates I created a complete PKI following Jamie's Guide [1];
could use a self-signed cert. Attaching xxlarge.devstack.p12 for
convenience (as this test plan does not require the cert to function).
$ openssl pkcs12 -export \
-inkey xxlarge.devstack.key.pem \
-in xxlarge.devstack.cert.pem \
-certfile ca-chain.cert.pem \
-passout pass: -out xxlarge.devstack.p12
$ openstack secret store \
--name="xxlarge.devstack.p12" \
-t 'application/octet-stream' \
-e 'base64' \
--payload="$(base64 < xxlarge.devstack.p12)"
+---------------+-------------------------------------------------------------------+
| Field | Value |
+---------------+-------------------------------------------------------------------+
| Secret href | https://None:9312/v1/secrets/86f4113d-e9f6-43e3-aa10-459a6acc14b3 |
| Name | xxlarge.devstack.p12 |
| Created | None |
| Status | None |
| Content types | {'default': 'application/octet-stream'} |
| Algorithm | aes |
| Bit length | 256 |
| Secret type | opaque |
| Mode | cbc |
| Expiration | None |
+---------------+-------------------------------------------------------------------+
Create an LB (provisions the Amphora):
$ openstack loadbalancer create --wait \
--name lb1-xxlarge \
--vip-subnet-id ext_net_subnet \
--flavor o1.xxlarge
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone | None |
| created_at | 2026-02-02T22:16:05 |
| description | |
| flavor_id | 350f607c-9189-49ac-a585-84960322137a |
| id | 19d431d8-3ce7-4942-af7f-8fca6bd8e88c |
| listeners | |
| name | lb1-xxlarge |
| operating_status | ONLINE |
| pools | |
| project_id | 3ac0c8861bee423b9983a57f73bed0da |
| provider | amphora |
| provisioning_status | ACTIVE |
| updated_at | 2026-02-02T22:17:03 |
| vip_address | 10.149.109.90 |
| vip_network_id | adaf097d-c89a-4df4-b97a-4a09105c0c75 |
| vip_port_id | df4bd66e-a61b-4da4-810f-9e98112cb081 |
| vip_qos_policy_id | None |
| vip_subnet_id | 3a1853de-5caa-46ce-a5eb-7e2ca70898d3 |
| vip_vnic_type | normal |
| tags | |
| additional_vips | [] |
+---------------------+--------------------------------------+
In my test environment (using nested virt) the allocation of the
haproxy TLS session cache takes so long that the systemd unit times
out. To avoid this, ssh into the amphora and set
DefaultTimeoutStartSec=infinity and DefaultTimeoutStopSec=infinity in
/etc/systemd/system.conf:
Get the amphora's management IP (using your `service` project's UUID):
$ openstack server list --project 0086b84a8dcf44b5b425abbddbbbd8c3
+--------------------------------------+--------------------------------------------+--------+--------------------------------------------+----------------------------------------------+------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------------------------------------+--------+--------------------------------------------+----------------------------------------------+------------+
| 7d00e939-41e4-4017-9757-b7656961b114 | amphora-f9436cbb- | ACTIVE | ext_net=10.149.109.99; lb-mgmt-net=fc00:8c | amphora-haproxy-x86_64-ubuntu-22.04-20250702 | m1.xxlarge |
| | bfa8-41e0-b1fc-a4bd7855bf3d | | 8e:8682:b866:f816:3eff:fe15:4398 | | |
+--------------------------------------+--------------------------------------------+--------+--------------------------------------------+----------------------------------------------+------------+
Log in to the machine hosting the Octavia API services, SSH to the
amphora IP, modify the systemd config and reload it:
$ ssh -i .ssh/id_amphora ubuntu at fc00:8c8e:8682:b866:f816:3eff:fe15:4398
$ sudo sed -i 's/#DefaultTimeoutStartSec=90s/DefaultTimeoutStartSec=infinity/' /etc/systemd/system.conf
$ sudo sed -i 's/#DefaultTimeoutStopSec=90s/DefaultTimeoutStopSec=infinity/' /etc/systemd/system.conf
$ sudo systemctl daemon-reload
Create the listener (this configures haproxy in the Amphora):
$ openstack loadbalancer listener create --wait \
--name thttps \
--protocol TERMINATED_HTTPS \
--protocol-port 443 \
--default-tls-container-ref https://None:9312/v1/secrets/86f4113d-e9f6-43e3-aa10-459a6acc14b3 \
lb1-xxlarge
+-----------------------------+-----------------------------------------------------------------------------+
| Field | Value |
+-----------------------------+-----------------------------------------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2026-02-02T22:22:59 |
| default_pool_id | None |
| default_tls_container_ref | https://None:9312/v1/secrets/86f4113d-e9f6-43e3-aa10-459a6acc14b3 |
| description | |
| id | 7799303a-fb43-4e2f-b3f8-6f6e187f2002 |
| insert_headers | None |
| l7policies | |
| loadbalancers | 19d431d8-3ce7-4942-af7f-8fca6bd8e88c |
| name | thttps |
| operating_status | OFFLINE |
| project_id | 3ac0c8861bee423b9983a57f73bed0da |
| protocol | TERMINATED_HTTPS |
| protocol_port | 443 |
| provisioning_status | ACTIVE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | 2026-02-02T22:24:03 |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256: |
| | DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM- |
| | SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128- |
| | SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256 |
| tls_versions | ['TLSv1.2', 'TLSv1.3'] |
| alpn_protocols | ['h2', 'http/1.1', 'http/1.0'] |
| tags | |
| hsts_max_age | None |
| hsts_include_subdomains | False |
| hsts_preload | False |
+-----------------------------+-----------------------------------------------------------------------------+
And create a pool (triggers haproxy restart in the amphora):
$ openstack loadbalancer pool create \
--name xxlarge_p1 \
--lb-algorithm ROUND_ROBIN \
--listener thttps \
--protocol HTTP \
--wait
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2026-02-02T22:34:15 |
| description | |
| healthmonitor_id | |
| id | 6a76ea3a-c4eb-4f1f-9277-b3bf76247462 |
| lb_algorithm | ROUND_ROBIN |
| listeners | 7799303a-fb43-4e2f-b3f8-6f6e187f2002 |
| loadbalancers | 19d431d8-3ce7-4942-af7f-8fca6bd8e88c |
| members | |
| name | xxlarge_p1 |
| operating_status | OFFLINE |
| project_id | 3ac0c8861bee423b9983a57f73bed0da |
| protocol | HTTP |
| provisioning_status | ACTIVE |
| session_persistence | None |
| updated_at | 2026-02-02T22:35:02 |
| tls_container_ref | None |
| ca_tls_container_ref | None |
| crl_container_ref | None |
| tls_enabled | False |
| tls_ciphers | None |
| tls_versions | None |
| tags | |
| alpn_protocols | None |
+----------------------+--------------------------------------+
Expected behavior:
Checking the journal from the amphora's shell should show no OOM
Actual behavior:
Observe 1 or more OOM-killer invocations in the amphora. The process
list is likely to contain two haproxy processes with slightly more
than 16GB in total_vm:
Feb 02 22:34:24 amphora-f9436cbb-bfa8-41e0-b1fc-a4bd7855bf3d kernel: [ 2055] 65534 2055 4739383 2824 126976 0 0 haproxy
Feb 02 22:34:24 amphora-f9436cbb-bfa8-41e0-b1fc-a4bd7855bf3d kernel: [ 2074] 0 2074 4687328 3373674 27123712 0 0 haproxy
[1] https://jamielinux.com/docs/openssl-certificate-authority/introduction.html
[2] https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutSec=
[ Where problems could occur ]
The patch affects only the code that produces the `ssl.tune.cachesize`
haproxy configuration option. `ssl.tune.cachesize` is only included in
the haproxy configuration for listeners with protocol
TERMINATED_HTTPS.
If the change is wrong, we'd expect to see failures related memory
consumption in Amphora that host TERMINATED_HTTPS listeners; most
likely these would be OOM-killer invocations visible on the system
log, but issues could also manifest as haproxy failing as a result of
invalid configuration.
Under very specific circumstances (recurring SSL connections balanced
well with the existing `ssl.tune.cachesize` on a TERMINATED_HTTPS
loadbalancer) it's also possible that this change could moderately
regress performance. A performance regression of this nature can be
prevented by provisioning an Amphora with more available memory.
[ Root Cause ]
454cff5 (in Zed+ IIUC) introduces the use of haproxy's
`tune.ssl.cachesize` for TERMINATED_HTTPS listeners [1][2].
The commit does not make clear that during a reload of haproxy
(SIGUSR2), the old worker process stays running until the new worker
process is ready [3][4]. This means that two TLS session caches are
allocated/held simultaneously during a reload of the service [5].
For small Amphorae, this works fine. The default connection limit is
50000, which takes enough of a chunk out of the 50% allocation that
there is enough wiggle room for the new haproxy worker to allocate its
cache and coexist with the old worker for some time.
However, as the available memory in the system increases, the memory
consumed by the session cache approaches 50%, and increases the
worker's memory usage beyond 50% (as something else in the worker is
also using memory in proportion to the configured cachesize).
I tested 10 values of tune.ssl.cachesize in an amphora with 32GiB of
RAM, reloading the haproxy service each time:
- vsz here is the value reported by `ps -ax -o pid,vsz,rss,uss,pmem,args | grep haproxy`
- overhead is `tune.ssl.cachesize_MiB - vsz_MiB - 261`
- overhead% is `floor((overhead / tune.ssl.cachesize_MiB) * 100)`
tune.ssl.cachesize | tune.ssl.cachesize_MiB | vsz | vsz_MiB | overhead | overhead%
0 | 0 | 267416 | 261 | 0 | 0%
7741606 | 1476 | 2142472 | 2092 | 355 | 24%
15483212 | 2953 | 4017260 | 3923 | 709 | 24%
23224818 | 4429 | 5892180 | 5754 | 1064 | 24%
30966424 | 5906 | 7767100 | 7585 | 1418 | 24%
38708030 | 7382 | 9642020 | 9416 | 1773 | 24%
46449636 | 8859 | 11516940 | 11247 | 2127 | 24%
54191242 | 10336 | 13391860 | 13077 | 2480 | 23%
61932848 | 11812 | 15266780 | 14908 | 2835 | 24%
69674454 | 13289 | 17141700 | 16739 | 3189 | 23%
77416060 | 14765 | 19016744 | 18571 | 3545 | 24%
Note that this listener was not configured with a pool, so there was
no load on the system when I gathered this data.
As shown, haproxy to consumes additional memory proportional to the
size of the TLS session cache. The allocation for the cache occurs at
[6], referring to [7].
I verified the documentation's assertion that tune.ssl.cachesize is
200 bytes on amd64; sizeof(struct shared_block) is 48 bytes on the
same hardware [8].
Octavia should allocate closer to 1/3 than 1/2 for the TLS session
cache. I'll test and propose a patch against master shortly.
[1] https://opendev.org/openstack/octavia/commit/454cff587ed10b5e504da93b074b77cb85055b13
[2] https://www.haproxy.com/documentation/haproxy-configuration-manual/new/2-8r1/#section-3.2.-tunesslcachesize
[3] https://github.com/haproxy/haproxy/issues/217#issuecomment-544515990
[4] https://manpages.ubuntu.com/manpages/jammy/en/man1/haproxy.1.html
[5] https://opendev.org/openstack/octavia/src/branch/master/octavia/amphorae/backends/agent/api_server/templates/systemd.conf.j2
[6] https://git.launchpad.net/ubuntu/+source/haproxy/tree/src/ssl_sock.c?h=applied/ubuntu/noble-devel#n5346
[7] https://git.launchpad.net/ubuntu/+source/haproxy/tree/src/shctx.c?h=applied/ubuntu/noble-devel#n300
[8] https://git.launchpad.net/ubuntu/+source/haproxy/tree/include/haproxy/shctx-t.h?h=applied/ubuntu/noble-devel#n38
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2119987/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list