[Bug 2077348] Re: [SRU] Loadbalacer stuck in status PENDING_DELETE if TLS storage unavailable in cascade deletion

[Hua Zhang]

I don't think Zed will impact the upgrade because 1) this issue only
occurs when deleting a LB with cascade and ssl mode during upgrade
period 2) even if this issue does occur, it can be resolved
automatically upon upgrading from Zed to antelope. Considering that Zed
has been EOL, I will mark Zed as "won't fix".

** Changed in: cloud-archive/zed
       Status: New => Won't Fix

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2077348

Title:
  [SRU] Loadbalacer stuck in status PENDING_DELETE if TLS storage
  unavailable in cascade deletion

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  Fix Committed
Status in Ubuntu Cloud Archive bobcat series:
  Fix Committed
Status in Ubuntu Cloud Archive caracal series:
  Fix Committed
Status in Ubuntu Cloud Archive dalmatian series:
  Fix Released
Status in Ubuntu Cloud Archive epoxy series:
  Fix Released
Status in Ubuntu Cloud Archive yoga series:
  Fix Committed
Status in Ubuntu Cloud Archive zed series:
  Won't Fix
Status in octavia:
  Fix Released
Status in octavia package in Ubuntu:
  Fix Released
Status in octavia source package in Focal:
  Won't Fix
Status in octavia source package in Jammy:
  Fix Committed
Status in octavia source package in Noble:
  Fix Committed
Status in octavia source package in Oracular:
  Fix Released
Status in octavia source package in Plucky:
  Fix Released

Bug description:
  [Impact]

  Loadbalacer stuck in status PENDING_DELETE if TLS cert unavailable

  [Test Case]

  Pls refer to [Test steps] section below.

  [Regression Potential]

  The fix is already in the upstream main, stable/2024.1, stable/2023.2,
  stable/2023.1 branches, so it is a clean backport and might be helpful
  for deployments using octavia.

  I also test this fix, it works well -
  https://paste.ubuntu.com/p/s4MsMjV6mP/

  [Others]

  Original Bug Description Below
  ===========

  Loadbalacer stuck in status PENDING_DELETE if TLS cert unavailable

  1. Create load balancer with TERMINATED_HTTPS listener
  2. Disable your TLS storage, or delete cert from storage
  3. Try to delete loadbalancer with cascade flag

  Error on logs:

  ```
  Unable to retrieve certificate(s) due to Could not retrieve certificate: <some id>
  Exception during message handling
  ```

  ```
  Traceback (most recent call last):
  File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming, res = self.dispatcher.dispatch(message),
  File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch, return self._do_dispatch(endpoint, method, ctxt, args),
  File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch, result = func(ctxt, **new_args),
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/queue/v2/endpoints.py", line 56, in delete_load_balancer, self.worker.delete_load_balancer(loadbalancer, cascade),
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/worker/v2/controller_worker.py", line 387, in delete_load_balancer, listeners = flow_utils.get_listeners_on_lb(db_lb),
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/worker/v2/flows/flow_utils.py", line 52, in get_listeners_on_lb, prov_listener = provider_utils.db_listener_to_provider_listener(),
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 182, in db_listener_to_provider_listener, new_listener_dict = listener_dict_to_provider_dict(),
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 261, in listener_dict_to_provider_dict, with excutils.save_and_reraise_exception() as ctxt:,
  File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__, self.force_reraise(),
  File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise, raise self.value,
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 258, in listener_dict_to_provider_dict, cert_dict = cert_parser.load_certificates_data(cert_manager)
  File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/tls_utils/cert_parser.py", line 381, in load_certificates_data, raise exceptions.CertificateRetrievalException(, octavia.common.exceptions.CertificateRetrievalException: Could not retrieve certificate: ]

  ```

  [Test steps]

  1. Create load balancer with TERMINATED_HTTPS listener, eg:

  secret1_id=$(openstack secret store --name='lb_tls_secret_1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < www.server1.com.p12)" -f value -c "Secret href")
  octavia_user_id=$(openstack user show octavia --domain service_domain -f value -c id); echo $octavia_user_id;
  openstack acl user add -u $octavia_user_id $secret1_id
  subnetid=$(openstack subnet show private_subnet -f value -c id); echo $subnetid
  lb_id=$(openstack loadbalancer create --name lb1 --vip-subnet-id $subnetid -f value -c id); echo $lb_id
  listener_id=$(openstack loadbalancer listener create $lb_id --name https_listener --protocol-port 80 --protocol TERMINATED_HTTPS --default-tls-container=$secret1_id --sni-container-refs $secret1_id $secret2_id -f value -c id); echo $listener_id

  2. Disable your TLS storage, or delete cert from storage, eg:

  openstack secret delete $secret1_id

  3. Try to delete loadbalancer with cascade flag

  openstack loadbalancer delete lb1 --cascade

  Here are the detailed steps for me to reproduce the problem -
  https://paste.ubuntu.com/p/wh3dJpJR9B/

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2077348/+subscriptions