[Bug 2067441] Re: [SRU] Loadbalancer is stuck with PENDING_UPDATE state on member update API

[Edward Hope-Morley]

It seems the target for Bobcat was skipped (see request in comment #12)
so I have added it. We need to make sure that fix is released in order
i.e. Dalmation, Caracal, Bobcat, Antelope etc. For releases that already
have the fix it helps if you can mark the target as Fix Released so that
it is clear we don't need to check that release.

** Also affects: cloud-archive/bobcat
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/zed
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to octavia in Ubuntu.
https://bugs.launchpad.net/bugs/2067441

Title:
  [SRU] Loadbalancer is stuck with PENDING_UPDATE state on member update
  API

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  Fix Committed
Status in Ubuntu Cloud Archive bobcat series:
  New
Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive yoga series:
  New
Status in Ubuntu Cloud Archive zed series:
  New
Status in octavia:
  In Progress
Status in octavia package in Ubuntu:
  Fix Released
Status in octavia source package in Jammy:
  In Progress
Status in octavia source package in Mantic:
  Won't Fix
Status in octavia source package in Noble:
  In Progress
Status in octavia source package in Oracular:
  Fix Released

Bug description:
  [Impact]

  Loadbalancer is stuck with PENDING_UPDATE state on batch member update
  API.

  [Test Case]

  Please refer to [Test steps] section below.

  [Regression Potential]

  The fix is already in the upstream main, stable/2024.1, stable/2023.2,
  stable/2023.1 branches, so it is a clean backport and might be helpful
  for deployments using octavia.

  I also test this fix, it works well -
  https://paste.ubuntu.com/p/wPy7pB3SR6/  and
  https://paste.ubuntu.com/p/zpPDScQCtK/

  and I also test debdiff for this fix, it works well -
  https://paste.ubuntu.com/p/nS6c3QYRGn/

  
  [Others]

  Original Bug Description Below
  ===========

  By mistake, I sent wrong request with duplicated ip, port compbination through the Batch Update Members API(ver 2023.1).
  https://docs.openstack.org/api-ref/load-balancer/v2/#batch-update-members

  For example :
  192.0.2.16:80 Member already exists, and request data like follows
  {
      "members": [
          {
              "subnet_id": "xxxxxxx",
              "address": "192.0.2.16",
              "protocol_port": 80
          }, {
              "subnet_id": "xxxxxxx",
              "address": "192.0.2.16",
              "protocol_port": 80
          }
      ]
  }

  After the request, the status of Loadbalancer does not change from
  PENDING_UPDATE.

  When checking the source code, there is no logic to check for
  duplicates.

  In the controller logic(member.py), members are classified into
  new_members/updated_members/deleted_member, but the updated_members
  data is being passed as is with duplicates, so this is suspected to be
  the cause of the problem.

  ## log : 33fe25ab-5477-4787-a8e1-f657376b0ead is duplicated
  May 29 04:14:32 ubuntu octavia-worker[123317]: INFO octavia.controller.queue.v2.endpoints [-] Batch updating members: old='[]', new='[]', updated='['825dbebc-da79-4f88-bf48-0e3e63a09d90', '33fe25ab-5477-4787-a8e1-f657376b0ead', '33fe25ab-5477-4787-a8e1-f657376b0ead']'...
  May 29 04:14:32 ubuntu octavia-worker[123317]: ERROR oslo_messaging.rpc.server [-] Exception during message handling: taskflow.exceptions.Duplicate: Atoms with duplicate names found: ['octavia-mark-member-active-indb-33fe25ab-5477-4787-a8e1-f657376b0ead']

  FYI, There is validation logic for new_members.

  [Test steps]

  1, set up a openstack env with octavia deployment

  2, create a test lb

  3, add a member into lb pool

  openstack loadbalancer member create --subnet-id private_subnet --address 192.168.21.226 --protocol-port 80 lb1-pool
  $ openstack loadbalancer member list lb1-pool |grep ACTIVE
  | b36bb21e-8eed-40bc-a1cb-e69da070c0b9 | | 4f1016d73ae245fe8c5c6a637930f3d2 | ACTIVE | 192.168.21.226 | 80 | ONLINE | 1 |

  3, run test.py (https://paste.ubuntu.com/p/38vPW5R5S8/) to call batch
  member update API to add the same member (eg: 192.168.21.226 above)

  4, then we will reproduce the problem, lb will be stuck with
  PENDING_UPDATE state.

  $ openstack loadbalancer member list lb1-pool |grep 192
  | b36bb21e-8eed-40bc-a1cb-e69da070c0b9 | | 4f1016d73ae245fe8c5c6a637930f3d2 | PENDING_UPDATE | 192.168.21.226 | 80 | ONLINE | 40 |

  5, This is error log I saw - https://paste.ubuntu.com/p/K5s7knNmWw/

  [Some Analyses]

  You can see some analysis from the bugs I created earlier -
  https://bugs.launchpad.net/octavia/+bug/2070348

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2067441/+subscriptions