[Bug 2091071] Re: [OVN] Exception when deleting agent

Tue Mar 17 18:03:28 UTC 2026

Adding Dalmatian debdiff

** Patch added: "lp2091071-neutron-dalmatian.debdiff"
   https://bugs.launchpad.net/cloud-archive/+bug/2091071/+attachment/5953443/+files/lp2091071-neutron-dalmatian.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/2091071

Title:
  [OVN] Exception when deleting agent

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive dalmatian series:
  New
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  New
Status in neutron source package in Noble:
  New

Bug description:
  [ Impact ]

  In environments using OVN, the neutron-api service fails when
  attempting to delete an agent from the cache. This results in a
  TypeError: unhashable type: 'list' exception. This bug prevents the
  removal of decommissioned OVN agents. Users will see zombie agents in
  the openstack network agent list that cannot be cleared.

  The fix resolves the issue by passing the agent's ID directly instead
  of wrapping it in a list. The AgentCache().delete() method performs a
  dictionary lookup; because lists are mutable and unhashable in Python,
  they cannot be used as dictionary keys. This fix has already been
  accepted upstream [0].

  [ Test Plan ]

  To trigger the OVN agent cache deletion logic, we will decommission a
  compute node.

  ** Setup OpenStack Environment **

  1.  Deploy a test cloud in your reproducer bastion using stsstack-bundles:
      $ cd stsstack-bundles/openstack
      $ ./generate-bundle.sh --series <UBUNTU_RELEASE> --release <OPENSTACK_RELEASE> --name neutron-test --run
      # wait until all units are active/idle (some will be blocked/waiting)
      $ ./tools/vault-unseal-and-authorise.sh
      # wait until all units are active/idle
      $ ./configure
      $ source novarc

  2.  Add extra compute capacity to allow for decommissioning tests:
      $ juju add-unit nova-compute -n 2
      # wait until all units are active/idle

  ** Verify the Bug **

  3.  Decommission a compute unit:
      $ juju run nova-compute/2 disable 
      $ juju run nova-compute/2 remove-from-cloud 
      $ juju remove-unit nova-compute/2
      $ juju status nova-compute --watch 2s
      # wait until the unit has been removed

  4.  Attempt to delete the agent from Neutron:
      $ openstack network agent list
      $ openstack network agent delete {AGENT_ID}

  5.  Confirm the failure:
      The agent will still appear in the output of the following command despite the deletion attempt:
      $ openstack network agent list

  6.  Verify the traceback in the logs:
      On the neutron-api unit, check the neutron-server log:
      $ juju ssh neutron-api/0 "sudo cat /var/log/neutron/neutron-server.log | grep -B 15 TypeError | tail -n 9"
      Observed results:
      - Log shows: TypeError: unhashable type: 'list'

  ** Verify the Fix **

  7.  Install the patched package on the neutron-api unit:
      $ juju ssh neutron-api/0 
      # upgrade to patched neutron
      $ sudo apt update 
      $ sudo apt install neutron-common

  8.  Repeat the decommissioning process with a different unit:
      $ juju run nova-compute/1 disable
      $ juju run nova-compute/1 remove-from-cloud
      $ juju remove-unit nova-compute/1
      $ juju status nova-compute --watch 2s
      # wait until the unit has been removed

  9.  Delete the agent:
      $ openstack network agent list
      $ openstack network agent delete {AGENT_ID}

  10. Confirm the fix:
      Verify the agent is successfully removed:
      $ openstack network agent list
      Verify no new TypeError was observed during the removal process:
      $ juju ssh neutron-api/0 "sudo cat /var/log/neutron/neutron-server.log | grep -B 15 TypeError | tail -n 9"
      Observed results:
      - Agent is no longer present in the agent list
      - No new TypeErrors are generated in the log

  [ Where problems could occur ]

  The change modifies the input type of AgentCache.delete from a list to
  a single ID. If other parts of the code (outside this specific OVN
  logic) call this method and expect it to handle a list, those calls
  could fail or fail to find the correct key in the cache.

  If the external_ids['delete_agent'] value is somehow missing or
  malformed, the method would attempt to delete a non-existent key. This
  could lead to stale entries remaining in the agent cache.

  [ Other Info ]

  This bug was introduced in Bobcat with [1]. Therefore, releases before
  that (Yoga for the UCA and Jammy for Ubuntu in this case) do not need
  to be fixed.

  Packages in Questing and Resolute already have the fix. Similarly, in
  the UCA, Epoxy and Flamingo have the fix too.

  [0] - https://review.opendev.org/c/openstack/neutron/+/937092
  [1] - https://review.opendev.org/c/openstack/neutron/+/883607

  Original Description: Find the original description of the bug below:

  Call neutron API to delete ovn agent, it returns 204 but the agent
  isn't deleted. Check neutron-server.log find out there is an exception
  when deleting agent:

  ```
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: TypeError: unhashable type: 'list'
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event Traceback (most recent call last):
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event   File "/var/lib/kolla/venv/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event     match.run(event, row, updates)
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 330, in run
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event     n_agent.AgentCache().delete([row.external_ids['delete_agent']])
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py", line 282, in delete
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event     del self.agents[agent_id]
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event TypeError: unhashable type: 'list'
  2024-12-04 09:21:15.303 838 ERROR ovsdbapp.event
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: TypeError: unhashable type: 'list'
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event Traceback (most recent call last):
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event   File "/var/lib/kolla/venv/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event     match.run(event, row, updates)
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 330, in run
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event     n_agent.AgentCache().delete([row.external_ids['delete_agent']])
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py", line 282, in delete
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event     del self.agents[agent_id]
  2024-12-04 09:21:15.303 799 ERROR ovsdbapp.event TypeError: unhashable type: 'list'
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2091071/+subscriptions