[Bug 2103413] Re: Python3.13: Garbage collection happens too early on greenthread switches
Guillaume Boutry
2103413 at bugs.launchpad.net
Wed Apr 2 08:15:56 UTC 2025
After more investigation, I'm now certain the core of the issue is the
Garbage Collector freeing up object much before their last use.
Quickly setup a reproducer VM: https://pastebin.ubuntu.com/p/gzMwyCZSct/
`ppa:gboutry/plucky-proposed-openstack` is a ppa containing the RC1 packaging for most openstack projects, but the nova package is carrying the following patch to help track object finalization: https://pastebin.ubuntu.com/p/cj7tb3kmGV/
I have also reverse applied this patch out of bound to have the greenthread identity (the same error happens with and without this patch):
https://github.com/openstack/nova/commit/cd980cdd1ea957a9aa2220014046ea53079a623f
The carried patch for finalization tracing is tracing: https://opendev.org/openstack/nova/src/commit/98226b60f3fe7b20e8d7f208c12f8d0086cd83d0/nova/network/neutron.py#L1216
This object gets it's attributes freed up very early in the method
runtime. I can't pin point a precis LOC where this happens, as it
changes all the time. (meaning it's not a subsequent call that frees up
the internal objects).
Note that the `admin_client` object itself does NOT get freed, only its
inner objects.
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] /usr/lib/python3.13/weakref.py:590 - finalizer base_client
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] base_client - Monotonic delta, 0.158562s
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] base_client - end of finalizer
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - finalizer called: <Thread(Thread-1 (poll), started daemon 123734175578944)>
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - finalizer called: {123734291935488: <_MainThread(MainThread, started 123734291935488)>, 123734175578944: <Thread(Thread-1 (poll), started daemon 123734175578944)>, 123734174920256: <Thread(Rabbit-heartbeat, started daemon 123734174920256)>, 123734127568576: <Thread(Thread-2 (_native_thread), started daemon 123734127568576)>, 123734048372416: <Thread(tpool_thread_0, started daemon 123734048372416)>, 123734039979712: <Thread(tpool_thread_1, started daemon 123734039979712)>, 123734031587008: <Thread(tpool_thread_2, started daemon 123734031587008)>, 123734023194304: <Thread(tpool_thread_3, started daemon 123734023194304)>, 123734014801600: <Thread(tpool_thread_4, started daemon 123734014801600)>, 123734006408896: <Thread(tpool_thread_5, started daemon 123734006408896)>, 123733998016192: <Thread(tpool_thread_6, started daemon 123733998016192)>, 123733444392640: <Thread(tpool_thread_7, started daemon 123733444392640)>, 123733435999936: <Thread(tpool_thread_8, started daemon 123733435999936)>, 123733427607232: <Thread(tpool_thread_9, started daemon 123733427607232)>, 123733419214528: <Thread(tpool_thread_10, started daemon 123733419214528)>, 123733410821824: <Thread(tpool_thread_11, started daemon 123733410821824)>, 123733402429120: <Thread(tpool_thread_12, started daemon 123733402429120)>, 123733394036416: <Thread(tpool_thread_13, started daemon 123733394036416)>, 123732907521728: <Thread(tpool_thread_14, started daemon 123732907521728)>, 123732899129024: <Thread(tpool_thread_15, started daemon 123732899129024)>, 123732890736320: <Thread(tpool_thread_16, started daemon 123732890736320)>, 123732882343616: <Thread(tpool_thread_17, started daemon 123732882343616)>, 123732873950912: <Thread(tpool_thread_18, started daemon 123732873950912)>, 123732865558208: <Thread(tpool_thread_19, started daemon 123732865558208)>, 123734129856768: <Thread(Thread-3 (_runner), started daemon 123734129856768)>}
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] /usr/lib/python3.13/weakref.py:590 - finalizer httpclient
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - Monotonic delta, 0.171590s
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - end of finalizer
PS: `admin_client` is the object where this happens the most, but not at all the runs, some of the other failures, it's neutron_client (https://opendev.org/openstack/nova/src/commit/98226b60f3fe7b20e8d7f208c12f8d0086cd83d0/nova/network/neutron.py#L1212) that gets its internal object freed.
using `gc.disable()` makes the issue disappear (yay! disable gc!) or
actually holding a hardref to `admin_client.baseclient.httpclient` makes
the method pass most of the time...
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/2103413
Title:
Python3.13: Garbage collection happens too early on greenthread
switches
Status in OpenStack Compute (nova):
New
Status in nova package in Ubuntu:
New
Bug description:
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] Traceback (most recent call last):
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2648, in _build_and_run_instance
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] self.driver.spawn(context, instance, image_meta,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] injected_files, admin_password,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] allocs, network_info=network_info,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] block_device_info=block_device_info,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] accel_info=accel_info)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4832, in spawn
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] xml = self._get_guest_xml(context, instance, network_info,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] disk_info, image_meta,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] block_device_info=block_device_info,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] mdevs=mdevs, accel_info=accel_info)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7997, in _get_guest_xml
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] network_info_str = str(network_info)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/model.py", line 622, in __str__
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return self._sync_wrapper(fn, *args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/model.py", line 605, in _sync_wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] self.wait()
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/model.py", line 637, in wait
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] self[:] = self._gt.wait()
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 232, in wait
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return self._exit_event.wait()
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~~~~~^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/eventlet/event.py", line 131, in wait
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] current.throw(*self._exc)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 272, in main
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] result = function(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/utils.py", line 663, in context_wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return func(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2018, in _allocate_network_async
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] raise e
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1996, in _allocate_network_async
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] nwinfo = self.network_api.allocate_for_instance(
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] context, instance,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ...<3 lines>...
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] resource_provider_mapping=resource_provider_mapping,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] network_arqs=network_arqs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1287, in allocate_for_instance
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] nw_info = self.get_instance_nw_info(
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] context, instance, networks=ordered_nets,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] port_ids=ordered_port_ids,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] admin_client=admin_client,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] preexisting_port_ids=preexisting_port_ids)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 2047, in get_instance_nw_info
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] result = self._get_instance_nw_info(context, instance, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 2073, in _get_instance_nw_info
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] nw_info = self._build_network_info_model(context, instance, networks,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] port_ids, admin_client,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] preexisting_port_ids,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] refresh_vif_id,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] force_refresh=force_refresh)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 3496, in _build_network_info_model
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] data = client.list_ports(**search_opts)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ret = obj(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 815, in list_ports
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return self.list('ports', self.ports_path, retrieve_all,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] **_params)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ret = obj(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 372, in list
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] for r in self._pagination(collection, path, **params):
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 387, in _pagination
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] res = self.get(path, params=params)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ret = obj(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 356, in get
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return self.retry_request("GET", action, body=body,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] headers=headers, params=params)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ret = obj(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 333, in retry_request
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return self.do_request(method, action, body=body,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] headers=headers, params=params)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ret = obj(*args, **kwargs)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 284, in do_request
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] resp, replybody = self.httpclient.do_request(action, method, body=body,
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] headers=headers)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/client.py", line 341, in do_request
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] self._check_uri_length(url)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~~~~~~^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/client.py", line 334, in _check_uri_length
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] uri_len = len(self.endpoint_url) + len(url)
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/neutronclient/client.py", line 348, in endpoint_url
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] return self.get_endpoint()
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ~~~~~~~~~~~~~~~~~^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 330, in get_endpoint
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] if self.endpoint_override:
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] ^^^^^^^^^^^^^^^^^^^^^^
2025-03-17 11:08:00.390 38463 ERROR nova.compute.manager [instance: 5c6786d9-15c5-414f-990f-eff2bf9f4d1f] AttributeError: 'SessionClient' object has no attribute 'endpoint_override'
This using Plucky Proposed - 25.04
nova-compute: 3:30.0.0+git2025030311.e6b8b051a9-0ubuntu1
This nova compute version is a snapshot of current master branch from
the Nova project, RC1 will soon be available:
https://review.opendev.org/c/openstack/releases/+/943941
This happens during testing with Tempest, in tests such as:
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
Nova should successfully create a VM
Nova failed to communicate with neutron because of an attribute error
I've tested the same version of Nova using the Epoxy Cloud archive
from proposed, and it worked correctly.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2103413/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list