[Bug 2095364] Re: List server API v2.96 returns 500 if there is missing request spec object
Wesley Hershberger
2095364 at bugs.launchpad.net
Wed Oct 22 19:53:17 UTC 2025
### Verification Done Caracal ###
wesley at j-caracal:~/regress-stack$ apt-cache policy python3-nova
python3-nova:
Installed: 3:29.2.0-0ubuntu1.2~cloud0
Candidate: 3:29.2.0-0ubuntu1.2~cloud0
Version table:
*** 3:29.2.0-0ubuntu1.2~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-proposed/caracal/main amd64 Packages
100 /var/lib/dpkg/status
3:29.2.0-0ubuntu1.1~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/caracal/main amd64 Packages
3:25.2.1-0ubuntu2.9 500
500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
3:25.2.1-0ubuntu2.6 500
500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
3:25.0.0-0ubuntu1 500
500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
wesley at j-caracal:~/regress-stack$ . ~/auth.rc
wesley at j-caracal:~/regress-stack$ os endpoint list
+----------------------------------+------------+--------------+--------------+---------+-----------+------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+------------+--------------+--------------+---------+-----------+------------------------------------------+
| 0596d5aea5684f238fdf64f40aeeec18 | AutoPkgOne | glance | image | True | public | http://10.148.1.7:9292/ |
| 07faa334e4d04703a1389673fb83d69f | AutoPkgOne | glance | image | True | admin | http://10.148.1.7:9292/ |
| 15ea8a207610411cb8f384053eafa692 | AutoPkgOne | nova | compute | True | admin | http://10.148.1.7:8774/v2.1 |
| 1601346eb9c041f1994ee20b3b8d4960 | AutoPkgOne | nova | compute | True | internal | http://10.148.1.7:8774/v2.1 |
| 17db39c64c28420787ea62ac5e5c698b | AutoPkgOne | cinder | volumev3 | True | public | http://10.148.1.7:8776/v3/%(project_id)s |
| 19edf588959a475ea4801c4856f0a863 | AutoPkgOne | keystone | identity | True | public | http://10.148.1.7:5000/v3/ |
| 28288ac4a9504672b0d648025e844ba7 | AutoPkgOne | placement | placement | True | public | http://10.148.1.7:8778/ |
| 30d51ceede3d4db7bd62a15bd9ef082f | AutoPkgOne | nova | compute | True | public | http://10.148.1.7:8774/v2.1 |
| 3d6d6ce0263a493bbe6196a7119bd7f0 | AutoPkgOne | keystone | identity | True | admin | http://10.148.1.7:5000/v3/ |
| 4056042131e441cbabd37d6c2df3b7b0 | AutoPkgOne | cinder | volumev3 | True | admin | http://10.148.1.7:8776/v3/%(project_id)s |
| 5184d5d31ce04a35920c031e162cc564 | AutoPkgOne | placement | placement | True | admin | http://10.148.1.7:8778/ |
| 62b1def1d6344390b4f3cd720ea10c3d | AutoPkgOne | cinder | volumev3 | True | internal | http://10.148.1.7:8776/v3/%(project_id)s |
| 84f8d69ba440495b94f01834d5b1b104 | AutoPkgOne | neutron | network | True | public | http://10.148.1.7:9696/ |
| 890c2dc2e5ed41af9cec53fc880eceb6 | AutoPkgOne | glance | image | True | internal | http://10.148.1.7:9292/ |
| 98155d1ff03e403090b0756b4f023a96 | AutoPkgOne | neutron | network | True | admin | http://10.148.1.7:9696/ |
| c71df94f623547158ae7d64c66e4e492 | AutoPkgOne | neutron | network | True | internal | http://10.148.1.7:9696/ |
| e1b67c952c2b4a40ad526e4573c421d9 | AutoPkgOne | placement | placement | True | internal | http://10.148.1.7:8778/ |
| f51764fe3bcc472a8174ad11c9185b51 | AutoPkgOne | keystone | identity | True | internal | http://10.148.1.7:5000/v3/ |
+----------------------------------+------------+--------------+--------------+---------+-----------+------------------------------------------+
wesley at j-caracal:~/regress-stack$ openstack server create --image ubuntu-jammy --flavor m1.small --network private-network demo-test1
+--------------------------------------+-----------------------------------------------------+
| Field | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | 8xuzc6AfX93w |
| config_drive | |
| created | 2025-10-22T19:50:56Z |
| flavor | m1.small (916cd4b1-a617-46b8-a830-30aa6d852cf8) |
| hostId | |
| id | d1f7711b-4170-48ff-9295-15d431d422de |
| image | ubuntu-jammy (2394208d-8841-4b42-aae1-2338577c9221) |
| key_name | None |
| name | demo-test1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | f142b20f722649c282dac4259ec76608 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2025-10-22T19:50:56Z |
| user_id | f02f266c2bc346ddab6fa567ed7cf698 |
+--------------------------------------+-----------------------------------------------------+
wesley at j-caracal:~/regress-stack$ sudo mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 316
Server version: 8.0.43-0ubuntu0.22.04.2 (Ubuntu)
Copyright (c) 2000, 2025, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input
statement.
mysql> use nova_api;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> delete from request_specs where instance_uuid = "d1f7711b-4170-48ff-9295-15d431d422de";
Query OK, 1 row affected (0.09 sec)
mysql> exit
Bye
wesley at j-caracal:~/regress-stack$ openstack --os-compute-api-version 2.96 server list --long
+-------------------+------------+--------+------------+-------------+-------------------+--------------+-------------------+----------+-------------------+---------------+------------+-------------+
| ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Flavor | Availability Zone | Host | Properties | Host Status |
+-------------------+------------+--------+------------+-------------+-------------------+--------------+-------------------+----------+-------------------+---------------+------------+-------------+
| d1f7711b-4170- | demo-test1 | ACTIVE | None | Running | private-network=1 | ubuntu-jammy | 2394208d-8841- | m1.small | nova | j-caracal.lxd | | UP |
| 48ff-9295- | | | | | 92.168.133.157 | | 4b42-aae1- | | | | | |
| 15d431d422de | | | | | | | 2338577c9221 | | | | | |
+-------------------+------------+--------+------------+-------------+-------------------+--------------+-------------------+----------+-------------------+---------------+------------+-------------+
### Verification Done Caracal ###
** Tags removed: verification-caracal-needed verification-needed
** Tags added: verification-caracal-done verification-done
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2095364
Title:
List server API v2.96 returns 500 if there is missing request spec
object
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive caracal series:
Fix Committed
Status in Ubuntu Cloud Archive dalmatian series:
Won't Fix
Status in Ubuntu Cloud Archive epoxy series:
Fix Committed
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) 2024.1 series:
Fix Released
Status in OpenStack Compute (nova) 2024.2 series:
Fix Released
Status in OpenStack Compute (nova) 2025.1 series:
Fix Released
Status in OpenStack Compute (nova) 2025.2 series:
Fix Released
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Noble:
Fix Committed
Status in nova source package in Plucky:
Fix Committed
Bug description:
[ Impact ]
The user who reported this in Caracal is experiencing intermittent VM build
failures in their environment; depending on when the build fails, Nova may not
have created the VM's record in the nova_api:request_specs table yet, leading
to the failure when the user requests details about the VM.
We do not have a consistent organic reproducer that produces the missing record
during machine build. However, it's possible to remove the record from the
database for an existing server to reproduce the failure.
As noted in the upstream commit and original bug description, Nova should be
able to handle the missing record [1].
I've included the related unit tests in the patch and verified that the patch
fixes the issue in Noble & Plucky using regress-stack.
[1]
https://opendev.org/openstack/nova/commit/509820f156e339e825f799d84b06cb11ac6b6096
[ Test Plan ]
Run the test plan in a VM with sufficient disk (25GiB+) and RAM
(4GiB+).
For plucky, apply the following commit to regress-stack:
https://github.com/MggMuggins/regress-
stack/commit/748cb81fae39796fc90ca6a6d4427eb6eae4a0c0
Deploy regress-stack and set up a playground environment:
```
sudo snap install openstackclients --channel=2024.1/stable
sudo snap install astral-uv --classic
sudo apt-get install --yes dpkg-dev python3-dev python-apt-dev
git clone https://github.com/canonical/regress-stack.git
cd regress-stack
uv sync
sudo apt-get install --yes $(uv run regress-stack packages nova)
sudo uv run regress-stack setup
# Grab the openrc since regress-stack was set up as root
sudo cp /root/auth.rc ~/
sudo chown $(id -u):$(id -g) ~/auth.rc
uv run regress-stack playground
```
```
. ~/auth.rc
openstack server create --image ubuntu-{noble,plucky} --flavor m1.small --network private-network demo-test1
```
```
sudo mysql -u root
use nova_api;
delete from request_specs where instance_uuid = "<UUID>";
exit
```
Actual behavior:
```
$ openstack --os-compute-api-version 2.96 server list --long
HttpException: 500: Server Error for url: http://10.148.1.2:8774/v2.1/servers/detail?deleted=False, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
```
Expected behavior:
```
$ openstack --os-compute-api-version 2.96 server list --long
+-------------------+------------+--------+------------+-------------+-------------------+---------------+-------------------+----------+-------------------+--------+------------+-------------+
| ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Flavor | Availability Zone | Host | Properties | Host Status |
+-------------------+------------+--------+------------+-------------+-------------------+---------------+-------------------+----------+-------------------+--------+------------+-------------+
| 24ef1693-2375- | demo-test1 | ACTIVE | None | Running | private-network=1 | ubuntu-plucky | 1348517e-a509- | m1.small | nova | p0.lxd | | UP |
| 4061-98ec- | | | | | 92.168.133.201 | | 47f2-91e2- | | | | | |
| 106cb4f7d877 | | | | | | | e26d3064f549 | | | | | |
+-------------------+------------+--------+------------+-------------+-------------------+---------------+-------------------+----------+-------------------+--------+------------+-------------+
```
[ Where problems could occur ]
The fix for this issue touches code that is only run when handling requests to
the /servers/detail endpoint. Failures in this code would show up when directly
querying that endpoint or using the `--long` option of the openstack CLI.
[ Original Description ]
The List Server API and List Server detail API with v2.96 microversion
hits 500 error if one or more instance's request spec object don't
exist by any reason.
The error reason is v2.96 and later version requires the request_spec
information for the requested_az information, but the latest list view
builder code assumes all request_spec objects for all instances exist
correctly.
## Stacktrace
2025-01-16T06:39:41.756 10 ERROR nova.api.openstack.wsgi [F] [None req-0605cf7e-09db-4ba1-9914-68d34d935587 9d5b1aa120675648abcb2cb156f8020b 792bb91fb3a254b4b5cbf61bed57854e - - default default] Unexpected exception in API method: KeyError: '5d9070aa-33da-46bc-94bc-604746ecfbd2'
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] Traceback (most recent call last):
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/openstack/wsgi.py", line 665, in wrapped
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] return f(*args, **kwargs)
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/validation/__init__.py", line 195, in wrapper
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] return func(*args, **kwargs)
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/validation/__init__.py", line 195, in wrapper
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] return func(*args, **kwargs)
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/validation/__init__.py", line 195, in wrapper
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] return func(*args, **kwargs)
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] [Previous line repeated 2 more times]
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 146, in detail
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] servers = self._get_servers(req, is_detail=True)
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 360, in _get_servers
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] response = self._view_builder.detail(
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] servers_dict = self._list_view(self.show, request, instances,
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 529, in _list_view
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] server_list = [func(request, server,
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] File "/opt/nova/lib64/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in <listcomp>
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] provided_az=req_specs_dict[server.uuid])["server"]
2025-01-16T06:39:41.756 10 EXCEPTION nova.api.openstack.wsgi [P] [-] KeyError: '5d9070aa-33da-46bc-94bc-604746ecfbd2'
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2095364/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list