[Bug 2097539] Re: Heat stack updates fail when "reauthentication_auth_method=trusts"
Matthew Ruffell
2097539 at bugs.launchpad.net
Fri Dec 12 03:25:30 UTC 2025
Sponsored for plucky, thanks.
The only (minor) nitpick is I changed your buglink from the full URL to
the shortened URL and rebased correctly on the OpenStack git repo.
I split the commits out in the way the OpenStack Team likes them, and
used the correct release "plucky" vs "noble-epoxy". "noble-epoxy" isn't
wrong, but it isn't right either. While plucky is still supported, it
should be plucky, and the openstack robot takes care of epoxy. Once
plucky goes EOL though, thats when we need to use "noble-epoxy".
Git repo pushed:
https://code.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/heat/+git/heat/+ref/stable/2025.1
All uploaded for plucky now.
Uploading heat_24.0.0-0ubuntu1.1.dsc
Uploading heat_24.0.0-0ubuntu1.1.debian.tar.xz
Uploading heat_24.0.0-0ubuntu1.1_source.buildinfo
Uploading heat_24.0.0-0ubuntu1.1_source.changes
BTW, the OpenStack robot will automatically make noble-epoxy once this
reaches plucky-proposed.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2097539
Title:
Heat stack updates fail when "reauthentication_auth_method=trusts"
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
New
Status in Ubuntu Cloud Archive bobcat series:
New
Status in Ubuntu Cloud Archive caracal series:
New
Status in Ubuntu Cloud Archive dalmatian series:
New
Status in Ubuntu Cloud Archive epoxy series:
New
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
New
Status in Ubuntu Cloud Archive zed series:
New
Status in heat package in Ubuntu:
Fix Released
Status in heat source package in Jammy:
In Progress
Status in heat source package in Noble:
In Progress
Status in heat source package in Plucky:
In Progress
Status in heat source package in Questing:
Fix Released
Bug description:
[ Impact ]
* This bug causes a DB column to be wrongly set to NULL, rendering
any subsequent stack update actions impossible. This effectively
renders Heat unusable for scenarios where nested stacks are used
and "reauthentication_auth_method=trusts" is required.
* The fix ensures that the related DB column is correctly set with
the value from the parent stack.
[ Test Plan ]
1) Set up an Openstack cloud with Heat and set up a test user
2) Log in as admin or any other user able to do stack operations:
$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_AUTH_URL=https://<ip>:5000/v3
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=<some password>
OS_IDENTITY_API_VERSION=3
$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 95108ce766424c7aa92b20c92dd61e8f | admin |
| 415a1d3f6d324187a15391c8205ce131 | demo |
| 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
| c7ac272709164a79b2b1833db40696da | testadmin |
+----------------------------------+------------+
3) Create a stack with userfrom step 2
$ openstack stack create -t ./stack.yaml teststack --wait
2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
4) Update the stack, this would work:
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | 2025-02-06T11:40:20Z |
| stack_status | UPDATE_COMPLETE |
| stack_status_reason | Stack UPDATE completed successfully |
+---------------------+--------------------------------------+
5) Change users and re-run the stack update:
$ export OS_USERNAME=testadmin; export OS_PASSWORD=<somepassword>
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
Stack teststack UPDATE_FAILED
Notice below the simple test stacks I used:
stack.yaml:
heat_template_version: 2021-04-16
resources:
test_res:
type: OS::Heat::ResourceGroup
#update_policy:
#batch_create:
# max_batch_size: 1
#rolling_update:
# max_batch_size: 1
properties:
count: 2
resource_def:
type: nestedstack.yaml
properties:
index_name: "%index%"
nestedstack.yaml:
heat_template_version: 2015-04-30
parameters:
index_name:
type: string
description: ""
resources:
test-user:
type: OS::Keystone::User
properties:
default_project: "admin"
domain: admin_domain
enabled: True
name: { list_join : ["",["test", { get_param: index_name }]] }
password: "test"
[ Where problems could occur ]
* If the creds from the parent stack cannot be read for any particular
reason then the related DB would still be set to NULL. I have not found
any scenario were this would happend though.
* There may be performance implications for very complex and heavily
nested stack deployments as we need to retrieve the parent stack data
every time which is somewhat costly.
[ Other Info ]
When a stack is created with a certain user,
"reauthentication_auth_method=trusts" and an update is triggered, the
update will fail with the below error:
$ openstack stack resource list -n 5 teststack
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
| 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
| 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
| test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
| test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
The above happens because the user_creds_id column in the stack table
of the Heat DB is set to NULL for all nested resources (butnot the
main stack):
mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| id | name | action | status | user_creds_id |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
| 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
| 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
| 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
I would argue that:
- if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
- the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
Versions affected:
------------------
Openstack Yoga (no other versions were tested)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2097539/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list