[Bug 2097539] Re: Heat stack updates fail when "reauthentication_auth_method=trusts"
Alejandro Santoyo Gonzalez
2097539 at bugs.launchpad.net
Wed Jan 21 10:06:33 UTC 2026
Marking the intermediate (Openstack/UCA) releases as Won't Fix as they
are EOL.
I also tested upgrades to verify the patch does not affect them and
found no issues (e.g., went from Caracal to Dalmatian and to Epoxy,
tested stack creation and updates, no issues). We should be good to
release to Yoga and Caracal.
** Changed in: cloud-archive/zed
Status: New => Won't Fix
** Changed in: cloud-archive/epoxy
Status: New => Won't Fix
** Changed in: cloud-archive/dalmatian
Status: New => Won't Fix
** Changed in: cloud-archive/bobcat
Status: New => Won't Fix
** Changed in: cloud-archive/antelope
Status: New => Won't Fix
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2097539
Title:
Heat stack updates fail when "reauthentication_auth_method=trusts"
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
Won't Fix
Status in Ubuntu Cloud Archive bobcat series:
Won't Fix
Status in Ubuntu Cloud Archive caracal series:
New
Status in Ubuntu Cloud Archive dalmatian series:
Won't Fix
Status in Ubuntu Cloud Archive epoxy series:
Won't Fix
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
New
Status in Ubuntu Cloud Archive zed series:
Won't Fix
Status in heat package in Ubuntu:
Fix Released
Status in heat source package in Jammy:
Fix Committed
Status in heat source package in Noble:
Fix Committed
Status in heat source package in Plucky:
Won't Fix
Status in heat source package in Questing:
Fix Released
Bug description:
[ Impact ]
* This bug causes a DB column to be wrongly set to NULL, rendering
any subsequent stack update actions impossible. This effectively
renders Heat unusable for scenarios where nested stacks are used
and "reauthentication_auth_method=trusts" is required.
* The fix ensures that the related DB column is correctly set with
the value from the parent stack.
[ Test Plan ]
1) Set up an Openstack cloud with Heat and set up a test user
2) Log in as admin or any other user able to do stack operations:
$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_AUTH_URL=https://<ip>:5000/v3
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=<some password>
OS_IDENTITY_API_VERSION=3
$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 95108ce766424c7aa92b20c92dd61e8f | admin |
| 415a1d3f6d324187a15391c8205ce131 | demo |
| 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
| c7ac272709164a79b2b1833db40696da | testadmin |
+----------------------------------+------------+
3) Create a stack with userfrom step 2
$ openstack stack create -t ./stack.yaml teststack --wait
2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
4) Update the stack, this would work:
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | 2025-02-06T11:40:20Z |
| stack_status | UPDATE_COMPLETE |
| stack_status_reason | Stack UPDATE completed successfully |
+---------------------+--------------------------------------+
5) Change users and re-run the stack update:
$ export OS_USERNAME=testadmin; export OS_PASSWORD=<somepassword>
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
Stack teststack UPDATE_FAILED
Notice below the simple test stacks I used:
stack.yaml:
heat_template_version: 2021-04-16
resources:
test_res:
type: OS::Heat::ResourceGroup
#update_policy:
#batch_create:
# max_batch_size: 1
#rolling_update:
# max_batch_size: 1
properties:
count: 2
resource_def:
type: nestedstack.yaml
properties:
index_name: "%index%"
nestedstack.yaml:
heat_template_version: 2015-04-30
parameters:
index_name:
type: string
description: ""
resources:
test-user:
type: OS::Keystone::User
properties:
default_project: "admin"
domain: admin_domain
enabled: True
name: { list_join : ["",["test", { get_param: index_name }]] }
password: "test"
[ Where problems could occur ]
* If the creds from the parent stack cannot be read for any particular
reason then the related DB would still be set to NULL. I have not found
any scenario were this would happend though.
* There may be performance implications for very complex and heavily
nested stack deployments as we need to retrieve the parent stack data
every time which is somewhat costly.
[ Other Info ]
When a stack is created with a certain user,
"reauthentication_auth_method=trusts" and an update is triggered, the
update will fail with the below error:
$ openstack stack resource list -n 5 teststack
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
| 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
| 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
| test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
| test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
The above happens because the user_creds_id column in the stack table
of the Heat DB is set to NULL for all nested resources (butnot the
main stack):
mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| id | name | action | status | user_creds_id |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
| 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
| 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
| 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
I would argue that:
- if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
- the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
Versions affected:
------------------
Openstack Yoga (no other versions were tested)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2097539/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list