[Bug 2097539] Re: Heat stack updates fail when "reauthentication_auth_method=trusts"
Timo Aaltonen
2097539 at bugs.launchpad.net
Fri Dec 12 09:28:26 UTC 2025
Hello Alejandro, or anyone else affected,
Accepted heat into plucky-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/heat/1:24.0.0-0ubuntu1.1 in a few
hours, and then in the -proposed repository.
Please help us by testing this new package. See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed. Your feedback will aid us getting this
update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
plucky to verification-done-plucky. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-plucky. In either case, without details of your testing we will
not be able to proceed.
Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance for helping!
N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.
** Changed in: heat (Ubuntu Plucky)
Status: In Progress => Fix Committed
** Tags added: verification-needed verification-needed-plucky
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2097539
Title:
Heat stack updates fail when "reauthentication_auth_method=trusts"
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
New
Status in Ubuntu Cloud Archive bobcat series:
New
Status in Ubuntu Cloud Archive caracal series:
New
Status in Ubuntu Cloud Archive dalmatian series:
New
Status in Ubuntu Cloud Archive epoxy series:
New
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
New
Status in Ubuntu Cloud Archive zed series:
New
Status in heat package in Ubuntu:
Fix Released
Status in heat source package in Jammy:
In Progress
Status in heat source package in Noble:
In Progress
Status in heat source package in Plucky:
Fix Committed
Status in heat source package in Questing:
Fix Released
Bug description:
[ Impact ]
* This bug causes a DB column to be wrongly set to NULL, rendering
any subsequent stack update actions impossible. This effectively
renders Heat unusable for scenarios where nested stacks are used
and "reauthentication_auth_method=trusts" is required.
* The fix ensures that the related DB column is correctly set with
the value from the parent stack.
[ Test Plan ]
1) Set up an Openstack cloud with Heat and set up a test user
2) Log in as admin or any other user able to do stack operations:
$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_AUTH_URL=https://<ip>:5000/v3
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=<some password>
OS_IDENTITY_API_VERSION=3
$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 95108ce766424c7aa92b20c92dd61e8f | admin |
| 415a1d3f6d324187a15391c8205ce131 | demo |
| 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
| c7ac272709164a79b2b1833db40696da | testadmin |
+----------------------------------+------------+
3) Create a stack with userfrom step 2
$ openstack stack create -t ./stack.yaml teststack --wait
2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
4) Update the stack, this would work:
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | 2025-02-06T11:40:20Z |
| stack_status | UPDATE_COMPLETE |
| stack_status_reason | Stack UPDATE completed successfully |
+---------------------+--------------------------------------+
5) Change users and re-run the stack update:
$ export OS_USERNAME=testadmin; export OS_PASSWORD=<somepassword>
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
Stack teststack UPDATE_FAILED
Notice below the simple test stacks I used:
stack.yaml:
heat_template_version: 2021-04-16
resources:
test_res:
type: OS::Heat::ResourceGroup
#update_policy:
#batch_create:
# max_batch_size: 1
#rolling_update:
# max_batch_size: 1
properties:
count: 2
resource_def:
type: nestedstack.yaml
properties:
index_name: "%index%"
nestedstack.yaml:
heat_template_version: 2015-04-30
parameters:
index_name:
type: string
description: ""
resources:
test-user:
type: OS::Keystone::User
properties:
default_project: "admin"
domain: admin_domain
enabled: True
name: { list_join : ["",["test", { get_param: index_name }]] }
password: "test"
[ Where problems could occur ]
* If the creds from the parent stack cannot be read for any particular
reason then the related DB would still be set to NULL. I have not found
any scenario were this would happend though.
* There may be performance implications for very complex and heavily
nested stack deployments as we need to retrieve the parent stack data
every time which is somewhat costly.
[ Other Info ]
When a stack is created with a certain user,
"reauthentication_auth_method=trusts" and an update is triggered, the
update will fail with the below error:
$ openstack stack resource list -n 5 teststack
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
| 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
| 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
| test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
| test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
The above happens because the user_creds_id column in the stack table
of the Heat DB is set to NULL for all nested resources (butnot the
main stack):
mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| id | name | action | status | user_creds_id |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
| 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
| 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
| 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
I would argue that:
- if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
- the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
Versions affected:
------------------
Openstack Yoga (no other versions were tested)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2097539/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list