[Bug 2097539] Re: Heat stack updates fail when "reauthentication_auth_method=trusts"
Alejandro Santoyo Gonzalez
2097539 at bugs.launchpad.net
Thu Feb 20 17:14:43 UTC 2025
Some extra analysis:
I believe the issue was introduced by commit [1] which seems to have
landed in Wallaby.
When trusts are used and based on the changes added by that commit when
the trust is getting regenerated due to the use of a different user, a
new set of creds is generated but only for the parent stack effectively
making it impossible for updates operations to succeed unless the DB is
manually cleared or the stack rebuilt (and the same stack owner user
always used for any further update operations).
Details:
- Initially when a user different to the stack owner is used to do an update, this is the error logged:
2025-02-20 11:43:57.112 2879 DEBUG heat.engine.stack [req-2fd1989f-48d3-473a-b40e-59b79098f18a testadmin - - - -] Context user_id doesn't match trustor, using stored context _delete_user_cred /usr/lib/python3/dist-packages/heat/engin
e/stack.py:1925
- this is triggered by function store which() calls _delete_user_cred()
wich effectively removes the credentials and requests new credentials to
be generated and saved.
- when the update reaches the first nested resource, update_stack() is
called which invokes parser.Stack.load() and this function invokes
_from_db() to get the nested stack data from the DB to make the
requested changes if any.
- the problem is that by now the user_creds_id column for all nested
resources are set to NULL due to the creds deletion done in store() and
the new credentials were not propagated to the nested stacks/resources.
This then triggers the below error:
2025-02-20 16:23:13.341 822603 ERROR heat.engine.resource
oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to
use stored_context with no user_creds
which comes from stored_context() called during the nested stack loading
s it tries to return an instance of the Stack class
- there is no logic that I can see to either copy the new creds saved
for the parent stack onto the nested stacks nor to create a new set of
creds for the nested stacks/resources
- the issue seems to be present in the Heat master branch code as well
Potential fixes:
- Make sure we propagate the credentials to all nested resources upon creation.
- When loading the nested stack data, figure out a way to get the new user creds from the parent stack and inject those into the DB
Both methods will likely require major code changes.
[1] https://github.com/openstack/heat/commit/d8efcd17808ebbb6eb6d88e592635bdd1ebb3d92
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2097539
Title:
Heat stack updates fail when "reauthentication_auth_method=trusts"
Status in heat package in Ubuntu:
New
Bug description:
Summary:
--------
When a stack is created with a certain user,
"reauthentication_auth_method=trusts" and an update is triggered, the
update will fail with the below error:
$ openstack stack resource list -n 5 teststack
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
| 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
| 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
| test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
| test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
The above happens because the user_creds_id column in the stack table of the Heat DB is set to NULL for all nested resources (butnot the main stack):
mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| id | name | action | status | user_creds_id |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
| 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
| 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
| 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
I would argue that:
- if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
- the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
Versions affected:
------------------
Openstack Yoga (no other versions were tested
Reproducer:
-----------
1) Set up an Openstack cloud with Heat and set up a test user
2) Log in as admin or any other user able to do stack operations:
$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_AUTH_URL=https://<ip>:5000/v3
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=<some password>
OS_IDENTITY_API_VERSION=3
$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 95108ce766424c7aa92b20c92dd61e8f | admin |
| 415a1d3f6d324187a15391c8205ce131 | demo |
| 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
| c7ac272709164a79b2b1833db40696da | testadmin |
+----------------------------------+------------+
3) Create a stack with userfrom step 2
$ openstack stack create -t ./stack.yaml teststack --wait
2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
4) Update the stack, this would work:
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | 2025-02-06T11:40:20Z |
| stack_status | UPDATE_COMPLETE |
| stack_status_reason | Stack UPDATE completed successfully |
+---------------------+--------------------------------------+
5) Change users and re-run the stack update:
$ export OS_USERNAME=testadmin; export OS_PASSWORD=<somepassword>
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
Stack teststack UPDATE_FAILED
Notice below the simple test stacks I used:
stack.yaml:
heat_template_version: 2021-04-16
resources:
test_res:
type: OS::Heat::ResourceGroup
#update_policy:
#batch_create:
# max_batch_size: 1
#rolling_update:
# max_batch_size: 1
properties:
count: 2
resource_def:
type: nestedstack.yaml
properties:
index_name: "%index%"
nestedstack.yaml:
heat_template_version: 2015-04-30
parameters:
index_name:
type: string
description: ""
resources:
test-user:
type: OS::Keystone::User
properties:
default_project: "admin"
domain: admin_domain
enabled: True
name: { list_join : ["",["test", { get_param: index_name }]] }
password: "test"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/heat/+bug/2097539/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list