[Bug 2097539] Re: Heat stack updates fail when "reauthentication_auth_method=trusts"
Alejandro Santoyo Gonzalez
2097539 at bugs.launchpad.net
Tue Sep 23 08:21:37 UTC 2025
** Description changed:
- Summary:
- --------
-
- When a stack is created with a certain user,
- "reauthentication_auth_method=trusts" and an update is triggered, the
- update will fail with the below error:
-
- $ openstack stack resource list -n 5 teststack
- +---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
- | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
- +---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
- | test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
- | 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
- | 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
- | test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
- | test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
- +---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
-
- 2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
- 2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
- roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
- 2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
- 2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
- 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
-
-
- The above happens because the user_creds_id column in the stack table of the Heat DB is set to NULL for all nested resources (butnot the main stack):
-
- mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
- +--------------------------------------+------------------------------------------------+--------+----------+---------------+
- | id | name | action | status | user_creds_id |
- +--------------------------------------+------------------------------------------------+--------+----------+---------------+
- | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
- | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
- | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
- | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
- +--------------------------------------+------------------------------------------------+--------+----------+---------------+
-
-
- I would argue that:
-
- - if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
- - the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
-
-
- Versions affected:
- ------------------
-
- Openstack Yoga (no other versions were tested
-
-
- Reproducer:
- -----------
+ [ Impact ]
+
+ * This bug causes a DB column to be wrongly set to NULL, rendering
+ any subsequent stack update actions impossible. This effectively
+ renders Heat unusable for scenarios where nested stacks are used
+ and "reauthentication_auth_method=trusts" is required.
+
+ * The fix ensures that the related DB column is correctly set with
+ the value from the parent stack.
+
+ [ Test Plan ]
1) Set up an Openstack cloud with Heat and set up a test user
2) Log in as admin or any other user able to do stack operations:
$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_AUTH_URL=https://<ip>:5000/v3
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=<some password>
OS_IDENTITY_API_VERSION=3
$ openstack user list
+----------------------------------+------------+
- | ID | Name |
+ | ID | Name |
+----------------------------------+------------+
- | 95108ce766424c7aa92b20c92dd61e8f | admin |
- | 415a1d3f6d324187a15391c8205ce131 | demo |
- | 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
- | c7ac272709164a79b2b1833db40696da | testadmin |
+ | 95108ce766424c7aa92b20c92dd61e8f | admin |
+ | 415a1d3f6d324187a15391c8205ce131 | demo |
+ | 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
+ | c7ac272709164a79b2b1833db40696da | testadmin |
+----------------------------------+------------+
3) Create a stack with userfrom step 2
$ openstack stack create -t ./stack.yaml teststack --wait
- 2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
- 2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
- 2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
- 2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
- +---------------------+--------------------------------------+
- | Field | Value |
- +---------------------+--------------------------------------+
- | id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
- | stack_name | teststack |
- | description | No description |
- | creation_time | 2025-02-06T11:40:01Z |
- | updated_time | None |
- | stack_status | CREATE_COMPLETE |
- | stack_status_reason | Stack CREATE completed successfully |
+ 2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
+ 2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
+ 2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
+ 2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
+ +---------------------+--------------------------------------+
+ | Field | Value |
+ +---------------------+--------------------------------------+
+ | id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
+ | stack_name | teststack |
+ | description | No description |
+ | creation_time | 2025-02-06T11:40:01Z |
+ | updated_time | None |
+ | stack_status | CREATE_COMPLETE |
+ | stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
4) Update the stack, this would work:
- $ openstack stack update -t stack.yaml teststack --wait
- 2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
- 2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
- 2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
- 2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
- +---------------------+--------------------------------------+
- | Field | Value |
- +---------------------+--------------------------------------+
- | id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
- | stack_name | teststack |
- | description | No description |
- | creation_time | 2025-02-06T11:40:01Z |
- | updated_time | 2025-02-06T11:40:20Z |
- | stack_status | UPDATE_COMPLETE |
- | stack_status_reason | Stack UPDATE completed successfully |
+ $ openstack stack update -t stack.yaml teststack --wait
+ 2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
+ 2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
+ 2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
+ 2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
+ +---------------------+--------------------------------------+
+ | Field | Value |
+ +---------------------+--------------------------------------+
+ | id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
+ | stack_name | teststack |
+ | description | No description |
+ | creation_time | 2025-02-06T11:40:01Z |
+ | updated_time | 2025-02-06T11:40:20Z |
+ | stack_status | UPDATE_COMPLETE |
+ | stack_status_reason | Stack UPDATE completed successfully |
+---------------------+--------------------------------------+
5) Change users and re-run the stack update:
$ export OS_USERNAME=testadmin; export OS_PASSWORD=<somepassword>
- $ openstack stack update -t stack.yaml teststack --wait
- 2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
- 2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
- 2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
- ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
- 2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
- ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
+ $ openstack stack update -t stack.yaml teststack --wait
+ 2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
+ 2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
+ 2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
+ ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
+ 2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
+ ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
Stack teststack UPDATE_FAILED
Notice below the simple test stacks I used:
stack.yaml:
heat_template_version: 2021-04-16
-
+
resources:
test_res:
type: OS::Heat::ResourceGroup
#update_policy:
#batch_create:
- # max_batch_size: 1
+ # max_batch_size: 1
#rolling_update:
- # max_batch_size: 1
-
+ # max_batch_size: 1
+
properties:
count: 2
resource_def:
type: nestedstack.yaml
properties:
index_name: "%index%"
-
nestedstack.yaml:
heat_template_version: 2015-04-30
parameters:
- index_name:
+ index_name:
type: string
- description: ""
+ description: ""
resources:
test-user:
type: OS::Keystone::User
properties:
default_project: "admin"
domain: admin_domain
enabled: True
name: { list_join : ["",["test", { get_param: index_name }]] }
password: "test"
+
+ [ Where problems could occur ]
+
+ * If the creds from the parent stack cannot be read for any particular
+ reason then the related DB would still be set to NULL. I have not found
+ any scenario were this would happend though.
+
+ * There may be performance implications for very complex and heavily
+ nested stack deployments as we need to retrieve the parent stack data
+ every time which is somewhat costly.
+
+ [ Other Info ]
+
+ When a stack is created with a certain user,
+ "reauthentication_auth_method=trusts" and an update is triggered, the
+ update will fail with the below error:
+
+ $ openstack stack resource list -n 5 teststack
+ +---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+ +---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
+ | test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
+ | 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
+ | 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
+ | test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
+ | test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
+ +---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
+
+ 2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
+ 2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
+ roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
+ 2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
+ 2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
+ 2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
+
+ The above happens because the user_creds_id column in the stack table of
+ the Heat DB is set to NULL for all nested resources (butnot the main
+ stack):
+
+ mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
+ +--------------------------------------+------------------------------------------------+--------+----------+---------------+
+ | id | name | action | status | user_creds_id |
+ +--------------------------------------+------------------------------------------------+--------+----------+---------------+
+ | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
+ | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
+ | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
+ | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
+ +--------------------------------------+------------------------------------------------+--------+----------+---------------+
+
+ I would argue that:
+
+ - if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
+ - the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
+
+ Versions affected:
+ ------------------
+
+ Openstack Yoga (no other versions were tested)
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2097539
Title:
Heat stack updates fail when "reauthentication_auth_method=trusts"
Status in Ubuntu Cloud Archive:
New
Status in Ubuntu Cloud Archive caracal series:
New
Status in Ubuntu Cloud Archive dalmatian series:
New
Status in Ubuntu Cloud Archive epoxy series:
New
Status in Ubuntu Cloud Archive flamingo series:
New
Status in heat package in Ubuntu:
New
Status in heat source package in Jammy:
New
Status in heat source package in Noble:
New
Status in heat source package in Plucky:
New
Bug description:
[ Impact ]
* This bug causes a DB column to be wrongly set to NULL, rendering
any subsequent stack update actions impossible. This effectively
renders Heat unusable for scenarios where nested stacks are used
and "reauthentication_auth_method=trusts" is required.
* The fix ensures that the related DB column is correctly set with
the value from the parent stack.
[ Test Plan ]
1) Set up an Openstack cloud with Heat and set up a test user
2) Log in as admin or any other user able to do stack operations:
$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_AUTH_URL=https://<ip>:5000/v3
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=<some password>
OS_IDENTITY_API_VERSION=3
$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 95108ce766424c7aa92b20c92dd61e8f | admin |
| 415a1d3f6d324187a15391c8205ce131 | demo |
| 190c8cbd62734b9eb1f82ab482b9441e | alt_demo |
| c7ac272709164a79b2b1833db40696da | testadmin |
+----------------------------------+------------+
3) Create a stack with userfrom step 2
$ openstack stack create -t ./stack.yaml teststack --wait
2025-02-06 11:40:02Z [teststack]: CREATE_IN_PROGRESS Stack CREATE started
2025-02-06 11:40:02Z [teststack.test_res]: CREATE_IN_PROGRESS state changed
2025-02-06 11:40:08Z [teststack.test_res]: CREATE_COMPLETE state changed
2025-02-06 11:40:09Z [teststack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
4) Update the stack, this would work:
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:40:20Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:40:20Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:40:29Z [teststack.test_res]: UPDATE_COMPLETE state changed
2025-02-06 11:40:29Z [teststack]: UPDATE_COMPLETE Stack UPDATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c |
| stack_name | teststack |
| description | No description |
| creation_time | 2025-02-06T11:40:01Z |
| updated_time | 2025-02-06T11:40:20Z |
| stack_status | UPDATE_COMPLETE |
| stack_status_reason | Stack UPDATE completed successfully |
+---------------------+--------------------------------------+
5) Change users and re-run the stack update:
$ export OS_USERNAME=testadmin; export OS_PASSWORD=<somepassword>
$ openstack stack update -t stack.yaml teststack --wait
2025-02-06 11:41:35Z [teststack]: UPDATE_IN_PROGRESS Stack UPDATE started
2025-02-06 11:41:35Z [teststack.test_res]: UPDATE_IN_PROGRESS state changed
2025-02-06 11:41:36Z [teststack.test_res]: UPDATE_FAILED RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx
2025-02-06 11:41:36Z [teststack]: UPDATE_FAILED Resource UPDATE failed: RemoteError: resources.test_res: Remote error: Error Attempt to use stored_context with no user_creds
['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/heat/common/context.py", line 410, in wrapped\n
Stack teststack UPDATE_FAILED
Notice below the simple test stacks I used:
stack.yaml:
heat_template_version: 2021-04-16
resources:
test_res:
type: OS::Heat::ResourceGroup
#update_policy:
#batch_create:
# max_batch_size: 1
#rolling_update:
# max_batch_size: 1
properties:
count: 2
resource_def:
type: nestedstack.yaml
properties:
index_name: "%index%"
nestedstack.yaml:
heat_template_version: 2015-04-30
parameters:
index_name:
type: string
description: ""
resources:
test-user:
type: OS::Keystone::User
properties:
default_project: "admin"
domain: admin_domain
enabled: True
name: { list_join : ["",["test", { get_param: index_name }]] }
password: "test"
[ Where problems could occur ]
* If the creds from the parent stack cannot be read for any particular
reason then the related DB would still be set to NULL. I have not found
any scenario were this would happend though.
* There may be performance implications for very complex and heavily
nested stack deployments as we need to retrieve the parent stack data
every time which is somewhat costly.
[ Other Info ]
When a stack is created with a certain user,
"reauthentication_auth_method=trusts" and an update is triggered, the
update will fail with the below error:
$ openstack stack resource list -n 5 teststack
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
| test_res | 969e8d08-ad3d-4fae-923b-1badecdebdf3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2025-02-06T12:11:11Z | teststack |
| 1 | 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:19Z | teststack-test_res-tany4qj3iums |
| 0 | 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | file:///home/ubuntu/nestedstack.yaml | UPDATE_COMPLETE | 2025-02-06T12:08:18Z | teststack-test_res-tany4qj3iums |
| test-user | c1d165409ebe4f2193682252c8f11b2e | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:06Z | teststack-test_res-tany4qj3iums-1-lceepegkpdnz |
| test-user | 5695cd71bbc545a586e50dd21ff1a7e5 | OS::Keystone::User | CREATE_COMPLETE | 2025-02-06T11:40:05Z | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 |
+---------------+--------------------------------------+--------------------------------------+-----------------+----------------------+------------------------------------------------+
2025-02-06 12:11:11.043 279650 INFO heat.engine.service [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Updating stack teststack-test_res-tany4qj3iums
2025-02-06 12:11:11.045 279650 WARNING oslo_config.cfg [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is b
roken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
2025-02-06 12:11:11.049 279651 WARNING oslo_messaging._drivers.common [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] Failed to rebuild remote exception due to error: Error.__init__() got an unexpected keyword argument 'message'
2025-02-06 12:11:11.050 279651 INFO heat.engine.resource [req-4aa4d063-e789-4fa5-ae88-02b910b1d8ec - - - - -] UPDATE: ResourceGroup "test_res" [969e8d08-ad3d-4fae-923b-1badecdebdf3] Stack "teststack" [3ea4ee9d-1686-4a97-ae92-c1f6232cd16c]
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource Traceback (most recent call last):
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1693, in update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource yield from self.action_handler_task(action,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource handler_data = handler(*args)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 485, in handle_update
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource checkers[0].start()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 183, in start
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.step()
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/scheduler.py", line 210, in step
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource poll_period = next(self._runner)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/heat/resource_group.py", line 438, in _run_to_completion
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource updater = self.update_with_template(template, {},
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/stack_resource.py", line 533, in update_with_template
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource self.rpc_client()._update_stack(self.context, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 343, in _update_stack
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self.call(ctxt,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/rpc/client.py", line 89, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource result = self.transport._send(
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource raise result
2025-02-06 12:11:11.050 279651 ERROR heat.engine.resource oslo_messaging.rpc.client.RemoteError: Remote error: Error Attempt to use stored_context with no user_creds
The above happens because the user_creds_id column in the stack table
of the Heat DB is set to NULL for all nested resources (butnot the
main stack):
mysql> select id,name,action,status,user_creds_id from heat.stack where name like "%teststack%" and deleted_at is NULL;
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| id | name | action | status | user_creds_id |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
| 1713eb91-93dd-47a9-b0ac-cf3247ee5ebd | teststack-test_res-tany4qj3iums-1-lceepegkpdnz | UPDATE | COMPLETE | NULL |
| 3ea4ee9d-1686-4a97-ae92-c1f6232cd16c | teststack | UPDATE | FAILED | 49 |
| 4d99bfcb-9fe8-4c35-83f6-d3e09de11e39 | teststack-test_res-tany4qj3iums-0-y4vpheh75oi4 | UPDATE | COMPLETE | NULL |
| 969e8d08-ad3d-4fae-923b-1badecdebdf3 | teststack-test_res-tany4qj3iums | UPDATE | COMPLETE | NULL |
+--------------------------------------+------------------------------------------------+--------+----------+---------------+
I would argue that:
- if different users are not to be used then the operation should be denied or error out in a more graceful and controlled way without "corrupting" the DB
- the user_creds_id column should not be set to NULL preventing further updates (regardless of the user) to succeed
Versions affected:
------------------
Openstack Yoga (no other versions were tested)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2097539/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list