[Bug 2086520] [NEW] Heat Appends Duplicate '/v3' to Keystone Endpoint URL, Causing Authorization Failure
sowmya
2086520 at bugs.launchpad.net
Sat Nov 2 15:00:10 UTC 2024
Public bug reported:
Description
==========
Heat erroneously appends /v3 to the Keystone endpoint URL, even when the version is already included, resulting in a malformed URL and subsequent communication issues.
When creating a Kubernetes cluster using Magnum, the VMs created as part
of the Heat stack attempt to communicate with the Keystone URL, which is
determined by the server_keystone_endpoint_type setting in heat.conf.
The issue arises in the file
heat/engine/clients/os/keystone/heat_keystoneclient.py, where the
Keystone URL is fetched from the service catalog. The following code
snippet demonstrates the problem:
def server_keystone_endpoint_url(self, fallback_endpoint):
ks_endpoint_type = cfg.CONF.server_keystone_endpoint_type
if (ks_endpoint_type in ['public', 'internal', 'admin']):
if (hasattr(self.context, 'auth_plugin') and
hasattr(self.context.auth_plugin, 'get_access')):
try:
auth_ref = self.context.auth_plugin.get_access(self.session)
if hasattr(auth_ref, "service_catalog"):
unversioned_sc_auth_uri = (
auth_ref.service_catalog.get_urls(
service_type='identity',
interface=ks_endpoint_type))
if len(unversioned_sc_auth_uri) > 0:
sc_auth_uri = (
unversioned_sc_auth_uri[0] + "/v3")
return sc_auth_uri
The issue leads to the Heat stack creation process failing, as the VMs
try to connect to a Keystone URL with a duplicated /v3, resulting in
authorization errors. The following error message is logged in the VM:
Sep 30 05:19:40 new-cluster1-taypswwfmte6-master-0 heat-container-agent[2624]: Authorization failed: Not Found (HTTP 404) (Request-ID: req-108d6dda-f180-493a-ba10-4afb59ecfd56)
Sep 30 05:19:40 new-cluster1-taypswwfmte6-master-0 podman[2605]: /var/lib/os-collect-config/local-data not found. Skipping
This issue specifically occurs when the Keystone endpoint URL already
ends with /v3.
Here is the commit : https://opendev.org/openstack/heat/commit/c79e1db
Steps to Reproduce
===============
Create or deploy a Kubernetes cluster using Magnum.
Below are the commands for template and cluster creation
1. openstack coe cluster template create new-cluster-template1 \
--image magnum-fedora-coreos-40 \
--external-network PUBLICNET \
--dns-nameserver 8.8.8.8 \
--master-flavor gp.0.4.8 \
--flavor gp.0.4.8 \
--network-driver calico \
--volume-driver cinder \
--docker-volume-size 3 \
--coe kubernetes
2. openstack coe cluster create new-cluster1 \
--cluster-template new-cluster-template1 \
--master-count 1 \
--node-count 1 \
--master-flavor gp.0.4.8 --keypair test-mykey --labels kube_tag=v1.27.8-rancher2,container_runtime=containerd,containerd_version=1.6.28,containerd_tarball_sha256=f70736e52d61e5ad225f4fd21643b5ca1220013ab8b6c380434caeefb572da9b,cloud_provider_tag=v1.27.3,cinder_csi_plugin_tag=v1.27.3,k8s_keystone_auth_tag=v1.27.3,magnum_auto_healer_tag=v1.27.3,octavia_ingress_controller_tag=v1.27.3,calico_tag=v3.26.4
Expected Result
===========
The cluster creation process is successful without any errors.
Actual Result
===========
Cluster creation fails due to the VM's inability to communicate with the malformed Keystone URL containing duplicate /v3.
Environment
===========
OpenStack Heat 2024.1
** Affects: heat (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2086520
Title:
Heat Appends Duplicate '/v3' to Keystone Endpoint URL, Causing
Authorization Failure
Status in heat package in Ubuntu:
New
Bug description:
Description
==========
Heat erroneously appends /v3 to the Keystone endpoint URL, even when the version is already included, resulting in a malformed URL and subsequent communication issues.
When creating a Kubernetes cluster using Magnum, the VMs created as
part of the Heat stack attempt to communicate with the Keystone URL,
which is determined by the server_keystone_endpoint_type setting in
heat.conf. The issue arises in the file
heat/engine/clients/os/keystone/heat_keystoneclient.py, where the
Keystone URL is fetched from the service catalog. The following code
snippet demonstrates the problem:
def server_keystone_endpoint_url(self, fallback_endpoint):
ks_endpoint_type = cfg.CONF.server_keystone_endpoint_type
if (ks_endpoint_type in ['public', 'internal', 'admin']):
if (hasattr(self.context, 'auth_plugin') and
hasattr(self.context.auth_plugin, 'get_access')):
try:
auth_ref = self.context.auth_plugin.get_access(self.session)
if hasattr(auth_ref, "service_catalog"):
unversioned_sc_auth_uri = (
auth_ref.service_catalog.get_urls(
service_type='identity',
interface=ks_endpoint_type))
if len(unversioned_sc_auth_uri) > 0:
sc_auth_uri = (
unversioned_sc_auth_uri[0] + "/v3")
return sc_auth_uri
The issue leads to the Heat stack creation process failing, as the VMs
try to connect to a Keystone URL with a duplicated /v3, resulting in
authorization errors. The following error message is logged in the VM:
Sep 30 05:19:40 new-cluster1-taypswwfmte6-master-0 heat-container-agent[2624]: Authorization failed: Not Found (HTTP 404) (Request-ID: req-108d6dda-f180-493a-ba10-4afb59ecfd56)
Sep 30 05:19:40 new-cluster1-taypswwfmte6-master-0 podman[2605]: /var/lib/os-collect-config/local-data not found. Skipping
This issue specifically occurs when the Keystone endpoint URL already
ends with /v3.
Here is the commit : https://opendev.org/openstack/heat/commit/c79e1db
Steps to Reproduce
===============
Create or deploy a Kubernetes cluster using Magnum.
Below are the commands for template and cluster creation
1. openstack coe cluster template create new-cluster-template1 \
--image magnum-fedora-coreos-40 \
--external-network PUBLICNET \
--dns-nameserver 8.8.8.8 \
--master-flavor gp.0.4.8 \
--flavor gp.0.4.8 \
--network-driver calico \
--volume-driver cinder \
--docker-volume-size 3 \
--coe kubernetes
2. openstack coe cluster create new-cluster1 \
--cluster-template new-cluster-template1 \
--master-count 1 \
--node-count 1 \
--master-flavor gp.0.4.8 --keypair test-mykey --labels kube_tag=v1.27.8-rancher2,container_runtime=containerd,containerd_version=1.6.28,containerd_tarball_sha256=f70736e52d61e5ad225f4fd21643b5ca1220013ab8b6c380434caeefb572da9b,cloud_provider_tag=v1.27.3,cinder_csi_plugin_tag=v1.27.3,k8s_keystone_auth_tag=v1.27.3,magnum_auto_healer_tag=v1.27.3,octavia_ingress_controller_tag=v1.27.3,calico_tag=v3.26.4
Expected Result
===========
The cluster creation process is successful without any errors.
Actual Result
===========
Cluster creation fails due to the VM's inability to communicate with the malformed Keystone URL containing duplicate /v3.
Environment
===========
OpenStack Heat 2024.1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/heat/+bug/2086520/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list