[Bug 2086520] Re: Heat Appends Duplicate '/v3' to Keystone Endpoint URL, Causing Authorization Failure

[sowmya]

Fix has been merged into the upstream repository:
https://review.opendev.org/c/openstack/heat/+/933986

** Changed in: heat (Ubuntu)
     Assignee: (unassigned) => sowmya (sowmya.nethi)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/2086520

Title:
  Heat Appends Duplicate '/v3' to Keystone Endpoint URL, Causing
  Authorization Failure

Status in heat package in Ubuntu:
  New

Bug description:
  Description
  ==========
  Heat erroneously appends /v3 to the Keystone endpoint URL, even when the version is already included, resulting in a malformed URL and subsequent communication issues.

  When creating a Kubernetes cluster using Magnum, the VMs created as
  part of the Heat stack attempt to communicate with the Keystone URL,
  which is determined by the server_keystone_endpoint_type setting in
  heat.conf. The issue arises in the file
  heat/engine/clients/os/keystone/heat_keystoneclient.py, where the
  Keystone URL is fetched from the service catalog. The following code
  snippet demonstrates the problem:

  def server_keystone_endpoint_url(self, fallback_endpoint):
      ks_endpoint_type = cfg.CONF.server_keystone_endpoint_type
      if (ks_endpoint_type in ['public', 'internal', 'admin']):
          if (hasattr(self.context, 'auth_plugin') and
                  hasattr(self.context.auth_plugin, 'get_access')):
              try:
                  auth_ref = self.context.auth_plugin.get_access(self.session)
                  if hasattr(auth_ref, "service_catalog"):
                      unversioned_sc_auth_uri = (
                          auth_ref.service_catalog.get_urls(
                              service_type='identity',
                              interface=ks_endpoint_type))
                      if len(unversioned_sc_auth_uri) > 0:
                          sc_auth_uri = (
                              unversioned_sc_auth_uri[0] + "/v3")
                          return sc_auth_uri

  The issue leads to the Heat stack creation process failing, as the VMs
  try to connect to a Keystone URL with a duplicated /v3, resulting in
  authorization errors. The following error message is logged in the VM:

  Sep 30 05:19:40 new-cluster1-taypswwfmte6-master-0 heat-container-agent[2624]: Authorization failed: Not Found (HTTP 404) (Request-ID: req-108d6dda-f180-493a-ba10-4afb59ecfd56)
  Sep 30 05:19:40 new-cluster1-taypswwfmte6-master-0 podman[2605]: /var/lib/os-collect-config/local-data not found. Skipping

  This issue specifically occurs when the Keystone endpoint URL already
  ends with /v3.

  Here is the commit : https://opendev.org/openstack/heat/commit/c79e1db

  Steps to Reproduce
  ===============
  Create or deploy a Kubernetes cluster using Magnum. 
  Below are the commands for template and cluster creation
  1. openstack coe cluster template create new-cluster-template1 \
            --image magnum-fedora-coreos-40  \
            --external-network  PUBLICNET \
            --dns-nameserver 8.8.8.8 \
            --master-flavor gp.0.4.8 \
            --flavor gp.0.4.8  \
            --network-driver calico \
            --volume-driver cinder \
            --docker-volume-size 3 \
            --coe kubernetes

  2. openstack coe cluster create new-cluster1 \
            --cluster-template new-cluster-template1 \
            --master-count 1 \
            --node-count 1 \
            --master-flavor gp.0.4.8 --keypair test-mykey --labels kube_tag=v1.27.8-rancher2,container_runtime=containerd,containerd_version=1.6.28,containerd_tarball_sha256=f70736e52d61e5ad225f4fd21643b5ca1220013ab8b6c380434caeefb572da9b,cloud_provider_tag=v1.27.3,cinder_csi_plugin_tag=v1.27.3,k8s_keystone_auth_tag=v1.27.3,magnum_auto_healer_tag=v1.27.3,octavia_ingress_controller_tag=v1.27.3,calico_tag=v3.26.4

  Expected Result
  ===========
  The cluster creation process is successful without any errors.

  Actual Result
  ===========
  Cluster creation fails due to the VM's inability to communicate with the malformed Keystone URL containing duplicate /v3.

  Environment
  ===========
  OpenStack Heat 2024.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/heat/+bug/2086520/+subscriptions