[Bug 2115144] Re: [SRU] Improve application credential validation speed

Wed Oct 15 12:42:45 UTC 2025

Hello Trent, or anyone else affected,

Accepted keystone into jammy-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/keystone/2:21.0.1-0ubuntu2 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
jammy to verification-done-jammy. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-jammy. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: keystone (Ubuntu Jammy)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-jammy

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to keystone in Ubuntu.
https://bugs.launchpad.net/bugs/2115144

Title:
  [SRU] Improve application credential validation speed

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  Confirmed
Status in Ubuntu Cloud Archive bobcat series:
  Fix Committed
Status in Ubuntu Cloud Archive caracal series:
  Fix Released
Status in Ubuntu Cloud Archive dalmatian series:
  Fix Released
Status in Ubuntu Cloud Archive epoxy series:
  Fix Released
Status in Ubuntu Cloud Archive flamingo series:
  Fix Released
Status in Ubuntu Cloud Archive yoga series:
  Confirmed
Status in Ubuntu Cloud Archive zed series:
  Confirmed
Status in keystone package in Ubuntu:
  In Progress
Status in keystone source package in Jammy:
  Fix Committed
Status in keystone source package in Noble:
  Fix Released

Bug description:
  [ Impact ]

  Validating an application credential token is very slow, taking at
  least 400ms+ in a simple devstack environment, 5-10x longer than
  validating a user/password project token.

  The primary bottleneck during a token validation request
  (/v3/auth/tokens) is that token.roles is evaluated at least 5 times.
  validate_token is called twice, first during RBAC to populate the
  subject token context and again to actually validate the token. Each
  call to validate_token then called token.roles twice because it first
  checks if it is None, before calling it again to use the result.
  Lastly token.roles is evaluated a fifth time during
  render_token_response_from_model.

  Each evaluation of token.roles calls through
  _get_application_credential_roles into list_role_assignments which
  then makes multiple round-trip SQL queries to the database.

  Unlike the related get_roles_for_user_and_project function, none of
  these calls are currently cached/memoized. We memoize
  list_role_assignments to get the same-speedup.

  Reduce the number of token.roles calls to only 3 by storing and re-
  using the token.roles result in validate_token, then memoize
  list_role_assignments so the 2nd and 3rd call fetch from the cache
  instead of repeating many SQL queries.

  This provides a substantial performance improvement bringing
  validation time in-line with user/password tokens.

  This bug is being opened to track the Ubuntu SRU of this fix, which was merged upstream without a corresponding bug report:
  https://review.opendev.org/c/openstack/keystone/+/880456

  [ Test Plan ]

  source novarc

  export TOKEN_userpass=$(openstack token issue -f value -c id)

  openstack application credential delete test1
  eval $(openstack application credential create -f shell -c id -c secret test1)
  export OS_APPLICATION_CREDENTIAL_ID=${id}
  export OS_APPLICATION_CREDENTIAL_SECRET=${secret}
  export OS_AUTH_TYPE="v3applicationcredential"

  unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME
  OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME
  OS_USER_DOMAIN_ID

  export TOKEN_appcred=$(openstack token issue -f value -c id)

  curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H
  "X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token:
  ${TOKEN_userpass}" -H "Accept-Encoding: gzip" -H "Accept:
  application/json" --write-out "%{time_total}\n"

  curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H
  "X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token:
  ${TOKEN_appcred}" -H "Accept-Encoding: gzip" -H "Accept:
  application/json" --write-out "%{time_total}\n"

  # Timing is approximately ~0.340s tor token1, ~0.061s for token2

  # Alternate test, using 'wrk', runs as many requests as possible for 10 seconds and prints a benchmark result
  wrk -t4 -c4 -d10s ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_userpass}" -H "Accept-Encoding: gzip" -H "Accept: application/json" --latency

  wrk -t4 -c4 -d10s ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token:
  ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_appcred}" -H "Accept-
  Encoding: gzip" -H "Accept: application/json" --latency

  # Reset
  unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME OS_USER_DOMAIN_ID OS_APPLICATION_CREDENTIAL_ID OS_APPLICATION_CREDENTIAL_SECRET OS_AUTH_TYPE

  [ Where problems could occur ]

  Improving this performance required two changes.

  The first simply removed immediately duplicate calls to expensive SQL
  queries, regression in that part of the code is less likely.

  The second change however was to start caching the result of some
  queries, and ensuring correct cache invalidation is an important
  concern.  A thorough audit of locations where cache invalidation is
  required was taken and those locations were addressed. Additionally
  the upstream unit tests fortunately already tested these code paths
  and also highlighted the lack of cache invalidation, and was resolved
  once cache invalidation was in place.

  This patch has been merged upstream for over 18 months since the
  Caracal release and in-use in a relevant production environment for
  the same amount of time. I conducted a search of all commits since
  then and have not found any fixes either for this commit or related
  code. This reduces regression potential.

  We are back porting this to 4 previous OpenStack releases. This was
  also done upstream and the upstream tests all pass.

  [ Other ]

  Affected Versions

  24.04 Noble / 2024.1 Caracal - Merged into Caracal during development,
  exists in 25.0.0-0ubuntu1 which is already in the archive. No action
  required.

  23.10 Mantic / 2023.2 Bobcat - Backported upstream, exists in the
  latest point release 24.1.0, but is not in the archive version 24.0.0.
  SRU Required. https://review.opendev.org/c/openstack/keystone/+/909256

  23.04 Lunar / 2023.1 Antelope - Backported upstream, not in the latest
  point release 23.0.2. SRU Required.
  https://review.opendev.org/c/openstack/keystone/+/936329

  22.10 Kinetic / Zed - Backported upstream, not in the latest point
  release 22.0.2. SRU Required.
  https://review.opendev.org/c/openstack/keystone/+/940285

  22.04 Jammy / Yoga - Backported upstream, not in the latest point
  release 21.0.1. SRU Required.
  https://review.opendev.org/c/openstack/keystone/+/940286

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2115144/+subscriptions