[Bug 2115144] Re: [SRU] Improve application credential validation speed
Trent Lloyd
2115144 at bugs.launchpad.net
Mon Oct 20 03:23:50 UTC 2025
Jammy (Yoga) Verified
[Before]
ubuntu at stg-reproducer-lathiat-project-bastion:~/stsstack-bundles/openstack$ ./test-ks.sh
INFO: installing certificate authority for this deployment...done.
Failed to delete application credential with name or ID 'test1': No applicationcredential with a name or ID of 'test1' exists.
1 of 1 application credentials failed to delete.
==== userpass
Running 10s test @ https://10.149.131.212:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 49.55ms 14.81ms 152.88ms 73.44%
Req/Sec 20.31 5.75 40.00 67.76%
Latency Distribution
50% 47.71ms
75% 57.39ms
90% 67.64ms
99% 94.95ms
810 requests in 10.01s, 3.61MB read
Requests/sec: 80.88
Transfer/sec: 369.03KB
==== appcred
Running 10s test @ https://10.149.131.212:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 436.69ms 89.41ms 649.80ms 69.66%
Req/Sec 1.94 0.76 5.00 51.69%
Latency Distribution
50% 424.74ms
75% 487.95ms
90% 575.95ms
99% 649.80ms
89 requests in 10.02s, 418.40KB read
Requests/sec: 8.88
Transfer/sec: 41.76KB
[After upgrade to 2:21.0.1-0ubuntu2 from jammy-proposed]
ubuntu at stg-reproducer-lathiat-project-bastion:~/stsstack-bundles/openstack$ ./test-ks.sh
INFO: installing certificate authority for this deployment...done.
==== userpass
Running 10s test @ https://10.149.131.212:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 44.53ms 12.02ms 97.59ms 70.03%
Req/Sec 22.42 6.16 40.00 60.00%
Latency Distribution
50% 43.05ms
75% 51.31ms
90% 60.12ms
99% 76.36ms
897 requests in 10.01s, 4.00MB read
Requests/sec: 89.57
Transfer/sec: 408.66KB
==== appcred
Running 10s test @ https://10.149.131.212:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 42.16ms 10.08ms 77.34ms 68.22%
Req/Sec 23.67 5.60 40.00 57.00%
Latency Distribution
50% 41.62ms
75% 48.56ms
90% 54.92ms
99% 68.39ms
947 requests in 10.02s, 4.35MB read
Requests/sec: 94.54
Transfer/sec: 444.46KB
** Tags removed: verification-needed verification-needed-jammy
** Tags added: verification-done verification-done-jammy
** Changed in: cloud-archive/bobcat
Assignee: (unassigned) => Trent Lloyd (lathiat)
** Changed in: cloud-archive/antelope
Assignee: (unassigned) => Trent Lloyd (lathiat)
** Changed in: cloud-archive/yoga
Assignee: (unassigned) => Trent Lloyd (lathiat)
** Changed in: cloud-archive/zed
Assignee: (unassigned) => Trent Lloyd (lathiat)
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to keystone in Ubuntu.
https://bugs.launchpad.net/bugs/2115144
Title:
[SRU] Improve application credential validation speed
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
Fix Committed
Status in Ubuntu Cloud Archive bobcat series:
Fix Committed
Status in Ubuntu Cloud Archive caracal series:
Fix Released
Status in Ubuntu Cloud Archive dalmatian series:
Fix Released
Status in Ubuntu Cloud Archive epoxy series:
Fix Released
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
Confirmed
Status in Ubuntu Cloud Archive zed series:
Fix Committed
Status in keystone package in Ubuntu:
In Progress
Status in keystone source package in Jammy:
Fix Committed
Status in keystone source package in Noble:
Fix Released
Bug description:
[ Impact ]
Validating an application credential token is very slow, taking at
least 400ms+ in a simple devstack environment, 5-10x longer than
validating a user/password project token.
The primary bottleneck during a token validation request
(/v3/auth/tokens) is that token.roles is evaluated at least 5 times.
validate_token is called twice, first during RBAC to populate the
subject token context and again to actually validate the token. Each
call to validate_token then called token.roles twice because it first
checks if it is None, before calling it again to use the result.
Lastly token.roles is evaluated a fifth time during
render_token_response_from_model.
Each evaluation of token.roles calls through
_get_application_credential_roles into list_role_assignments which
then makes multiple round-trip SQL queries to the database.
Unlike the related get_roles_for_user_and_project function, none of
these calls are currently cached/memoized. We memoize
list_role_assignments to get the same-speedup.
Reduce the number of token.roles calls to only 3 by storing and re-
using the token.roles result in validate_token, then memoize
list_role_assignments so the 2nd and 3rd call fetch from the cache
instead of repeating many SQL queries.
This provides a substantial performance improvement bringing
validation time in-line with user/password tokens.
This bug is being opened to track the Ubuntu SRU of this fix, which was merged upstream without a corresponding bug report:
https://review.opendev.org/c/openstack/keystone/+/880456
[ Test Plan ]
source novarc
export TOKEN_userpass=$(openstack token issue -f value -c id)
openstack application credential delete test1
eval $(openstack application credential create -f shell -c id -c secret test1)
export OS_APPLICATION_CREDENTIAL_ID=${id}
export OS_APPLICATION_CREDENTIAL_SECRET=${secret}
export OS_AUTH_TYPE="v3applicationcredential"
unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME
OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME
OS_USER_DOMAIN_ID
export TOKEN_appcred=$(openstack token issue -f value -c id)
curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H
"X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token:
${TOKEN_userpass}" -H "Accept-Encoding: gzip" -H "Accept:
application/json" --write-out "%{time_total}\n"
curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H
"X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token:
${TOKEN_appcred}" -H "Accept-Encoding: gzip" -H "Accept:
application/json" --write-out "%{time_total}\n"
# Timing is approximately ~0.340s tor token1, ~0.061s for token2
# Alternate test, using 'wrk', runs as many requests as possible for 10 seconds and prints a benchmark result
wrk -t4 -c4 -d10s ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_userpass}" -H "Accept-Encoding: gzip" -H "Accept: application/json" --latency
wrk -t4 -c4 -d10s ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token:
${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_appcred}" -H "Accept-
Encoding: gzip" -H "Accept: application/json" --latency
# Reset
unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME OS_USER_DOMAIN_ID OS_APPLICATION_CREDENTIAL_ID OS_APPLICATION_CREDENTIAL_SECRET OS_AUTH_TYPE
[ Where problems could occur ]
Improving this performance required two changes.
The first simply removed immediately duplicate calls to expensive SQL
queries, regression in that part of the code is less likely.
The second change however was to start caching the result of some
queries, and ensuring correct cache invalidation is an important
concern. A thorough audit of locations where cache invalidation is
required was taken and those locations were addressed. Additionally
the upstream unit tests fortunately already tested these code paths
and also highlighted the lack of cache invalidation, and was resolved
once cache invalidation was in place.
This patch has been merged upstream for over 18 months since the
Caracal release and in-use in a relevant production environment for
the same amount of time. I conducted a search of all commits since
then and have not found any fixes either for this commit or related
code. This reduces regression potential.
We are back porting this to 4 previous OpenStack releases. This was
also done upstream and the upstream tests all pass.
[ Other ]
Affected Versions
24.04 Noble / 2024.1 Caracal - Merged into Caracal during development,
exists in 25.0.0-0ubuntu1 which is already in the archive. No action
required.
23.10 Mantic / 2023.2 Bobcat - Backported upstream, exists in the
latest point release 24.1.0, but is not in the archive version 24.0.0.
SRU Required. https://review.opendev.org/c/openstack/keystone/+/909256
23.04 Lunar / 2023.1 Antelope - Backported upstream, not in the latest
point release 23.0.2. SRU Required.
https://review.opendev.org/c/openstack/keystone/+/936329
22.10 Kinetic / Zed - Backported upstream, not in the latest point
release 22.0.2. SRU Required.
https://review.opendev.org/c/openstack/keystone/+/940285
22.04 Jammy / Yoga - Backported upstream, not in the latest point
release 21.0.1. SRU Required.
https://review.opendev.org/c/openstack/keystone/+/940286
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2115144/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list