[Bug 2115144] Re: [SRU] Improve application credential validation speed
Trent Lloyd
2115144 at bugs.launchpad.net
Thu Oct 16 07:56:59 UTC 2025
Bobcat verified
[Before]
ubuntu at stg-reproducer-lathiat-project-bastion:~/stsstack-bundles/openstack$ ./test-ks.sh
INFO: installing certificate authority for this deployment...done.
Failed to delete application credential with name or ID 'test1': No applicationcredential with a name or ID of 'test1' exists.
1 of 1 application credentials failed to delete.
==== userpass
Running 10s test @ https://10.149.131.146:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 52.08ms 13.71ms 100.61ms 68.58%
Req/Sec 19.16 5.40 40.00 71.00%
Latency Distribution
50% 51.85ms
75% 60.98ms
90% 70.30ms
99% 91.47ms
767 requests in 10.02s, 3.46MB read
Requests/sec: 76.54
Transfer/sec: 353.92KB
==== appcred
Running 10s test @ https://10.149.131.146:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 495.49ms 110.05ms 806.94ms 65.38%
Req/Sec 1.65 0.58 3.00 55.13%
Latency Distribution
50% 472.66ms
75% 577.25ms
90% 657.02ms
99% 806.94ms
78 requests in 10.02s, 371.49KB read
Requests/sec: 7.78
Transfer/sec: 37.07KB
[After installing 2:24.0.0-0ubuntu1~cloud1 from bobcat-proposed]
ubuntu at stg-reproducer-lathiat-project-bastion:~/stsstack-bundles/openstack$ ./test-ks.sh
INFO: installing certificate authority for this deployment...done.
==== userpass
Running 10s test @ https://10.149.131.146:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 56.23ms 14.00ms 106.07ms 68.92%
Req/Sec 17.76 5.57 30.00 64.25%
Latency Distribution
50% 55.82ms
75% 64.85ms
90% 74.55ms
99% 91.84ms
711 requests in 10.02s, 3.21MB read
Requests/sec: 70.96
Transfer/sec: 328.14KB
==== appcred
Running 10s test @ https://10.149.131.146:5000/v3/auth/tokens
4 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 58.25ms 14.27ms 119.50ms 66.42%
Req/Sec 17.09 5.35 30.00 63.16%
Latency Distribution
50% 57.07ms
75% 68.25ms
90% 77.86ms
99% 89.66ms
684 requests in 10.02s, 3.18MB read
Requests/sec: 68.25
Transfer/sec: 325.03KB
** Tags removed: verification-bobcat-needed
** Tags added: verification-bobcat-doen
** Tags removed: verification-bobcat-doen
** Tags added: verification-bobcat-done
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to keystone in Ubuntu.
https://bugs.launchpad.net/bugs/2115144
Title:
[SRU] Improve application credential validation speed
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
Fix Committed
Status in Ubuntu Cloud Archive bobcat series:
Fix Committed
Status in Ubuntu Cloud Archive caracal series:
Fix Released
Status in Ubuntu Cloud Archive dalmatian series:
Fix Released
Status in Ubuntu Cloud Archive epoxy series:
Fix Released
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
Confirmed
Status in Ubuntu Cloud Archive zed series:
Fix Committed
Status in keystone package in Ubuntu:
In Progress
Status in keystone source package in Jammy:
Fix Committed
Status in keystone source package in Noble:
Fix Released
Bug description:
[ Impact ]
Validating an application credential token is very slow, taking at
least 400ms+ in a simple devstack environment, 5-10x longer than
validating a user/password project token.
The primary bottleneck during a token validation request
(/v3/auth/tokens) is that token.roles is evaluated at least 5 times.
validate_token is called twice, first during RBAC to populate the
subject token context and again to actually validate the token. Each
call to validate_token then called token.roles twice because it first
checks if it is None, before calling it again to use the result.
Lastly token.roles is evaluated a fifth time during
render_token_response_from_model.
Each evaluation of token.roles calls through
_get_application_credential_roles into list_role_assignments which
then makes multiple round-trip SQL queries to the database.
Unlike the related get_roles_for_user_and_project function, none of
these calls are currently cached/memoized. We memoize
list_role_assignments to get the same-speedup.
Reduce the number of token.roles calls to only 3 by storing and re-
using the token.roles result in validate_token, then memoize
list_role_assignments so the 2nd and 3rd call fetch from the cache
instead of repeating many SQL queries.
This provides a substantial performance improvement bringing
validation time in-line with user/password tokens.
This bug is being opened to track the Ubuntu SRU of this fix, which was merged upstream without a corresponding bug report:
https://review.opendev.org/c/openstack/keystone/+/880456
[ Test Plan ]
source novarc
export TOKEN_userpass=$(openstack token issue -f value -c id)
openstack application credential delete test1
eval $(openstack application credential create -f shell -c id -c secret test1)
export OS_APPLICATION_CREDENTIAL_ID=${id}
export OS_APPLICATION_CREDENTIAL_SECRET=${secret}
export OS_AUTH_TYPE="v3applicationcredential"
unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME
OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME
OS_USER_DOMAIN_ID
export TOKEN_appcred=$(openstack token issue -f value -c id)
curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H
"X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token:
${TOKEN_userpass}" -H "Accept-Encoding: gzip" -H "Accept:
application/json" --write-out "%{time_total}\n"
curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H
"X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token:
${TOKEN_appcred}" -H "Accept-Encoding: gzip" -H "Accept:
application/json" --write-out "%{time_total}\n"
# Timing is approximately ~0.340s tor token1, ~0.061s for token2
# Alternate test, using 'wrk', runs as many requests as possible for 10 seconds and prints a benchmark result
wrk -t4 -c4 -d10s ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token: ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_userpass}" -H "Accept-Encoding: gzip" -H "Accept: application/json" --latency
wrk -t4 -c4 -d10s ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token:
${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_appcred}" -H "Accept-
Encoding: gzip" -H "Accept: application/json" --latency
# Reset
unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME OS_USER_DOMAIN_ID OS_APPLICATION_CREDENTIAL_ID OS_APPLICATION_CREDENTIAL_SECRET OS_AUTH_TYPE
[ Where problems could occur ]
Improving this performance required two changes.
The first simply removed immediately duplicate calls to expensive SQL
queries, regression in that part of the code is less likely.
The second change however was to start caching the result of some
queries, and ensuring correct cache invalidation is an important
concern. A thorough audit of locations where cache invalidation is
required was taken and those locations were addressed. Additionally
the upstream unit tests fortunately already tested these code paths
and also highlighted the lack of cache invalidation, and was resolved
once cache invalidation was in place.
This patch has been merged upstream for over 18 months since the
Caracal release and in-use in a relevant production environment for
the same amount of time. I conducted a search of all commits since
then and have not found any fixes either for this commit or related
code. This reduces regression potential.
We are back porting this to 4 previous OpenStack releases. This was
also done upstream and the upstream tests all pass.
[ Other ]
Affected Versions
24.04 Noble / 2024.1 Caracal - Merged into Caracal during development,
exists in 25.0.0-0ubuntu1 which is already in the archive. No action
required.
23.10 Mantic / 2023.2 Bobcat - Backported upstream, exists in the
latest point release 24.1.0, but is not in the archive version 24.0.0.
SRU Required. https://review.opendev.org/c/openstack/keystone/+/909256
23.04 Lunar / 2023.1 Antelope - Backported upstream, not in the latest
point release 23.0.2. SRU Required.
https://review.opendev.org/c/openstack/keystone/+/936329
22.10 Kinetic / Zed - Backported upstream, not in the latest point
release 22.0.2. SRU Required.
https://review.opendev.org/c/openstack/keystone/+/940285
22.04 Jammy / Yoga - Backported upstream, not in the latest point
release 21.0.1. SRU Required.
https://review.opendev.org/c/openstack/keystone/+/940286
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2115144/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list