[Bug 2115144] Re: [SRU] Improve application credential validation speed

[Trent Lloyd]

** Changed in: keystone (Ubuntu)
     Assignee: (unassigned) => Trent Lloyd (lathiat)

** Changed in: keystone (Ubuntu)
       Status: New => In Progress

** Changed in: keystone (Ubuntu)
   Importance: Undecided => Critical

** Changed in: keystone (Ubuntu)
   Importance: Critical => Medium

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to keystone in Ubuntu.
https://bugs.launchpad.net/bugs/2115144

Title:
  [SRU] Improve application credential validation speed

Status in keystone package in Ubuntu:
  In Progress

Bug description:
  [ Impact ]

  Validating an application credential token is very slow, taking at
  least 400ms+ in a simple devstack environment, 5-10x longer than
  validating a user/password project token.

  The primary bottleneck during a token validation request
  (/v3/auth/tokens) is that token.roles is evaluated at least 5 times.
  validate_token is called twice, first during RBAC to populate the
  subject token context and again to actually validate the token. Each
  call to validate_token then called token.roles twice because it first
  checks if it is None, before calling it again to use the result.
  Lastly token.roles is evaluated a fifth time during
  render_token_response_from_model.

  Each evaluation of token.roles calls through
  _get_application_credential_roles into list_role_assignments which
  then makes multiple round-trip SQL queries to the database.

  Unlike the related get_roles_for_user_and_project function, none of
  these calls are currently cached/memoized. We memoize
  list_role_assignments to get the same-speedup.

  Reduce the number of token.roles calls to only 3 by storing and re-
  using the token.roles result in validate_token, then memoize
  list_role_assignments so the 2nd and 3rd call fetch from the cache
  instead of repeating many SQL queries.

  This provides a substantial performance improvement bringing
  validation time in-line with user/password tokens.

  This bug is being opened to track the Ubuntu SRU of this fix, which was merged upstream without a corresponding bug report:
  https://review.opendev.org/c/openstack/keystone/+/880456

  
  [ Test Plan ]

  source novarc

  export TOKEN_userpass=$(openstack token issue -f value -c id)

  openstack application credential delete test1
  eval $(openstack application credential create -f shell -c id -c secret test1)
  export OS_APPLICATION_CREDENTIAL_ID=${id}
  export OS_APPLICATION_CREDENTIAL_SECRET=${secret}
  export OS_AUTH_TYPE="v3applicationcredential"

  unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME
  OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME
  OS_USER_DOMAIN_ID

  export TOKEN_appcred=$(openstack token issue -f value -c id)

  time curl ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token:
  ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_userpass}" -H "Accept-
  Encoding: gzip" -H "Accept: application/json"

  time curl ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token:
  ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_appcred}" -H "Accept-
  Encoding: gzip" -H "Accept: application/json"

  # Timing is approximately ~0.340s tor token1, ~0.061s for token2

  
  [ Where problems could occur ]

  Improving this performance required two changes.

  The first simply removed immediately duplicate calls to expensive SQL
  queries, regression in that part of the code is less likely.

  The second change however was to start caching the result of some
  queries, and ensuring correct cache invalidation is an important
  concern.  A thorough audit of locations where cache invalidation is
  required was taken and those locations were addressed. Additionally
  the upstream unit tests fortunately already tested these code paths
  and also highlighted the lack of cache invalidation, and was resolved
  once cache invalidation was in place.

  This patch has been merged upstream for over 18 months since the
  Caracal release and in-use in a relevant production environment for
  the same amount of time. I conducted a search of all commits since
  then and have not found any fixes either for this commit or related
  code. This reduces regression potential.

  We are back porting this to 4 previous OpenStack releases. This was
  also done upstream and the upstream tests all pass.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/keystone/+bug/2115144/+subscriptions