[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope

[Timo Aaltonen]

Hello Bryan, or anyone else affected,

Accepted python-neutron-lib into jammy-proposed. The package will build
now and be available at https://launchpad.net/ubuntu/+source/python-
neutron-lib/2.20.0-0ubuntu1.1 in a few hours, and then in the -proposed
repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
jammy to verification-done-jammy. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-jammy. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: python-neutron-lib (Ubuntu Jammy)
       Status: Confirmed => Fix Committed

** Tags added: verification-needed verification-needed-jammy

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2092217

Title:
  [SRU] Missing Group-By Optimization Backport in Yoga and Antelope

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive antelope series:
  New
Status in Ubuntu Cloud Archive bobcat series:
  Fix Released
Status in Ubuntu Cloud Archive caracal series:
  Fix Released
Status in Ubuntu Cloud Archive yoga series:
  New
Status in Ubuntu Cloud Archive zed series:
  Won't Fix
Status in python-neutron-lib package in Ubuntu:
  Fix Released
Status in python-neutron-lib source package in Jammy:
  Fix Committed

Bug description:
  [ Impact ]

    * There is a missing performance optimization in the Yoga
  (2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-
  lib caused by a missing database query optimization at [1], which is
  present upstream. In particular, the commit at [1] minimizes the
  number of objects returned from the database (grouping by resource id
  to prevent returning duplicated rows). The commit is a follow up to
  the commit at [2] in neutron. Both commits are needed to get the
  performance benefit. Fortunately said neutron commit [2] is present in
  Yoga (2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1)

    * To summarize, commit [1] is present in Ubuntu's releases of Zed
  (3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal
  (3.11.0-0ubuntu1) but not Yoga (2.20.0-0ubuntu1) or Antelope
  (3.4.0-0ubuntu1)

    * This proposed SRU patch is a cherry-pick backport of the upstream
  commit [1]

  [ Test Plan ]

    * Deploy openstack yoga/antelope on jammy and create as many RBAC
  rules as possible (the original report had 15000 to 30000 on various
  clouds). A simple and programmatic way to generate thousands of rules
  would be to generate thousands of projects, a few admin security
  groups and networks and share them with all projects

    * Measure the time elapsed when executing:
      openstack network rbac list
      openstack security group list
      openstack network list
      openstack network show <name>

    * The expected behaviour is that without the patch the performance
  will degrade with added RBAC groups while there will be no significant
  performance change with the patch.

    [ Where problems could occur ]

    * While it is not expected to introduce a regression as the Ubuntu
  releases surrounding Yoga and Antelope contain the patch, the patch
  inherently affects performance. There may be edge cases with a
  reduction in performance. For example, this patch reduces the number
  of returned values of the query to the set of rows (eliminating
  duplicates) - if there are few RBAC rules, this may be unnecessary
  overhead as the query may not have produced many duplicates results to
  begin with.

    * This affects the aggregation of sql queries, if there is an
  integration that doesn't expect this grouping (for some reason expects
  duplicates), it may be affected

  [ Other Info ]

    * There may be questions regarding whether there are accompanying
  tests to prove that there is no regression. Those tests are present in
  the associated neutron patch (as per the commit message [3]).

    * Note that I have included an additional patch in the Yoga debdiff.
  This additional patch removes a failing test related to tenant_id.
  It's not actually related to the neutron patch but I incidentally
  discovered it when addressing the missing optimization that is the
  primary concern of this SRU. If one clones the source of the Yoga
  release and then tries to debuild it, the build fails with a single
  error associated with a tenant_id test. This test was removed upstream
  in March 2022 [4], before the Yoga release, shortly after the request
  context in oslo deprecated the same field in January 2022 [5], which
  manifested the test failure.

    * The original report of the poor scaling with RBAC rules that lead
  to the development of the change is located at the LP bug [6], which
  may provide some additional context.

  Neutron RBAC performance optimization:
  [1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8
  [2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b
  [3] https://review.opendev.org/c/openstack/neutron/+/884877
  [6] https://bugs.launchpad.net/neutron/+bug/1918145

  Removal of tenant_id test:
  [4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7
  [5] https://review.opendev.org/c/openstack/oslo.context/+/815938

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions