[Bug 2141119] Re: [SRU] Don't fail if a provider driver cannot be loaded in Octavia API

[Zachary Raines]

** Description changed:

  [ Impact ]
  
  Currently, when the enabled_provider_drivers setting is not correctly
  configured, e.g., it contains a non-existing driver, or when a provider
  driver fails to load, octavia as a whole errors out and the API becomes
  unreachable. This means that if there is a transient problem with a
  provider, or a misconfiguration, even if that provider is not being
  actively used, the other functioning drivers cannot be used.
  
  [ Test Plan ]
  
  1. Deploy openstack with charmed openstack including charmed octavia.
  
  2. Ensure that the octavia endpoint is working by calling `openstack
  loadbalancer provider list`
  
  Expected Output
  ---------------
  +---------+--------------------+
  | name | description |
  +---------+--------------------+
  | ovn | Octavia OVN driver |
  | amphora | Amphora |
  +---------+--------------------+
  ---------------
  
  3. Manually break the ovn-provider by editing `/usr/lib/python3/dist-
  packages/ovn_octavia_provider/driver.py` on the octavia unit to raise an
  error in the `__init__` method of `OvnProviderDriver` i.e.
  
  ```
  --- driver.py
  +++ driver.py
  @@ -38,6 +38,7 @@
       def __init__(self):
           super().__init__()
  +        raise RuntimeError("internal error")
  ```
  
- 4. Restart the octavia api service with `juju run octavia/leader
+ 4. Update the `/etc/octavia/octavia.conf` on `octavia/leader` so the the
+ enabled_provider_drivers list contains an invalid driver
+ 
+ ---------
+ enabled_provider_drivers = amphora:The Octavia Amphora driver,ovn:Octavia OVN driver,dne:Does not exist
+ ---------
+ 
+ 5. Restart the octavia api service with `juju run octavia/leader
  restart-services` and wait a little for the unit to settle.
  
- 5. Call the API with `openstack loadbalancer provider list`
+ 6. Call the API with `openstack loadbalancer provider list`
  
  Expected Output
  ---------------
  +---------+--------------------+
  | name | description |
  +---------+--------------------+
  | amphora | Amphora |
  +---------+--------------------+
  ---------------
  
  Expected Behavior: The Octavia API is available.
  
  Actual Output
  -------------
  Service Unavailable (HTTP 503) (Request-ID: None)
  -------------
  
  Actual Behavior: The Octavia API does not become available.
  
  [ Where problems could occur ]
  
  This patch implements the strategy described in [0], skipping over
  drivers which error while allowing others to proceed with
  initialization. The Octavia API service now removes the failing
  driver(s) from the enabled list, and the other provider drivers remain
  functional.
  
  Because driver initialization errors are no longer fatal, it is easier
  for user to miss problems with their configuration. Errors are still
  logged, however, and misconfigured drivers will be missing from the
  provider list.
  
  [ Other Info ]
  
  This patch has been merged upstream into master, stable/*, and
  unmaintained/*.
  
  Story for this patch
  [0]: https://storyboard.openstack.org/#!/story/2008710
  
  Upstream patches:
  [1]: https://review.opendev.org/c/openstack/octavia/+/780215
  [2]: https://review.opendev.org/c/openstack/octavia/+/976490

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2141119

Title:
  [SRU] Don't fail if a provider driver cannot be loaded in Octavia API

Status in Ubuntu Cloud Archive:
  In Progress
Status in Ubuntu Cloud Archive antelope series:
  In Progress
Status in Ubuntu Cloud Archive bobcat series:
  In Progress
Status in Ubuntu Cloud Archive caracal series:
  In Progress
Status in Ubuntu Cloud Archive dalmatian series:
  In Progress
Status in Ubuntu Cloud Archive epoxy series:
  In Progress
Status in Ubuntu Cloud Archive flamingo series:
  In Progress
Status in Ubuntu Cloud Archive gazpacho series:
  Fix Committed
Status in Ubuntu Cloud Archive yoga series:
  In Progress
Status in Ubuntu Cloud Archive zed series:
  In Progress
Status in octavia package in Ubuntu:
  Fix Released
Status in octavia source package in Jammy:
  Fix Committed
Status in octavia source package in Noble:
  Fix Committed
Status in octavia source package in Questing:
  Fix Committed
Status in octavia source package in Resolute:
  Fix Released

Bug description:
  [ Impact ]

  Currently, when the enabled_provider_drivers setting is not correctly
  configured, e.g., it contains a non-existing driver, or when a
  provider driver fails to load, octavia as a whole errors out and the
  API becomes unreachable. This means that if there is a transient
  problem with a provider, or a misconfiguration, even if that provider
  is not being actively used, the other functioning drivers cannot be
  used.

  [ Test Plan ]

  1. Deploy openstack with charmed openstack including charmed octavia.

  2. Ensure that the octavia endpoint is working by calling `openstack
  loadbalancer provider list`

  Expected Output
  ---------------
  +---------+--------------------+
  | name | description |
  +---------+--------------------+
  | ovn | Octavia OVN driver |
  | amphora | Amphora |
  +---------+--------------------+
  ---------------

  3. Manually break the ovn-provider by editing `/usr/lib/python3/dist-
  packages/ovn_octavia_provider/driver.py` on the octavia unit to raise
  an error in the `__init__` method of `OvnProviderDriver` i.e.

  ```
  --- driver.py
  +++ driver.py
  @@ -38,6 +38,7 @@
       def __init__(self):
           super().__init__()
  +        raise RuntimeError("internal error")
  ```

  4. Update the `/etc/octavia/octavia.conf` on `octavia/leader` so the
  the enabled_provider_drivers list contains an invalid driver

  ---------
  enabled_provider_drivers = amphora:The Octavia Amphora driver,ovn:Octavia OVN driver,dne:Does not exist
  ---------

  5. Restart the octavia api service with `juju run octavia/leader
  restart-services` and wait a little for the unit to settle.

  6. Call the API with `openstack loadbalancer provider list`

  Expected Output
  ---------------
  +---------+--------------------+
  | name | description |
  +---------+--------------------+
  | amphora | Amphora |
  +---------+--------------------+
  ---------------

  Expected Behavior: The Octavia API is available.

  Actual Output
  -------------
  Service Unavailable (HTTP 503) (Request-ID: None)
  -------------

  Actual Behavior: The Octavia API does not become available.

  [ Where problems could occur ]

  This patch implements the strategy described in [0], skipping over
  drivers which error while allowing others to proceed with
  initialization. The Octavia API service now removes the failing
  driver(s) from the enabled list, and the other provider drivers remain
  functional.

  Because driver initialization errors are no longer fatal, it is easier
  for user to miss problems with their configuration. Errors are still
  logged, however, and misconfigured drivers will be missing from the
  provider list.

  [ Other Info ]

  This patch has been merged upstream into master, stable/*, and
  unmaintained/*.

  Story for this patch
  [0]: https://storyboard.openstack.org/#!/story/2008710

  Upstream patches:
  [1]: https://review.opendev.org/c/openstack/octavia/+/780215
  [2]: https://review.opendev.org/c/openstack/octavia/+/976490

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2141119/+subscriptions