[Bug 2146964] [NEW] [SRU] Prevent masakari HA from creating duplicated notifications

Tue Mar 31 21:20:21 UTC 2026

Public bug reported:

[ Impact ]

 * Older versions of Masakari are lacking a proper mechanism to prevent
   concurrent record inserts into the database, resulting duplicated
   event processing that can potentially lead to disastrous outcomes.

 * Affected versios include:

   - Focal/Ussuri 9.0.0-0ubuntu0.20.04.5 / 9.0.0-0ubuntu0.20.04.5~cloud0 (UCA)
   - Jammy/Yoga 13.0.0-0ubuntu1 / 13.0.0-0ubuntu1~cloud0 (UCA)
   - Jammy/Caracal 17.0.0-0ubuntu1 /	17.0.0-0ubuntu1~cloud0 (UCA)
   - Noble 17.0.0-0ubuntu1
   - Resolute 21.0.0~rc1-0ubuntu1 (Proposed)

 * This is a known issue and reported by many users also presented in
   LP#2028450.

 * This requires an HA Masakari deployment without a coordinator
   configured.

 * None of the current Charmed Masakari deployments support using a
   coordinator and are all affected.

[ Test Plan ]

 * Juju bundles will be crafted and attached to this SRU bug to make
   reproducer deployments easier. The plan is to cover all affected
   versions. Detailed instructions will be provided.

 * The LP#2028450 bug includes a reproducer script (2), which is handy
   to reproduce the issue as well as validate the fix.

 * The reproducer script simulates concurrent insertion situations
   using OpenStack CLI.

[ Where problems could occur ]

 * The nature if this change is very simple. In the HA API, the
   create notification function will be modified to introduce an
   artificial pause, significantly reducing the chance of
   concurrent insertion if event records in database. 

   This will only be invoked if coordination back-end is not 
   configured. Please see (3) for more details:

   if not CONF.coordination.backend_url:
           time.sleep(random.uniform(1, 5))

 * The probable risk here is a delay between 1-5 seconds during
   event creation and further processing.

[ Other Info ]

 * There is currently an effort to merge this change upstream (4).

(1) https://bugs.launchpad.net/masakari/+bug/2028450
(2) https://launchpadlibrarian.net/849768758/lp2028450-reproducer.bash
(3) https://review.opendev.org/c/openstack/masakari/+/978343/3/masakari/ha/api.py
(4) https://review.opendev.org/c/openstack/masakari/+/978343

** Affects: masakari
     Importance: Undecided
     Assignee: Alan Baghumian (alanbach)
         Status: In Progress

** Affects: masakari (Ubuntu)
     Importance: High
     Assignee: Alan Baghumian (alanbach)
         Status: In Progress

** Affects: masakari (Ubuntu Focal)
     Importance: High
     Assignee: Alan Baghumian (alanbach)
         Status: In Progress

** Affects: masakari (Ubuntu Jammy)
     Importance: High
     Assignee: Alan Baghumian (alanbach)
         Status: In Progress

** Affects: masakari (Ubuntu Noble)
     Importance: High
     Assignee: Alan Baghumian (alanbach)
         Status: In Progress

** Affects: masakari (Ubuntu Resolute)
     Importance: High
     Assignee: Alan Baghumian (alanbach)
         Status: In Progress

** Changed in: masakari
     Assignee: (unassigned) => Alan Baghumian (alanbach)

** Changed in: masakari
       Status: New => In Progress

** Also affects: masakari (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: masakari (Ubuntu Resolute)
   Importance: Undecided
       Status: New

** Also affects: masakari (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Also affects: masakari (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: masakari (Ubuntu Noble)
   Importance: Undecided
       Status: New

** Changed in: masakari (Ubuntu Focal)
     Assignee: (unassigned) => Alan Baghumian (alanbach)

** Changed in: masakari (Ubuntu Jammy)
     Assignee: (unassigned) => Alan Baghumian (alanbach)

** Changed in: masakari (Ubuntu Noble)
     Assignee: (unassigned) => Alan Baghumian (alanbach)

** Changed in: masakari (Ubuntu Resolute)
     Assignee: (unassigned) => Alan Baghumian (alanbach)

** Changed in: masakari (Ubuntu Focal)
       Status: New => In Progress

** Changed in: masakari (Ubuntu Jammy)
       Status: New => In Progress

** Changed in: masakari (Ubuntu Noble)
       Status: New => In Progress

** Changed in: masakari (Ubuntu Resolute)
       Status: New => In Progress

** Changed in: masakari (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: masakari (Ubuntu Jammy)
   Importance: Undecided => High

** Changed in: masakari (Ubuntu Noble)
   Importance: Undecided => High

** Changed in: masakari (Ubuntu Resolute)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to masakari in Ubuntu.
https://bugs.launchpad.net/bugs/2146964

Title:
  [SRU] Prevent masakari HA from creating duplicated notifications

Status in masakari:
  In Progress
Status in masakari package in Ubuntu:
  In Progress
Status in masakari source package in Focal:
  In Progress
Status in masakari source package in Jammy:
  In Progress
Status in masakari source package in Noble:
  In Progress
Status in masakari source package in Resolute:
  In Progress

Bug description:
  [ Impact ]

   * Older versions of Masakari are lacking a proper mechanism to prevent
     concurrent record inserts into the database, resulting duplicated
     event processing that can potentially lead to disastrous outcomes.

   * Affected versios include:

     - Focal/Ussuri 9.0.0-0ubuntu0.20.04.5 / 9.0.0-0ubuntu0.20.04.5~cloud0 (UCA)
     - Jammy/Yoga 13.0.0-0ubuntu1 / 13.0.0-0ubuntu1~cloud0 (UCA)
     - Jammy/Caracal 17.0.0-0ubuntu1 /	17.0.0-0ubuntu1~cloud0 (UCA)
     - Noble 17.0.0-0ubuntu1
     - Resolute 21.0.0~rc1-0ubuntu1 (Proposed)

   * This is a known issue and reported by many users also presented in
     LP#2028450.

   * This requires an HA Masakari deployment without a coordinator
     configured.

   * None of the current Charmed Masakari deployments support using a
     coordinator and are all affected.

  [ Test Plan ]

   * Juju bundles will be crafted and attached to this SRU bug to make
     reproducer deployments easier. The plan is to cover all affected
     versions. Detailed instructions will be provided.

   * The LP#2028450 bug includes a reproducer script (2), which is handy
     to reproduce the issue as well as validate the fix.

   * The reproducer script simulates concurrent insertion situations
     using OpenStack CLI.

  [ Where problems could occur ]

   * The nature if this change is very simple. In the HA API, the
     create notification function will be modified to introduce an
     artificial pause, significantly reducing the chance of
     concurrent insertion if event records in database. 

     This will only be invoked if coordination back-end is not 
     configured. Please see (3) for more details:

     if not CONF.coordination.backend_url:
             time.sleep(random.uniform(1, 5))

   * The probable risk here is a delay between 1-5 seconds during
     event creation and further processing.

  [ Other Info ]

   * There is currently an effort to merge this change upstream (4).

  (1) https://bugs.launchpad.net/masakari/+bug/2028450
  (2) https://launchpadlibrarian.net/849768758/lp2028450-reproducer.bash
  (3) https://review.opendev.org/c/openstack/masakari/+/978343/3/masakari/ha/api.py
  (4) https://review.opendev.org/c/openstack/masakari/+/978343

To manage notifications about this bug go to:
https://bugs.launchpad.net/masakari/+bug/2146964/+subscriptions