[Bug 1878548] Re: There are cases when masakari-hostmonitor will recognize online nodes as offline and send (in)appropriate notifications to Masakari

[Corey Bryant]

** Also affects: masakari-monitors (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: masakari-monitors (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Changed in: masakari-monitors (Ubuntu)
       Status: New => Fix Released

** Changed in: masakari-monitors (Ubuntu Focal)
       Status: New => Triaged

** Changed in: masakari-monitors (Ubuntu Focal)
   Importance: Undecided => High

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/wallaby
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/ussuri
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/victoria
   Importance: Undecided
       Status: New

** Changed in: cloud-archive
       Status: New => Fix Released

** Changed in: cloud-archive/ussuri
   Importance: Undecided => High

** Changed in: cloud-archive/ussuri
       Status: New => Triaged

** Changed in: cloud-archive/victoria
   Importance: Undecided => High

** Changed in: cloud-archive/victoria
       Status: New => Triaged

** Changed in: cloud-archive/wallaby
   Importance: Undecided => High

** Changed in: cloud-archive/wallaby
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to masakari-monitors in Ubuntu.
https://bugs.launchpad.net/bugs/1878548

Title:
  There are cases when masakari-hostmonitor will recognize online nodes
  as offline and send (in)appropriate notifications to Masakari

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in Ubuntu Cloud Archive wallaby series:
  Triaged
Status in masakari-monitors:
  Fix Released
Status in masakari-monitors ussuri series:
  Fix Released
Status in masakari-monitors victoria series:
  Fix Released
Status in masakari-monitors wallaby series:
  Fix Released
Status in masakari-monitors xena series:
  Fix Released
Status in masakari-monitors package in Ubuntu:
  Fix Released
Status in masakari-monitors source package in Focal:
  Triaged

Bug description:
  [Issue]
  ComputeNodes are managed by pacemaker_remote in my environment.
  When one ComputeNode is isolated in the network, masakari-hostmonitors on the other ComputeNodes will send failure notification about the isolated ComputeNode to masakari-api.
  At that time, the isolated masakari-hostomonitor will recognize other ComputeNodes as offline. So it sends failure notification about online ComputeNodes.
  As a result, masakari-engine runs the recovery procedure to online ComputeNodes.

  [Cause]
  The current masakari-hostmonitor can't determine whether or not it is isolated in the network if ComputeNodes are managed by pacemaker_remote.

  masakari-hostmonitor with pacemaker(not remote) will wait until it is killed if it is isolated in the network. It is implemented in the following code.
  <https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/host_handler/handle_host.py#L398-L402>

  But masakari-hostmonitor with pacemaker_remote won't determine if it is isolated.
  <https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/host_handler/handle_host.py#L93-L95>

  [Solution]
  The ComputeNode managed by pacemaker_remote should determine recognize itself as offline when it is isolated.
  The state monitoring process should be skipped in that case.

  See comment #11 for how yoctozepto managed to reproduce something
  similar to the described.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1878548/+subscriptions