[Bug 2095432] [NEW] Ping health-monitor does not work

Tue Jan 21 17:03:16 UTC 2025

Public bug reported:

Creating a ping health monitor always results in ERROR state. This
happens because the haproxy service running in the amphora image does
not have enough permissions to run the file /var/lib/octavia/ping-
wrapper.sh.

In Octavia Logs:

2025-01-17 17:47:06.714 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] member 8c4c9a83-96ff-4533-867b-6b451acdc255 status has changed from OFFLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
2025-01-17 17:47:06.719 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] pool 00d6affb-3e1e-46df-9162-7b47c21da98f status has changed from ONLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
2025-01-17 17:47:06.724 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] loadbalancer 8f720506-28fc-4589-a33b-de1cba13c93f status has changed from ONLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229

In the amphora:

Jan 21 16:53:14 amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145 ip[19071]: [ALERT]    (19071) : Failed to exec process for external health check: Permission denied. Aborting.
Jan 21 16:53:19 amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145 ip[19073]: [ALERT]    (19073) : Failed to exec process for external health check: Permission denied. Aborting.

Looking at the permissions for /var/lib/octavia:

root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# ls -lha octavia
total 36K
drwxr-x---  4 octavia octavia 4.0K Jan 21 15:49 .
drwxr-xr-x 43 root    root    4.0K Jan 20 19:42 ..
drwxr-xr-x  3 root    root    4.0K Jan 20 19:42 certs
drwxr-xr-x  2 root    root    4.0K Jan 21 16:52 d0a9cf91-1c0c-499d-a86c-3553099fa43d
srw-rw-rw-  1 root    root       0 Jan 20 19:43 d0a9cf91-1c0c-499d-a86c-3553099fa43d.sock
-rw-r--r--  1 root    root      25 Jan 20 19:11 haproxy-default-user-group.conf
-rwxr-xr-x  1 root    root     207 Jan 20 20:05 ping-wrapper.sh
-rw-r--r--  1 root    root      23 Jan 20 19:42 plugged_interfaces
-rw-r--r--  1 root    root      23 Jan 20 19:42 plugged_interfaces.sorted
-rw-r-----  1 root    root      85 Jan 21 16:54 stats_counters.json
root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# 

The haproxy runs in a separate namespace as the nobody:nogroup config:

root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# grep user octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg 
    user nobody
    stats socket /var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d.sock mode 0666 level user

root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# grep group octavia/haproxy-default-user-group.conf 
    group nogroup

Service config:

[Service]
# Force context as we start haproxy under "ip netns exec"
SELinuxContext=system_u:system_r:haproxy_t:s0

Environment="CONFIG=/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg"
"USERCONFIG=/var/lib/octavia/haproxy-default-user-group.conf"
"PIDFILE=/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/d0a9cf91-1c0c-499d-a86c-3553099fa43d.pid"

ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $USERCONFIG -c -q -L
PhDoY7WOeBqhwysZ0Dzu6CxNZgU

ExecReload=/usr/sbin/haproxy -c -f $CONFIG -f $USERCONFIG -L
PhDoY7WOeBqhwysZ0Dzu6CxNZgU

process:

nobody      1728  0.0  1.3 113372 13716 ?        S    Jan20   0:10
/usr/sbin/haproxy -sf 1651 -Ws -f
/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/d0a9cf91-1c0c-499d-a86c-3553099fa43d.pid
-L PhDoY7WOeBqhwysZ0Dzu6CxNZgU

Given that the haproxy user configuration is set in the octavia source
files, some possible solutions to run the ping-wrapper.sh are:

1) Change the /var/lib/octavia folder permissions to 755 instead of 750.
This will allow the haproxy service to run the ping-wrapper.sh file
which already has a 755 permission. This however exposes all the folder
contents. This can be done either during the octavia-diskimage-retrofit
build process or in the octavia package sources files.

2) Change the /var/lib/octavia ownership to nobody:nogroup. This allows
access to the folder but also gives permission to create files in the
folder. This can be done either during the octavia-diskimage-retrofit
build process or in the octavia package sources files.

3) Move ping-wrapper.sh out of the /var/lib/octavia folder, to a public
place. This will also require changing the octavia source haproxy
template files which have the location of the file hardcoded to
/var/lib/octavia.

** Affects: snap-octavia-diskimage-retrofit
     Importance: Undecided
         Status: New

** Affects: octavia (Ubuntu)
     Importance: Undecided
         Status: New

** Also affects: octavia (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to octavia in Ubuntu.
https://bugs.launchpad.net/bugs/2095432

Title:
  Ping health-monitor does not work

Status in snap-octavia-diskimage-retrofit:
  New
Status in octavia package in Ubuntu:
  New

Bug description:
  Creating a ping health monitor always results in ERROR state. This
  happens because the haproxy service running in the amphora image does
  not have enough permissions to run the file /var/lib/octavia/ping-
  wrapper.sh.

  In Octavia Logs:

  2025-01-17 17:47:06.714 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] member 8c4c9a83-96ff-4533-867b-6b451acdc255 status has changed from OFFLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
  2025-01-17 17:47:06.719 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] pool 00d6affb-3e1e-46df-9162-7b47c21da98f status has changed from ONLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
  2025-01-17 17:47:06.724 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] loadbalancer 8f720506-28fc-4589-a33b-de1cba13c93f status has changed from ONLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229

  In the amphora:

  Jan 21 16:53:14 amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145 ip[19071]: [ALERT]    (19071) : Failed to exec process for external health check: Permission denied. Aborting.
  Jan 21 16:53:19 amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145 ip[19073]: [ALERT]    (19073) : Failed to exec process for external health check: Permission denied. Aborting.

  Looking at the permissions for /var/lib/octavia:

  root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# ls -lha octavia
  total 36K
  drwxr-x---  4 octavia octavia 4.0K Jan 21 15:49 .
  drwxr-xr-x 43 root    root    4.0K Jan 20 19:42 ..
  drwxr-xr-x  3 root    root    4.0K Jan 20 19:42 certs
  drwxr-xr-x  2 root    root    4.0K Jan 21 16:52 d0a9cf91-1c0c-499d-a86c-3553099fa43d
  srw-rw-rw-  1 root    root       0 Jan 20 19:43 d0a9cf91-1c0c-499d-a86c-3553099fa43d.sock
  -rw-r--r--  1 root    root      25 Jan 20 19:11 haproxy-default-user-group.conf
  -rwxr-xr-x  1 root    root     207 Jan 20 20:05 ping-wrapper.sh
  -rw-r--r--  1 root    root      23 Jan 20 19:42 plugged_interfaces
  -rw-r--r--  1 root    root      23 Jan 20 19:42 plugged_interfaces.sorted
  -rw-r-----  1 root    root      85 Jan 21 16:54 stats_counters.json
  root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# 

  The haproxy runs in a separate namespace as the nobody:nogroup config:

  root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# grep user octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg 
      user nobody
      stats socket /var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d.sock mode 0666 level user

  root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# grep group octavia/haproxy-default-user-group.conf 
      group nogroup

  Service config:

  [Service]
  # Force context as we start haproxy under "ip netns exec"
  SELinuxContext=system_u:system_r:haproxy_t:s0

  Environment="CONFIG=/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg"
  "USERCONFIG=/var/lib/octavia/haproxy-default-user-group.conf"
  "PIDFILE=/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/d0a9cf91-1c0c-499d-a86c-3553099fa43d.pid"

  ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $USERCONFIG -c -q -L
  PhDoY7WOeBqhwysZ0Dzu6CxNZgU

  ExecReload=/usr/sbin/haproxy -c -f $CONFIG -f $USERCONFIG -L
  PhDoY7WOeBqhwysZ0Dzu6CxNZgU

  process:

  nobody      1728  0.0  1.3 113372 13716 ?        S    Jan20   0:10
  /usr/sbin/haproxy -sf 1651 -Ws -f
  /var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg -f
  /var/lib/octavia/haproxy-default-user-group.conf -p
  /var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/d0a9cf91-1c0c-499d-a86c-3553099fa43d.pid
  -L PhDoY7WOeBqhwysZ0Dzu6CxNZgU

  Given that the haproxy user configuration is set in the octavia source
  files, some possible solutions to run the ping-wrapper.sh are:

  1) Change the /var/lib/octavia folder permissions to 755 instead of
  750. This will allow the haproxy service to run the ping-wrapper.sh
  file which already has a 755 permission. This however exposes all the
  folder contents. This can be done either during the octavia-diskimage-
  retrofit build process or in the octavia package sources files.

  2) Change the /var/lib/octavia ownership to nobody:nogroup. This
  allows access to the folder but also gives permission to create files
  in the folder. This can be done either during the octavia-diskimage-
  retrofit build process or in the octavia package sources files.

  3) Move ping-wrapper.sh out of the /var/lib/octavia folder, to a
  public place. This will also require changing the octavia source
  haproxy template files which have the location of the file hardcoded
  to /var/lib/octavia.

To manage notifications about this bug go to:
https://bugs.launchpad.net/snap-octavia-diskimage-retrofit/+bug/2095432/+subscriptions