[Bug 1784757] Re: [SRU] rabbitmq-server does not properly shutdown

[Launchpad Bug Tracker]

This bug was fixed in the package rabbitmq-server - 3.6.10-1ubuntu0.4

---------------
rabbitmq-server (3.6.10-1ubuntu0.4) bionic; urgency=medium

  * Improve systemd service file in order to avoid a 90-second hang
    when restarting rabbitmq-server. (LP: #1784757) (Closes #902136)
    - d/rabbitmq-server.service: Make sure the service starts after
      epmd.  Use "Type=notify" for the service.
    - d/control: Depend on socat, which is needed for the communication
      between rabbitmq-server and systemd when "Type=notify" is used.

 -- Sergio Durigan Junior <sergio.durigan at canonical.com>  Tue, 07 Jul
2020 10:26:18 -0400

** Changed in: rabbitmq-server (Ubuntu Bionic)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1784757

Title:
  [SRU] rabbitmq-server does not properly shutdown

Status in rabbitmq-server package in Ubuntu:
  Fix Released
Status in rabbitmq-server source package in Bionic:
  Fix Released
Status in rabbitmq-server source package in Cosmic:
  Won't Fix
Status in rabbitmq-server package in Debian:
  Fix Released

Bug description:
  [Impact]

  The systemd file rabbitmq-server.service on Bionic uses "Type=simple"
  when defining the service, but unfortunately this doesn't work very
  well for rabbitmq-server. In certain situations, systemd will fail to
  keep track of a start/stop/restart event, and will hang for 90 seconds
  before giving the prompt back to the user. Another problem is that
  rabbitmq-server must start after the epmd service, so we need to
  explicitly declare this dependency in the service file.

  [Test Case]

  Although I was able to reproduce this almost 100% of the time, there
  were rare occasions when the restart procedure finished normally. I
  was also only able to reproduce it using a bionic VM, not a container.
  If you have multipass or lxd configured to launch VMs, that should be
  easy.

  The steps are:

  $ lxc launch ubuntu-daily:bionic --vm bug1784757-rabbitmq-server # or use multipass
  $ lxc shell bug1784757-rabbitmq-server
  # apt update
  # apt install rabbitmq-server -y
  # systemctl restart rabbitmq-server.service

  In a normal scenario, the restart should take around 3 seconds or
  less. With the bug, it takes around 90 seconds. If you can't reproduce
  it, try running "systemctl restart" again. A quick way to trigger it
  is to run a for loop like:

  # for i in $(seq 10); do time systemctl restart rabbitmq-
  server.service ; done

  [Regression Potential]

  * Because rabbitmq-server implements systemd's "Type=notify" using
  socat to communicate with systemd-notify over a socket, we will be
  introducing another point of failure (socat) in the mix.

  * So, albeit unlikely, there may be a problem when using socat.  That
  would not be a regression, though, since the outcome would be the same
  as we have today: "systemctl restart" would not properly work, even
  though the service did restart.

  [Original Report]

  When I run `systemctl restart rabbitmq-server` it waits for 90 seconds
  then systemd sends SIGKILL to it.

  Presumably the `epmd` process does not receive SIGTERM, since if I run
  `kill 1493` (or whatever pid it currently is) then restart happens
  straight after that successfully

  ● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
    Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
    Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
    Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
   Main PID: 177 (code=killed, signal=TERM)
      Tasks: 1 (limit: 4915)
     CGroup: /system.slice/rabbitmq-server.service
             └─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon

  Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
  Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit at rmq-1'
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
  Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
  Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
  Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit at rmq-1'

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1784757/+subscriptions