[Bug 1784757] Re: [SRU] rabbitmq-server does not properly shutdown

[Sergio Durigan Junior]

First, verifying that the current package fails:

root at bug1784757-rabbitmq-server:~# apt policy rabbitmq-server
rabbitmq-server:
  Installed: 3.6.10-1ubuntu0.3
  Candidate: 3.6.10-1ubuntu0.3
  Version table:
 *** 3.6.10-1ubuntu0.3 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     3.6.10-1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root at bug1784757-rabbitmq-server:~# time systemctl restart rabbitmq-server.service

real    1m33.535s
user    0m0.005s
sys     0m0.007s

Now, installing rabbitmq-server from -proposed, and verifying that the
bug is fixed:

root at bug1784757-rabbitmq-server:~# apt policy rabbitmq-server
rabbitmq-server:
  Installed: 3.6.10-1ubuntu0.4
  Candidate: 3.6.10-1ubuntu0.4
  Version table:
 *** 3.6.10-1ubuntu0.4 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     3.6.10-1ubuntu0.3 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     3.6.10-1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root at bug1784757-rabbitmq-server:~# for i in $(seq 10); do time systemctl restart rabbitmq-server.service ; done

real    0m2.980s
user    0m0.006s
sys     0m0.004s

real    0m3.010s
user    0m0.004s
sys     0m0.000s

real    0m4.907s
user    0m0.004s
sys     0m0.000s

real    0m3.012s
user    0m0.002s
sys     0m0.002s

real    0m2.899s
user    0m0.004s
sys     0m0.000s

real    0m2.975s
user    0m0.004s
sys     0m0.000s

real    0m3.036s
user    0m0.002s
sys     0m0.003s

real    0m3.013s
user    0m0.004s
sys     0m0.000s

real    0m3.003s
user    0m0.004s
sys     0m0.000s

real    0m3.040s
user    0m0.004s
sys     0m0.000s

Therefore, I'm marking this bug as verified.

** Tags removed: verification-needed-bionic
** Tags added: verification-done-bionic

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1784757

Title:
  [SRU] rabbitmq-server does not properly shutdown

Status in rabbitmq-server package in Ubuntu:
  Fix Released
Status in rabbitmq-server source package in Bionic:
  Fix Committed
Status in rabbitmq-server source package in Cosmic:
  Won't Fix
Status in rabbitmq-server package in Debian:
  Fix Released

Bug description:
  [Impact]

  The systemd file rabbitmq-server.service on Bionic uses "Type=simple"
  when defining the service, but unfortunately this doesn't work very
  well for rabbitmq-server. In certain situations, systemd will fail to
  keep track of a start/stop/restart event, and will hang for 90 seconds
  before giving the prompt back to the user. Another problem is that
  rabbitmq-server must start after the epmd service, so we need to
  explicitly declare this dependency in the service file.

  [Test Case]

  Although I was able to reproduce this almost 100% of the time, there
  were rare occasions when the restart procedure finished normally. I
  was also only able to reproduce it using a bionic VM, not a container.
  If you have multipass or lxd configured to launch VMs, that should be
  easy.

  The steps are:

  $ lxc launch ubuntu-daily:bionic --vm bug1784757-rabbitmq-server # or use multipass
  $ lxc shell bug1784757-rabbitmq-server
  # apt update
  # apt install rabbitmq-server -y
  # systemctl restart rabbitmq-server.service

  In a normal scenario, the restart should take around 3 seconds or
  less. With the bug, it takes around 90 seconds. If you can't reproduce
  it, try running "systemctl restart" again. A quick way to trigger it
  is to run a for loop like:

  # for i in $(seq 10); do time systemctl restart rabbitmq-
  server.service ; done

  [Regression Potential]

  * Because rabbitmq-server implements systemd's "Type=notify" using
  socat to communicate with systemd-notify over a socket, we will be
  introducing another point of failure (socat) in the mix.

  * So, albeit unlikely, there may be a problem when using socat.  That
  would not be a regression, though, since the outcome would be the same
  as we have today: "systemctl restart" would not properly work, even
  though the service did restart.

  [Original Report]

  When I run `systemctl restart rabbitmq-server` it waits for 90 seconds
  then systemd sends SIGKILL to it.

  Presumably the `epmd` process does not receive SIGTERM, since if I run
  `kill 1493` (or whatever pid it currently is) then restart happens
  straight after that successfully

  ● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
    Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
    Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
    Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
   Main PID: 177 (code=killed, signal=TERM)
      Tasks: 1 (limit: 4915)
     CGroup: /system.slice/rabbitmq-server.service
             └─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon

  Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
  Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit at rmq-1'
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
  Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
  Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
  Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit at rmq-1'

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1784757/+subscriptions