[Bug 1749959] Re: Upgrade of erlang-base left rabbitmq-server dead

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: erlang (Ubuntu)
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to erlang in Ubuntu.
https://bugs.launchpad.net/bugs/1749959

Title:
  Upgrade of erlang-base left rabbitmq-server dead

Status in erlang package in Ubuntu:
  Confirmed

Bug description:
  Last night, unattended-upgrades upgraded the erlang packages on our
  RabbitMQ server from 1:18.3-dfsg-1ubuntu3 to 1:18.3-dfsg-1ubuntu3.1.
  erlang-base's prerm script successfully found and stopped rabbitmq-
  server, and the postinst script tried to start it again:

  # From /var/log/apt/term.log, same in /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
  #...
  Preparing to unpack .../erlang-base_1%3a18.3-dfsg-1ubuntu3.1_amd64.deb ...
  Searching for services which depend on erlang and should be stopped...found: rabbitmq-server.
  Stopping services which depend on erlang
    rabbitmq-server: stopping...done.
  Services stopped successfully.
  Killing epmd...it is not running.
  Unpacking erlang-base (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ...
  Preparing to unpack .../erlang-nox_1%3a18.3-dfsg-1ubuntu3.1_all.deb ...
  Unpacking erlang-nox (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ...
  Processing triggers for man-db (2.7.5-1) ...
  Setting up erlang-base (1:18.3-dfsg-1ubuntu3.1) ...
  Searching for services which depend on erlang and should be started...found: rabbitmq-server.
  Starting services which depend on erlang
    rabbitmq-server: starting...done.
  Services started successfully.
  #...

  I looked into erlang-base.prerm and erlang-base.postinst and ran the
  relevant snippets manually, and it looks like they would have used
  "invoke-rc.d":

     invoke-rc.d rabbitmq-server stop  > /dev/null 2>&1   # from erlang-base.prerm
     invoke-rc.d rabbitmq-server start > /dev/null 2>&1   # from erlang-base.postinst

  But while the stop command succeeded, the start command did not, and
  there isn't any evidence in the logs that it was ever issued. This is
  the reason why I think this is a bug in the erlang package (or
  possibly invoke-rc.d/init-system-helpers?), and not in rabbitmq-
  server:

  # journalctl
  Feb 16 06:30:03 rabbit-b systemd[1]: Starting Daily apt upgrade and clean activities...
  Feb 16 06:30:48 rabbit-b systemd[1]: Reloading.
  Feb 16 06:30:49 rabbit-b systemd[1]: Started ACPI event daemon.
  Feb 16 06:30:49 rabbit-b systemd[1]: Stopping RabbitMQ Messaging Server...
  Feb 16 06:30:50 rabbit-b rabbitmq[6596]: Stopping and halting node 'rabbit at rabbit-b' ...
  Feb 16 06:30:51 rabbit-b systemd[1]: Stopped RabbitMQ Messaging Server.
  Feb 16 06:30:52 rabbit-b systemd[1]: Reloading.
  Feb 16 06:30:52 rabbit-b systemd[1]: Started ACPI event daemon.
  Feb 16 06:30:54 rabbit-b systemd[1]: Reloading.
  Feb 16 06:30:54 rabbit-b systemd[1]: Started ACPI event daemon.
  # ... and the next log entry is from after unattended-upgrades had ended at 06:30:56

  # /var/log/rabbitmq/rabbit at rabbit-b.log
  =INFO REPORT==== 16-Feb-2018::06:30:50 ===
  Stopping RabbitMQ
  #... lots of AMQP connection closures
  =INFO REPORT==== 16-Feb-2018::06:30:51 ===
  Halting Erlang VM
  # ... and the next log entry is from 2h later when we manually started RabbitMQ again

  This is Ubuntu 16.04.2 LTS with all security patches and the latest
  rabbitmq-server (3.5.7-1ubuntu0.16.04.2)

  PS: Where are the sources for erlang-base.prerm and erlang-
  base.postinst? https://packages.debian.org/source/stretch/erlang
  points to http://anonscm.debian.org/viewvc/pkg-erlang/erlang/trunk/,
  but that link is dead

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1749959/+subscriptions