[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"

Corey Bryant corey.bryant at canonical.com
Thu Jul 25 19:45:56 UTC 2019 

Reproducing with designate, the problem appears after we have one
designate unit that is at bionic-rocky (py3) and one that is at bionic-
queens (py2). It is resolved once all units are at bionic-rocky. More
details on that:

# upgrade one unit to bionic-rocky
juju config designate openstack-origin=cloud:bionic-rocky
juju run-action designate/0 openstack-upgrade

juju status
Unit                      Workload  Agent  Machine  Public address  Ports     Message
designate/0               error     idle   7        10.5.0.198      9001/tcp  Services not running that should be: designate-producer
  hacluster-designate/0   active    idle            10.5.0.198                Unit is ready and clustered
designate/1*              active    idle   8        10.5.0.6        9001/tcp  Unit is ready
  hacluster-designate/1*  active    idle            10.5.0.6                  Unit is ready and clustered

juju ssh designate/0

2019-07-25 18:04:27.452 24807 DEBUG designate.coordination [-] CCB:
list(self._get_members(self._group_id))=[b'juju-3d28eb-
coreycb2-8:86c49114-a37a-4e1a-8654-72bf1ccbde1f', 'juju-3d28eb-
coreycb2-7:d64cde68-d9f0-4910-9f1a-6d9724ea77b9'] _update_partitions
/usr/lib/python3/dist-packages/designate/coordination.py:202

where:
- machine 8 (b'juju-3d28eb-coreycb2-8:86c49114-a37a-4e1a-8654-72bf1ccbde1f') hasn't been upgraded yet and is still bionic-queens
- machine 7 ('juju-3d28eb-coreycb2-7:d64cde68-d9f0-4910-9f1a-6d9724ea77b9') is the machine that was upgraded to bionic-rocky

Note: designate coordinator backend gets the juju strings by calling
get_members() from the tooz memcached backend. /usr/lib/python3/dist-
packages/tooz/drivers/memcached.py

# now try to upgrade the other unit to bionic-rocky
juju run-action designate/1 openstack-upgrade

# restart memcache and designate-producer on designate/0 and resolve
juju resolved designate/0

# all better

Unit                      Workload  Agent  Machine  Public address  Ports     Message
designate/0               active    idle   7        10.5.0.198      9001/tcp  Unit is ready
  hacluster-designate/0   active    idle            10.5.0.198                Unit is ready and clustered
designate/1*              active    idle   8        10.5.0.6        9001/tcp  Unit is ready
  hacluster-designate/1*  active    idle            10.5.0.6                  Unit is ready and clustered

juju ssh designate/0

2019-07-25 19:31:11.365 12217 DEBUG designate.coordination [-] CCB:
list(self._get_members(self._group_id))=['juju-3d28eb-
coreycb2-7:0ceca5d8-9eee-492d-bead-f99ee990a21b', 'juju-3d28eb-
coreycb2-8:134e6142-4364-43a5-96fc-b9f125ba87a0'] _update_partitions
/usr/lib/python3/dist-packages/designate/coordination.py:202

Both strings are str type now and able to be sorted.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to designate in Ubuntu.
https://bugs.launchpad.net/bugs/1828534

Title:
  [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not
  running that should be: designate-producer"

Status in OpenStack Designate Charm:
  Invalid
Status in designate package in Ubuntu:
  Triaged

Bug description:
  Designate has to use Worker and Producer as of Rocky as the support
  for zone manager and pool manager was up for removal in Rocky. This
  was addressed in https://bugs.launchpad.net/charm-
  designate/+bug/1773190

  During a Queens to Rocky upgrade I ran into an issue with designate-
  producer being down (see the details below).

  I was able to successfully start designate-producer by hand. It looks
  like there were several attempts to restart the service by systemd
  itself ("Scheduled restart job, restart counter is at 5.") which
  eventually failed when the failcount became higher than 5. Debugging
  mode was disabled so I did not see anything in the producer log.

  designate/0*                  blocked   idle       2/lxd/2  10.232.46.153   9001/tcp            Services not running that should be: designate-producer
    hacluster-designate/0*      active    idle                10.232.46.153                       Unit is ready and clustered

  systemctl list-unit-files | grep designate-
  designate-agent.service                  enabled
  designate-api.service                    enabled
  designate-central.service                enabled
  designate-mdns.service                   enabled
  designate-producer.service               enabled
  designate-sink.service                   enabled
  designate-worker.service                 enabled

  ubuntu at juju-eeda89-2-lxd-2:~$ journalctl -u designate-producer.service
  https://paste.ubuntu.com/p/WRXYvfynnd/

  ubuntu at juju-eeda89-2-lxd-2:~$ pgrep -af designate
  172877 bash /lib/systemd/system/jujud-unit-hacluster-designate-0/exec-start.sh
  172878 bash /lib/systemd/system/jujud-unit-designate-0/exec-start.sh
  172884 /var/lib/juju/tools/unit-designate-0/jujud unit --data-dir /var/lib/juju --unit-name designate/0 --debug
  172886 /var/lib/juju/tools/unit-hacluster-designate-0/jujud unit --data-dir /var/lib/juju --unit-name hacluster-designate/0 --debug
  392388 /usr/bin/python3.6 /usr/bin/designate-mdns --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-mdns.log
  392399 /usr/bin/python3.6 /usr/bin/designate-agent --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-agent.log
  392405 /usr/bin/python3.6 /usr/bin/designate-worker --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-worker.log
  392411 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
  392415 /usr/bin/python3.6 /usr/bin/designate-sink --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-sink.log
  392423 /usr/bin/python3.6 /usr/bin/designate-api --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-api.log
  392481 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
  392482 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
  392483 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
  392484 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log

  ----------

  designate/0 unit log:

  2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
  2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.

  # journalctl -u designate-producer
  journalctl -u designate-producer | grep start
  May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
  May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 1.
  May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
  May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 2.
  May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
  May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 3.
  May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
  May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 4.
  May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
  May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 5.
  May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.

  designate/0 unit log (continued):

  2019-05-09 00:20:58 INFO juju-log Purging ['designate-pool-manager',
  'designate-zone-manager', 'python-designate', 'python-memcache']

  2019-05-09 00:20:59 DEBUG openstack-upgrade Removing designate-pool-manager (1:7.0.0-0ubuntu1~cloud0) ...
  2019-05-09 00:21:15 DEBUG openstack-upgrade Removing designate-zone-manager (1:7.0.0-0ubuntu1~cloud0) ...
  2019-05-09 00:21:31 DEBUG openstack-upgrade Removing python-designate (1:7.0.0-0ubuntu1~cloud0) ...

  2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using
  /usr/bin/python3-designate-producer to provide /usr/bin/designate-
  producer (designate-producer) in auto mode

  -------------------

  grep producer /var/log/juju/unit-designate-0.log
  2019-05-09 00:18:43 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python2-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
  2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-central', 'designate-common', 'designate-mdns', 'designate-worker', 'designate-sink', 'designate-producer', 'bind9utils', 'python3-designate', 'python-apt', 'memcached', 'python3-memcache', 'haproxy', 'apache2'] with options: ['--option', 'Dpkg::Options::=--force-confnew', '--option', 'Dpkg::Options::=--force-confdef']
  2019-05-09 00:19:58 DEBUG openstack-upgrade   designate-producer designate-worker python3-amqp python3-anyjson
  2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 designate-producer all 1:7.0.0-0ubuntu1~cloud0 [10.6 kB]
  2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
  2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-designate-producer_1%3a7.0.0-0ubuntu1~cloud0_all.deb ...
  2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
  2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
  2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
  2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list