[Bug 1827690] Re: [19.04][stein] barbican-worker is down: Requested revision 1a0c2cdafb38 overlaps with other requested revisions 39cf2e645cba

[OpenStack Infra]

Reviewed:  https://review.opendev.org/c/openstack/charm-barbican/+/756931
Committed: https://opendev.org/openstack/charm-barbican/commit/ca80b266ed831813a2a9ecec84d288ec3f8f5104
Submitter: "Zuul (22348)"
Branch:    master

commit ca80b266ed831813a2a9ecec84d288ec3f8f5104
Author: David Ames <david.ames at canonical.com>
Date:   Thu Oct 8 22:55:30 2020 +0000

    Run db sync manually by leader only
    
    Without db_auto_create = False each barbican-worker attempts to run
    alembic upgrades on the database whenever they start. This leads to
    race conditions that can leave the sate of the database broken
    (multiple alembic version that "overlap").
    
    This change runs the barbican-manage db upgrade by the leader only
    avoiding the race condition. charms.openstack also handles the
    openstack upgrade process with a call to instance.db_sync.
    
    Change-Id: I6b9498059c7057b73b1c3db0e355456c38b0510e
    Closes-Bug: #1827690


** Changed in: charm-barbican
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to barbican in Ubuntu.
https://bugs.launchpad.net/bugs/1827690

Title:
  [19.04][stein] barbican-worker is down: Requested revision
  1a0c2cdafb38 overlaps with other requested revisions 39cf2e645cba

Status in OpenStack Barbican Charm:
  Fix Committed
Status in barbican package in Ubuntu:
  Invalid

Bug description:
  After deploying barbican (stein, 19.04 charms) I got into a situation
  where both workers (two-node HA) are down due to a failed alembic
  migration:

  https://private-fileshare.canonical.com/~dima/charm-dumps/04-05-2019-barbican-0-var-log.tar.gz
  https://private-fileshare.canonical.com/~dima/charm-dumps/04-05-2019-barbican-1-var-log.tar.gz

  https://paste.ubuntu.com/p/J6GvwkXMWc/ - bundle.yaml

  May 03 20:51:11 juju-35b20e-3-lxd-0 systemd[1]: Started OpenStack Barbican Key Management Workers.
  May 03 20:51:39 juju-35b20e-3-lxd-0 systemd[1]: barbican-worker.service: Main process exited, code=exited, status=1/FAILURE
  May 03 20:51:39 juju-35b20e-3-lxd-0 systemd[1]: barbican-worker.service: Failed with result 'exit-code'.

  juju status barbican
  Model      Controller  Cloud/Region  Version  SLA          Timestamp
  openstack  samaas      samaas        2.6-rc1  unsupported  23:27:23Z

  App                 Version  Status   Scale  Charm           Store       Rev  OS      Notes
  barbican            8.0.0    blocked      2  barbican        jujucharms   18  ubuntu
  barbican-vault      1.2.2    active       2  barbican-vault  jujucharms    2  ubuntu
  hacluster-barbican           active       2  hacluster       jujucharms   54  ubuntu

  Unit                     Workload  Agent  Machine  Public address  Ports              Message
  barbican/0               blocked   idle   3/lxd/0  10.232.46.135   9311/tcp,9312/tcp  Services not running that should be: barbican-worker
    barbican-vault/1       active    idle            10.232.46.135                      Unit is ready
    hacluster-barbican/1   active    idle            10.232.46.135                      Unit is ready and clustered
  barbican/1*              blocked   idle   2/lxd/0  10.232.46.130   9311/tcp,9312/tcp  Services not running that should be: barbican-worker
    barbican-vault/0*      active    idle            10.232.46.130                      Unit is ready
    hacluster-barbican/0*  active    idle            10.232.46.130                      Unit is ready and clustered

  https://paste.ubuntu.com/p/BS3fHw287r/
  2019-05-03 21:13:46.582 115638 ERROR barbican alembic.util.exc.CommandError: Requested revision 1a0c2cdafb38 overlaps with other requested revisions 39cf2e645cba

  mysql> use barbican;
  mysql> select * from alembic_version;
  +--------------+
  | version_num  |
  +--------------+
  | 1a0c2cdafb38 |
  | 39cf2e645cba |
  +--------------+
  2 rows in set (0.00 sec)

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-barbican/+bug/1827690/+subscriptions