[Bug 2059721] Re: ovn db upgrade

[Vadim Kuznetsov]

Here is my explanation:
1. According to ovn docs https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html
"The only step required after upgrading the packages is to restart ovn-northd, which automatically restarts the databases and upgrades the database schema, as well.
if you’re using a Linux distribution with systemd:
$ sudo systemctl restart ovn-northd

which is not true in Ubuntu case
```
ExecStart=/usr/share/ovn/scripts/ovn-ctl start_northd --ovn-manage-ovsdb=no --no-monitor $OVN_CTL_OPTS
```
ovn-manage-ovsdb=no, so it will not restart the DBs.

2. DB restarts by ovn-ovsdb-server-nb.service and ovn-ovsdb-server-sb.service
run_nb_ovsdb and run_sb_ovsdb options in ovn-ctl 
```
ExecStart=/usr/share/ovn/scripts/ovn-ctl run_nb_ovsdb $OVN_CTL_OPTS
```

3. This commit fixed Cluster OVN database upgrade https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c
in this bug https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1907081

This is the code that is responsible for the upgrade
https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L293
```
if test X$detach = Xno && test $mode = cluster && test -z "$cluster_remote_addr" ; then
```
First two checks are true.
Third check means upgrade_cluster (L313) will run on leader node, i.e  --db-nb-cluster-remote-addr= and --db-sb-cluster-remote-addr= is not set.

4. During rolling upgrades of controller nodes, the leader will likely move.
OVS_CTL_OPTS generated from this template
https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/2023.1/templates/ovn-northd-opts.j2
but only when new node joins the cluster https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/2023.1/tasks/providers/ovn_cluster_setup.yml#L105
so after upgrade all 3 nodes will have --db-nb-cluster-remote-addr and --db-sb-cluster-remote-addr set to the some value that was leader node at that time.

What we need is --db-nb-cluster-remote-addr= not set on leader node, so
during ovn-central restart on leader node upgrade_cluster will run.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/2059721

Title:
  ovn db upgrade

Status in OpenStack-Ansible:
  Triaged
Status in ovn package in Ubuntu:
  New

Bug description:
  After distro upgrade of the openstack-ansible 2023.1 site from Ubuntu
  20.04 to Ubuntu 22.04 OVN cluster DB was not upgraded.

  root at b-mgmt-neutron-ovn-northd-container-cb715707:/var/log/ovn# ovn-nbctl --version
  ovn-nbctl 23.03.1
  Open vSwitch Library 3.1.2
  DB Schema 7.0.0

  root at b-mgmt-neutron-ovn-northd-container-cb715707:/var/log/ovn# ovn-sbctl --version
  ovn-sbctl 23.03.1
  Open vSwitch Library 3.1.2
  DB Schema 20.27.0

  However

  root at b-mgmt-neutron-ovn-northd-container-cb715707:/var/log/ovn# ovsdb-client get-schema-version unix:/var/run/ovn/ovnnb_db.sock OVN_Northbound
  6.1.0
  root at b-mgmt-neutron-ovn-northd-container-cb715707:/var/log/ovn# ovsdb-client get-schema-version unix:/var/run/ovn/ovnsb_db.sock OVN_Southbound
  20.21.0

  Restrarting ovn-central did not help.

  ovn-northd.log:
  2024-03-28T17:28:52.521Z|00053|ovsdb_idl|WARN|OVN_Southbound database lacks Chassis_Template_Var table (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00054|ovsdb_idl|WARN|Load_Balancer table in OVN_Southbound database lacks datapath_group column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00055|ovsdb_idl|WARN|MAC_Binding table in OVN_Southbound database lacks timestamp column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00056|ovsdb_idl|WARN|OVN_Southbound database lacks Mirror table (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00057|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks additional_chassis column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00058|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks additional_encap column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00059|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks mirror_rules column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00060|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks port_security column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00061|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks requested_additional_chassis column (database needs upgrade?)
  2024-03-28T17:28:52.521Z|00062|ovsdb_idl|WARN|OVN_Southbound database lacks Static_MAC_Binding table (database needs upgrade?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/openstack-ansible/+bug/2059721/+subscriptions