[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup

[Peter]

We have the same issue with Rocky.
One of my SQL wizard colleague helped me with some query, which can change the block_device_mapping table, and the RBD host/username/ports (if you change the number of ceph monitors, you'll need it)

Since we have multiple Zones, and our change will only affect Zone1, and
since we have iSCSI storage too, we needed a bit more precise query.

Also my colleague pointed out, that the connection_info is JSON, and
since the MariaDB 10.2.3 have support for json, he used them, just to be
sure not to mess up the syntax.

So the three query (use with caution, and - of course - your own risk!):

update block_device_mapping as b set connection_info =
json_replace(connection_info, '$.data.auth_username', 'dev-r1z1-c4e')
where instance_uuid in (select i.uuid from instances as i where
i.deleted_at is null and i.availability_zone = 'Zone1') AND
JSON_EXISTS(b.connection_info, '$.data.hosts') = 1 and b.deleted_at is
NULL;


update block_device_mapping as b set connection_info = json_replace(connection_info, '$.data.hosts', JSON_ARRAY("10.1.58.156", "10.1.58.157", "10.1.58.158")) where instance_uuid in (select i.uuid from instances as i where i.deleted_at is null and i.availability_zone = 'Zone1') AND JSON_EXISTS(b.connection_info, '$.data.hosts') = 1 and b.deleted_at is NULL;


update block_device_mapping as b set connection_info = json_replace(connection_info, '$.data.ports', JSON_ARRAY("6789", "6789", "6789")) where instance_uuid in (select i.uuid from instances as i where i.deleted_at is null and i.availability_zone = 'Zone1') AND JSON_EXISTS(b.connection_info, '$.data.hosts') = 1 and b.deleted_at is NULL;

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1452641

Title:
  Static Ceph mon IP addresses in connection_info can prevent VM startup

Status in OpenStack Compute (nova):
  In Progress
Status in nova package in Ubuntu:
  Triaged

Bug description:
  The Cinder rbd driver extracts the IP addresses of the Ceph mon servers from the Ceph mon map when the instance/volume connection is established. This info is then stored in nova's block-device-mapping table and is never re-validated down the line. 
  Changing the Ceph mon servers' IP adresses will prevent the instance from booting as the stale connection info will enter the instance's XML. One idea to fix this would be to use the information from ceph.conf, which should be an alias or a loadblancer, directly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions