[Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup

[Alan Baghumian]

I confirm shelving then unshelving the instance does do the trick:

1) Launched a volume backed instance and checked the connection_info
data in the database:

mysql> select connection_info from volume_attachment where volume_id='132f6fb0-1119-4e8a-a555-18e4dfc8cbbf';
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| connection_info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"name": "cinder-ceph-nvme/volume-132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "hosts": ["10.1.15.222", "10.1.15.140", "10.1.15.138"], "ports": ["6789", "6789", "6789"], "cluster_name": "ceph", "auth_enabled": true, "auth_username": "cinder-ceph-nvme", "secret_type": "ceph", "secret_uuid": "1ffc33c5-d23d-4355-9297-f26aae327a49", "volume_id": "132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "discard": true, "qos_specs": null, "access_mode": "rw", "encrypted": false, "cacheable": false, "driver_volume_type": "rbd", "attachment_id": "70405775-a24c-4235-a342-fdd0c64ece93"} |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


2) Artificially updated the records with invalid IP addresses:

mysql> update volume_attachment set connection_info='{"name": "cinder-ceph-nvme/volume-132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "hosts": ["10.1.15.225", "10.1.15.142", "10.1.15.139"], "ports": ["6789", "6789", "6789"], "cluster_name": "ceph", "auth_enabled": true, "auth_username": "cinder-ceph-nvme", "secret_type": "ceph", "secret_uuid": "1ffc33c5-d23d-4355-9297-f26aae327a49", "volume_id": "132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "discard": true, "qos_specs": null, "access_mode": "rw", "encrypted": false, "cacheable": false, "driver_volume_type": "rbd", "attachment_id": "70405775-a24c-4235-a342-fdd0c64ece93"}' where volume_id='132f6fb0-1119-4e8a-a555-18e4dfc8cbbf' limit 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

3) Shelved then Unshelved the instance, I see a fresh entry in the
connection_info table with the correct mon IP addresses:

mysql> select connection_info from volume_attachment where volume_id='132f6fb0-1119-4e8a-a555-18e4dfc8cbbf';
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| connection_info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"name": "cinder-ceph-nvme/volume-132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "hosts": ["10.1.15.222", "10.1.15.140", "10.1.15.138"], "ports": ["6789", "6789", "6789"], "cluster_name": "ceph", "auth_enabled": true, "auth_username": "cinder-ceph-nvme", "secret_type": "ceph", "secret_uuid": "1ffc33c5-d23d-4355-9297-f26aae327a49", "volume_id": "132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "discard": true, "qos_specs": null, "access_mode": "rw", "encrypted": false, "cacheable": false, "driver_volume_type": "rbd", "attachment_id": "0ef8a98c-9f61-4864-bf41-eb8ccce88ccf"} |
| {"name": "cinder-ceph-nvme/volume-132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "hosts": ["10.1.15.225", "10.1.15.142", "10.1.15.139"], "ports": ["6789", "6789", "6789"], "cluster_name": "ceph", "auth_enabled": true, "auth_username": "cinder-ceph-nvme", "secret_type": "ceph", "secret_uuid": "1ffc33c5-d23d-4355-9297-f26aae327a49", "volume_id": "132f6fb0-1119-4e8a-a555-18e4dfc8cbbf", "discard": true, "qos_specs": null, "access_mode": "rw", "encrypted": false, "cacheable": false, "driver_volume_type": "rbd", "attachment_id": "70405775-a24c-4235-a342-fdd0c64ece93"} |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

4) Verified the instance is running and was able to SSH to it

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1452641

Title:
  Static Ceph mon IP addresses in connection_info can prevent VM startup

Status in OpenStack Compute (nova):
  Confirmed
Status in nova package in Ubuntu:
  Triaged

Bug description:
  The Cinder rbd driver extracts the IP addresses of the Ceph mon servers from the Ceph mon map when the instance/volume connection is established. This info is then stored in nova's block-device-mapping table and is never re-validated down the line. 
  Changing the Ceph mon servers' IP adresses will prevent the instance from booting as the stale connection info will enter the instance's XML. One idea to fix this would be to use the information from ceph.conf, which should be an alias or a loadblancer, directly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions