[Bug 1374999] [NEW] iSCSI volume detach does not correctly remove the multipath device descriptors
Launchpad Bug Tracker
1374999 at bugs.launchpad.net
Wed Oct 29 16:39:59 UTC 2014
You have been subscribed to a public bug by Ubuntu Foundations Team Bug Bot (crichton):
iSCSI volume detach does not correctly remove the multipath device
descriptors
tested environment:
nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.
I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths.
Here is the terminal log for last volume detach.
openstack at W1DEV103:~/devstack$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
|
ID
| Status | Name | Size | Volume Type | Bootable |
Attached to
|
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
None
| false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
openstack at W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:38:13 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- 4:0:0:42 sdb 8:16 active undef running
| |- 5:0:0:42 sdd 8:48 active undef running
| |- 6:0:0:42 sdf 8:80 active undef running
| `- 7:0:0:42 sdh 8:112 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- 11:0:0:42 sdp 8:240 active undef running
|- 8:0:0:42 sdj 8:144 active undef running
|- 9:0:0:42 sdl 8:176 active undef running
`- 10:0:0:42 sdn 8:208 active undef running
openstack at W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:38:19 JST 2014
tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6
openstack at W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
56a63288-5cc0-4f5c-9197-cde731172dd8
openstack at W1DEV103:~/devstack$
openstack at W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
|
ID
| Status | Name | Size | Volume Type | Bootable |
Attached to
|
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
None
| false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
openstack at W1DEV103:~/devstack$
openstack at W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
ID
| Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
None
| false |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
openstack at W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:39:23 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 ,
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| `- #:#:#:# - #:# active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
`- #:#:#:# - #:# active undef running
openstack at W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:39:27 JST 2014
iscsiadm: No active sessions.
Then I manually removed the multipaths,
openstack at W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
openstack at W1CN103:/etc/iscsi$ sudo multipath -l
openstack at W1CN103:/etc/iscsi$
I think the problem is in,
virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):
End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
self._remove_multipath_device_descriptor(multipath_device)
return
Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_multipath_device_descriptor due to following if statement.
if not in_use:
# disconnect if no other multipath devices with same iqn
self._disconnect_mpath(iscsi_properties, ips_iqns)
return
It just disconnect them but not remove them.
One of the reasons why we have to remove them is,
https://bugs.launchpad.net/nova/+bug/1223975
IMO, we should call _remove_multipath_device_descriptor in above if
statement before return.
** Affects: nova
Importance: Undecided
Assignee: Rafael David Tinoco (inaddy)
Status: Confirmed
** Affects: nova (Ubuntu)
Importance: Undecided
Status: New
** Affects: nova (Ubuntu Trusty)
Importance: Undecided
Status: New
** Tags: cts patch
--
iSCSI volume detach does not correctly remove the multipath device descriptors
https://bugs.launchpad.net/bugs/1374999
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.
More information about the Ubuntu-sponsors
mailing list