[Bug 1157599] [NEW] Using an iscsi device in the nova-volume VG lets nova-volume crash on system boot

[Tobias Bindhammer]

Public bug reported:

I have a RAID attached via iscsi to a node that runs nova-volume. This
device is in the nova-volumes VG.

root at vs-node4:/root# vgdisplay nova-volumes
  --- Volume group ---
  VG Name               nova-volumes
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  23
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3,19 TiB
  PE Size               4,00 MiB
  Total PE              836819
  Alloc PE / Size       514560 / 1,96 TiB
  Free  PE / Size       322259 / 1,23 TiB
  VG UUID               51A9z4-gdpn-FX9M-uH2T-qMM0-5Z90-bI6B29

However when booting the whole system, nova-volume crashes, as the
volume-group is not appearing in time:

2013-03-20 07:46:35 DEBUG nova.utils [req-9b8655d3-5471-48a9-946b-3b87e8d7280a None None] Führe Kommando (subprocess) aus: sudo nova-rootwrap vgs --noheadings -o name from (pid=1010) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
2013-03-20 07:46:37 CRITICAL nova [-] volume group nova-volumes doesn't exist
2013-03-20 07:46:37 TRACE nova Traceback (most recent call last):
2013-03-20 07:46:37 TRACE nova   File "/usr/bin/nova-volume", line 49, in <module>
2013-03-20 07:46:37 TRACE nova     service.wait()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait
2013-03-20 07:46:37 TRACE nova     _launcher.wait()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait
2013-03-20 07:46:37 TRACE nova     service.wait()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-03-20 07:46:37 TRACE nova     return self._exit_event.wait()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-03-20 07:46:37 TRACE nova     return hubs.get_hub().switch()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-03-20 07:46:37 TRACE nova     return self.greenlet.switch()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-03-20 07:46:37 TRACE nova     result = function(*args, **kwargs)
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
2013-03-20 07:46:37 TRACE nova     server.start()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start
2013-03-20 07:46:37 TRACE nova     self.manager.init_host()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 93, in init_host
2013-03-20 07:46:37 TRACE nova     self.driver.check_for_setup_error()
2013-03-20 07:46:37 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 107, in check_for_setup_error
2013-03-20 07:46:37 TRACE nova     % FLAGS.volume_group)
2013-03-20 07:46:37 TRACE nova Error: volume group nova-volumes doesn't exist

When issuing a 'service nova-volume start' afterwards, nova-volume is up
and running well. So the service is started too early as it seems. The
crash brings me into serious trouble, as then instances with attached
volumes don't get access to their volumes. Terminating instances then
leads to zombie instances and attached volumes that can't be detached
anymore. Sorting out things manually in the nova database enabled me to
start all instances again and properly attach all volumes. But nothing i
want to do on every reboot of that node. Any suggestions? With
pohysically attached disks thinsg work fine of course.

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: nova (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: iscsi lvm nova-volume nova-volumes upstart volume-grop

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1157599

Title:
  Using an iscsi device in the nova-volume VG lets nova-volume crash on
  system boot

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1157599/+subscriptions