[Bug 1718133] Re: Live block migration in Pike fails due to qemu-img

Wed Sep 20 09:18:16 UTC 2017

This patch set was never accepted that way, but yeah reviving that might
be an option to work around this. Also this was long ago, but the lock
rewrite/enablement happened only recently so a totally new approach
might be needed (or is present without us knowing).

We might ping qemu-devel about that to be clear of too much assumptions.

To help with that I found that repro could be as easy as:
1. start a guest
2. get the blockinfo and pick a disk
3. run qemu-img info on it while running

Example:
#1 (any other kvm guest will do as well)
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
uvt-kvm create --password=ubuntu xenial-test release=xenial arch=amd64 label=daily
#2 Depending on how you started it you might know the path already
virsh domblklist xenial-test
Target     Source
------------------------------------------------
vda        /var/lib/uvtool/libvirt/images/xenial-test.qcow
vdb        /var/lib/uvtool/libvirt/images/xenial-test-ds.qcow
#3 check img info
$ qemu-img info /var/lib/uvtool/libvirt/images/xenial-test.qcow
qemu-img: Could not open '/var/lib/uvtool/libvirt/images/xenial-test.qcow': Failed to get shared "write" lock
Is another process using the image?

To simplify further - we don't care what the guest does:
$ qemu-img create -f qcow2 /tmp/test.qcow2 1M
$ qemu-system-x86_64 -hda /tmp/test.qcow2 -enable-kvm -nodefaults -nographic &
$ qemu-img info /tmp/test.qcow2
qemu-img: Could not open '/tmp/test.qcow2': Failed to get shared "write" lock
Is another process using the image?

I't won't get easier, but that might be great to re-trigger the old
nolock thread.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1718133

Title:
  Live block migration in Pike fails due to qemu-img

Status in nova package in Ubuntu:
  Confirmed
Status in qemu package in Ubuntu:
  Confirmed

Bug description:
  In Pike from Cloud Archive, Live Block Migrations fail:
  Error updating resources for node compute1.: InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk : Unexpected error while running command.
  Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk
  Exit code: 1
  Stdout: u''
  Stderr: u'qemu-img: Could not open \'/var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk\': Failed to get shared "write" lock\nIs another process using the image?\n'
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Traceback (most recent call last):
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6629, in update_available_resource_for_node
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 641, in update_available_resource
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5831, in get_available_resource
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7330, in _get_disk_over_committed_size_total
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     config, block_device_info)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7248, in _get_instance_disk_info_from_config
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     backing_file = libvirt_utils.get_disk_backing_file(path)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 200, in get_disk_backing_file
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     backing_file = images.qemu_img_info(path, format).backing_file
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 72, in qemu_img_info
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     raise exception.InvalidDiskInfo(reason=msg)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk : Unexpected error while running command.
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Exit code: 1
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Stdout: u''
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Stderr: u'qemu-img: Could not open \'/var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk\': Failed to get shared "write" lock\nIs another process using the image?\n'
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager 
  2017-09-18 19:56:44.775 11607 INFO nova.compute.manager [req-b6b3e84d-88fd-4052-9e56-bc89a1739ca3 3d4ca720acb84ba19cbbc7d5042d1f56 914a8d5ddac24f1c9b74e646633bab1c - default default] [instance: ccca487b-d5db-4324-81fb-2665e60da038] Terminating instance

  Maybe it is not a nova, but qemu issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1718133/+subscriptions