[Bug 1770295] Re: ceilometer compute cannot collection memory.usage meter for all vm in one compute node

[Christian Ehrhardt ]

It certainly doesn't match the initially reported issue about state lock and memory stats.
Please update the bug if this is still an issues with newer versions and the recent updates of buguldey turn out to be nice, but actually unrelated to the initial issue).

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1770295

Title:
  ceilometer compute cannot collection memory.usage meter for all vm in
  one compute node

Status in Ceilometer:
  In Progress
Status in Ubuntu Cloud Archive:
  New
Status in libvirt package in Ubuntu:
  Fix Released
Status in libvirt source package in Bionic:
  Incomplete

Bug description:
  when shutdown vm which long time, the libvirt while
  raise libvirtError: Timed out during operation: cannot
  acquire state change lock (held by
  remoteDispatchDomainShutdown), if occur, all the vm in
  the Physics host can not get memory.usage meter anymore!

  2018-05-03 07:57:02.211 82162 ERROR
  ceilometer.compute.virt.libvirt.inspector [-] Get used size of sda in
  instance-00008a39 failed. The reason is error: Guest agent is not
  responding: QEMU guest agent is not connected

  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk [-] Ignoring instance instance-00008c37 (a7a5433a-9958-4b3c-bb54-b1b693e1c3d7) : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainShutdown)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk Traceback (most recent call last):
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/disk.py", line 624, in get_samples
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     instance,
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/disk.py", line 560, in _populate_cache
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     for disk, info in disk_info:
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib/python2.7/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 242, in inspect_disk_info
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     block_info = domain.blockInfo(device)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 690, in blockInfo
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     if ret is None: raise libvirtError ('virDomainGetBlockInfo() failed', dom=self)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainShutdown)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk 
  2018-05-03 07:57:35.142 82162 ERROR ceilometer.compute.virt.libvirt.inspector [-] Get used size of vda in instance-00008a5d failed. The reason is error: Guest agent is not responding: QEMU guest agent is not connected

  
  2018-05-03 07:58:34.602 82162 WARNING ceilometer.compute.pollsters.memory [-] Cannot inspect data of MemoryUsagePollster for a7a5433a-9958-4b3c-bb54-b1b693e1c3d7: Failed to inspect memory usage of a7a5433a-9958-4b3c-bb54-b1b693e1c3d7, can not get info from libvirt: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainShutdown)
  2018-05-03 07:58:34.606 82162 ERROR ceilometer.agent.manager [-] Prevent pollster memory.usage from polling [<NovaLikeServer: VM-21A-01J14>, <NovaLikeServer: VM-21A-01J2H>, <NovaLikeServer: VM-21A003VB>, <NovaLikeServer: VM-21A-01HXN>, <NovaLikeServer: VM-21A-01HXJ>, <NovaLikeServer: VM-21A-01HXK>, <NovaLikeServer: VM-21A-01J2Z>, <NovaLikeServer: VM-21A-01J3C>, <NovaLikeServer: VM-21A-01HXQ>, <NovaLikeServer: VM-21A-01HXR>, <NovaLikeServer: VM-21A007BS>, <NovaLikeServer: VM-21A-01J3B>, <NovaLikeServer: VM-21A-01J2N>, <NovaLikeServer: VM-21A-01HXU>, <NovaLikeServer: VM-21A-01J3F>, <NovaLikeServer: VM-21A-01J3H>, <NovaLikeServer: VM-21A-01HY0>, <NovaLikeServer: VM-21A-01J3D>] on source memory_utilization anymore!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1770295/+subscriptions