[Bug 1770295] Re: ceilometer compute cannot collection memory.usage meter for all vm in one compute node

[Corey Bryant]

** Also affects: cloud-archive/queens
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/queens
       Status: New => Incomplete

** Changed in: cloud-archive/queens
   Importance: Undecided => Low

** Changed in: cloud-archive
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1770295

Title:
  ceilometer compute cannot collection memory.usage meter for all vm in
  one compute node

Status in Ceilometer:
  In Progress
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Incomplete
Status in libvirt package in Ubuntu:
  Fix Released
Status in libvirt source package in Bionic:
  Incomplete

Bug description:
  when shutdown vm which long time, the libvirt while
  raise libvirtError: Timed out during operation: cannot
  acquire state change lock (held by
  remoteDispatchDomainShutdown), if occur, all the vm in
  the Physics host can not get memory.usage meter anymore!

  2018-05-03 07:57:02.211 82162 ERROR
  ceilometer.compute.virt.libvirt.inspector [-] Get used size of sda in
  instance-00008a39 failed. The reason is error: Guest agent is not
  responding: QEMU guest agent is not connected

  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk [-] Ignoring instance instance-00008c37 (a7a5433a-9958-4b3c-bb54-b1b693e1c3d7) : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainShutdown)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk Traceback (most recent call last):
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/disk.py", line 624, in get_samples
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     instance,
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/disk.py", line 560, in _populate_cache
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     for disk, info in disk_info:
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib/python2.7/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 242, in inspect_disk_info
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     block_info = domain.blockInfo(device)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 690, in blockInfo
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk     if ret is None: raise libvirtError ('virDomainGetBlockInfo() failed', dom=self)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainShutdown)
  2018-05-03 07:57:32.474 82162 ERROR ceilometer.compute.pollsters.disk 
  2018-05-03 07:57:35.142 82162 ERROR ceilometer.compute.virt.libvirt.inspector [-] Get used size of vda in instance-00008a5d failed. The reason is error: Guest agent is not responding: QEMU guest agent is not connected

  
  2018-05-03 07:58:34.602 82162 WARNING ceilometer.compute.pollsters.memory [-] Cannot inspect data of MemoryUsagePollster for a7a5433a-9958-4b3c-bb54-b1b693e1c3d7: Failed to inspect memory usage of a7a5433a-9958-4b3c-bb54-b1b693e1c3d7, can not get info from libvirt: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainShutdown)
  2018-05-03 07:58:34.606 82162 ERROR ceilometer.agent.manager [-] Prevent pollster memory.usage from polling [<NovaLikeServer: VM-21A-01J14>, <NovaLikeServer: VM-21A-01J2H>, <NovaLikeServer: VM-21A003VB>, <NovaLikeServer: VM-21A-01HXN>, <NovaLikeServer: VM-21A-01HXJ>, <NovaLikeServer: VM-21A-01HXK>, <NovaLikeServer: VM-21A-01J2Z>, <NovaLikeServer: VM-21A-01J3C>, <NovaLikeServer: VM-21A-01HXQ>, <NovaLikeServer: VM-21A-01HXR>, <NovaLikeServer: VM-21A007BS>, <NovaLikeServer: VM-21A-01J3B>, <NovaLikeServer: VM-21A-01J2N>, <NovaLikeServer: VM-21A-01HXU>, <NovaLikeServer: VM-21A-01J3F>, <NovaLikeServer: VM-21A-01J3H>, <NovaLikeServer: VM-21A-01HY0>, <NovaLikeServer: VM-21A-01J3D>] on source memory_utilization anymore!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1770295/+subscriptions