[Bug 1943990] Re: After updating systemd packages to 245.4-4ubuntu3.13 and libvirt packages to 6.0.0-0ubuntu8.14, it is no longer possible to start kvm virtual machines

Fri Sep 17 19:51:16 UTC 2021

Some additional info:

The servers where I had problem were running a hyper convergent
solution, with ceph installed via ceph-ansible, including at least a
docker container (running node_exporter) - it is failing as well. These
servers were initially installed with general kernel (5.4.0-81-generic)
and a hwe kernel (5.8.0-34) was installed afterwards.

I succeeded in installing a new server with hwe kernel and updated
libraries, as well as upgrade a system without ceph nor docker; at the
upgrade, some errors concerned me, but didn't cause any bigger trouble:

Setting up libvirt-daemon-driver-qemu (6.0.0-0ubuntu8.14) ...
Setting up libvirt-daemon (6.0.0-0ubuntu8.14) ...
Setting up libvirt-daemon-driver-storage-rbd (6.0.0-0ubuntu8.14) ...
Setting up libvirt-daemon-system (6.0.0-0ubuntu8.14) ...
Job failed. See "journalctl -xe" for details.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
Setting up libvirt-daemon dnsmasq configuration.

-- A start job for unit libvirtd.service has begun execution.
-- 
-- The job identifier is 369924.
Sep 17 18:18:25 MGL1-SRV-0018 systemd[1]: Started Virtualization daemon.
-- Subject: A start job for unit libvirtd.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit libvirtd.service has finished successfully.
-- 
-- The job identifier is 369924.
Sep 17 18:18:25 MGL1-SRV-0018 nova-compute[3163250]: 2021-09-17 18:18:25.703 3163250 ERROR nova.scheduler.client.report [req-b80a57e5-3e9d-4dec-a861-d04363077b0c - - - - -] [req-38886c27-53ea-41b9-bf14-f68080cac535] Failed to update traits to [COMPUTE_TRUSTED_CERTS,COMPUTE_IMAGE_TYPE_AMI,HW_CPU_X86_VMX,HW_CPU_X86_MPX,COMPUTE_VOLUME_ATTACH_WITH_TAG,HW_CPU_X86_AVX512DQ,COMPUTE_GRAPHICS_MODEL_QXL,COMPUTE_NET_VIF_MODEL_E1000,COMPUTE_NET_VIF_MODEL_E1000E,HW_CPU_X86_AESNI,COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_DEVICE_TAGGING,HW_CPU_X86_SSE2,HW_CPU_X86_AVX2,COMPUTE_STORAGE_BUS_VIRTIO,HW_CPU_X86_SSSE3,COMPUTE_ACCELERATORS,COMPUTE_NET_VIF_MODEL_PCNET,HW_CPU_X86_AVX512CD,COMPUTE_GRAPHICS_MODEL_CIRRUS,HW_CPU_X86_FMA3,HW_CPU_X86_SSE41,HW_CPU_X86_AVX512F,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_STORAGE_BUS_SATA,COMPUTE_GRAPHICS_MODEL_VGA,HW_CPU_X86_BMI,COMPUTE_STORAGE_BUS_IDE,COMPUTE_IMAGE_TYPE_ARI,COMPUTE_STATUS_DISABLED,COMPUTE_NET_VIF_MODEL_SPAPR_VLAN,HW_CPU_X86_MMX,COMPUTE_NET_VIF_MODEL_RTL8139,COMPUTE_IMAGE_TYPE_RAW,COMPUTE_STORAGE_BUS_FDC,COMPUTE_GRAPHICS_MODEL_NONE,HW_CPU_X86_F16C,HW_CPU_X86_SSE,COMPUTE_RESCUE_BFV,HW_CPU_X86_AVX,COMPUTE_NET_VIF_MODEL_NE2K_PCI,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_STORAGE_BUS_SCSI,HW_CPU_X86_BMI2,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_VOLUME_MULTI_ATTACH,COMPUTE_VOLUME_EXTEND,HW_CPU_HYPERTHREADING,HW_CPU_X86_SSE42,HW_CPU_X86_AVX512VL,HW_CPU_X86_ABM,COMPUTE_NET_VIF_MODEL_VIRTIO,HW_CPU_X86_CLMUL,HW_CPU_X86_AVX512BW,HW_CPU_X86_AVX512VNNI,COMPUTE_GRAPHICS_MODEL_VMVGA,COMPUTE_GRAPHICS_MODEL_VIRTIO,COMPUTE_NODE,COMPUTE_STORAGE_BUS_USB] for resource provider with UUID f16fc1ab-3b42-431f-9d20-96a27224b90e.  Got 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Resource provider's generation already changed. Please update the generation and try again.  ", "request_id": "req-38886c27-53ea-41b9-bf14-f68080cac535"}]}
Sep 17 18:18:25 MGL1-SRV-0018 nova-compute[3163250]: 2021-09-17 18:18:25.703 3163250 WARNING nova.virt.libvirt.driver [req-b80a57e5-3e9d-4dec-a861-d04363077b0c - - - - -] An error occurred while updating compute node resource provider status to "disabled" for provider: f16fc1ab-3b42-431f-9d20-96a27224b90e: nova.exception.ResourceProviderUpdateConflict: A conflict was encountered attempting to update resource provider f16fc1ab-3b42-431f-9d20-96a27224b90e (generation 27): {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Resource provider's generation already changed. Please update the generation and try again.  ", "request_id": "req-38886c27-53ea-41b9-bf14-f68080cac535"}]}
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver Traceback (most recent call last):
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4434, in _update_compute_provider_status
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver     self.virtapi.update_compute_provider_status(
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 555, in update_compute_provider_status
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver     self.reportclient.set_traits_for_provider(
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 72, in wrapper
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver     return f(self, *a, **k)
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 1061, in set_traits_for_provider
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver     raise exception.ResourceProviderUpdateConflict(
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver nova.exception.ResourceProviderUpdateConflict: A conflict was encountered attempting to update resource provider f16fc1ab-3b42-431f-9d20-96a27224b90e (generation 27): {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Resource provider's generation already changed. Please update the generation and try again.  ", "request_id": "req-38886c27-53ea-41b9-bf14-f68080cac535"}]}
                                                     2021-09-17 18:18:25.703 3163250 ERROR nova.virt.libvirt.driver

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1943990

Title:
  After updating systemd packages to 245.4-4ubuntu3.13 and libvirt
  packages to 6.0.0-0ubuntu8.14, it is no longer possible to start kvm
  virtual machines

Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  # systemctl status systemd-machined.service 
  ● systemd-machined.service - Virtual Machine and Container Registration Service
       Loaded: loaded (/lib/systemd/system/systemd-machined.service; static; vendor preset: enabled)
       Active: inactive (dead)
         Docs: man:systemd-machined.service(8)
               https://www.freedesktop.org/wiki/Software/systemd/machined

  Sep 17 15:40:42 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
  Sep 17 15:40:42 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.
  Sep 17 15:42:26 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
  Sep 17 15:42:26 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.
  Sep 17 16:02:55 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
  Sep 17 16:02:55 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.
  Sep 17 16:04:39 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
  Sep 17 16:04:39 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.

  Trying to start a virtual machine from openstack, it defines the
  machine, but is not able to execute it.

  I used to have libvirt apparmor errors, but disabling libvirt in apparmor didn't solve the problem:
  [   27.785713] audit: type=1400 audit(1631888214.441:67): apparmor="DENIED" operation="capable" profile="libvirtd" pid=4817 comm="libvirtd" capability=17  capname="sys_rawio"

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: systemd-container 245.4-4ubuntu3.11
  ProcVersionSignature: Ubuntu 5.11.0-34.36~20.04.1-generic 5.11.22
  Uname: Linux 5.11.0-34-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27.20
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Fri Sep 17 16:23:39 2021
  SourcePackage: systemd
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1943990/+subscriptions