[Bug 1844879] Re: Behaviour change in systemd 243 breaks libvirt autopkgtest

[Christian Ehrhardt ]

To summarize (out of the noisy test log).
What happens with this patch applied is that libvirt looses its LXC-guest on the restart of the service.

With the XML from the test this is what happens:
$ virsh define debian/tests/smoke-lxc.xml
$ virsh start sl
$ virsh list
 Id     Name   State
------------------------
 2280   sl     running
$ virsh lxc-enter-namespace --noseclabel sl /bin/ls /bin/ls
/bin/ls
# At this point we know and have confirmed the guest works
# Now restarting libvirtd breaks it.
$ systemctl restart libvirtd
# Now the guest container is gone


Pre restart it looks like:
4     0    2426       1  20   0 138712 16132 poll_s Sl   ?          0:00 /usr/lib/libvirt/libvirt_lxc --name sl --console 20 --security=apparmor --handshake 23
4     0    2428    2426  20   0   4084  3216 poll_s Ss+  pts/0      0:00  \_ /bin/bash


The guest log in /var/log/libvirt/lxc/sl.log gets this entry in !BOTH! cases:
2019-09-23 10:51:46.004+0000: 2426: error : virNetSocketReadWire:1804 : End of file while reading data: Input/output error
So this is a common red-herring, the system with systemd 241-7ubuntu1 has the same as well.


Another red-herring is the libvirtd service on restart reporting the LXC processes as left-over. This also happens with old and new systemd:
Sep 23 12:51:46 autopkgtest systemd[1]: libvirtd.service: Found left-over process 2428 (bash) in control group while starting unit. Ignoring.
Sep 23 12:51:46 autopkgtest systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 23 12:51:46 autopkgtest systemd[1]: libvirtd.service: Found left-over process 2426 (libvirt_lxc) in control group while starting unit. Ignoring.
Sep 23 12:51:46 autopkgtest systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.


The error that is different between 241 and 243 is from the libvirtd daemon:
Sep 23 12:51:46 autopkgtest libvirtd[2448]: internal error: No valid cgroup for machine sl
Sep 23 12:51:46 autopkgtest libvirtd[2448]: End of file while reading data: Input/output error


This is not just a visibility problem, the process that was the guest really is gone at this point.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1844879

Title:
  Behaviour change in systemd 243 breaks libvirt autopkgtest

Status in libvirt package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  With systemd 243 libvirt autopkgtest starts failing:

  http://autopkgtest.ubuntu.com/packages/libv/libvirt/eoan/amd64

  https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan/eoan/amd64/libv/libvirt/20190921_104518_69667@/log.gz
  ...
  + virt-xml-validate debian/tests/smoke-lxc.xml
  debian/tests/smoke-lxc.xml validates
  + virsh define debian/tests/smoke-lxc.xml
  Domain sl defined from debian/tests/smoke-lxc.xml

  + rm -f /var/log/libvirt/lxc/sl.log
  + virsh start sl
  Domain sl started

  + grep -qs starting up /var/log/libvirt/lxc/sl.log
  + check_domain
  + grep/bin/ls

  
  Domain sl has been undefined

   -qs sl[[:space:]]\+running
  + virsh list
  + virsh lxc-enter-namespace --noseclabel sl /bin/ls /bin/ls
  + systemctl restart libvirtd
  + check_domain
  + grep -qs sl[[:space:]]\+running
  + virsh list
  + cleanup
  + [ -z  ]
  + virsh destroy sl
  error: Failed to destroy domain sl
  error: Requested operation is not valid: domain is not running
  + true
  + virsh undefine sl
  + CLEANED_UP=1
  autopkgtest [10:41:05]: test smoke-lxc: -----------------------]
  autopkgtest [10:41:07]: test smoke-lxc:  - - - - - - - - - - results - - - - - - - - - -
  smoke-lxc            FAIL non-zero exit status 1
  ...

  The behaviour change is still present in systemd master
  (v243-112-g984b96aa7a)

  I can revert the change in systemd, but it also fixes an issue reported at upstream.
  I will take another look later, but wanted to document my findings.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1844879/+subscriptions