[Bug 1925211] Re: Hot-unplug of disks leaves broken block devices around in Hirsute on s390x
Christian Ehrhardt
1925211 at bugs.launchpad.net
Mon May 17 06:11:03 UTC 2021
Testing former status with 5.11.0-17-generic
Journal:
Add:
May 17 05:56:26 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
May 17 05:56:26 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0
May 17 05:56:26 h kernel: virtio_blk virtio5: [vdc] 385 512-byte logical blocks (197 kB/193 KiB)
Remove:
May 17 05:56:35 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
May 17 05:56:35 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0
ubuntu at s1lp5:~$ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
Disk attached successfully
ubuntu at s1lp5:~$ virsh detach-disk h vdc
Disk detached successfully
ubuntu at s1lp5:~$ virsh domblklist h
Target Source
----------------------------------------------------
vda /var/lib/uvtool/libvirt/images/h.qcow
vdb /var/lib/uvtool/libvirt/images/h-ds.qcow
ubuntu at s1lp5:~$ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk
uvt-kvm: error: libvirt: Domain not found: no domain with matching name 'hirsute-2nd-zfs'
ubuntu at s1lp5:~$ uvt-kvm ssh --insecure h lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 47.9M 1 loop /snap/core18/1998
loop1 7:1 0 29.9M 1 loop /snap/snapd/11838
loop2 7:2 0 63.1M 1 loop /snap/lxd/20396
vda 252:0 0 8G 0 disk
└─vda1 252:1 0 8G 0 part /
vdb 252:16 0 372K 0 disk
vdc 252:32 0 192.5K 0 disk
The disk is still present despite not really being in the guest anymore.
Now testing the proposed kernel 5.11.0-18-generic
Upgrades fine, well triggering an unrelated bug 1928625 but fine in
regard to the issue here.
Then journal (as expected) looks the same:
# Attach
May 17 06:09:22 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
May 17 06:09:22 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0
May 17 06:09:22 h kernel: virtio_blk virtio5: [vdc] 385 512-byte logical blocks (197 kB/193 KiB)
# Detach
May 17 06:09:29 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
May 17 06:09:29 h kernel: crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0
But lsblk confirms that the new kernel now works:
ubuntu at s1lp5:~$ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
Disk attached successfully
ubuntu at s1lp5:~$ virsh detach-disk h vdc
Disk detached successfully
ubuntu at s1lp5:~$ virsh domblklist h
Target Source
----------------------------------------------------
vda /var/lib/uvtool/libvirt/images/h.qcow
vdb /var/lib/uvtool/libvirt/images/h-ds.qcow
ubuntu at s1lp5:~$ uvt-kvm ssh --insecure h lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 63.1M 1 loop /snap/lxd/20396
loop1 7:1 0 29.9M 1 loop /snap/snapd/11838
loop2 7:2 0 47.9M 1 loop /snap/core18/1998
vda 252:0 0 8G 0 disk
└─vda1 252:1 0 8G 0 part /
vdb 252:16 0 372K 0 disk
^^ no more vdc after the detach.
Setting verified
** Tags removed: verification-needed-hirsute
** Tags added: verification-done-hirsute
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1925211
Title:
Hot-unplug of disks leaves broken block devices around in Hirsute on
s390x
Status in Ubuntu on IBM z Systems:
Fix Committed
Status in linux package in Ubuntu:
Fix Committed
Status in systemd package in Ubuntu:
Invalid
Status in udev package in Ubuntu:
Invalid
Status in linux source package in Hirsute:
Fix Committed
Status in systemd source package in Hirsute:
Invalid
Status in udev source package in Hirsute:
Invalid
Bug description:
SRU Justification
[Impact]
Hot removal of disks under kvm on s390 does not result in the kernel
removing the block device, which can lead to hung tasks and other
issues.
[Test Plan]
See steps to reproduce the bug in the original description below. To
test, execute these steps and confirm that the block device gets
removed as expected.
[Where problems could occur]
The fix is a revert of the changes which introduced this regression.
The original commit was a removal of supposedly unused code, but it
seems a mistake was made in the logic around unregistering of disks.
Reverting the changes could have potential to introduce bugs related
to other virt devices, especially if it interacts badly with
subsequent driver changes. However, the patch reverted cleanly, and
reverting restores the code to the state which has been working well
in previous kernels and seems like the lowest risk option until a
proper fix is available upstream.
---
Repro:
#1 Get a guest
$ uvt-kvm create --disk 5 --password=ubuntu h release=hirsute arch=s390x label=daily
$ uvt-kvm wait h release=hirsute arch=s390x label=daily
#2 Attach and Detach disk
$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M
$ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
$ virsh detach-disk h vdc
From libvirts POV it is gone at this point
$ virsh domblklist h
Target Source
------------------------------------------------------------------
vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow
vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow
But the guest thinks still it is present
$ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk
...
vdc 252:32 0 20M 0 disk
This even remains a while after (not a race).
Any access to it in the guest will hang (as you'd expect of a non-existing blockdev)
4 0 1758 1739 20 0 12140 4800 - S+ pts/0 0:00 | \_ sudo mkfs.ext4 /dev/vdc
4 0 1759 1758 20 0 6924 1044 - D+ pts/0 0:00 | \_ mkfs.ext4 /dev/vdc
The result above was originally found with hirsute-guest at hirsute-host
on s390x
I do NOT see the same with groovy-guest at hirsute-host on s390x
I DO see the same with hirsute-guest at groovy-host on s390x
=> Guest version dependent not Host/Hipervisor dependent
I DO see the same with ZFS disks AND LVM disks being added&removed
=> not type dependent
I do NOT see the same on x86.
=> Arch dependent ??
... the evidence slowly points towards an issue in the guest, damn we are so
close to release - but non-fully detaching disks are critical in my POV :-/
Filing this as-is for awareness, but certainly this will need more debugging.
Unsure where this is going to eventually I'll now file it for kernel/udev/systemd.
If there are any known issues/components that are related let me know please!
---
ProblemType: Bug
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu65
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 21.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:
Lspci-vt: -[0000:00]-
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
Package: udev
PackageArchitecture: s390x
PciMultimedia:
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs
ProcVersionSignature: User Name 5.11.0-14.15-generic 5.11.12
RelatedPackageVersions:
linux-restricted-modules-5.11.0-14-generic N/A
linux-backports-modules-5.11.0-14-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: hirsute uec-images
Uname: Linux 5.11.0-14-generic s390x
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True
acpidump:
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1925211/+subscriptions
More information about the foundations-bugs
mailing list