[SRU][P/O][PATCH 0/2] WARNING: CPU: 18 PID: 3683 at arch/powerpc/kvm/../../../virt/kvm/vfio.c Call Traces seen when pci device is detached from the kvm guest (LP: #2104893)

[Massimiliano Pellizzer]

BugLink: https://bugs.launchpad.net/bugs/2104893

[Impact]

Currently on book3s-hv, the capability KVM_CAP_SPAPR_TCE_VFIO is only
available for KVM Guests running on PowerNV and not for the KVM guests
running on pSeries hypervisors. This prevents a pSeries L2 guest from
leveraging the in-kernel acceleration for H_PUT_TCE_INDIRECT and
H_STUFF_TCE hcalls that results in slow startup times for large memory
guests.

Support for VFIO on pSeries was restored in commit f431a8cde7f1
("powerpc/iommu: Reimplement the iommu_table_group_ops for pSeries"),
making it possible to re-enable this capability on pSeries hosts.

This change enables KVM_CAP_SPAPR_TCE_VFIO for nested PAPR guests on
pSeries, while maintaining the existing behavior on PowerNV. Booting an
L2 guest with 128GB of memory shows an average 11% improvement in
startup time.

[Fix]

Clean cherry pick of:
- b4392813bbc3b0 KVM: PPC: Enable CAP_SPAPR_TCE_VFIO on pSeries KVM guests
from linux-next.

[Test Plan]

Tested IBM side using the following procedure:
1. Start a guest
2. unbind the pci device from host using below command

root at ltcrain8og-lp6:~/NET# virsh nodedev-detach pci_0028_01_00_0
Device pci_0028_01_00_0 detached
root at ltcrain8og-lp6:~/NET# virsh nodedev-detach pci_0028_01_00_1
Device pci_0028_01_00_1 detached

3. Now attach the pci device to guest using virsh attach-device command. The xml
snippet used is given below

root at ltcrain8og-lp6:~/NET# vi pci2.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0028' bus='0x01' slot='0x00' function='0x0'/>
  </source>
</hostdev>

root at ltcrain8og-lp6:~/NET# virsh attach-device vm pci2.xml
Device attached successfully

4. On guest side the pci device gets attached sucesfully, and interface comes
up.
[root at localhost ~]# lspci
0000:00:01.0 Ethernet controller: Red Hat, Inc. Virtio network device
0000:00:02.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
0000:00:03.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
0000:00:04.0 Communication controller: Red Hat, Inc. Virtio console
0000:00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
0000:00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
0001:00:01.0 Ethernet controller: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx]

5. Now Detach the device using virsh detach command
root at ltcrain8og-lp6:~/NET# virsh detach-device vm pci2.xml
Device detached successfully

[Where problems could occur]

The fix affects the KVM subsystem on IBM Power platforms.
An issue with this fix may lead to incorrect exposure of the
KVM_CAP_SPAPR_TCE_VFIO capability on hosts or guests
that do not fully support it.