APPLIED[Unstable]/cmt: [SRU][F][PATCH 0/12] s390x/pci: enumerate pci functions per physical adapter (LP: 1874056)

[Seth Forshee]

On Mon, May 18, 2020 at 08:24:26PM +0200, frank.heimes at canonical.com wrote:
> Buglink: https://bugs.launchpad.net/bugs/1874056
> 
> SRU Justification:
> 
> [Impact]
> 
> * On s390x the enumeration of PCI functions does not reflect which functions belongs to which physical adapter.
> 
> * Layout of a PCI function address on Linux:
>   0000:00:00.0
>   <root complex>:<bus>:<device>.<function>
> 
> * On s390x, each function is presented as individual root complex today, e.g.:
>   PCHID 0100 VF1 0000:00:00.0
>   PCHID 0100 VF23 0001:00:00.0
>   PCHID 0200 VF1 0002:00:00.0
>   OCHID 0100 VF17 0003:00:00.0
> 
> * On other platforms, the addresses correctly reflect the actual HW configuration.
> 
> * Some device drivers (like mlx5, for Mellanox adapters) group functions of one physical adapter by checking which PCI functions have identical values for <root complex>:<bus>:<device>.
> 
> * We need to use the same enumeration scheme to achieve this functionality on s390x.
> 
> * In this case, the two physical functions of a Mellanox adapter need to get function number 0 and 1,
>   and all virtual functions need to use the same <root complex>:<bus> numbers with function/device numbers counting up.
> 
> * Required result (example with 4 VFs per PF):
>   PCHID 0100 PF 0 0000:00:00.0
>   PCHID 0100 PF 1 0000:00:00.1
>   PCHID 0100 PF 0 VF 0 0000:00:00.2
>   PCHID 0100 PF 0 VF 1 0000:00:00.3
>   PCHID 0100 PF 0 VF 2 0000:00:00.4
>   PCHID 0100 PF 0 VF 3 0000:00:00.5
>   PCHID 0100 PF 1 VF 0 0000:00:00.6
>   PCHID 0100 PF 1 VF 1 0000:00:00.7
>   PCHID 0100 PF 1 VF 2 0000:00:00.8
>   PCHID 0100 PF 1 VF 3 0000:00:00.9
>   PCHID 0200 PF 0 0001:00:00.0
> 
> [Fix]
> 
> * Backport 1: https://launchpadlibrarian.net/479699471/0001-s390-pci-Improve-handling-of-unset-UID.patch
> 
> * Backport 2: https://launchpadlibrarian.net/479699482/0002-s390-pci-embedding-hotplug_slot-in-zdev.patch
> 
> * Backport 3: https://launchpadlibrarian.net/479699492/0003-s390-pci-Expose-new-port-attribute-for-PCIe-function.patch
> 
> * Backport 4: https://launchpadlibrarian.net/479699497/0004-s390-pci-adaptation-of-iommu-to-multifunction.patch
> 
> * Backport 5: https://launchpadlibrarian.net/479700706/0005-s390-pci-define-kernel-parameters-for-PCI-multifunct.patch
> 
> * Backport 6: https://launchpadlibrarian.net/479700712/0006-s390-pci-define-RID-and-RID-available.patch
> 
> * Backport 7: https://launchpadlibrarian.net/479700739/0007-s390-pci-create-zPCI-bus.patch
> 
> * Backport 8: https://launchpadlibrarian.net/479700769/0008-s390-pci-adapt-events-for-zbus.patch
> 
> * Backport 9: https://launchpadlibrarian.net/479700786/0009-s390-pci-Handling-multifunctions.patch
> 
> * Backport 10: https://launchpadlibrarian.net/479700794/0010-s390-pci-Do-not-disable-PF-when-VFs-exist.patch
> 
> * Backport 11: https://launchpadlibrarian.net/479700798/0011-s390-pci-Documentation-for-zPCI.patch
> 
> * Backport 12: https://launchpadlibrarian.net/479700799/0012-s390-pci-removes-wrong-PCI-multifunction-assignment.patch
> 
> [Test Case]
> 
> * Prepare an IBM z13 or LinuxONE III (or newer) system with two or more RoCE Express PCI 2(.1) adapters.
> 
> * Assign the adapters (and it's virtual functions) to an LPAR.
> 
> * Verify whether the physical and virtual functions are grouped in arbitrary order or in consecutive order - physical first (for example with lspci -t ...)
> 
> [Regression Potential] 
> 
> * The regression potential can be considered as moderate, since:
> 
> * It is purely s390x specific code (arch/s390/* drivers/iommu/s390-iommu.c and drivers/pci/hotplug/s390_pci_hpc.c - and some doc adjustments, too).
> 
> * It largely affects zPCI, the s390x specific PCI code layer.
> 
> * PCI cards available for s390x are optional cards (RoCE and zEDC) and not very wide-spread.
> 
> * The situation described above affects the RoCE adapters only (Mellanox based).
> 
> * The patches are also upstream accepted and available via linux-next, but to apply them to focal kernel 5.4 the above backports are needed.
> 
> * However, the code is modified by several patches (12), hence there is a chance to break zPCI with them.
> 
> * For upfront testing a PPA got created with a focal (master-next) kernel that incl. all the above patches.

Several of these patches are from linux-next, which needs to be
indicated in the provenance, e.g. "(backported from commit xxx
linux-next)". Those patches should also be submitted for the development
release. I went ahead and cherry-picked the following patches from
linux-next to unstable/master.

 s390/pci: Expose new port attribute for PCIe functions
 s390/pci: adaptation of iommu to multifunction
 s390/pci: define kernel parameters for PCI multifunction
 s390/pci: define RID and RID available
 s390/pci: create zPCI bus
 s390/pci: adapt events for zbus
 s390/pci: Handling multifunctions
 s390/pci: Do not disable PF when VFs exist
 s390/pci: Documentation for zPCI
 s390/pci: removes wrong PCI multifunction assignment