ACK: [SRU][P][PATCH v2 0/1] I/O performance regression on NVMes under same bridge (dual port nvme) (LP: #2115738)

[Kuba Pawlak]

On 5.08.2025 22:48, Massimiliano Pellizzer wrote:
> BugLink: https://bugs.launchpad.net/bugs/2115738
>
> [ Impact ]
>
> iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes
>
> The iotlb_sync_map iommu ops allows drivers to perform necessary cache
> flushes when new mappings are established. For the Intel iommu driver,
> this callback specifically serves two purposes:
>
> - To flush caches when a second-stage page table is attached to a device
>    whose iommu is operating in caching mode (CAP_REG.CM==1).
> - To explicitly flush internal write buffers to ensure updates to memory-
>    resident remapping structures are visible to hardware (CAP_REG.RWBF==1).
>
> However, in scenarios where neither caching mode nor the RWBF flag is
> active, the cache_tag_flush_range_np() helper, which is called in the
> iotlb_sync_map path, effectively becomes a no-op.
>
> Despite being a no-op, cache_tag_flush_range_np() involves iterating
> through all cache tags of the iommu's attached to the domain, protected
> by a spinlock. This unnecessary execution path introduces overhead,
> leading to a measurable I/O performance regression. On systems with NVMes
> under the same bridge, performance was observed to drop from approximately
> ~6150 MiB/s down to ~4985 MiB/s.
>
> Introduce a flag in the dmar_domain structure. This flag will only be set
> when iotlb_sync_map is required (i.e., when CM or RWBF is set). The
> cache_tag_flush_range_np() is called only for domains where this flag is
> set. This flag, once set, is immutable, given that there won't be mixed
> configurations in real-world scenarios where some IOMMUs in a system
> operate in caching mode while others do not. Theoretically, the
> immutability of this flag does not impact functionality.
>
> [ Fix ]
>
> Backport the following commit:
> - 12724ce3fe1a iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF
> modes
> - cee686775f9c iommu/vt-d: Make iotlb_sync_map a static property of dmar_domain
> to Plucky, as well as dependencies of cee686775f9c.
>
> [ Test Plan ]
>
> Run fio against two NVMEs under the same pci bridge (dual port NVMe):
>
> $ sudo fio --readwrite=randread --blocksize=4k --iodepth=32 --numjobs=8
> --time_based --runtime=40 --ioengine=libaio --direct=1 --group_reporting
> --new_group --name=job1 --filename=/dev/nvmeXnY --new_group --name=job2
> --filename=/dev/nvmeWnZ
>
> verify that the speed reached with the two NVMEs under the same bridge is the
> same that would have been reached if the two NVMEs were not under the same
> bridge.
>
> [ Regression Potential ]
>
> This fix affects the Intel IOMMU (VT-d) driver.
> An issue with this fix may introduce problems such as
> incorrect omission of required IOTLB cache or write buffer flushes
> when attaching devices to a domain.
> This could result in memory remapping structures not being visible
> to hardware in configurations that actually require synchronization.
> As a consequence, devices performing DMA may exhibit data corruption,
> access violations, or inconsistent behavior due to stale or incomplete
> translations being used by the hardware.
>
> [ Changes between v1 and v2 ]
>
> Backported a followup (cee686775f9c) of the fix commit (12724ce3fe1a).
>
>
Acked-by: Kuba Pawlak <kuba.pawlak at canonical.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x216A9D7E3B63DCB4.asc
Type: application/pgp-keys
Size: 3139 bytes
Desc: OpenPGP public key
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20250806/cca9e843/attachment-0001.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20250806/cca9e843/attachment-0001.sig>