[SRU][N:gke][PATCH 026/106] arm64/mm: dplit __flush_tlb_range() to elide trailing DSB

[Tim Whisonant]

From: Ryan Roberts <ryan.roberts at arm.com>

BugLink: https://bugs.launchpad.net/bugs/2059316
BugLink: https://bugs.launchpad.net/bugs/2117098

Split __flush_tlb_range() into __flush_tlb_range_nosync() +
__flush_tlb_range(), in the same way as the existing flush_tlb_page()
arrangement.  This allows calling __flush_tlb_range_nosync() to elide the
trailing DSB.  Forthcoming "contpte" code will take advantage of this when
clearing the young bit from a contiguous range of ptes.

Ordering between dsb and mmu_notifier_arch_invalidate_secondary_tlbs() has
changed, but now aligns with the ordering of __flush_tlb_page().  It has
been discussed that __flush_tlb_page() may be wrong though.  Regardless,
both will be resolved separately if needed.

Link: https://lkml.kernel.org/r/20240215103205.2607016-12-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts <ryan.roberts at arm.com>
Reviewed-by: David Hildenbrand <david at redhat.com>
Tested-by: John Hubbard <jhubbard at nvidia.com>
Acked-by: Mark Rutland <mark.rutland at arm.com>
Acked-by: Catalin Marinas <catalin.marinas at arm.com>
Cc: Alistair Popple <apopple at nvidia.com>
Cc: Andrey Ryabinin <ryabinin.a.a at gmail.com>
Cc: Ard Biesheuvel <ardb at kernel.org>
Cc: Barry Song <21cnbao at gmail.com>
Cc: Borislav Petkov (AMD) <bp at alien8.de>
Cc: Dave Hansen <dave.hansen at linux.intel.com>
Cc: "H. Peter Anvin" <hpa at zytor.com>
Cc: Ingo Molnar <mingo at redhat.com>
Cc: James Morse <james.morse at arm.com>
Cc: Kefeng Wang <wangkefeng.wang at huawei.com>
Cc: Marc Zyngier <maz at kernel.org>
Cc: Matthew Wilcox (Oracle) <willy at infradead.org>
Cc: Thomas Gleixner <tglx at linutronix.de>
Cc: Will Deacon <will at kernel.org>
Cc: Yang Shi <shy828301 at gmail.com>
Cc: Zi Yan <ziy at nvidia.com>
Signed-off-by: Andrew Morton <akpm at linux-foundation.org>
(cherry picked from commit d9d8dc2bd3fb2689309f704fe85e6dde2b1bd73a)
Signed-off-by: dann frazier <dann.frazier at canonical.com>
Acked-by: Brad Figg <bfigg at nvidia.com>
Acked-by: Noah Wager <noah.wager at canonical.com>
Acked-by: Jacob Martin <jacob.martin at canonical.com>
Signed-off-by: Brad Figg <bfigg at nvidia.com>
Signed-off-by: Tim Whisonant <tim.whisonant at canonical.com>
---
 arch/arm64/include/asm/tlbflush.h | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index bfeb54f3a971f..a75de2665d844 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -424,7 +424,7 @@ do {									\
 #define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \
 	__flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false, kvm_lpa2_is_enabled());
 
-static inline void __flush_tlb_range(struct vm_area_struct *vma,
+static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma,
 				     unsigned long start, unsigned long end,
 				     unsigned long stride, bool last_level,
 				     int tlb_level)
@@ -458,10 +458,19 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
 		__flush_tlb_range_op(vae1is, start, pages, stride, asid,
 				     tlb_level, true, lpa2_is_enabled());
 
-	dsb(ish);
 	mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end);
 }
 
+static inline void __flush_tlb_range(struct vm_area_struct *vma,
+				     unsigned long start, unsigned long end,
+				     unsigned long stride, bool last_level,
+				     int tlb_level)
+{
+	__flush_tlb_range_nosync(vma, start, end, stride,
+				 last_level, tlb_level);
+	dsb(ish);
+}
+
 static inline void flush_tlb_range(struct vm_area_struct *vma,
 				   unsigned long start, unsigned long end)
 {
-- 
2.43.0