BACKPORT: FROMGIT: userfaultfd: opportunistic TLB-flush batching for present pages in MOVE
MOVE ioctl's runtime is dominated by TLB-flush cost, which is required for moving present pages. Mitigate this cost by opportunistically batching present contiguous pages for TLB flushing. Without batching, in our testing on an arm64 Android device with UFFD GC, which uses MOVE ioctl for compaction, we observed that out of the total time spent in move_pages_pte(), over 40% is in ptep_clear_flush(), and ~20% in vm_normal_folio(). With batching, the proportion of vm_normal_folio() increases to over 70% of move_pages_pte() without any changes to vm_normal_folio(). Furthermore, time spent within move_pages_pte() is only ~20%, which includes TLB-flush overhead. When the GC intensive benchmark, which was used to gather the above numbers, is run on cuttlefish (qemu android instance on x86_64), the completion time of the benchmark went down from ~45mins to ~20mins. Furthermore, system_server, one of the most performance critical system processes on android, saw over 50% reduction in GC compaction time on an arm64 android device. [lokeshgidra@google.com: make calculation of largest extent that can be batched unconditional on length, per Barry] Link: https://lkml.kernel.org/r/20250816191123.3601561-1-lokeshgidra@google.com Link: https://lkml.kernel.org/r/20250813193024.2279805-1-lokeshgidra@google.com Signed-off-by:Lokesh Gidra <lokeshgidra@google.com> Acked-by:
Peter Xu <peterx@redhat.com> Reviewed-by:
Barry Song <baohua@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: David Hildenbrand <david@redhat.com> Signed-off-by:
Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 50944692 git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm mm-stable) Conflicts: 1. Argument list of move_present_pte() doesn't have dst_pmd and dst_pmdval 2. Replace folio_mk_pte() with mk_pte() 3. Use WRITE_ONCE() for updating folio->index 4. Removed dst_pmd and dst_pmdval arguments to call of move_swap_pte() 5. Handle split_folio() return value change due to TAO patches Change-Id: I20c9b8aa30bff52c311b655b1513e99fe00dbfa7 Signed-off-by:
Lokesh Gidra <lokeshgidra@google.com> Bug: 422561794
Loading
Please sign in to comment