Commit 51405573 authored by Lokesh Gidra's avatar Lokesh Gidra
Browse files

BACKPORT: FROMGIT: userfaultfd: opportunistic TLB-flush batching for present pages in MOVE

MOVE ioctl's runtime is dominated by TLB-flush cost, which is required for
moving present pages.  Mitigate this cost by opportunistically batching
present contiguous pages for TLB flushing.

Without batching, in our testing on an arm64 Android device with UFFD GC,
which uses MOVE ioctl for compaction, we observed that out of the total
time spent in move_pages_pte(), over 40% is in ptep_clear_flush(), and
~20% in vm_normal_folio().

With batching, the proportion of vm_normal_folio() increases to over 70%
of move_pages_pte() without any changes to vm_normal_folio().
Furthermore, time spent within move_pages_pte() is only ~20%, which
includes TLB-flush overhead.

When the GC intensive benchmark, which was used to gather the above
numbers, is run on cuttlefish (qemu android instance on x86_64), the
completion time of the benchmark went down from ~45mins to ~20mins.

Furthermore, system_server, one of the most performance critical system
processes on android, saw over 50% reduction in GC compaction time on an
arm64 android device.

[lokeshgidra@google.com: make calculation of largest extent that can be batched unconditional on length, per Barry]
  Link: https://lkml.kernel.org/r/20250816191123.3601561-1-lokeshgidra@google.com
Link: https://lkml.kernel.org/r/20250813193024.2279805-1-lokeshgidra@google.com


Signed-off-by: default avatarLokesh Gidra <lokeshgidra@google.com>
Acked-by: default avatarPeter Xu <peterx@redhat.com>
Reviewed-by: default avatarBarry Song <baohua@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Kalesh Singh <kaleshsingh@google.com>
Cc: David Hildenbrand <david@redhat.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 50944692
 git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

 mm-stable)

Conflicts:
1. Argument list of move_present_pte() doesn't have dst_pmd and dst_pmdval
2. Replace folio_mk_pte() with mk_pte()
3. Use WRITE_ONCE() for updating folio->index
4. Removed dst_pmd and dst_pmdval arguments to call of move_swap_pte()
5. Handle split_folio() return value change due to TAO patches

Signed-off-by: default avatarLokesh Gidra <lokeshgidra@google.com>
Bug: 422561794
Cherrypick-From: https://android-review.googlesource.com/q/commit:85d39b10816ca0ef4a41d0de5641ffaccfcf5e54
Merged-In: I20c9b8aa30bff52c311b655b1513e99fe00dbfa7
Change-Id: I20c9b8aa30bff52c311b655b1513e99fe00dbfa7
Bug: 446709123
parent 148994e9
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment