Every process has a struct mm_struct describing its address space: the VMAs (virtual memory areas), page tables, and stack/heap boundaries. Any change or lookup against that structure — mmap(), munmap(), mprotect(), brk(), and crucially every page fault — must hold a lock called mmap_lock (renamed from mmap_sem in Linux 5.8). It's a per-process rw-semaphore, and it's the single most contended lock in most memory-heavy workloads.

Writers (mmap, munmap, mprotect) take it exclusively. Readers (page fault handler walking VMAs to find the right one) take it shared. Sounds fine — until you realize that a thread that touches a freshly-allocated page for the first time triggers a page fault, which grabs the reader lock, which blocks behind any pending writer. Rw-semaphores in Linux are write-preferring to avoid writer starvation, so a single mmap() in progress freezes every faulting thread.

Real-world example: A JVM heap-warming workload spawns 64 threads that all fault in fresh anonymous pages simultaneously. Meanwhile, a background thread calls madvise(MADV_DONTNEED) on a slab it's returning to the allocator — which takes mmap_lock for write. Every one of those 64 faulting threads stalls until the madvise completes. On perf you see enormous time in rwsem_down_read_slowpath and threads sleeping on UNINTERRUPTIBLE. The workload's tail latency spikes from 200μs to 40ms.

Kernel 5.12+ introduced per-VMA locks: page faults can now take a lock on just the specific VMA they're touching, using RCU to find it, avoiding mmap_lock entirely on the fast path. Look for CONFIG_PER_VMA_LOCK=y and the counter /proc/vmstat field pgfault_vma. On kernels without it, or when the VMA lookup fails and falls back, you're back to serializing.

Rule of thumb: if perf top shows >5% in rwsem_* functions or down_read/down_write under handle_mm_fault, you have mmap_lock contention. Mitigations, in order of effort: (1) pre-fault memory with MAP_POPULATE or madvise(MADV_POPULATE_WRITE) so faults happen once, single-threaded; (2) avoid frequent mmap/munmap churn — use a slab or arena allocator that reuses VMAs; (3) upgrade to a kernel with per-VMA locks; (4) shard the workload across processes so each has its own mm_struct.

The dark irony: malloc() arenas were designed to reduce heap-lock contention, but if glibc calls mmap() for large allocations, you've just moved the contention into the kernel.