When you mmap() a region, the kernel doesn't actually allocate physical pages. It just records the mapping in the VMA. The first time your code touches each page, you take a minor page fault: the fault handler runs, allocates a page, updates the page table, and returns. For a 10GB anonymous mapping, that's 2.6 million faults, each costing a few microseconds — and they hit your critical path at unpredictable times.

The classic workaround was MAP_POPULATE, which pre-faults at mmap time. But it has three problems: it's synchronous inside the mmap syscall (blocking your thread), it fails silently if any page can't be populated, and it doesn't work on file-backed regions you extended later. MADV_POPULATE_READ and MADV_POPULATE_WRITE (Linux 5.14+) fix all three.

The distinction matters. MADV_POPULATE_READ faults pages in as read-only — for anonymous memory, this maps them all to the shared zero page, using zero physical RAM. MADV_POPULATE_WRITE allocates real backing pages and makes them writable, avoiding a second copy-on-write fault when you actually write. If you're about to write the whole region, use WRITE. If you're about to read a file-backed mapping, use READ to pull data from the page cache without triggering readahead disruption elsewhere.

Concrete example: a low-latency trading process pre-allocates a 4GB ring buffer for order events. Without pre-faulting, the first pass through the buffer takes ~10 seconds of accumulated page fault time, spread across every message. With madvise(buf, 4*GB, MADV_POPULATE_WRITE) at startup, the faults happen once, upfront, on the initialization thread — and the hot path never sees a minor fault again. Measured with perf stat -e minor-faults, the steady-state fault count drops to zero.

Rule of thumb: a minor page fault costs roughly 1-3 μs on modern x86. Multiply by size / 4096 to estimate hidden latency. A 1GB mapping = 262,144 pages ≈ 500ms of scattered stalls if you fault lazily. If your workload can't tolerate that jitter, pre-populate.

Two gotchas: (1) MADV_POPULATE_WRITE on a file-backed region will trigger writeback pressure later — the kernel now has real dirty-capable pages to track. (2) Neither advise pins pages; the kernel can still reclaim them under memory pressure. If you need genuine residency, follow with mlock().