munmap and write combine flushing behavior

2026-07-11

Stack Overflow: View Question

Tags: linux, linux-kernel, mmap, pci-e

Score: 0 | Views: 65

The asker has a PCIe device with a BAR mapped write-combine (WC) into userspace. They issue a burst of memcpy writes with no explicit fence, then call munmap. Question: does munmap guarantee the CPU's WC buffers drain to the device? And what about abrupt process exit without a clean unmap?

This is a fun question because it sits at the intersection of three separate memory-ordering domains: the CPU's WC store buffers, the kernel's TLB/mapping teardown, and the PCIe fabric's posted-write ordering. Each has its own rules and none of them explicitly document munmap as a flushing operation.

Why it's interesting: WC memory is weakly ordered on x86 — stores can sit in the line-fill buffers indefinitely until something evicts them. The Intel SDM says WC buffers drain on serializing instructions (MFENCE, SFENCE, locked ops, CPUID) or on a strongly-ordered store to any other location. So the real question is: does the munmap path incidentally execute any of those before tearing the mapping down?

The likely answer is yes, but not because munmap promises it. Look at what munmap actually does in the kernel:

Any one of those atomic/serializing ops will drain WC buffers on the CPU that executes them. If the writing thread is the one calling munmap, you're fine. But WC buffers are per-CPU, so if the writes happened on CPU A and munmap runs on CPU B (unlikely in single-threaded code, common in multithreaded), CPU A's buffers won't flush from CPU B's serializing ops. The TLB shootdown IPI eventually forces CPU A into the kernel, which does drain them — but this is emergent behavior, not a documented guarantee.

Process termination: The exit path calls exit_mmap, which walks the same teardown path. Same emergent flushing occurs. Additionally, when the process is scheduled out for the last time, the context switch itself involves MOV CR3 (serializing) which drains WC.

The gotcha: "Eventually drains" is not the same as "drains before the device sees the transaction as complete." PCIe posted writes have their own ordering — even after WC flushes to the root complex, the device may not have observed them until a subsequent read (from the same BAR) forces the root complex to drain its write queues. If the driver needs completion semantics, an SFENCE plus a read-back of a device register is the only portable pattern.

Recommendation: Don't rely on munmap as a flush primitive. Add an explicit _mm_sfence() before munmap, and if you need to know the device saw the writes, do an MMIO read afterward.

The challenge: munmap incidentally drains WC buffers through its atomic ops and TLB flushes, but this is emergent behavior of the teardown path, not a documented API contract — and it says nothing about when the PCIe device actually observes the writes.

All newsletters