The Cache Prefetcher's Dead-on-Arrival Problem: Why Prefetched Lines Get Evicted Before They're Used

2026-06-17

A prefetcher's job is simple: bring data into the cache before the load asks for it. But a surprising fraction of prefetched lines are dead on arrival — fetched into L1 or L2, then evicted by other traffic before any demand load ever touches them. The bandwidth was spent, the cache slot was burned, and the program got zero benefit. Studies on commercial workloads (database scans, graph traversal, JIT'd interpreters) routinely find 20–40% of prefetched lines die unused.

Three failure modes dominate:

Modern CPUs fight this with prefetch usefulness tracking. Each prefetched line carries a bit indicating it was brought in by the prefetcher, not a demand. If it gets evicted with that bit still set, a counter in the prefetcher logic increments. When the unused-eviction rate crosses a threshold, the prefetcher throttles its lookahead distance or disables itself for that stream.

Real-world example: Linked-list traversal on a workload with a large per-node payload. The stride prefetcher sees the array of node pointers and prefetches forward — but each node dereference pulls in a 64-byte line that pushes earlier prefetches out of L1. Intel's perf counter L2_RQSTS.PF_HIT divided by L2_RQSTS.ALL_PF on this kind of code can drop below 30%. Meanwhile a tight memcpy on the same chip routinely hits 95%+ — the working set fits, the stride is perfect, and the prefetcher's lookahead matches consumption rate.

Rule of thumb: Prefetch distance should equal memory latency × consumption rate. If your loop consumes 4 cache lines per 100 cycles and DRAM latency is 200 cycles, prefetch ~8 lines ahead. Less and you stall; more and lines die before use. Software prefetch intrinsics (_mm_prefetch) let you tune this manually when the hardware predictor gets it wrong.

The deeper lesson: a prefetcher that's too aggressive isn't just neutral — it's actively harmful, because every dead-on-arrival line displaced something a demand load actually needed.

Key Takeaway: Prefetched lines that get evicted before use waste both bandwidth and cache capacity, which is why modern prefetchers track their own hit rate and throttle when too many of their fetches die unused.

All newsletters