Traditionally, when a NIC or storage controller wrote data via DMA, it went to DRAM. The CPU then had to fetch it back into cache when software touched it — a full memory round trip on every packet. DDIO (Data Direct I/O), introduced on Xeon E5 in 2012, changes this: PCIe writes land directly in the L3 cache, and reads pull from L3 when possible. DRAM is bypassed on the hot path.

The mechanism is deceptively simple. The uncore's caching agent treats DMA traffic almost like a core-issued write. Inbound writes allocate cache lines in L3 (write-allocate). Inbound reads that hit L3 are served without touching DRAM. But there's a catch: DDIO uses only a limited portion of L3 — historically 2 ways out of 11 (~10–20% of L3). This prevents I/O from thrashing CPU workloads.

The "leaky bucket" problem: If your NIC RX ring exceeds the DDIO way allocation, incoming packets evict earlier packets from L3 before software processes them. Now the CPU takes an L3 miss to DRAM to read a packet the NIC just wrote — you've paid the DMA cost plus the miss cost. This shows up as sudden latency cliffs at ~15% line rate on 100 GbE workloads.

Real-world example: DPDK applications on Intel are tuned around DDIO. If you size RX descriptor rings and mbuf pools to fit within DDIO's L3 slice (say, 2 MB per port on a 20 MB L3), you get L3-hit latencies (~15 ns) on every packet. Oversize the rings and your p99 latency jumps from 20 µs to 200 µs — the classic "buffer bloat inside the CPU" pattern. Kernel bypass frameworks like Snabb and VPP publish DDIO sizing guides for exactly this reason.

Rule of thumb: For high-rate PCIe I/O, keep your working DMA footprint (all in-flight descriptors + payload buffers being processed) under ~10% of L3 per socket. Above that, you're guaranteed to evict live data before touching it. A 30 MB L3 → budget ~3 MB for DMA-hot data.

AMD shipped its equivalent (sometimes called "PCIe Cache Injection") on newer EPYC parts, though allocation policy differs. On both, DDIO can be disabled in BIOS — occasionally worth doing for workloads where PCIe traffic is bulk-throughput but cache-cold, so DRAM DMA leaves L3 alone for CPU code.