A blocking cache is dead simple: on a miss, stall everything until DRAM answers. That's fine if you miss once every thousand cycles. Modern out-of-order cores miss constantly — and DRAM takes 200+ cycles. Stalling the L1 on every miss would idle the whole pipeline. MSHRs are the structure that lets a cache say "miss noted, I'll get back to you" and keep accepting new requests.

An MSHR is a small entry that tracks one outstanding miss. Each entry holds:

• The missing block address (so the fill knows where to land)
• A list of pending requests waiting on this block — load PC, destination register, byte offset, store data if it's a write
• Status bits: issued-to-next-level, fill-pending, ECC-pending

When a load misses, hardware checks the MSHR file first. Two cases:

• Primary miss — no MSHR for this block exists. Allocate one, send a refill request downstream.
• Secondary miss — an MSHR already exists for this block (a younger load wants the same line a prior miss fetched). Just append to the existing MSHR's request list. Don't send a duplicate fetch. This is called miss merging or MSHR coalescing, and it's huge for tight loops over arrays.

When the fill arrives, the MSHR walks its list, writes the line into the cache array, and forwards each waiting load's data to the register file (or wakeup network in OoO). Then the entry frees up.

Concrete example: Intel Sunny Cove's L1D has 22 fill buffers (MSHRs). AMD Zen 4 has 24. Apple M-series has substantially more — Firestorm cores measured around 148 outstanding L1 misses. That number directly bounds memory-level parallelism (MLP): a core with N MSHRs can have at most N independent cache misses in flight. Once full, the cache does block — and on a workload with chained pointer chases, this is often the actual bottleneck, not DRAM bandwidth.

Rule of thumb: peak achievable bandwidth = (MSHRs × cache line size) / memory latency. For 22 MSHRs × 64B / 80ns DRAM, that's ~17.6 GB/s per core. Want more? Add MSHRs, shrink the line, or cut latency. This is Little's Law applied to caches.

MSHRs also have a nasty edge case: if every MSHR holds requests targeting a single congested DRAM bank, you've burned your entire MLP budget on serialized traffic. Modern designs add per-bank limits and quality-of-service hints to keep one bad address from starving the rest.