A processor running a tight inner loop — say, a memcpy or a DSP filter tap — fetches the exact same 4-16 instructions billions of times. Every fetch burns energy in the L1 I-cache (tag lookup, way selection, data array read, ~30-50 pJ per access on a modern node) and consumes fetch bandwidth that could go elsewhere. A zero-overhead loop buffer (also called a loop cache or loop stream detector) is a tiny fully-associative buffer sitting between fetch and decode that captures the instructions of a detected loop and replays them from an SRAM one-tenth the size of the I-cache.

The mechanism has three phases. Detection: a small state machine watches for a backward-taken branch whose target lies within the last N fetched instructions (typically N=16 to 64). When it fires twice in a row to the same PC, the buffer is armed. Fill: on the next iteration, fetched instructions are mirrored into the loop buffer as they pass through. Streaming: once the buffer contains the full loop body and the branch predictor keeps predicting the backedge taken, the I-cache and even the branch predictor are gated off, and decode is fed straight from the loop buffer. A misprediction, exception, or any control flow leaving the buffered region tears down the mode.

Real example: Intel's Loop Stream Detector (LSD). Introduced in Nehalem (2008), refined through Skylake, it holds up to 64 micro-ops post-decode. When active on a tight loop, the entire front-end — fetch, predecode, decode — is clock-gated. Agner Fog's measurements show ~10% overall power reduction on DSP-like workloads and, more importantly, the removal of front-end bottlenecks: the LSD delivers 4 µops/cycle every cycle regardless of instruction-cache pressure. ARM's Cortex-A78 has a similar 64-entry "MOP cache" that plays the same role.

Rule of thumb for the energy win: a loop buffer access costs roughly the ratio of the two SRAM sizes. For a 64-entry × 32-bit loop buffer (256 bytes) versus a 32 KB I-cache, that's ~1/128 the dynamic energy per fetch. On a loop that iterates a million times, this is the difference between the front-end burning 30 mJ and 0.25 mJ.

The catch is fragility. Any of these break the mode: a function call inside the loop, a self-modifying store, a variable-length instruction crossing the buffer boundary, or a loop just one instruction too big. Compilers therefore align hot loops and use pragmas (#pragma unroll, or on ARM, __attribute__((loop_align))) specifically to keep them buffer-eligible. When you see a benchmark cliff after adding "one more line" to an inner loop, the loop buffer falling out of lock is often why.