x86 has a dirty secret: its variable-length, CISC-encoded instructions are brutal to decode. An instruction can be 1 to 15 bytes long, with prefixes, ModR/M bytes, SIB bytes, and displacements. To find where instruction N+1 starts, you must finish parsing instruction N. This serial dependency makes wide decode expensive — Intel's full decoder pipeline burns roughly 4-5 pipeline stages and significant power.

The fix: cache the decoded micro-ops themselves. Once an x86 instruction is cracked into µops, store those µops so you never decode the same instruction twice.

Two flavors evolved:

• Trace Cache (Pentium 4, 2000): Stored µops in execution order, following predicted branches. A single trace line could span multiple basic blocks. Brilliant idea, but cold-start misses were catastrophic — when a trace was evicted, the entire slow x86 decode pipeline had to rebuild it. P4's 12K-µop trace cache replaced the L1 I-cache entirely, which made misses devastating.
• Decoded Stream Buffer / µop Cache (Sandy Bridge, 2011 onward): Sits alongside the L1 I-cache, not replacing it. Indexed by instruction-pointer like a normal cache. Holds ~1500 µops in Sandy Bridge, ~4000 in modern Golden Cove. On hit, the legacy decoders (and the fetch pipeline upstream of them) are clock-gated off.

The hit-rate math: Modern Intel parts report µop cache hit rates of 80%+ on typical code. With legacy decode delivering 4 µops/cycle and the µop cache delivering 6-8 µops/cycle, that 80% hit rate translates to roughly 30-40% higher effective frontend throughput plus meaningful power savings — Intel has cited the µop cache as one of the largest single contributors to Sandy Bridge's perf-per-watt jump over Nehalem.

Real-world gotcha: The µop cache has alignment quirks. On Skylake-era parts, a 32-byte aligned instruction window must contain ≤18 µops to fit, and unconditional jumps end a µop cache line. Hot loops that straddle 32-byte boundaries can fall out of the µop cache entirely and tank by 20%+. This is why compilers like GCC have -falign-loops and why perf exposes the idq.dsb_uops counter — you can directly measure µop cache delivery.

Rule of thumb: If your hot loop fits in ~64 instructions and aligns to a 32-byte boundary, it'll live in the µop cache and run at peak frontend bandwidth. AMD's Zen has an analogous "op cache" with similar tradeoffs.

ARM mostly skipped this whole saga — fixed-length 32-bit instructions decode in parallel cheaply, so the cost/benefit never justified the silicon. Apple's M-series have small loop buffers but no full µop cache.