The Return Stack Buffer Underflow: Why Deep Call Chains Break Return Prediction

2026-08-21

The Return Address Stack (RAS) is a small hardware stack — typically 16 to 32 entries — that predicts where RET instructions will jump. Every CALL pushes the return address; every RET pops it. When it works, return prediction is essentially perfect: 100% accuracy on well-behaved code. But the RAS has a fixed depth, and when your call chain exceeds it, everything falls apart in ways that surprise even experienced engineers.

The underflow scenario: Suppose the RAS holds 16 entries and your program recurses 20 levels deep. On the way down, the first 16 CALLs fill the stack; the next 4 either overwrite the oldest entries (wrap-around) or get dropped (saturation), depending on the microarchitecture. On the way back up, the first 4 RETs find nothing useful — the predictor guesses, usually wrong, and each mispredicted return costs 15-20 cycles of pipeline flush. Then predictions work again... until the next unbalanced sequence.

Real-world example: Deeply recursive parsers (JSON, XML, recursive-descent compilers) routinely blow past 32 frames. A recursive Fibonacci at n=40 makes ~200M calls with peak depth 40 — the last 8 frames on each unwind mispredict every time. Measured impact: 2-3x slowdown vs. an iterative version, most of which is return misprediction, not call overhead. Tail-call optimization helps not just by saving stack space but by keeping the RAS balanced.

The corruption case is worse. If you use setjmp/longjmp, exception unwinding, or coroutine switches, you pop the software stack without executing RET. The RAS still thinks those frames are live. Now the next several RETs predict return addresses from functions that already unwound — guaranteed mispredictions until the RAS drains. Modern C++ exception paths can hit 50+ cycle penalties per catch just from this.

Rule of thumb: Keep hot recursive call chains under 16 deep. Every level beyond your CPU's RAS depth costs one guaranteed misprediction on unwind — call it ~18 cycles × (depth − RAS_size) per full recursion cycle. For a 32-entry RAS on Zen 4, recursion of depth D > 32 costs roughly (D − 32) × 18 extra cycles per full descent+ascent.

Mitigations: Convert to iteration with an explicit stack (the software stack lives in cache; the RAS doesn't help you anyway). Use tail calls where the ABI permits — GCC's -foptimize-sibling-calls turns some recursions into jumps, bypassing the RAS entirely. For unavoidable deep recursion, batching work per frame amortizes the misprediction cost across more useful work.

Key Takeaway: The Return Address Stack is a fixed-size hardware predictor — exceed its depth or corrupt its state with non-local control flow, and every unbalanced return becomes a guaranteed 15-20 cycle misprediction.

All newsletters