You've spent six hours hunting a heisenbug. It crashes once in twenty runs. You finally catch it in gdb — and then you fat-finger continue instead of step, blow past the moment of corruption, and have to start over. I've watched grown engineers cry over less.

rr (from Mozilla) is the answer. It records your program's execution — every syscall, signal, and thread interleaving — into a trace file, then replays it deterministically. Same memory addresses. Same scheduling. Same bug, every single time. And because it's deterministic, you can run it backwards.

Install (Debian/Ubuntu):

sudo apt install rr
# Needs hardware perf counters — enable for non-root:
sudo sysctl kernel.perf_event_paranoid=1

Record a flaky test, then replay it as many times as you like:

rr record ./my_flaky_test --seed 42
# ... eventually crashes ...
rr replay         # drops you into gdb on the recorded trace

Inside the replay session, you get every gdb command plus the reverse variants:

(rr) continue              # forward to crash
(rr) reverse-continue      # rewind to previous breakpoint
(rr) reverse-stepi         # back up one instruction
(rr) watch -l *0x7fff1234  # hardware watchpoint
(rr) reverse-continue      # rewinds to the LAST write to that address

That last trick is the killer feature. Find a corrupted value, set a watchpoint on its memory, reverse-continue, and rr drops you at the exact instruction that scribbled on it. No more "who freed this pointer?" guessing — you literally rewind to the call.

Real workflow for an intermittent bug:

# Loop until it crashes, keeping only the failing trace
rr record --chaos -- ./server --port 8080
# --chaos randomizes thread scheduling to surface races faster

# When it dies, the trace is saved. List traces:
rr ls

# Replay the latest:
rr replay

# Or replay a specific trace and start at an event:
rr replay -g 12450 ~/.local/share/rr/server-3

--chaos mode is its own little miracle: it deliberately picks unfair schedules to provoke data races that never trigger under normal load. I've shaken loose race conditions in minutes that had hidden in CI for months.

Why it beats the mainstream alternative: plain gdb gives you a microscope. rr gives you a microscope plus a TARDIS. Core dumps are corpses — rr gives you a living, scrubable timeline. Compared to printf-debugging across runs, the trace eliminates variance entirely: the bug you see in replay is the bug from the original run, byte for byte, register for register.

Caveats, because I won't lie to you: Linux/x86_64 only (Intel and recent AMD). Roughly 1.5–2× slowdown during record, near-native on replay. Doesn't handle programs that use rdtsc directly (rare) or certain CPU features without flags. Don't bother on virtualized hosts unless the hypervisor exposes PMU counters — rr will tell you immediately if it can't run.

Bonus trick — share a bug with a coworker by sending them the trace directory:

rr pack ~/.local/share/rr/myapp-0
tar czf bug.tar.gz ~/.local/share/rr/myapp-0
# They run: rr replay myapp-0  — and see exactly what you saw

A reproducible bug report. Imagine that.