2026-06-12
You've used perf record with sampling — it interrupts the CPU every N cycles, grabs a stack trace, and hopes the statistical picture is representative. Intel Processor Trace (PT) does something fundamentally different: it records every control-flow decision the CPU makes, with near-zero overhead, into a ring buffer in DRAM. You can replay the exact execution path of a program, instruction by instruction, after the fact.
The trick is what PT doesn't record. It doesn't store instruction pointers — those are derivable. Instead, it emits compressed packets only when control flow becomes non-obvious:
A decoder later walks the static binary alongside the packet stream: at each conditional branch it consumes one TNT bit, at each indirect branch it consumes a TIP target. The reconstruction is exact.
Overhead in practice: Intel quotes <5% on most workloads. A branch-heavy program at 4 GHz executes roughly 500M branches/sec; at ~0.2 bits/branch after compression, that's ~12 MB/sec of trace. A 1 GB ring buffer captures ~80 seconds of execution — enough to catch most rare bugs after the fact.
Real-world example: Mozilla's rr debugger uses PT to make crashes reversible. When Firefox segfaults in production, an engineer can load the trace, run the debugger backwards from the crash to find which store corrupted the pointer — no reproduction needed. Similarly, perf record -e intel_pt// followed by perf script --itrace=i100ns gives you cycle-accurate flame graphs without sampling bias; you see the tail latencies that perf record -F 999 misses entirely.
Rule of thumb for buffer sizing: assume 100 KB/sec per idle thread, 10 MB/sec per branch-heavy thread. Multiply by how many seconds you need to retain, then by core count. A 16-core server running a hot workload needs ~160 MB/sec of buffer drain — which is why PT supports writing directly to a ToPA (Table of Physical Addresses) ring rather than going through the kernel.
The catch: decoding is slow. Reconstructing a 1-second trace can take 30 seconds of CPU time, because the decoder must walk every instruction. PT trades recording cost for analysis cost — a deliberate, often correct, tradeoff for post-mortem debugging.
