A kprobe lets you attach a handler to almost any kernel instruction address at runtime. When that address executes, your handler runs, then the original instruction resumes. This is how bpftrace, perf probe, and ftrace's dynamic events work. The clever part is how the kernel diverts execution without stopping the world.

Registration, phase 1 — the INT3 trick. When you register a kprobe at address A:

• The kernel copies the original byte(s) at A into a per-probe insn_slot (an executable page of saved instructions).
• It overwrites the first byte of A with 0xCC — the single-byte INT3 breakpoint instruction. This write is a single byte, so it is atomic on x86: no other CPU can observe a half-patched instruction.
• When any CPU executes A, it traps into do_int3. The handler identifies the probe, calls your pre_handler, single-steps the saved instruction out of the insn_slot, calls your post_handler, then resumes at A + insn_len.

Phase 2 — optimization to a JMP. INT3 costs ~1000 cycles per hit (trap, save state, dispatch, single-step, IRET). If the probe survives long enough, a workqueue upgrades it: the kernel patches a 5-byte relative JMP at A that jumps to a trampoline calling your handler directly. Cost drops to ~50 cycles.

But you can't atomically write 5 bytes on x86. The trick: write the INT3 first, then patch bytes 2–5, then overwrite byte 1 with the JMP opcode 0xE9. Any CPU that races through mid-patch sees the INT3 and takes the slow path — correct, just slower. The text_poke_bp() API implements this dance and issues IPIs to serialize instruction streams (Intel requires a serializing instruction after cross-modifying code, or you risk stale prefetch).

Concrete example. Attach to vfs_read:

echo 'p:myprobe vfs_read' >> /sys/kernel/tracing/kprobe_events
echo 1 > /sys/kernel/tracing/events/kprobes/myprobe/enable

The first byte of vfs_read is now 0xCC. Within a few seconds, a JMP replaces it. Every read from every file on the system now takes a detour through your handler — and if that handler is empty, the overhead is roughly ~5ns per call.

Rule of thumb. An unoptimized kprobe adds ~1μs per hit; an optimized one adds ~50ns. If you probe a function called 10M times/sec, unoptimized = 10s of CPU/sec (fully saturated), optimized = 0.5s/sec (5% overhead). Always check /sys/kernel/debug/kprobes/list — probes marked [OPTIMIZED] are cheap; [DISABLED] or unmarked ones aren't.

What prevents optimization. The target must have 5 bytes of instructions with no jump target landing inside them (the JMP would clobber a branch destination). The kernel's decoder walks the function to check. Functions with tight backward branches in the prologue often stay stuck on the INT3 path forever.