SO_BUSY_POLL: Trading a CPU Core for Microsecond Network Latency

2026-08-31

Normal socket I/O follows a well-worn path: your thread calls recv(), finds the receive queue empty, blocks. The NIC receives a packet, fires an interrupt, softirq processes it, wakes your thread, scheduler eventually runs it, you return from the syscall. Best case: 15–50 microseconds of pure overhead. For HFT, RDMA fallback paths, or database replication, that's a lifetime.

SO_BUSY_POLL is a socket option that changes the contract. When you call recv() on an empty socket, instead of sleeping, the kernel directly polls the NIC driver for new packets for up to N microseconds before falling back to the normal wait path. The driver's NAPI poll function runs synchronously on your thread's CPU — no interrupt, no context switch, no wakeup latency.

How to enable it:

The driver must implement ndo_busy_poll (most modern 10G+ NICs do: ixgbe, mlx5, i40e). Under the hood it calls napi_busy_loop(), which grabs the NAPI context, drains descriptors from the RX ring, and pushes packets up the stack — all on your calling thread.

Real-world example: A market-data feed handler on a Mellanox ConnectX-5 with default settings shows ~12µs median wire-to-userspace latency, dominated by interrupt delivery and thread wakeup. Enable SO_BUSY_POLL=50 and pin the thread to an isolated core (isolcpus=, nohz_full=), and median drops to ~2µs with a tighter tail. The cost: that core sits at 100% CPU forever, and you lose the power savings from C-states.

Rule of thumb: Set the busy-poll window to roughly your packet inter-arrival time. If packets arrive every 3µs, poll for 5–10µs. Poll longer than that and you waste cycles; shorter and you fall back to interrupts and lose the whole benefit. Also disable interrupt coalescing (ethtool -C rx-usecs 0) — coalescing and busy-polling fight each other.

Gotchas: Only works with sockets whose NIC queue is on the same NUMA node as your thread — otherwise you're polling a remote NAPI context over QPI. And it silently degrades to normal behavior if the driver doesn't support it, so always measure — don't assume enabling the option did anything.

Key Takeaway: SO_BUSY_POLL turns a blocking recv() into a synchronous NIC poll, cutting wakeup latency from tens of microseconds to single digits — at the cost of one permanently-burning CPU core.

All newsletters