2026-08-31
Normal socket I/O follows a well-worn path: your thread calls recv(), finds the receive queue empty, blocks. The NIC receives a packet, fires an interrupt, softirq processes it, wakes your thread, scheduler eventually runs it, you return from the syscall. Best case: 15–50 microseconds of pure overhead. For HFT, RDMA fallback paths, or database replication, that's a lifetime.
SO_BUSY_POLL is a socket option that changes the contract. When you call recv() on an empty socket, instead of sleeping, the kernel directly polls the NIC driver for new packets for up to N microseconds before falling back to the normal wait path. The driver's NAPI poll function runs synchronously on your thread's CPU — no interrupt, no context switch, no wakeup latency.
How to enable it:
setsockopt(fd, SOL_SOCKET, SO_BUSY_POLL, &usecs, sizeof(usecs)) — per socketsysctl net.core.busy_read = 50 — system-wide default for readssysctl net.core.busy_poll = 50 — for poll()/epoll_wait()The driver must implement ndo_busy_poll (most modern 10G+ NICs do: ixgbe, mlx5, i40e). Under the hood it calls napi_busy_loop(), which grabs the NAPI context, drains descriptors from the RX ring, and pushes packets up the stack — all on your calling thread.
Real-world example: A market-data feed handler on a Mellanox ConnectX-5 with default settings shows ~12µs median wire-to-userspace latency, dominated by interrupt delivery and thread wakeup. Enable SO_BUSY_POLL=50 and pin the thread to an isolated core (isolcpus=, nohz_full=), and median drops to ~2µs with a tighter tail. The cost: that core sits at 100% CPU forever, and you lose the power savings from C-states.
Rule of thumb: Set the busy-poll window to roughly your packet inter-arrival time. If packets arrive every 3µs, poll for 5–10µs. Poll longer than that and you waste cycles; shorter and you fall back to interrupts and lose the whole benefit. Also disable interrupt coalescing (ethtool -C rx-usecs 0) — coalescing and busy-polling fight each other.
Gotchas: Only works with sockets whose NIC queue is on the same NUMA node as your thread — otherwise you're polling a remote NAPI context over QPI. And it silently degrades to normal behavior if the driver doesn't support it, so always measure — don't assume enabling the option did anything.
