C-States and the CPUIdle Governor: Why Your "Idle" CPU Costs You 50 Microseconds When It Wakes Up

2026-07-05

When a core has no runnable task, the kernel doesn't just spin — it executes MWAIT with a hint telling the CPU how deeply to sleep. That hint is a C-state: C0 is running, C1 halts the clock, C1E adds voltage reduction, C6 flushes and powers off the core's L1/L2 caches and saves architectural state to package RAM, C7/C8/C9/C10 progressively power off more of the uncore, LLC, and PLLs.

Deeper C-states save more power but cost more exit latency. Typical numbers on a Xeon or modern desktop part:

The menu governor (default on Linux) decides which state to enter by predicting how long the idle period will last. It multiplies the expected sleep time (from the next scheduled timer, adjusted by a decaying average of past idle durations) by a correction factor based on recent I/O wait, then picks the deepest state whose target residency (break-even point) fits. If the prediction was wrong and you exit early, the energy spent entering the state was wasted.

Real example — the 99.9th percentile problem. An HFT firm ran a UDP echo server averaging 5 µs response time. Under light load it suddenly showed 80 µs tail spikes. turbostat revealed cores dropping into C6 between packets. Every packet arriving after >600 µs of idle paid the C6 exit penalty plus a warm-cache reload. Fix: boot with intel_idle.max_cstate=1 processor.max_cstate=1, or write 1 to /sys/devices/system/cpu/cpu*/cpuidle/state[2..N]/disable. Tail latency dropped to 8 µs. Power draw per idle core rose from 3W to 12W — the cost of trading watts for microseconds.

Rule of thumb: a C-state is worth entering only if you'll stay there at least twice the target residency. For latency-sensitive threads, pin them to cores where you've disabled anything past C1, or use PM_QOS_CPU_DMA_LATENCY — writing a microsecond value to /dev/cpu_dma_latency caps the deepest state the governor will pick, system-wide, for as long as your fd stays open.

Also watch for package C-states: a single busy core keeps the whole package in PC0. But once all cores idle, the package can enter PC6/PC10, and now waking any core requires re-powering shared resources — the LLC, the ring bus, the memory controller PLL. This is why the first packet after quiescence is always the slowest.

Key Takeaway: Idle isn't free — deeper C-states trade watts for wake-up microseconds, and for latency-critical workloads you must either cap the max C-state or hold /dev/cpu_dma_latency open with a low value.

All newsletters