2026-06-11
Everyone installs sysstat for sar and iostat, then never discovers the gem hiding in the same package: pidstat. It is sampled, timestamped, per-process performance accounting across every dimension that matters — CPU, memory, disk, and scheduler contention — at whatever interval you choose, in plain greppable columns.
The mainstream alternatives all give you a slice of what you need:
top/htop — live, jittery, no history, painful to script.ps — cumulative since process birth. Useless for "what's spiking right now."vmstat — system-wide; can't isolate one PID.iotop — disk only, and needs root for anything useful.pidstat samples per-process metrics every N seconds for M iterations and prints them to stdout. That's it. The flags compose:
# CPU usage of every running process, every 1s, 5 samples
pidstat 1 5
# Disk I/O per process (includes iodelay — see below)
pidstat -d 1
# Memory: RSS, virtual size, AND minor/major page faults
pidstat -r 1
# Context switches — voluntary vs involuntary
pidstat -w 1
# Per-thread breakdown of one PID
pidstat -t -p 12345 1
# Filter by command-name regex
pidstat -C 'python|node' 1
# All four dimensions at once, with timestamps
pidstat -druhw 1
The features that earn it a permanent slot in my incident playbook:
-w separates cswch/s from nvcswch/s. High involuntary context switches = CPU-bound and getting preempted. High voluntary = blocking on I/O or syscalls. That single distinction answers "is this process slow because it's busy or because it's waiting?" without strace.-d exposes iodelay, the cumulative clock ticks the kernel charged this task for block I/O wait. If iodelay is climbing, the process is genuinely disk-bound — not just doing reads, but stalled on them.-r separates minflt/s from majflt/s. Minor faults are RSS growth (cheap). Major faults mean you're hitting disk to page memory in. A nonzero majflt/s on a "fast" service is the smoking gun for swap thrash.-h puts a unix timestamp on every line. Now your output joins by time against application logs, dmesg, anything.The standing one-liner for "prod is slow, I don't know what":
pidstat -druhw 1 > /tmp/pid-$(date +%s).log &
# ...let it run through the incident, then:
kill %1
awk '$8 > 50 || $14 > 0' /tmp/pid-*.log # CPU>50% or any iodelay
Every metric, every process, every second, timestamped, with no instrumentation, no perf events, and overhead under one percent. It reads /proc/[pid]/stat and /proc/[pid]/io — that's it.
A few extras the man page buries:
# Per-CPU breakdown for one process — find the hot core
pidstat -u -p 12345 -P ALL 1
# Include exited children's accounting (build systems, test runners)
pidstat -T CHILD 1 5
# Stack pressure / RSS for one PID, machine-readable
pidstat -rs -p 12345 1 | tail -n +4
Why this beats every modern alternative: APM agents tell you the application's view; pidstat tells you the kernel's view of the application. When those disagree — and they will — the kernel is right. Wire it into your incident runbook beside dmesg -T and you'll stop guessing.
pidstat is the sampled, timestamped, per-process performance table that top, ps, and vmstat together can't quite give you — and it's already on your box if sysstat is installed.
