You changed a hot loop. You ran the benchmark. Old code: 1.42s. New code: 1.38s. Ship it?

No. You just committed the most common performance sin in software: reading the mean of two noisy samples and declaring victory. That 3% "improvement" might vanish tomorrow — or reverse. Poul-Henning Kamp got tired of watching FreeBSD hackers do this in the early 2000s and wrote ministat, a ~300-line C program that lives in FreeBSD base and packages fine on Linux (apt install ministat, brew install ministat).

It takes columns of numbers and runs Student's t-test. That's it. That's the whole thing.

# Collect 20 samples per variant — more is better
for i in $(seq 20); do /usr/bin/time -f %e ./old 2>>&1; done > old.txt
for i in $(seq 20); do /usr/bin/time -f %e ./new 2>>&1; done > new.txt

ministat -c 95 old.txt new.txt

Output — actual, not paraphrased:

x old.txt
+ new.txt
+--------------------------------------------------------------------------+
|      x                                                                   |
|   x  x     x                        +                                    |
|xx xx xx  xxxx  x           +      + ++  +  +   +                         |
|  |____AM____|                     |______MA______|                       |
+--------------------------------------------------------------------------+
    N    Min      Max     Median      Avg      Stddev
x  20  1.401    1.462     1.421    1.4235   0.017429
+  20  1.352    1.418     1.379    1.3805   0.019204
Difference at 95.0% confidence
    -0.043 +/- 0.011729
    -3.020% +/- 0.824%
    (Student's t, pooled s = 0.018349)

Three things worth staring at:

• The ASCII box plot shows the distributions, not just means. You'll immediately spot the run where GC fired, or the outlier that was actually a background updatedb.
• "Difference at 95.0% confidence" is what you were guessing at with your eyeballs. If ministat says "No difference proven at 95% confidence," your speedup is noise, no matter how nice the mean looked.
• The reported delta comes with a confidence interval: -3.02% ± 0.824%. That's the honest answer.

Why not just use hyperfine? Hyperfine runs the benchmark. Ministat analyzes samples from anywhere — ab latency dumps, database query timings, GC pause logs, CI build times pulled with jq. Pipe two columns of numbers in, get significance out. Hyperfine and ministat are complements: hyperfine --export-csv, then feed the columns to ministat when you want more than a single mean.

Useful flags:

• -c 99.5 — tighter confidence (options: 80, 90, 95, 98, 99, 99.5). Papers usually want 99.
• -C 3 — pull column 3 out of whitespace-separated data. Skips the awk step.
• -w 74 — narrower plot for terminals or logs.
• -q — kill the plot, just print numbers. Good for CI gates.
• Passing 3+ files compares all against the first — regression bisects get compact.

The killer workflow: pipe your CI's benchmark job into a file per commit, then have a gate that runs ministat -q baseline.txt new.txt and fails only when the delta is significant and worse. You stop chasing 2% noise and stop shipping 8% regressions that hid inside variance.

Bonus history: PHK wrote it because he was tired of seeing benchmark arguments settled by whoever ran their test last. Twenty years later it's still 300 lines, still in FreeBSD base, still the correct answer.