SchedBlame: Who Ran While You Waited? Culprit-Attributed CPU Contention for Containers on Stock Kernels

2026-09-03

Authors: Hao Li, Tonghao Zhang, Honglei Wang

ArXiv: 2609.02052v1

PDF: Download PDF

Imagine you're running a busy restaurant kitchen where multiple chefs share the same stove. One chef's dish keeps coming out late, and you need to know why. Every existing tool can tell you "this chef waited 20 minutes for a burner" — but none can tell you which other chef was hogging it. That's the exact problem this paper tackles, except the "kitchen" is a Linux server and the "chefs" are containers (think Docker) competing for CPU time.

The problem in plain terms: When you pack many containers onto one machine to save money (which basically every cloud provider does), they fight for CPU. When one gets slow, operators today can only see victim-side signals — metrics that say "container A waited a lot" but not "container B is the reason." Finding the culprit currently requires either patching the Linux kernel (which most people can't do on cloud VMs), running heavyweight tracing (too expensive to leave on all the time), or making statistical guesses (unreliable when multiple victims exist at once).

What SchedBlame does: It's a lightweight monitoring tool that runs on a stock, unmodified Linux kernel and directly attributes CPU contention to the specific container causing it. Instead of just recording "you waited," it records "you waited, and here's exactly who was running instead of you." It does this by hooking into the kernel's scheduler using eBPF (a modern Linux feature that lets you safely run small programs inside the kernel without patching it) to capture the moment one task gets kicked off the CPU in favor of another — and then rolls those events up per-container.

Why the trick is clever: The naive approach — trace every context switch everywhere — generates so much data it slows the whole machine down. SchedBlame's insight is that you only need to record blame events when a container is actually being starved, and you can aggregate the culprits cheaply on-the-fly rather than post-processing terabytes of traces. That makes it cheap enough to leave running 24/7 in production.

Who cares:

Anyone who's ever stared at a Grafana dashboard showing high CPU wait and thought "yes, but why?" has felt this exact pain.

Why it matters: Turns a decades-old blind spot in Linux — knowing your container is slow but not knowing who's stealing your CPU — into a cheap, always-on production signal that works without kernel patches.

All newsletters