When you write() to a file, the kernel doesn't touch the disk. It copies your bytes into the page cache, marks those pages dirty, and returns. The actual I/O happens later, driven by a background flusher thread (kworker/flush) and gated by four sysctl knobs that most engineers never touch — until latency mysteries appear.

The knobs live in /proc/sys/vm/:

• dirty_background_ratio (default 10%): once this fraction of available memory is dirty, the flusher wakes up and starts writing in the background. Your writes stay fast.
• dirty_ratio (default 20%): once this fraction is dirty, your write() call is blocked and made to do the writeback itself. This is the "cliff."
• dirty_expire_centisecs (default 3000 = 30s): how old a dirty page must be before the flusher writes it, even if the background threshold hasn't been hit.
• dirty_writeback_centisecs (default 500 = 5s): how often the flusher wakes up to check.

The disaster scenario. Server with 128 GB RAM, default settings. A batch job dirties memory at 2 GB/s. Background threshold is 12.8 GB — hit in 6 seconds, flusher wakes up. But the disk sustains only 500 MB/s. Dirty pages keep growing. At 25.6 GB (20%), every subsequent write() blocks, throttled to disk speed. Meanwhile, an unrelated process calls fsync() on a 4 KB config file — and waits for the entire 25 GB of dirty pages belonging to other files to flush, because fsync() flushes the whole inode's dirty range and competes for the same congested block queue. Your "quick config update" takes 30+ seconds.

The rule of thumb. Set dirty_bytes (the absolute-value version of the ratio) to roughly 1–2 seconds of your disk's write bandwidth. For a 500 MB/s SSD: vm.dirty_bytes = 500000000, vm.dirty_background_bytes = 100000000. This caps worst-case latency at ~1 second instead of "however long it takes to flush 20% of RAM."

Real-world example. PostgreSQL and Redis both document this. PostgreSQL's checkpoint-tuning guide explicitly recommends lowering dirty_background_bytes because default settings let dirty pages accumulate until a checkpoint's fsync storm stalls all queries. Facebook's engineering blog documented multi-second p99 spikes traced entirely to the 20% cliff on machines with hundreds of GB of RAM.

Check yours right now: grep -E 'Dirty|Writeback' /proc/meminfo. If Dirty is in the gigabytes, your next fsync() is going to hurt.