This helper runs a child process and returns its stdout and stderr separately. It works flawlessly in unit tests, where commands produce a handful of lines. It works for months in production. Then one day someone runs it against a verbose tool and the entire worker pool freezes.

import subprocess

def run_capture(cmd):
    """Run cmd, return (stdout_bytes, stderr_bytes, exit_code)."""
    p = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    # Drain stdout first, then stderr, then wait for exit.
    stdout = p.stdout.read()
    stderr = p.stderr.read()
    exit_code = p.wait()
    return stdout, stderr, exit_code

# Works fine:
run_capture(["echo", "hello"])

# Hangs forever on a chatty child:
run_capture(["./build.sh"])   # writes ~200 KB of warnings to stderr

The Bug

OS pipes have a fixed kernel buffer — on Linux, typically 64 KiB. When a writer fills the buffer, the next write() blocks until a reader drains it. That's the time bomb.

Trace what happens with ./build.sh:

1. The parent calls p.stdout.read(), which blocks until stdout sees EOF — i.e., until the child exits.
2. The child starts emitting compiler warnings on stderr. After ~64 KiB the stderr pipe fills.
3. The child's next write() to stderr blocks, waiting for the parent to drain it.
4. But the parent won't touch stderr until p.stdout.read() returns. And p.stdout.read() won't return until the child exits. And the child can't exit because it's blocked on a stderr write.

Classic three-way deadlock between two pipes and a serial reader. The child is alive, the parent is alive, neither will ever make progress. strace shows the child wedged in write(2, ...); py-spy shows the parent wedged in read(). Everyone looks healthy in isolation.

The bug is invisible in tests because tiny outputs fit in the buffer, so the child finishes and closes both pipes before the parent's serial reads matter. It surfaces only when a real workload crosses the buffer threshold — and only on whichever stream the parent reads second.

The Fix

Don't drain pipes serially. Either read them concurrently (threads, select, asyncio) or let the standard library do it for you with communicate(), which spins up reader threads under the hood:

def run_capture(cmd):
    p = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    stdout, stderr = p.communicate()   # drains both pipes in parallel
    return stdout, stderr, p.returncode

If you genuinely need streaming access (say, to tee output to a log as it arrives), you must read both descriptors concurrently — one thread per stream, or a single selectors loop. A bare p.stdout.read() followed by p.stderr.read() is always wrong for unbounded children.

Two related traps worth knowing:

• p.wait() before read() is just as bad. The docs warn: "This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data."
• Merging streams with stderr=subprocess.STDOUT eliminates the second pipe and lets a single read() work safely — at the cost of interleaved output.