You wrote a distributed system. Then you ran it on a gigabit LAN and called it tested. Out in the wild — cellular tethering, transatlantic peers, a switch with a dying capacitor — your retry logic falls apart and your "exactly once" semantics become "yeah, three or four times, give or take." You need to simulate that hostile network without waiting for production to do it for you.

tc is part of iproute2 and has been in every Linux distro since the late 90s. Its netem qdisc — network emulator — lives in the kernel and mangles packets on the actual TCP stack. No userspace proxy, no client reconfiguration, no missed corner cases because someone forgot to route through Toxiproxy.

Basic invocations

Add 100ms of latency to every packet leaving eth0:

sudo tc qdisc add dev eth0 root netem delay 100ms

Latency with jitter, drawn from a normal distribution:

sudo tc qdisc add dev eth0 root netem delay 100ms 20ms distribution normal

Five percent packet loss, with correlation (loss tends to come in bursts on real links):

sudo tc qdisc add dev eth0 root netem loss 5% 25%

Pile it on — latency, loss, bit corruption, and reordering all at once:

sudo tc qdisc add dev eth0 root netem \
    delay 80ms 10ms \
    loss 1% \
    corrupt 0.1% \
    reorder 25% 50% \
    duplicate 0.5%

Inspect the current rules, then tear it all down:

tc qdisc show dev eth0
sudo tc qdisc del dev eth0 root

The killer feature: target specific traffic

Naive netem will eat your SSH session alive. Use a prio qdisc with a u32 filter so only the traffic you're actually testing gets abused. Here we punish Postgres on port 5432 and leave everything else alone:

sudo tc qdisc add dev eth0 root handle 1: prio
sudo tc qdisc add dev eth0 parent 1:3 handle 30: \
    netem delay 200ms 50ms loss 3%
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 \
    match ip dport 5432 0xffff flowid 1:3

Now your app's database connection gets the WAN treatment while your shell stays usable. Filter on source IP, destination IP, protocol, TOS, anything u32 can match.

Containers and CI

Inside a container with CAP_NET_ADMIN, you can netem the veth pair or even loopback. Spin up your service stack with docker-compose, attach to one container's network namespace, slap 300ms onto the link to the database container, and watch the connection-pool timeouts you swore couldn't happen:

docker run --cap-add=NET_ADMIN --rm -it myapp:test \
    sh -c 'tc qdisc add dev eth0 root netem delay 300ms loss 2% && \
           ./run-integration-tests.sh'

Why not a proxy?

• Toxiproxy / comcast / shadowsocks tricks need clients to point at a different host:port. Forget one config knob and you're testing the happy path again.
• iptables -j REJECT is binary — connected or dead. Real networks degrade.
• netem runs in the kernel against the real stack, so TCP retransmits, congestion control, and your app's socket timeouts all behave exactly as they will in production.

One footgun: netem only shapes egress. To garble inbound traffic, redirect it through an ifb (intermediate functional block) device first — modprobe ifb; ip link set ifb0 up — then apply the qdisc there. It's a one-liner once you've seen it; it's an afternoon of confusion the first time.