2026-07-25
Blue-green deployments flip 100% of traffic at once. That's fast rollback, but it's also fast blast radius — if the new version has a bug your tests missed, every user hits it simultaneously. Canary deployments route a small slice of traffic to the new version first, watch the metrics, and expand only if things look healthy.
The name comes from canaries in coal mines: send the fragile thing in first, and if it dies, you know to turn back before the humans get hurt.
The standard progression looks something like this:
What to watch during each stage: error rate (5xx responses), p50/p95/p99 latency, CPU/memory on the canary pods, and business metrics like conversion or checkout success. Compare canary metrics to the stable version side by side — absolute thresholds lie because traffic composition varies by hour.
Concrete example: A payments team ships a rewritten fraud check. At 1%, error rate looks fine but p99 latency on the canary is 340ms vs 180ms on stable. Absolute latency is under their 500ms SLO, so a naive alert wouldn't fire — but the relative regression (1.9x) trips the canary analyzer, which auto-rolls back. Investigation reveals a missing index on a new lookup table. Had they gone straight to 100%, every payment would have slowed down, users would have double-clicked "pay," and the duplicate-charge tickets would have flooded support.
Rule of thumb for canary duration: the canary needs to run long enough to observe your slowest-firing issue. If your worst production bugs historically surface within 30 minutes, a 10-minute canary catches most but not all. Multiply by 2-3x for safety. Traffic-based (not time-based) canaries are better when volume is spiky — "wait until we've seen 100k requests" is more meaningful than "wait 30 minutes."
Where canaries fail: stateful changes (schema migrations), long-lived connections (WebSockets that live for hours), and features that require critical mass (a chat feature is useless if only 1% of users have it). For these, canary the infrastructure, not the feature — ship dark, then flag on.