Reconciliation loops keep your system converging toward desired state. But a naive loop that retries every N seconds becomes a weapon when things go wrong. When a downstream API starts failing, every controller instance retries at the same interval, and your recovery attempt turns into a self-inflicted DDoS.

The naive loop: observe → diff → act → sleep 30s → repeat. When the API is healthy, this is fine. When the API returns 503, all 50 controller replicas hammer it every 30 seconds forever. The API can't recover because your "helpful" reconciliation is preventing it from catching its breath.

The fix has three parts:

• Exponential backoff on failure: after each failed reconcile, double the wait time. 30s → 60s → 120s → 240s, capped at some maximum (say 15 minutes).
• Jitter: add randomness so replicas don't synchronize. Instead of sleeping exactly 60s, sleep between 30s and 90s. This breaks the "thundering herd on the retry" problem.
• Reset on success: once a reconcile succeeds, drop back to the base interval. You want quick response to new changes, not a controller that's still backing off from a resolved outage.

Real example: A Kubernetes operator I worked with managed 800 custom resources, each triggering an external API call on reconcile. When the external API had a partial outage, the operator's fixed 15-second requeue turned into 3,200 requests per minute against a service already failing. The API team paged us, not the other way around. We switched to exponential backoff (15s base, 10min cap) with ±25% jitter. Next outage: the operator quietly backed off, the API recovered in 4 minutes, and nobody noticed.

Rule of thumb: your maximum backoff should be roughly (target recovery time) / (number of retries you're willing to attempt during recovery). If you want to recover within 30 minutes and are okay with ~5 retry attempts during that window, cap backoff at 6 minutes. Never cap below 1 minute — a controller retrying every 30 seconds during a real outage is indistinguishable from an attack.

What to watch out for: don't apply backoff to new events. If a user changes a resource, you want immediate reconciliation, not a 10-minute wait because the previous reconcile of a different resource failed. Track backoff per resource, not per controller.

Also: log the backoff duration. When someone asks "why isn't my change taking effect?", you need to be able to say "we're in a 4-minute backoff window because of these three failures" — not shrug.