A reconciliation loop watches desired state and drives actual state toward it. Add backoff and jitter and you handle failure retries gracefully. But there's a subtler problem: successful reconciliations can still overwhelm downstream systems when events fire faster than the API can absorb them.

Consider a Kubernetes controller managing 5,000 Pods. A ConfigMap change triggers a reconcile for every Pod that references it. Each reconcile issues an API call to update the Pod's annotation. Without rate limiting, your controller floods the API server with 5,000 requests in under a second — the API server throttles you, some requests fail, your controller retries them, and now you're in a feedback loop where the retry storm prevents recovery.

Rate limiting inside the reconciler solves this by decoupling event arrival rate from work processing rate. Controller-runtime uses a workqueue with two layers:

• Per-item rate limiter: if item X is reconciled and requeued, delay its next processing (exponential backoff based on failure count)
• Bucket rate limiter: token bucket capping overall queue drain rate (e.g., 10 QPS with burst 100)

The default in controller-runtime is a max-of rate limiter: for each item, take the larger of the item-specific backoff and the global bucket delay. This means a healthy item never waits on the global bucket unless total throughput exceeds capacity, but a failing item gets its own escalating penalty.

Sizing rule of thumb: set your bucket QPS to roughly 10-20% of the downstream API's rate limit, leaving headroom for other controllers, kubectl users, and admission webhooks. If the kube-apiserver allows 400 QPS per client, cap your controller at 40-80 QPS. Set burst to 5-10x QPS to handle legitimate spikes without smoothing away all reactivity.

The trap: too-aggressive rate limiting creates its own outage. If you cap at 5 QPS and a legitimate config change requires reconciling 5,000 resources, that's 1,000 seconds — over 16 minutes — before the last Pod converges. Users see stale state, alerts fire, someone restarts the controller thinking it's stuck. Rate limiting must be tuned to the ratio of steady-state churn to burst churn, not just the peak.

Also watch for priority inversion: a burst of low-priority reconciles (e.g., a label sync) can starve high-priority ones (e.g., a Pod deletion) if they share the same queue. Split queues by priority class when the workload is heterogeneous.