You're writing a Kubernetes controller. A watch stream fires an event every time a Pod changes. Your naive first draft calls reconcile(pod) directly from the watch callback. Then a rolling update ships and 500 Pods change in two seconds. Your reconcile logic — which calls the API server, updates status, maybe patches other objects — runs 500 times concurrently, hammers the API server, and half the calls fail with 429s. The next event arrives while the previous reconcile is still in flight for the same object, and now you have two goroutines fighting over the same resource version.

The work queue pattern puts a bounded, deduplicating queue between the event source and the reconciler. The watch handler does one thing: extract the object key (namespace/name) and call queue.Add(key). A fixed pool of workers pulls keys off the queue and runs reconcile. That's it.

The queue gives you four properties the direct-call model can't:

• Deduplication. If a key is already in the queue, adding it again is a no-op. 500 events for the same Pod collapse into one reconcile.
• Per-key serialization. The queue guarantees the same key is never processed by two workers simultaneously. No resource-version conflicts, no lost updates.
• Bounded concurrency. N workers means at most N API server calls in flight, regardless of event burst size.
• Rate-limited requeues. When reconcile fails, you call queue.AddRateLimited(key), which backs off exponentially per key. A failing object doesn't block others.

Real example: client-go's workqueue.RateLimitingInterface. The Deployment controller watches Pods, ReplicaSets, and Deployments — three streams that can all touch the same Deployment. Every handler resolves to the owning Deployment's key and enqueues it. One reconcile pass reads current state and reconciles. If a Pod flaps 20 times during that reconcile, the next pass sees one enqueued key, not 20.

Rule of thumb: workers ≈ 2–5 for most controllers. More workers doesn't help when the bottleneck is the API server, and it does hurt when a single-object bug now floods logs 20× faster. Start at 2, raise only when you measure queue depth staying nonzero under normal load.

The anti-pattern to avoid: enqueueing the full object instead of the key. Objects go stale between enqueue and process. Always enqueue the key, then re-read from the informer cache inside reconcile. The cache reflects the latest observed state; the object you saw at enqueue time is already history.