When multiple controllers reconcile the same resource, you get a race: two controllers read the same version, both compute an update, both write. Without coordination, the second write silently clobbers the first. The controller with optimistic concurrency pattern solves this by attaching a version number (Kubernetes calls it resourceVersion) to every read, and requiring writes to include that version. If the stored version has changed since you read it, the write is rejected with a 409 Conflict, and you re-reconcile from scratch.

The mechanics are simple: read returns {spec, status, resourceVersion: 42}. You compute a new status. You write back with resourceVersion: 42. If someone else wrote in the meantime, the stored version is now 43, and the API server rejects your write. You re-read, re-compute, retry.

Real-world example: Two controllers watch a Deployment. The HPA (Horizontal Pod Autoscaler) wants to bump replicas from 3 to 5. The deployment controller wants to update the pod template hash after a rolling update completes. Both read version 42. HPA writes first — version becomes 43. Deployment controller's write with version 42 gets rejected. It re-reads version 43 (which now has replicas=5), re-applies its template hash change, and writes version 44. No lost update. No lock. No coordination service.

Rule of thumb: if your conflict rate exceeds 5% of writes, optimistic concurrency is the wrong tool — you're paying retry cost on every hot resource. Either shard the work (different controllers own different fields via server-side apply field ownership) or serialize through a work queue with a single worker per key.

The pattern's real trap is the retry storm. A controller that retries immediately on conflict will hammer the API server when contention spikes. Always combine optimistic concurrency with:

• Exponential backoff with jitter — don't retry in a tight loop.
• Fresh reads on retry — never retry with the stale object; the reconciliation logic must run again against current state.
• Bounded retry count — after N failures, requeue via the work queue instead of blocking the worker.
• Idempotent computation — because you will run it multiple times.

A common bug: mutating the cached object before writing. If the write fails and you retry with the same mutated object, you've now applied your delta twice. Always deep-copy before mutating, and re-fetch on conflict.