2026-08-30
Grace periods let ownership transfer smoothly. But when a field flaps between managers — HPA sets replicas to 5, then a human sets it to 10, then HPA claws it back to 3 — you get post-mortem chaos. Who owned replicas at 14:23 when the outage started? Without an audit trail, you're guessing.
The pattern: on every ownership transition, append an immutable record of who owned the field, when they took it, when they released it, and what the value was at handoff. Store it separately from the live object so it survives deletes.
Real-world example: A fintech runs a Kubernetes controller managing a PaymentProcessor CRD. The maxConcurrentTransactions field is normally owned by an autoscaler, but during incidents an on-call engineer takes ownership via kubectl. At 03:47 UTC, transactions start failing. The autoscaler had set the value to 200 at 03:15. At 03:31, the on-call bumped it to 500 with a 10-minute grace period. At 03:42, the grace expired and the autoscaler reclaimed ownership, dropping it back to 200 mid-incident — starving the payment queue. Without the audit trail, the team blamed the autoscaler. With it, they saw the exact handoff timeline and added a rule: grace periods extend automatically if error rates are elevated.
What each audit entry needs:
spec.maxConcurrentTransactions)Rule of thumb: keep audit entries for at least 3× your longest grace period, and never less than 7 days. If your grace periods top out at 30 minutes, retain 90 minutes minimum — but you'll want a week for post-mortems. A field flapping between two managers 10 times an hour generates ~1,700 entries per week per field. At ~500 bytes per entry, that's under 1MB per field per week — cheap.
Don't audit reads. Only ownership transitions. Auditing every reconciliation write balloons storage and drowns the signal. And don't store audit trails in the object's status — status gets rewritten constantly. Use a separate audit stream (a dedicated CRD, a log aggregator, or an event sink).