Every operation has two cost components: fixed overhead (network round-trip, TCP handshake, query parsing, transaction setup) and marginal cost (the actual work per item). When fixed cost dominates, doing one item at a time is wasteful. Batching collects N operations and pays the fixed cost once.

The classic trap is the N+1 query problem. Fetching 500 orders and then querying each order's customer individually is 501 round trips. If your database is 2ms away, that's ~1 second of pure network latency. Batching the customer lookups into one WHERE id IN (...) query drops it to ~4ms — a 250x speedup without changing a single index.

Real-world example: A payments service was writing audit events to Postgres one row at a time — 8,000 rows/second saturating a connection pool. Wrapping writes in a 50ms tumbling window and using COPY for bulk insert dropped CPU by 70% and let a single connection handle 40,000 rows/second. The 50ms added latency was invisible to users; the throughput unlock was massive.

Rule of thumb — the crossover formula:

• Let F = fixed cost per call, M = marginal cost per item
• Batching helps when F / M > 10. Below that, complexity outweighs savings.
• For a typical DB call: F ≈ 1ms, M ≈ 10µs → ratio of 100. Batch aggressively.
• For an in-process hash map lookup: F ≈ M. Don't bother.

Three batching triggers, pick based on latency tolerance:

• Size-based: flush at N items. Predictable memory, unbounded latency if traffic is slow.
• Time-based: flush every T milliseconds. Bounded latency, unpredictable batch size.
• Hybrid (the correct answer 90% of the time): flush at N items or T ms, whichever comes first. Bounds both dimensions.

What batching costs you:

• Latency floor: the first item in a batch waits for the batch to fill. Never batch synchronous user-facing reads unless the wait is under ~5ms.
• Blast radius: one poison-pill item can fail the whole batch. Isolate errors — return per-item results, not a single boolean.
• Memory pressure: unbounded queues become OOM incidents. Always cap the buffer and apply backpressure.
• Debuggability: a stack trace now covers 500 operations, not one. Log batch IDs and per-item correlation IDs.

Anti-pattern: batching writes to a system that already batches internally (Kafka producers, most SDKs). You're adding latency for savings the client library already gave you. Read the docs before wrapping.