Imagine you're running a service like Claude Code or Cursor at scale. Thousands of developers are using AI coding agents simultaneously, and each session isn't a quick chat — it's a long, sprawling conversation where the model reads files, runs tools, gets results back, edits code, and repeats this loop dozens or hundreds of times. The infrastructure powering this is straining under workloads it was never designed for, and nobody had really measured why until now.

This paper does something refreshingly grounded: the authors collected real-world traces from actual coding agent sessions and studied how the serving infrastructure behaves. The villain of the story is something called the KVCache — short for "key-value cache." When an LLM processes your prompt, it does expensive math on every token and stores intermediate results in memory so it doesn't have to redo that work on the next turn. For chatbots, this cache is small and short-lived. For coding agents, it's enormous and sticky.

The authors found two patterns that break standard serving systems:

• Massive shared prefixes: Every turn in a coding session re-sends the same gigantic context — system prompt, tool definitions, file contents, conversation history. Sessions repeatedly reuse these large prefixes, so naively recomputing them wastes huge amounts of compute.
• Sustained cache pressure: Unlike chat, where users come and go in seconds, coding sessions stay alive for minutes or hours while the agent thinks, runs tools, and waits. The cache has to be kept warm across long idle gaps, and existing eviction policies (which assume "if it's been idle, dump it") throw away exactly the data that's about to be reused.

Current LLM serving systems were tuned for chatbot-style workloads: short prompts, quick replies, predictable turnover. Coding agents violate every one of those assumptions. The result is wasted GPU memory, redundant recomputation, and higher latency for every user on the box.

The paper proposes CacheWise, a smarter management layer that recognizes the structure of agent sessions. It keeps shared prefixes around longer, predicts when a session is genuinely done versus just paused waiting on a tool call, and makes eviction decisions based on how coding workloads actually behave rather than chat heuristics.

The reason this matters beyond the technical weeds: coding agents are one of the fastest-growing use cases for LLMs, and the cost of serving them is a major bottleneck for both providers and the price users pay. Squeezing more efficiency out of the same hardware translates directly into cheaper, faster, more responsive agents — and into providers being able to support longer, more capable sessions without melting their GPUs.