When you type a question into ChatGPT and watch the answer stream back, two very different things are happening inside the datacenter. First, the model has to read and understand your prompt — this is called the prefill phase, and it's compute-heavy: the GPU's math units are working flat out. Then, the model has to generate the answer one token at a time — the decode phase — which is memory-heavy: each new token requires shuttling huge piles of cached data in and out of memory.

Modern datacenter GPUs (think H100s and B200s) are built with HBM — High Bandwidth Memory. HBM is spectacular but extremely expensive: it's stacked vertically on top of the GPU, requires exotic packaging, and is in short supply. The dirty secret: during prefill, all that fancy HBM bandwidth sits mostly idle. You're paying premium prices for a feature you barely use half the time.

This paper proposes a clever fix. Since recent systems already split prefill and decode onto separate machines (a trick called disaggregation), why not use different kinds of hardware for each? Specifically:

• Prefill runs on cheaper accelerators using GDDR memory — the same kind of memory in consumer gaming GPUs. It has less bandwidth than HBM but plenty for compute-bound work.
• Decode stays on premium HBM-based GPUs, where the bandwidth actually pays off.

The authors call this setup MemHA (memory-heterogeneous accelerators). On paper, it should give you the same performance at a fraction of the cost. In practice, gluing two different hardware types together raises hard questions: how do you move the cached intermediate state ("KV cache") between machines without choking on the network? How do you schedule requests so neither side becomes a bottleneck? How do you handle the awkward mismatch when a fast prefill server hands off to a slower-to-receive decode server?

The paper works through these problems and shows that a well-engineered MemHA system can serve large language models at substantially lower cost than an all-HBM deployment without meaningfully degrading user-facing latency or throughput.