KVEraser – Learning to Steer KV Cache for Efficient Context Erasing

Researchers propose KVEraser, a method to selectively erase or redirect key-value cache entries in transformer inference, reducing memory bandwidth and compute during long-context processing without retraining.

The KV cache is the primary bottleneck in long-context LLM inference—memory bandwidth, not arithmetic, limits throughput. Steering which tokens remain in cache directly improves token-per-second rates and reduces serving infrastructure cost per inference. This becomes material at scale: providers operating 100K+ concurrent users see direct OPEX reduction and higher margins per request.

For builders: long-context applications (RAG, document analysis, code repositories) become cheaper to serve, shifting economics toward larger context windows as feasible deployment strategy. For operators: inference efficiency improvements reduce GPU/memory requirements per user, enabling denser packing of workloads on existing hardware. Second-order effect: commoditization pressure increases on inference providers competing on cost-per-token, favoring those with hardware efficiency gains baked into serving stacks.