KVBoost reuses the KV cache without a shared prefix

The prefill stage of an LLM recomputes key value (KV) tensors on every request, and the prefix caching systems that mitigate that cost today come with an awkward condition: they only hit when two prompts share a contiguous prefix from the first token. If the repeated fragment shows up in the middle of the context, there is no reuse at all. KVBoost, posted to arXiv on 25 August, proposes reusing the KV cache at chunk level regardless of where the shared content sits.

The scenario is more common than the theoretical framing suggests. In a RAG system the same documents are retrieved over and over, but rarely in the same order. In a coding assistant, the same files appear interleaved with different conversation. In both cases the repeated content is there, just not at the beginning.

Two hashes and two repairs

The core piece is a dual hash keying scheme that separates positional identity (prefix hash) from content identity (content hash). With that, the system supports both exact and approximate matches, something a classic prefix cache cannot offer.

The price of splitting the context is attention boundary errors. A chunk cached independently was computed without seeing its neighbours, so the tokens at the edge carry attention that does not match the real context. KVBoost puts forward two repair strategies: SelectiveRecompute, which re encodes the boundary regions, and CacheBlendRecompute, which runs a probe pass, locates the tokens with the highest deviation and recomputes only those. The name nods to CacheBlend, the earlier line of work on merging partial KV caches.

Around that, the system adds three more mechanisms: asymmetric KV quantisation (int8 and int4), adaptive chunk boundary splitting and importance weighted eviction under a fixed memory budget. It is a pragmatic combination, aimed at letting the technique survive outside the test bench.

What the paper measures and what it does not

The evaluation runs on Qwen/Qwen2.5-3B with 1,000 bug localisation samples, a use case where the same repository is sent again and again with different questions. The abstract published on arXiv is truncated and does not include the final latency or quality figures, so anyone who wants to quote concrete speedups will have to go to the PDF. This is a freshly announced v1, with no peer review.

There is also a scope limitation worth reading closely: KVBoost is designed for HuggingFace compatible decoder models, meaning for whoever serves open weights on their own infrastructure. It is not something you can switch on from the outside against a closed API.

Who it is useful for

If your team serves its own models, this goes straight onto the list of things to try. Prefill is where time to first token goes when the context is long, and the contiguous prefix restriction leaves a fair amount of savings on the table.

If you work against the Claude API, the technique does not apply but the lesson does. Anthropic's prompt caching also works by prefix, with explicit breakpoints, and that forces you to order the prompt from the most stable to the most volatile: instructions and fixed documents first, then the history, and the current question at the end. Work like this is a reminder that the restriction is an implementation detail and not a law of physics, even if it is still the rule you have to design around today.

We care less about the speedup number than about the direction of the work: moving the unit of reuse from the prefix to the chunk is the kind of boring optimisation that ends up changing the real cost of serving long context. It remains to be seen whether the cost of the repair passes stays under control as the number of chunks grows.