KV Sharing, MHC and Compressed Attention: the architectural bets shaping LLMs in 2026

Inference cost remains the most concrete bottleneck in large model deployment. It's not a benchmark problem: it's real money on compute bills. That's why advances in how LLMs manage attention memory, especially the key-value cache (KV cache), carry such weight in applied research this year.

Sebastian Raschka, a researcher and technical communicator with established credibility in the ML community, published a detailed analysis of three architectural development lines gaining traction this week on his Substack: KV Sharing, Multi-Head Compression (MHC), and Compressed Attention. The thread on Hacker News didn't accumulate many comments, but the article itself circulated strongly among ML engineers, which says quite a bit about the content's profile: dense, technical, and uncompromising in its approach.

What each approach proposes

KV Sharing starts from an empirical observation: in many transformer layers, the key (K) and value (V) matrices generated by each attention head are redundant with one another. If multiple heads share KV representations instead of computing them independently, cache size shrinks without appreciable quality loss. The idea isn't new—variants like Multi-Query Attention (MQA) or Grouped-Query Attention (GQA) have been in production for years—but recent work pushes sharing to more aggressive levels and across layers, not just across heads.

Multi-Head Compression (MHC) addresses the problem from another angle: rather than removing or merging heads, it projects K and V vectors into a lower-dimensional space before storing them in cache. During generation, they're retrieved and expanded back. The tradeoff is explicit: some projection cost in exchange for significantly smaller memory footprint. Raschka notes that some recent experiments show compression can be quite aggressive—4x to 8x factors—without perplexity spiking on standard tasks.

Compressed Attention is perhaps the most ambitious of the three. Instead of operating on the full sequence of past tokens, it applies compression mechanisms to the context before building the cache. Some recent implementations combine this with sliding attention or anchor token selection, so the model maintains a compressed representation of distant context without discarding it entirely. For long context windows—Claude Opus 4.7's million tokens is an extreme example—this type of technique could be the difference between economic viability and infeasibility.

Why it matters beyond the paper

These three lines converge on a practical problem: the larger the context window, the more the KV cache grows in memory, and the more expensive maintaining long sessions becomes in production. Teams deploying models as a service know this well: cost doesn't scale linearly with token count, but worse, because the cache takes GPU memory that competes with batch size.

For those working with Claude Code and MCP servers in long-running pipelines—processing extensive documents, analyzing entire repositories, agents with persistent history—inference efficiency has direct consequences for latency and cost per call. It's not disconnected laboratory research: advances in KV cache compression are what eventually make those million-token windows economically sustainable at scale.

Raschka's analysis is also useful as a navigation map. Literature on attention efficiency is abundant and sometimes contradictory in its claims. Having a synthesis that distinguishes what works in controlled benchmarks from what has reached production models has real value for engineers who need to make architecture or model selection decisions.

Who it's relevant for

The article is written for a reader with technical grounding in transformers. It's not onboarding material, but it doesn't require having read the original papers either. It's especially useful for:

• ML engineers evaluating architecture options for their own models or fine-tuning.
• Infrastructure teams optimizing inference cost in production.
• Developers of Claude integrations who need to understand the real constraints of long context.

Our take: The fact that these techniques advance in parallel across several labs suggests the economic pressure on inference is real and shared. Raschka does good work articulating the state of the art without overselling; it's worth having as a reference.