RBS-Attention cuts time to first token sixfold at 128K without retraining

A 128,000-token prompt on a Qwen3-30B-A3B-Instruct-2507-FP8 model running on an H100 GPU produces its first token almost six times faster with RBS-Attention applied. That is the headline number of a paper published today on arXiv by a group tackling the least glamorous bottleneck in long-context inference: prefill.

Prefill is the phase in which the model processes the full prompt before generating anything. With dense attention, its cost grows quadratically with input length, so at 128K tokens the wait for the first token becomes the dominant problem. The authors report a 20.65x speedup in prefill attention measured in isolation, 11.92x inside vLLM and 5.97x in total time to first token.

The problem: mean dilution

The idea of trimming prefill with block-sparse attention is not new. Keys are grouped into blocks, a centroid is computed per block, and only the blocks whose centroid looks relevant to the query are kept. It works well when useful information is spread out, but it fails in one very specific case: a block with a single highly relevant token surrounded by dozens of irrelevant ones has a bland centroid and gets discarded.

The paper names this failure mean dilution. It is a serious problem for needle-in-a-haystack retrieval tasks, where the fact the model needs may live at a single position in the context.

Two selection branches instead of one

RBS-Attention, short for Radius-Bounded Sparse, answers with two complementary branches that run without any training:

Centroid base branch: captures the average relevance of each block, like existing methods.
Radius rescue branch: uses the maximum radius of each key block, that is, how far the most extreme token sits from its centroid. A large radius signals that the centroid may be hiding something. The distribution of that radius depends on the prompt, the layer and the attention head, and the method models it to decide which blocks are at risk of being underestimated.

Each branch is thresholded independently and the two masks are combined. That controls how much weight rescued blocks get without breaking the standard block-sparse FlashAttention execution, which is exactly where the speed gain comes from. No new kernel and no prior fine-tuning are needed.

What it loses in quality

On the dense Qwen3-32B model, RBS-Attention scores 88.65 overall accuracy on RULER, the reference benchmark for long context, versus 89.52 with dense attention. Less than one point of difference in exchange for an order-of-magnitude cost reduction in prefill attention. The abstract does not break down RULER by subtask or report behaviour beyond 128K, so it is worth reading the full paper before extrapolating.

Why it matters and for whom

Models with large context windows have become common. Claude Opus 4.8 offers an optional 1M-token window, and open models such as the Qwen3 variants in the paper are already evaluated at 128K. The cost of using those windows is not only the price per token but the latency until the first response, which is what the user perceives.

RBS-Attention is mainly of interest to three profiles:

Teams serving open models with vLLM or their own infrastructure who want lower latency on long-prompt workloads: RAG over many documents, code repository analysis, transcript summarisation.
Inference efficiency researchers working on sparse attention who need a clear explanation of why centroid-based methods fail at pinpoint retrieval.
* Anyone designing agents with accumulated context, where every turn reprocesses a prompt that keeps growing and prefill runs dozens of times per session.

For those consuming models through a closed API, such as Anthropic's, the impact is indirect: techniques like this end up integrated on the server and show up as lower latency, not as an option to toggle.

What remains to be checked

The figures were measured on H100s with an FP8-quantised MoE model, a favourable setup for this kind of speedup. Results on more modest hardware and on large dense models beyond the quality evaluation are missing. We also do not know yet how much computing per-block radii costs in practice, or whether the method plays well with the prefix caching production servers rely on.

At ElephantPink we have seen in RAG projects that prefill latency limits more than price per token once context passes 50K, so any training-free technique that reduces it deserves attention. But one RULER point is one point, and in fine-grained retrieval tasks that difference can be exactly the fact the client needed.