Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-$p$ selection more suitable than fixed top-$k$ sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36$\times$ prefill speedup at 1M context and about a 2.01$\times$ decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.

📋 한 줄 요약

**[Sparse Attention / Full→Sparse Transfer]** RTPurbo가 full-attention LLM의 intrinsic sparsity 활용 — retrieval head만 full KV cache·16D indexer로 token retrieval·동적 top-p 선택, 수백 step으로 1M 컨텍스트에서 9.36× prefill·2.01× decode speedup·near-lossless.

🎯 핵심 기여도

LLM long-context inference가 full attention의 quadratic 비용에 bottleneck됨을 지적.
기존 efficient 대안이 native sparse training이나 heuristic token eviction에 의존해 효율·학습 비용·정확도 간 undesirable trade-off를 만든다는 한계 진단.
Full-attention LLM이 이미 intrinsically sparse하며 minimal adaptation만으로 highly sparse model로 변환 가능함을 보임.
세 관찰 기반 접근: (1) 일부 attention head만 full long-context processing을 진정으로 요구, (2) Long-range retrieval이 주로 low-dimensional subspace로 통제되어 16-dimensional indexer로 relevant token 효율 검색 가능, (3) Useful token budget이 strongly query-dependent — 고정 top-k 대신 dynamic top-p 선택이 더 적합. 이에 기반해 RTPurbo 제안 — retrieval head에 대해서만 full KV cache 보유, sparse attention용 경량 token indexer 도입. 모델의 intrinsic sparsity 활용으로 단 수백 training step만으로 sparsification 달성.

💡 핵심 아이디어

Long-context LLM의 효율은 native sparse pretraining이나 heuristic eviction 없이도, full-attention 모델의 intrinsic sparsity(소수 retrieval head·16D subspace·query-dependent budget)를 활용해 수백 step의 minimal adaptation만으로 near-lossless·9× 수준 prefill speedup을 달성할 수 있다.

🔬 기술적 접근법

**방법론**: RTPurbo — full→sparse transfer via intrinsic sparsity.
**핵심 기법**: (1) Retrieval head 식별 — 소수 head만 full long-context 필요, (2) Long-range retrieval의 16D low-dimensional indexer로 효율 검색, (3) Dynamic top-p 선택 — 고정 top-k 대신 query-dependent budget, (4) Retrieval head에 대해서만 full KV cache 보유, (5) 단 수백 training step의 minimal adaptation으로 sparsification.

📊 주요 결과

Long-context benchmark·reasoning task에서 near-lossless accuracy 보존.
1M context에서 최대 9.36× prefill speedup.
약 2.01× decode speedup.
단 수백 step training만으로 full→sparse 변환.
Expensive native sparse pretraining 없이 strong sparse inference 가능.

💭 의의 및 한계

**의의**: Full-attention LLM의 intrinsic sparsity를 명시적 활용하는 새 관점, 표준 full-attention 학습 후 sparsification으로 학습 비용·정확도 trade-off 해소, 1M 컨텍스트의 9.36× prefill speedup의 실용적 임팩트. **한계**: 16D subspace·retrieval head 식별의 모델·task별 검증 필요, dynamic top-p hyperparameter 튜닝, abstract에서 정확도 손실 정량은 "near-lossless"로만 표현.

🚀 실용적 활용

1M 컨텍스트 LLM의 효율적 추론.
Long-context RAG·document QA·코드베이스 분석.
Full→sparse 변환의 학습 비용 절감 레시피.