DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes

Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from incorrect reasoning traces by converting them into opportunities for improvement, making training more scalable and less dependent on external resources. This yields a richer and more diverse learning signal, improving exploration efficiency from imperfect model behavior. As a result, DenoiseRL improves reasoning performance and overall training efficiency while reducing the need for expensive data curation or stronger teacher models. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical and general reasoning benchmarks and promotes stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable alternative pathway for improving reasoning in large language models.

📋 한 줄 요약

**[Reasoning RL / Self-Bootstrapping]** DenoiseRL이 weak model의 incorrect reasoning trace로부터 recovery-oriented optimization을 수행 — stronger teacher·heavy curation 불필요, 수학·일반 reasoning benchmark에서 strong on-policy RL baseline 일관 능가.

🎯 핵심 기여도

강화학습이 LLM의 reasoning 발전 핵심 paradigm이지만 대부분 기존 방법이 stronger teacher model·heavily curated difficult dataset에 의존해 scalable capability improvement를 제한함을 진단.
DenoiseRL 도입 — external supervision을 weak model의 실패에 대한 recovery-oriented optimization으로 substitute하는 RL framework.
Stronger supervision·carefully engineered data 의존 대신 incorrect reasoning trace로부터 직접 학습 — incorrect trace를 improvement 기회로 변환해 training을 scalable·external resource 의존 적게 만듦.
풍부·다양한 learning signal을 산출, imperfect model behavior로부터 exploration efficiency 향상; expensive data curation·stronger teacher 필요 감소.

💡 핵심 아이디어

Reasoning RL의 scalability bottleneck은 stronger supervision·data curation 의존이며, weak model의 incorrect reasoning trace를 recovery-oriented optimization 대상으로 활용하면 external resource 없이 self-bootstrapping이 가능해 imperfect behavior가 풍부한 learning signal로 전환된다.

🔬 기술적 접근법

**방법론**: DenoiseRL — recovery-oriented RL from weak model failures.
**핵심 기법**: (1) External teacher·supervision 회피, (2) Weak model의 incorrect reasoning trace 수집, (3) 각 incorrect trace를 recovery 기회로 변환, (4) Recovery objective로 RL 학습, (5) Imperfect behavior에서 diverse learning signal·exploration efficiency 추출.

📊 주요 결과

경쟁 수학·일반 reasoning benchmark 전반에서 DenoiseRL이 strong on-policy RL baseline 일관 outperform.
Training difficulty 증가 시 stronger self-corrective behavior 촉진.
Stronger teacher·data curation 비용 없이 reasoning·training efficiency 향상.

💭 의의 및 한계

**의의**: Reasoning RL의 self-bootstrapping 가능성 입증, external 의존 회피로 scalable, self-corrective behavior 강화로 reasoning quality 향상. **한계**: Weak model 실패 trace의 quality에 의존, recovery objective 설계의 task 의존성, 매우 어려운 frontier task에서의 한계는 후속 검증.

🚀 실용적 활용

LLM reasoning capability의 저비용 향상.
Open-source 모델의 self-improvement 파이프라인.
Frontier teacher 없이도 학습 가능한 RL 레시피.