Efficient Agentic Reasoning Through Self-Regulated Simulative Planning

How should an agent decide when and how to plan? A dominant approach builds agents as reactive policies with adaptive computation (e.g., chain-of-thought), trained end-to-end expecting planning to emerge implicitly. Without control over the presence, structure, or horizon of planning, these systems dramatically increase reasoning length, yielding inefficient token use without reliable accuracy gains. We argue efficient agentic reasoning benefits from decomposing decision-making into three systems: simulative reasoning (System II) grounding deliberation in future-state prediction via a world model; self-regulation (System III) deciding when and how deeply to plan via a learned configurator; and reactive execution (System I) handling fine-grained action. Simulative reasoning provides unified planning across diverse tasks without per-domain engineering, while self-regulation ensures the planner is invoked only when needed. To test this, we develop SR^2AM (Self-Regulated Simulative Reasoning Agentic LLM), realizing both as distinct stages within an LLM's chain-of-thought, with the LLM as world model. We explore two instantiations: recording decisions from a prompted multi-module system (v0.1) and reconstructing structured plans from traces of pretrained reasoning LLMs (v1.0), trained via supervised then reinforcement learning (RL). Across math, science, tabular analysis, and web information seeking, v0.1-8B and v1.0-30B achieve Pass@1 competitive with 120-355B and 685B-1T parameter systems respectively, while v1.0-30B uses 25.8-95.3% fewer reasoning tokens than comparable agentic LLMs. RL increases average planning horizon by 22.8% while planning frequency grows only 2.0%, showing it learns to plan further ahead rather than more often. More broadly, learned self-regulation instantiates a principle we expect to extend beyond planning to how agents govern their own learning and adaptation.

한 줄 요약

**[Agentic Reasoning / Self-Regulated Planning]** SR^2AM이 simulative reasoning(System II)·self-regulation(System III)·reactive execution(System I)의 3 system 분해로 plan 시점·깊이 학습 — v1.0-30B가 685B-1T 시스템과 Pass@1 경쟁하며 reasoning token 25.8-95.3% 절감, RL이 planning 빈도(+2.0%)가 아닌 horizon(+22.8%)을 늘림.

핵심 기여도

Agent가 언제·어떻게 plan할지의 결정 문제 제기. Dominant approach가 agent를 adaptive computation(예: chain-of-thought)의 reactive policy로 end-to-end 학습하고 planning이 implicit emerge하길 기대.
Planning의 존재·구조·horizon 통제 없이 이러한 시스템이 reasoning 길이를 dramatically 증가시켜 신뢰할 만한 정확도 향상 없는 비효율 token 사용을 야기함을 진단.
Efficient agentic reasoning이 decision-making을 3 system 분해에서 이득을 본다고 주장 — simulative reasoning(System II)이 world model 통한 future-state prediction에 deliberation을 grounding, self-regulation(System III)이 learned configurator로 plan 시점·깊이 결정, reactive execution(System I)이 fine-grained action 처리.
SR^2AM(Self-Regulated Simulative Reasoning Agentic LLM) 개발로 검증 — LLM의 chain-of-thought 내 distinct stage로 둘 다 실현, LLM을 world model로. v0.1(prompted multi-module 기록)·v1.0(pretrained reasoning LLM trace 재구성·SFT 후 RL) 두 instantiation 탐색.

핵심 아이디어

효율적 agentic reasoning은 planning을 implicit emergence에 맡기지 말고 simulative reasoning·self-regulation·reactive execution 3 system으로 explicit 분해해야 하며, RL이 self-regulation을 학습시키면 planning을 "더 자주"가 아닌 "더 멀리" 하도록 학습한다 — token 효율과 정확도의 동시 최적화.

기술적 접근법

**방법론**: SR^2AM — 3 system 분해(simulative + self-regulation + reactive) agentic LLM.
**핵심 기법**: (1) Simulative reasoning(System II) — LLM을 world model로 future-state 예측, (2) Self-regulation(System III) — learned configurator가 plan 시점·깊이 결정, (3) Reactive execution(System I) — fine-grained action, (4) v0.1: prompted multi-module 결정 기록, (5) v1.0: pretrained reasoning LLM trace에서 structured plan 재구성·SFT 이후 RL.

주요 결과

수학·과학·tabular 분석·web information seeking 평가.
v0.1-8B가 120-355B parameter 시스템과 Pass@1 경쟁.
v1.0-30B가 685B-1T parameter 시스템과 Pass@1 경쟁하며 reasoning token 25.8-95.3% 절감.
RL이 평균 planning horizon 22.8% 증가, planning frequency는 2.0%만 증가 — "더 멀리" plan 학습.

의의 및 한계

**의의**: Agentic reasoning에 System I/II/III 인지심리학 framework 적용해 efficient·controllable 학습 paradigm 제시, parameter 효율(30B vs 685B-1T) 극적 개선, self-regulation이 일반 agent 학습·적응 원리로 확장 가능하다는 비전. **한계**: World model 품질이 LLM 자체에 의존, v1.0 RL의 안정·재현성, 평가 task의 다양성 한계, planning과 reactive 사이 latency overhead의 실제 응용 영향.

실용적 활용

효율적 agentic LLM·복잡 task 자동화.
Reasoning token cost·latency 절감.
자율 agent의 학습·적응 일반 framework.