Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement

Agentic reinforcement learning (RL) has proven effective for training LLM-based agents with external tool-use capabilities. However, we identify that agentic RL training induces increasing redundant tool calls and blurs the model's intrinsic knowledge boundary, where the model fails to distinguish when tools are needed versus when parametric knowledge suffices. Existing solutions based on reward shaping create coarse-grained optimization targets that tend to incentivize indiscriminate tool-call suppression, leading to reward hacking. In this paper, we propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary through dual-path (with-tool and no-tool) rollouts during training. We define the knowledge boundary as the per-instance determination of whether tools are required and the minimum tool calls necessary. By comparing correctness across paths, AKBE categorizes trajectories and constructs targeted supervisory signals that guide efficient tool-use patterns for each question. These signals are integrated seamlessly into the agentic RL training loop. Experiments on seven QA benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL, yielding 25% higher tool productivity without any accuracy-efficiency trade-off. Further analysis suggests its plug-and-play compatibility across different RL algorithms and the mechanism of each signal category. Our code is available at https://github.com/CuSO4-Chen/AKBE.

📋 한 줄 요약

**[Agentic RL / Knowledge Boundary]** AKBE가 dual-path(with-tool/no-tool) rollout으로 모델 intrinsic knowledge boundary 동적 probing, 7 QA 벤치마크에서 정확도 +1.85·tool call 18% 감소·tool productivity 25% 향상.

🎯 핵심 기여도

Agentic RL이 external tool-use 능력을 갖춘 LLM agent 학습에 효과적임 입증되었지만, agentic RL 학습이 redundant tool call 증가와 모델의 intrinsic knowledge boundary blur를 유발 — tool 필요 시 vs parametric knowledge 충분 시 구분 실패 진단.
기존 reward shaping 솔루션이 coarse-grained 최적화 target을 만들어 indiscriminate tool-call suppression incentivize·reward hacking 초래함 지적.
AKBE(Agentic Knowledge Boundary Enhancement) 제안 — on-policy 방법으로 학습 중 dual-path(with-tool·no-tool) rollout을 통해 모델의 intrinsic knowledge boundary 동적 probing.
Knowledge boundary를 per-instance — tool 필요 여부 결정과 필요 최소 tool call 수 — 로 정의, path 간 정확성 비교로 trajectory 분류, 각 질문마다 efficient tool-use pattern을 guide하는 targeted supervisory signal 구성 — agentic RL 학습 loop에 seamlessly 통합.

💡 핵심 아이디어

Agentic RL의 redundant tool call 문제는 coarse-grained reward shaping의 한계에서 발생하며, dual-path rollout으로 모델의 intrinsic knowledge boundary를 per-instance 동적 probing해 targeted supervisory signal을 구성하면 정확도-효율 trade-off 없이 둘 다 개선할 수 있다.

🔬 기술적 접근법

**방법론**: AKBE — Agentic Knowledge Boundary Enhancement, on-policy 방법.
**핵심 기법**: (1) 학습 중 dual-path(with-tool·no-tool) rollout 실행, (2) Knowledge boundary를 per-instance(tool 필요 여부·최소 tool call 수)로 정의, (3) Path 간 정확성 비교로 trajectory categorization, (4) 각 카테고리별 targeted supervisory signal 구성, (5) Agentic RL 학습 loop에 seamlessly 통합.

📊 주요 결과

7 QA 벤치마크에서 standard agentic RL 대비 평균 task accuracy +1.85.
Tool call 18% 감소.
Tool productivity 25% 향상.
Accuracy-efficiency trade-off 없이 양측 동시 개선.
다양 RL 알고리즘 간 plug-and-play 호환성 확인.
Code: github.com/CuSO4-Chen/AKBE.

💭 의의 및 한계

**의의**: Agentic RL의 redundant tool call 문제를 명확 진단·해결, intrinsic knowledge boundary 개념의 RL 학습 활용, plug-and-play 호환성으로 다양 agentic RL 파이프라인에 적용 가능. **한계**: Dual-path rollout으로 학습 cost 약 2배 증가 가능, 7 QA 벤치마크 중심으로 다른 agentic task 일반화는 추가 검증, knowledge boundary 정의의 task 종속성.

🚀 실용적 활용

Tool-augmented LLM agent의 효율 학습.
RAG·검색·계산기 등 tool 사용 최적화.
Production agent 시스템의 비용 절감·신뢰성 향상.