Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.

📋 한 줄 요약

**[Multimodal Agentic Reasoning / RL]** AXPO가 all-wrong tool-using subgroup에서 thinking prefix 고정·tool call 재샘플링으로 Thinking-Acting Gap 해결 — Qwen3-VL-Thinking 8B가 SFT+GRPO 대비 +1.8pp Pass@1·@4, 32B Base를 4× 적은 파라미터로 능가.

🎯 핵심 기여도

Extended reasoning을 가진 VLM이 복잡 문제에 성공하지만 많은 실세계 문제가 internal reasoning만으로 해결 안 되는 external tool을 요구함을 진단.
Agentic reasoning이 두 behavior — thinking(self-contained default)와 tool use(high-variance auxiliary acting) — 를 interleave하며 structural asymmetry 가짐을 "Thinking-Acting Gap"으로 명명.
표준 RL recipe인 GRPO 하에서 두 진단 증상 관찰 — tool use가 rollout의 ~30%만 시도되며, 시도 시 그룹 내 tool-using rollout이 ~40%의 question에서 all-wrong → tool call에서 필요한 learning signal 억제.
AXPO(Agent eXplorative Policy Optimization) 제안 — 각 all-wrong tool-using subgroup에 대해 thinking prefix를 고정하고 tool call·continuation을 재샘플링·uncertainty 기반 prefix selection과 결합.

💡 핵심 아이디어

Multimodal agentic reasoning의 학습 비효율은 tool-use rollout이 그룹 내 all-wrong일 때 learning signal이 사라지는 데서 비롯되며, 그런 case에서 thinking prefix를 고정한 채 tool call만 재샘플링하면 작은 보강만으로 large performance gain을 얻을 수 있다.

🔬 기술적 접근법

**방법론**: AXPO — selective resampling on all-wrong tool-using subgroups.
**핵심 기법**: (1) Thinking-Acting Gap 진단 — tool use 30% 시도·tool-using rollout 40% all-wrong, (2) All-wrong tool-using subgroup 식별, (3) Thinking prefix 고정 후 tool call·continuation 재샘플링, (4) Uncertainty-based prefix selection, (5) SFT+AXPO로 Qwen3-VL-Thinking 학습·9 multimodal benchmark·3 scale 비교.

📊 주요 결과

9 multimodal benchmark·3 scale Qwen3-VL-Thinking에서 SFT+AXPO가 SFT+GRPO를 평균 +1.8pp Pass@1·+1.8pp Pass@4(8B) 능가.
8B with SFT+AXPO가 32B Base를 Pass@4 기준 4× 적은 파라미터로 surpass.
All-wrong subgroup에 집중적 보강이 tool call의 learning signal 회복.

💭 의의 및 한계

**의의**: Multimodal agentic RL의 구조적 비효율(Thinking-Acting Gap) 정량 진단, selective resampling이라는 단순·효과적 해법, parameter efficiency 4× 입증으로 실용성. **한계**: Qwen3-VL-Thinking 중심 — 다른 VLM 일반화 추가 검증, all-wrong subgroup 정의의 threshold 의존, resampling 비용 증가.

🚀 실용적 활용

Tool-using multimodal agent의 효율적 RL 학습.
VLM agent의 inference cost 절감.
Agentic reasoning RL 일반 sample efficiency 가이드.