Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning

Equipping large language models with explicit skills has emerged as a promising paradigm for enabling autonomous agents to solve complex tasks. Agent skills can be inherently divided into general skills for broad cognitive transfer and task-specific skills for dynamic execution. However, existing skill-based reinforcement learning (RL) methods typically force a rigid choice between full externalization, which incurs prohibitive context overhead, and full internalization, which risks overfitting and knowledge conflicts. To address this dilemma, we propose Skill0.5, a novel agentic RL framework that explicitly differentiates skill treatments by combining general skill internalization with task-specific skill utilization. Driven by a dynamic, difficulty-aware router, Skill0.5 streams tasks into distinct mastery tiers to apply tailored optimization strategies: it internalizes general skills via privileged distillation to build a cognitive foundation for hard tasks, while using diagnostic probing on easy tasks to penalize shortcuts and enforce specific skill utilization. Experiments on ALFWorld and WebShop demonstrate that Skill0.5 outperforms both memory-based and skill-based RL baselines, yielding performance improvements across both in-distribution and out-of-distribution scenarios.

📋 한 줄 요약

**[Agentic RL / Skill Routing]** Skill0.5가 general skill 내재화 + task-specific skill 활용을 difficulty-aware router로 결합 — privileged distillation·diagnostic probing으로 hard/easy task에 다른 최적화, ALFWorld·WebShop의 ID·OOD 동시 향상.

🎯 핵심 기여도

LLM에 explicit skill을 부여하는 것이 자율 agent의 복잡 task 해결에 유망 paradigm으로 부상함을 배경 — agent skill은 broad cognitive transfer 위한 general skill과 동적 실행 위한 task-specific skill로 inherently 구분.
기존 skill-based RL이 보통 rigid 선택 강제 — full externalization(prohibitive context overhead) vs full internalization(overfitting·knowledge conflict 위험) — 의 dilemma 진단.
Skill0.5 제안 — agentic RL 프레임워크로 general skill internalization과 task-specific skill utilization 결합으로 skill treatment를 명시적으로 차별화.
Dynamic·difficulty-aware router가 task를 distinct mastery tier로 stream — hard task에는 privileged distillation으로 general skill 내재화해 인지적 토대 구축, easy task에는 diagnostic probing으로 shortcut 패널티·specific skill 활용 강제.

💡 핵심 아이디어

Agent의 skill 학습은 모든 skill을 동일하게 다루지 말고 general(내재화)과 task-specific(외부 활용)으로 분리하되, task 난이도에 따라 router로 동적 라우팅해 hard task는 privileged distillation·easy task는 diagnostic probing이라는 차별화 최적화를 적용해야 한다.

🔬 기술적 접근법

**방법론**: Skill0.5 — difficulty-aware router 기반 dual skill treatment.
**핵심 기법**: (1) Skill을 general(broad transfer)·task-specific(dynamic execution)로 분리, (2) Dynamic·difficulty-aware router가 task를 mastery tier로 stream, (3) Hard task — privileged distillation으로 general skill 내재화(cognitive foundation), (4) Easy task — diagnostic probing으로 shortcut 패널티·specific skill 활용 강제, (5) 두 skill 처리의 명시적 분리·tailored optimization.

📊 주요 결과

ALFWorld·WebShop에서 memory-based·skill-based RL baseline 모두 outperform.
In-distribution·out-of-distribution 시나리오 양쪽에서 성능 향상.
General skill 내재화로 인지 토대 구축 확인.
Diagnostic probing이 shortcut을 효과적으로 패널티함 입증.

💭 의의 및 한계

**의의**: Skill-based agentic RL의 internalization vs externalization dilemma 해소, difficulty-aware router로 동적 최적화 차별화, OOD generalization 동시 향상, ALFWorld·WebShop의 광범위 검증. **한계**: Router의 difficulty 추정 정확도 의존, privileged distillation에 teacher 필요, 더 복잡 환경(웹·로봇)으로 일반화 추가 검증.

🚀 실용적 활용

자율 agent의 skill 학습·라우팅.
Agentic RL의 OOD generalization 향상.
LLM agent의 hard·easy task 차별화 학습.