Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents

Computer-use agents (CUAs) have recently made substantial progress, but deploying a separate large expert for each software domain remains expensive. Small open computer-use agents are more practical specialization targets, but they remain substantially weaker and exhibit uneven domain-specific failures. A straightforward remedy is to synthesize large-scale training data for the target domain, yet we find that this naive approach yields only marginal improvements. Building on this observation, we introduce LearnWeak, an annotation-free specialization framework for small computer-use agents that uses a stronger reference agent to identify the student's weaknesses in the target domain, synthesize targeted tasks, and construct supervision automatically. LearnWeak further introduces an error-aware specialization objective that disentangles planning and execution errors, enabling more behaviorally precise updates than broad uniform supervision. On OSWorld, LearnWeak achieves average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively, across eight domains. We also validate that our student-aware dataset generation and training approaches outperform existing autonomous trajectory generation and training baselines. Our work highlights the importance of student awareness in both data synthesis and agent training, pointing toward a more principled and efficient path for specializing small computer-use agents in diverse domains.

📋 한 줄 요약

**[Computer-Use Agent / Specialization]** LearnWeak가 stronger reference로 student 약점 식별·targeted task 합성·error-aware supervision 자동화 — OSWorld 8 도메인에서 EvoCUA-8B·OpenCUA-7B 대비 평균 +11.6·+11.1 pp 향상.

🎯 핵심 기여도

Computer-use agent(CUA)가 최근 substantial 진전을 이뤘지만 각 software domain마다 별도의 large expert를 배포하는 것이 비싸다는 한계 진단.
Small open computer-use agent가 더 practical specialization target이지만 substantially 약하고 uneven domain-specific failure 보임을 지적.
Straightforward remedy로 target domain의 대규모 학습 데이터 합성을 시도했지만 marginal improvement만 얻는다는 관찰 — 이 위에 LearnWeak 제안.
Annotation-free specialization framework — stronger reference agent로 student의 weakness in target domain을 식별, targeted task 합성, supervision 자동 구성; error-aware specialization objective로 planning·execution error를 disentangle해 broad uniform supervision보다 behaviorally precise update 가능.

💡 핵심 아이디어

Small CUA의 specialization은 대규모 데이터 합성보다 student의 weakness에 targeted된 학습 데이터·supervision이 결정적이며, planning vs execution error를 disentangle해 error-aware objective로 학습하면 student-aware 보강이 자동·annotation-free로 가능하다.

🔬 기술적 접근법

**방법론**: LearnWeak — student-aware data synthesis + error-aware supervision.
**핵심 기법**: (1) Stronger reference agent로 student의 target domain weakness 식별, (2) Targeted task 자동 합성, (3) Annotation-free supervision 자동 구성, (4) Error-aware specialization objective — planning·execution error disentangle, (5) 8 domain·OSWorld 평가로 baseline 비교.

📊 주요 결과

OSWorld 8 도메인 평균 +11.6 pp (vs EvoCUA-8B) 및 +11.1 pp (vs OpenCUA-7B) 향상.
Student-aware dataset generation·training이 기존 autonomous trajectory generation·training baseline outperform.
Naïve large-scale 합성 대비 LearnWeak의 우월성 입증.

💭 의의 및 한계

**의의**: Small CUA specialization의 student-awareness 중요성을 정량 입증, annotation-free 자동 파이프라인의 실용성, planning·execution error disentangle 원리 제시. **한계**: OSWorld 중심 평가 — 다른 환경 일반화 추가 검증, stronger reference agent 가용성 의존, error attribution의 정확도 의존성.

🚀 실용적 활용

다양 software domain의 small CUA 배포.
Annotation 부담 없는 agent specialization 파이프라인.
오픈소스 agent 모델의 도메인 적응.