DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning

Reinforcement Learning has become a standard paradigm for aligning Large Language Models with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.

📋 한 줄 요약

**[GRPO 다중 보상]** DVAO가 rollout group 내 empirical reward variance로 combination weight를 동적 조정 — Reward·Advantage Combination의 instability·static-weight 한계 해소, bounded advantage·self-adaptive regularization, Qwen3/2.5에서 Pareto frontier 우월.

🎯 핵심 기여도

RL이 LLM을 인간 의도·task 요구에 정렬하는 표준 패러다임이 되었고, GRPO가 PPO 대비 value-model-free·효율적 대안을 제공하지만, 실세계 multi-reward 설정 적응이 challenging함을 진단.
표준 scalarization 방식의 결함 — Reward Combination은 advantage의 squared magnitude가 과도하게 커져 학습 불안정, Advantage Combination은 static hyperparameter에 의존·cross-objective correlation 무시.
DVAO 제안 — Dynamic Variance-adaptive Advantage Optimization: rollout group 내 각 objective의 empirical reward variance에 기반해 combination weight를 dynamically 조정, 강한 learning signal objective는 up-weight·noisy objective는 suppress.
수학적으로 DVAO가 bounded advantage magnitude를 유지해 안정 학습을 보장, self-adaptive cross-objective regularization 메커니즘 도입.

💡 핵심 아이디어

Multi-reward RL의 안정성·효과 trade-off는 정적 가중치가 아닌 rollout 내 empirical variance 기반 동적 가중치로 해결 가능하며, variance가 큰 objective에 더 큰 가중치를 부여하면 학습 신호가 강한 reward를 자연스럽게 따라가면서 noisy reward로 인한 instability를 억제할 수 있다.

🔬 기술적 접근법

**방법론**: DVAO — Dynamic Variance-adaptive Advantage Optimization (GRPO 확장).
**핵심 기법**: (1) Rollout group 내 각 objective의 empirical reward variance 측정, (2) Variance 기반 combination weight 동적 조정 — 강한 signal up-weight·noisy suppress, (3) Bounded advantage magnitude 이론적 보장, (4) Self-adaptive cross-objective regularization, (5) Qwen3·Qwen2.5 모델 mathematical reasoning·tool-use 벤치에서 baseline 비교.

📊 주요 결과

수학적 reasoning·tool-use 벤치마크에서 DVAO가 baseline 일관 outperform.
Superior multi-objective Pareto frontier 달성.
Robust training stability 확보.
Qwen3·Qwen2.5 모델에서 광범위 실험 검증.

💭 의의 및 한계

**의의**: GRPO의 multi-reward 확장 표준화, variance 기반 동적 가중치라는 직관·이론·실험이 정렬된 방법, Pareto frontier 향상으로 실용 가치. **한계**: Reward 개수가 매우 많거나 reward 간 강한 상관관계가 있을 때 variance 추정 안정성, hyperparameter-free 주장이지만 regularization parameter 의존성, abstract 외 구체 수치 미명시.

🚀 실용적 활용

RLHF·tool-use·multi-reward LLM 학습.
Reasoning model의 다목적 reward 통합.
Pareto-optimal LLM 정렬 일반 도구.