RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains

Pointwise reward modeling offers critical signals for LLM post-training, yet struggles with absolute scoring in subjective, non-verifiable settings. Rubric-based methods address this by decomposing evaluation into explicit criteria, but existing approaches typically depend on frontier LLMs and suffer from ties caused by hard Boolean aggregation. We present RUBRIC-ARROW, an alternating framework that jointly trains a rubric generator and a rubric-conditioned judge, with its RL stage using only pairwise preference data. Our method couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and an alternating GRPO scheme that together train the pointwise evaluator. Extensive experiments show that RUBRIC-ARROW achieves competitive reward-modeling accuracy and yields consistent gains for downstream policy post-training.

📋 한 줄 요약

**[Reward Modeling / Rubric]** RUBRIC-ARROW가 pairwise preference만으로 rubric generator·rubric-conditioned judge를 alternating 학습 — probability scoring + phase-specific reward + alternating GRPO로 tie 감소·non-verifiable 도메인 reward 정확도·downstream policy 향상.

🎯 핵심 기여도

Pointwise reward modeling이 LLM post-training에 critical signal을 제공하지만 subjective·non-verifiable 환경의 absolute scoring에 struggle함을 진단.
Rubric 기반 방법이 평가를 explicit criteria로 decompose해 이를 해결하지만 기존 접근이 frontier LLM에 의존하고 hard Boolean aggregation으로 인한 tie를 겪음.
RUBRIC-ARROW 제안 — alternating framework로 rubric generator와 rubric-conditioned judge를 jointly 학습, RL stage가 pairwise preference data만 사용.
본 방법은 tie를 줄이는 probability-based scoring rule + phase-specific preference-based reward + pointwise evaluator를 함께 학습하는 alternating GRPO 결합.

💡 핵심 아이디어

Non-verifiable 도메인 reward modeling에서 rubric 기반 평가의 hard Boolean aggregation tie 문제는 probability-based scoring과 phase-specific reward로 완화 가능하며, pairwise preference만으로 rubric generator·judge를 alternating GRPO로 jointly 학습하면 frontier LLM 의존 없이도 SOTA 수준 reward signal 확보 가능.

🔬 기술적 접근법

**방법론**: RUBRIC-ARROW — alternating rubric generator + rubric-conditioned judge.
**핵심 기법**: (1) Rubric generator와 rubric-conditioned judge의 joint alternating 학습, (2) RL stage가 pairwise preference data만 사용, (3) Probability-based scoring rule로 hard Boolean aggregation의 tie 감소, (4) Phase-specific preference-based reward로 단계별 학습 안정화, (5) Alternating GRPO scheme으로 pointwise evaluator 학습.

📊 주요 결과

Reward-modeling accuracy가 경쟁 수준 달성.
Downstream policy post-training에서 일관 gain.
Pointwise scoring의 tie 비율 감소.
Pairwise preference data만으로 frontier LLM 의존 없이 학습.

💭 의의 및 한계

**의의**: Non-verifiable 도메인 reward modeling의 실용 framework 제공, rubric의 hard Boolean aggregation 한계 정식 해결, frontier LLM 의존 없이 경쟁력 있는 pointwise reward 학습. **한계**: Rubric generator 품질이 전체 성능 좌우, alternating 학습의 hyperparameter·수렴 안정성, pairwise preference data 확보 비용.

🚀 실용적 활용

Subjective·non-verifiable 도메인 LLM post-training.
Open-ended writing·creative·assistant 응답 reward modeling.
Rubric 기반 평가의 자동화.