Unsupervised Process Reward Models

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

📋 한 줄 요약

**[Process Reward Model / Unsupervised]** uPRM이 LLM next-token 확률에서 첫 erroneous step의 batch-wise scoring function 정의로 step·answer 라벨 없이 학습 — ProcessBench에서 LLM-as-Judge 대비 +15%, supervised PRM에 comparable.

🎯 핵심 기여도

PRM(Process Reward Model)이 LLM 추론을 step-level fine-grained supervision으로 steer하는 강력한 메커니즘이지만 모든 reasoning step에 expert annotation을 요구해 비싸고 scale 어려움을 진단.
uPRM 제안 — human supervision(step annotation·final answer ground truth) 없이 학습하는 unsupervised PRM 방법.
핵심 아이디어: LLM next-token 확률에서 derive한 scoring function이 batch of reasoning trajectory 전반의 first erroneous step 후보 position을 jointly 평가.
3 시나리오에서 효과 입증 — (1) ProcessBench에서 first erroneous step 식별 LLM-as-Judge 대비 최대 15% absolute 향상, (2) test-time scaling verifier로 supervised PRM에 comparable·majority voting 대비 +6.9%, (3) RL reward signal로 사용 시 ground-truth supervised PRM 대비 robust policy 최적화.

💡 핵심 아이디어

PRM은 expert annotation 없이도 LLM 자체의 next-token 확률을 batch-wise jointly 분석해 first erroneous step을 식별할 수 있으며, 이로부터 도출한 unsupervised reward signal이 supervised PRM에 comparable한 verifier·더 robust한 RL reward를 제공한다.

🔬 기술적 접근법

**방법론**: uPRM — unsupervised PRM training.
**핵심 기법**: (1) LLM next-token probability를 활용한 scoring function 정의, (2) Batch of reasoning trajectory 전반의 first erroneous step 후보 jointly 평가, (3) Step-level annotation·final answer 라벨 모두 불필요, (4) uPRM을 step identifier·test-time verifier·RL reward signal로 3중 평가, (5) ProcessBench·test-time scaling·RL policy 학습에서 baseline 비교.

📊 주요 결과

ProcessBench에서 first erroneous step identification — LLM-as-Judge 대비 최대 15% absolute accuracy 향상.
Test-time scaling verifier — supervised PRM에 comparable, majority voting 대비 최대 6.9% outperform.
RL reward signal로 사용 시 ground-truth label로 학습한 supervised PRM보다 robust policy optimization.
Scalable reward modeling for complex reasoning task로의 경로 제시.

💭 의의 및 한계

**의의**: PRM의 annotation bottleneck 제거로 scaling 가능, LLM 자체 확률을 활용한 self-supervised reward의 일반 패턴 정립, supervised PRM에 comparable·일부 환경에서 우수. **한계**: LLM의 next-token 확률 품질에 의존, 매우 복잡 reasoning에서 erroneous step 정의의 ambiguity, batch-wise jointly evaluation의 batch composition 의존성.

🚀 실용적 활용

LLM 수학·코드 추론 학습 강화.
Step-level reward 없이 RLHF·RLVR 확장.
Test-time scaling의 저비용 verifier.