PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

The rapid growth in submissions to machine learning venues has strained the scientific peer-review system and intensified interest in LLM-based automated peer reviewers. However, how good these systems are actually, especially compared to human reviewers at catching scientific gaps, remains poorly understood. In this work, we introduce PRISM (Peer Review Intelligence via Structured Multi-dimensional assessment), a benchmarking framework that evaluates review quality across four dimensions: Depth of Analysis, Novelty Assessment,Flaw Identification & Major Issues Prioritization, and Multi-dimensional Constructiveness. Unlike most existing evaluations based on surface-level metrics like ROUGE and BLEU, or unconstrained LLM-as-a-judge prompting that conflates fluency with rigor, PRISM grounds each dimension in argument mining, retrieval-augmented verification, and consensus-based scoring. We apply PRISM to benchmark five leading automated reviewer systems and human reviewers on a stratified corpus of reviews from ICLR, ICML, and NeurIPS. The results reveal that LLMs can match or beat human reviewers on individual dimensions: comparable depth of analysis, stronger novelty verification, and highly accurate critique prioritization. However, no single system consistently matches the balanced performance of the human baseline across all dimensions at once. Each exhibits a distinct specialization profile with characteristic blind spots -- failure modes that aggregate metrics miss entirely. The implication is that LLM reviewers are best understood as targeted supplements to human review, effective within specific dimensions, but unreliable as standalone replacements. Our demo and key results can be found at https://khanhthanhdev.github.io/prism-page/.

📋 한 줄 요약

**[LLM Peer Review / Benchmark]** PRISM이 review를 4 차원(Depth·Novelty·Flaw·Constructiveness)으로 평가 — LLM이 개별 차원에서 human reviewer와 동등·우수하지만 종합 balanced 성능은 부재, 표적 supplement로 가치.

🎯 핵심 기여도

ML venue 제출 폭증으로 peer-review 시스템 strain·LLM 기반 자동 reviewer 관심 강화, 그러나 이들 시스템이 실제 얼마나 좋고 human reviewer 대비 scientific gap 포착에서 어떤지 poorly understood.
PRISM(Peer Review Intelligence via Structured Multi-dimensional assessment) 도입 — review 품질을 4 차원으로 평가하는 벤치마킹 프레임워크.
4 차원: Depth of Analysis, Novelty Assessment, Flaw Identification & Major Issues Prioritization, Multi-dimensional Constructiveness.
기존 평가가 ROUGE·BLEU 같은 surface metric 또는 fluency와 rigor를 conflate하는 unconstrained LLM-as-a-judge prompting에 의존했지만, PRISM은 argument mining·retrieval-augmented verification·consensus-based scoring으로 각 차원 ground.

💡 핵심 아이디어

LLM peer reviewer 평가는 surface metric이나 unconstrained judge prompting 한계를 넘어야 하며, argument mining·retrieval-augmented verification·consensus scoring으로 ground된 4 차원 구조 평가로 LLM이 각 차원에서 human과 비교 가능하지만 종합 balanced 성능은 부재함을 드러낸다.

🔬 기술적 접근법

**방법론**: PRISM — 4-dimensional structured assessment framework.
**핵심 기법**: (1) 4 차원 — Depth of Analysis, Novelty Assessment, Flaw Identification & Prioritization, Multi-dimensional Constructiveness, (2) Argument mining으로 각 차원 ground, (3) Retrieval-augmented verification, (4) Consensus-based scoring, (5) ICLR·ICML·NeurIPS의 stratified review corpus에 5 leading 자동 reviewer 시스템·human reviewer 적용.

📊 주요 결과

LLM이 개별 차원에서 human reviewer와 매치·beat 가능 — comparable depth of analysis, stronger novelty verification, highly accurate critique prioritization.
그러나 어떤 단일 시스템도 모든 차원에서 동시에 human baseline의 balanced 성능을 consistent 매치 못 함.
각 시스템이 distinct specialization profile과 characteristic blind spot 보유 — aggregate metric이 완전 놓치는 failure mode.
Implication: LLM reviewer는 표적 supplement로 valuable하지만 standalone replacement로는 unreliable.

💭 의의 및 한계

**의의**: LLM peer review 평가의 structured·grounded 표준 정립, LLM의 dimension-specific 강점·약점 정량화, supplement-vs-replacement 정책 가이드 제공, ICLR·ICML·NeurIPS의 실제 review로 외적 타당성. **한계**: 4 차원의 구조가 모든 review 측면 커버하지 못할 수 있음, consensus-based scoring의 평가자 의존, 5 leading 시스템 cohort의 future 시스템 일반화 추가 필요.

🚀 실용적 활용

LLM peer reviewer 시스템의 평가·진단 표준.
학회 review process의 LLM 보조 정책 설계.
AI safety의 critique·assessment 능력 평가 일반.