EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation

The rapid evolution of generative video foundation models has propelled the field toward professional-grade cinematic synthesis. To achieve such demanding quality, the community transitions towards Reinforcement Learning (RL) and agentic workflows. However, reliable evaluation has emerged as a critical bottleneck. Existing benchmarks predominantly evaluate ''whether it is right'' (basic prompt-following) while fundamentally neglecting ''whether it is good'' (cinematic quality, acting, and aesthetics). Furthermore, current automated metrics lack the domain-specific rigor required to provide trustworthy signals, creating a severe credibility gap between human aesthetic perception and machine scoring. To bridge this gap, we introduce EvalVerse, a comprehensive, pipeline-aware, and expert-calibrated evaluation framework. We treat video generation assessment not merely as an engineering task, but as a core scientific problem: the systematic digitization of subjective cinematic expertise. First, we organize domain knowledge into an evaluation taxonomy aligned with the professional filmmaking workflow (pre-production, production, and post-production). Second, we distill human expert judgments into a curated dataset with large-scale human annotations. Third, we inject this knowledge into Vision-Language Models (VLMs) through an expert-calibrated fine-tuning strategy, enabling the VLM to perform explicit Chain-of-Thought reasoning. Compared to previous works, EvalVerse not only retains compatibility with foundational ''rightness'' metrics, but also significantly expands the criteria to ''goodness'' and broaden the task coverage to complex multi-shot sequencing and audio-visual integration. Consequently, by providing granular diagnostic signals, EvalVerse transcends a static leaderboard and establishes a fundamental infrastructure for future work, such as reward models and evaluator agent.

📋 한 줄 요약

**[Cinematic Video 평가 / Expert-Calibrated]** EvalVerse가 영화 제작 워크플로우 정렬 taxonomy·expert annotation·CoT 가능 VLM fine-tuning으로 "rightness" 너머 "goodness" 평가 — multi-shot sequencing·audio-visual integration까지 커버, 미래 reward·evaluator agent 인프라.

🎯 핵심 기여도

생성 video foundation model의 rapid evolution으로 분야가 professional-grade cinematic synthesis 방향으로 가속, RL·agentic workflow로 전환 중이지만, 신뢰성 있는 평가가 critical 병목으로 부상함을 진단.
기존 벤치마크가 주로 "whether it is right"(basic prompt-following)만 평가하고 "whether it is good"(cinematic quality·acting·aesthetic) 평가는 fundamental 결여.
현재 자동화 메트릭이 domain-specific rigor 부족해 신뢰성 있는 signal 미제공, human aesthetic perception과 machine scoring 간 심각한 credibility gap 형성.
EvalVerse 도입 — comprehensive·pipeline-aware·expert-calibrated 평가 프레임워크. 평가를 engineering이 아닌 core scientific 문제로 다룸 — subjective cinematic expertise의 systematic digitization.

💡 핵심 아이디어

Cinematic video 평가는 단순 prompt 추종이 아니라 영화 제작 (pre/production/post-production) 워크플로우에 정렬된 taxonomy 위에 expert judgment를 distillation해 VLM에 expert-calibrated fine-tuning으로 주입해야 하며, 명시적 CoT reasoning으로 "goodness"까지 평가 가능하게 만들어야 한다.

🔬 기술적 접근법

**방법론**: EvalVerse — 3-step pipeline (taxonomy + expert 데이터 + CoT VLM).
**핵심 기법**: (1) Professional 영화 제작 워크플로우(pre/production/post-production) 정렬 evaluation taxonomy 조직, (2) Human expert judgment를 distill해 large-scale human annotation 큐레이션 데이터셋 구성, (3) Expert-calibrated fine-tuning으로 VLM에 도메인 지식 주입해 explicit CoT reasoning 가능, (4) "Rightness" 메트릭 호환성 유지하면서 "goodness" 기준 확장, (5) Multi-shot sequencing·audio-visual integration까지 task coverage.

📊 주요 결과

영화 제작 워크플로우 정렬 evaluation taxonomy 정립.
Expert annotation 기반 large-scale 데이터셋 큐레이션.
Expert-calibrated VLM이 explicit CoT reasoning으로 cinematic 품질 평가.
"Rightness" 호환 + "goodness" 평가 + multi-shot·audio-visual 통합 커버리지.
Static leaderboard 너머 reward model·evaluator agent 인프라로 기능.

💭 의의 및 한계

**의의**: Video 생성 평가의 패러다임을 "right"에서 "good"으로 확장, expert knowledge digitization을 과학적 문제로 정립, multi-shot·audio-visual 통합 커버리지로 응용 폭, 미래 RL reward·evaluator agent 인프라 제공. **한계**: Expert annotation의 cost·subjective bias, 도메인 특화로 다른 video 도메인(과학·교육) 일반화 추가 검증, VLM fine-tuning의 evaluator 신뢰성 보장 한계.

🚀 실용적 활용

영화·광고·콘텐츠 산업의 video 생성 모델 평가.
RL 학습용 reward model 구축.
Evaluator agent의 신뢰성 있는 평가 시스템.