LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Changrui Chen, Xiuwei Zhao, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Kaichen Zhang, Wenkang Zhang, Zheng Cheng, Nansen Zhang, Chunsheng Wu, Chunjiang Ge, Zimin Ran, Dehua Song, Chunyuan Li, Shikun Feng, Ming Hu, Zhangquan Chen, Junbo Niu, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, Jiankang Deng
arXiv:2605.25979 · 2026-05-27 공개 · arXiv · PDF
vlm video-understanding temporal-grounding jump-score llava-ov-2 spatiotemporal-coordinate motion-residual codec-stream-tokenization
Abstract
We introduce LLaVA-OneVision-2 (LLaVA-OV-2), the most capable vision-language model in the LLaVA-OneVision series to date, achieving superior performance across a broad range of multimodal benchmarks. The model builds on a native OneVision-Encoder and incorporates Windowed Attention for efficient local computation while maintaining native resolution. Its key advance is codec-stream tokenization: it treats compressed video as a continuous bit-cost stream, where bit-cost dynamics determine adaptive temporal groups, and motion-residual cues select salient spatial evidence into compact visual canvases. This allocation concentrates a limited token budget on event-bearing content, enabling more stable long-video token compression than fixed groups of pictures. A shared 3D RoPE further places codec canvases, sampled frames, and images in a unified spatiotemporal coordinate system. Furthermore, we build the LLaVA-OV-2 data and training stack around large-scale open supervision: approximately 8M re-captioned video samples for pretraining, a 4M-sample spatial corpus for fine-tuning. We also introduce JumpScore, a temporal-localization benchmark targeting fine-grained grounding in high-frequency, densely repeated motion, a regime underrepresented by existing video evaluations. A standout capability of LLaVA-OV-2 is its unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-trace reasoning. On JumpScore, LLaVA-OneVision-2-8B reaches 74.9 JumpScore mAP, surpassing Qwen3-VL-8B (30.1) by +44.8 points; under matched visual-token budgets on the same benchmark, codec-stream inputs improve temporal grounding over frame sampling by +9.7 points. Across standard benchmarks, LLaVA-OneVision-2-8B further outperforms Qwen3-VL-8B by +4.3 average points on video tasks, +5.3 on spatial tasks, and +15.6 average J&F on tracking tasks.
한국어 요약
📋 한 줄 요약
**[비전-언어 모델 / Codec-Stream 토큰화]** LLaVA-OV-2가 codec-stream 토큰화·Windowed Attention·3D RoPE로 video·image·spatial·tracking 통합 — 8B 모델이 JumpScore 74.9 mAP로 Qwen3-VL-8B(30.1) +44.8 능가, 표준 video 벤치마크 +4.3 우위.
🎯 핵심 기여도
- LLaVA-OneVision-2(LLaVA-OV-2) 도입 — LLaVA-OneVision 시리즈에서 가장 capable한 VLM, 광범위 multimodal benchmark에서 superior 성능.
- Native OneVision-Encoder 기반에 Windowed Attention 통합으로 native resolution 유지하면서 효율적 local computation.
- 핵심 진보 — codec-stream 토큰화 — 압축된 video를 continuous bit-cost stream으로 다루며, bit-cost dynamics가 adaptive temporal group 결정, motion-residual cue가 salient spatial 증거를 compact visual canvas로 선택. 제한 token budget을 event-bearing content에 집중해 fixed GOP보다 안정적인 long-video 토큰 압축.
- Shared 3D RoPE로 codec canvas·sampled frame·image를 통합 spatiotemporal 좌표계에 배치. 8M re-captioned video pretraining + 4M spatial corpus fine-tuning. JumpScore — high-frequency·dense repeated motion의 fine-grained grounding 평가하는 새 temporal-localization 벤치마크 도입.
💡 핵심 아이디어
Long video VLM의 토큰 압축은 fixed GOP 대신 codec의 bit-cost dynamics에 따라 event-bearing content에 적응적으로 token budget을 할당하면 long-video 이해의 안정성과 성능을 동시에 끌어올릴 수 있으며, 3D RoPE의 통합 spatiotemporal 좌표계가 multi-modal evidence를 자연스럽게 fusion한다.
🔬 기술적 접근법
- **방법론**: Native OneVision-Encoder + codec-stream 토큰화 + Windowed Attention + 3D RoPE.
- **핵심 기법**: (1) Codec-stream 토큰화 — bit-cost dynamics 기반 adaptive temporal grouping, motion-residual로 salient spatial evidence 선택, (2) Windowed Attention으로 native resolution 유지한 효율 local computation, (3) Shared 3D RoPE로 codec canvas·sampled frame·image 통합, (4) 8M re-captioned video pretraining + 4M spatial corpus fine-tuning, (5) JumpScore 벤치마크로 high-frequency dense motion temporal localization 평가.
📊 주요 결과
- LLaVA-OV-2-8B JumpScore: 74.9 mAP (Qwen3-VL-8B 30.1 대비 +44.8 points).
- Matched visual-token budget 하 codec-stream 입력이 frame sampling 대비 temporal grounding +9.7 points.
- 표준 벤치마크 평균: video tasks +4.3, spatial tasks +5.3, tracking tasks +15.6 J&F (Qwen3-VL-8B 대비).
- Video·temporal grounding·spatial grounding·manipulation-trace reasoning 통합 인지.
💭 의의 및 한계
**의의**: 비디오 VLM의 codec-domain 토큰화라는 새 접근, long-video 이해의 안정성·성능 동시 향상, JumpScore로 fine-grained temporal localization 평가의 공백 해소, 통합 인지(video·spatial·tracking)의 generality. **한계**: Codec encoder 종속(인코딩 사전 처리 필요), 8M+4M 데이터 규모 학습 비용, JumpScore의 다른 모델 평가 일반성 추가 검증.
🚀 실용적 활용
- 장시간 video 이해·temporal grounding.
- AR/VR·로봇의 멀티모달 인지.
- 스포츠·고속 모션 영상 분석.