GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation

Video world models can generate realistic futures from a single instruction, but they often fail to preserve consistent point-level motion over time. As a result, the generated videos appear plausible, yet lack the physical grounding required for reliable action execution, such as robot manipulation. We present GEM-4D, a geometry-grounded video world model that resolves this limitation by injecting dense 4D correspondence supervision, distilled from a pretrained geometry foundation model, into the video generative backbone during training. This supervision enables the model to jointly capture appearance and geometric structure while retaining a single-stream architecture with no additional inference cost. We further introduce an inverse dynamics module that converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-art performance on both video prediction and geometric consistency across simulation and realistic scenarios and improves real-world manipulation success from 61% to 81%. Additional results are available at the project page: https://anonymous-submission-20.github.io/gem.github.io/.

📋 한 줄 요약

**[Video World Model / 로봇 조작]** GEM-4D가 pretrained geometry foundation model의 dense 4D correspondence supervision을 video generative backbone에 distill, single-stream 유지하면서 real-world manipulation 성공률 61%→81% 향상.

🎯 핵심 기여도

Video world model이 single instruction에서 현실적 미래를 생성할 수 있지만 시간에 걸친 consistent point-level motion 보존 실패로 robot manipulation 같은 reliable action 실행에 필요한 physical grounding이 부족함을 진단.
GEM-4D — geometry-grounded video world model로 한계 해소, pretrained geometry foundation model에서 distill한 dense 4D correspondence supervision을 학습 중 video generative backbone에 주입.
Single-stream architecture 유지·추가 추론 비용 없이 appearance와 geometric structure를 jointly 캡처.
Inverse dynamics module 도입 — correspondence-consistent video rollout을 실행 가능 로봇 trajectory로 변환, simulation과 real-world에 직접 배포.

💡 핵심 아이디어

Video world model의 physical grounding 부족은 dense 4D correspondence supervision을 pretrained geometry foundation model로부터 distill해 generative backbone에 주입함으로써 single-stream·추가 추론 비용 없이 해결할 수 있으며, inverse dynamics가 video rollout과 실 로봇 control을 연결한다.

🔬 기술적 접근법

**방법론**: GEM-4D — geometry-distilled video world model + inverse dynamics.
**핵심 기법**: (1) Pretrained geometry foundation model에서 dense 4D correspondence supervision distill, (2) 학습 시 video generative backbone에 주입, (3) Single-stream architecture 유지·추가 추론 비용 없음, (4) Appearance·geometric structure 동시 캡처, (5) Inverse dynamics module로 correspondence-consistent rollout을 실행 가능 trajectory로 변환.

📊 주요 결과

Video prediction·geometric consistency에서 simulation·realistic scenario 모두 SOTA.
Real-world manipulation 성공률 61% → 81% 향상.
Single-stream으로 inference cost 증가 없음.
Simulation·real-world 모두에 직접 배포 가능.

💭 의의 및 한계

**의의**: Video world model의 physical grounding 한계를 distillation으로 우아하게 해소, geometry foundation model의 zero-cost 활용 패턴 정립, real-world manipulation 큰 폭 개선으로 실용성 입증. **한계**: 4D correspondence distill의 품질이 pretrained geometry model에 의존, single-stream 단순성과 더 복잡 dynamics 표현력 trade-off, manipulation task 외 일반화 추가 검증.

🚀 실용적 활용

로봇 조작 정책 학습의 world model.
시뮬레이션·실 환경 통합 로봇 학습.
Video generative model의 physical grounding 일반 기법.