GEM: Generative Supervision Helps Embodied Intelligence

Embodied Vision-Language Models (VLMs) have demonstrated impressive performance and generalization in robotics, particularly within Vision-Language-Action frameworks. However, a significant gap remains between the high-level semantic focus of standard text-guided pre-training paradigms and the low-level spatial and physical knowledge critical for execution in embodied environments. In this paper, we introduce GEM, a Generative-supervised Embodied vision-language Model designed to bridge this divide. We propose integrating a depth map generation task directly into the VLM pre-training phase. By training this generative objective jointly with the main model, we observe substantial improvements in embodied intelligence, significantly enhancing both semantic understanding and physical operation capabilities. To support this paradigm, we curate and release GEM-4M, a comprehensive large-scale dataset featuring a mixture of grounding, reasoning, and planning data paired with high-quality depth supervision. Extensive experiments demonstrate that GEM achieves state-of-the-art results across diverse embodied benchmarks. Furthermore, our deployed action model, GEM-VLA, exhibits vastly superior task execution abilities in both simulation environments and real-world evaluations. Code, models, and datasets are available at https://zhaorw02.github.io/GEM/

📋 한 줄 요약

**[Embodied VLM / Depth Generative Supervision]** GEM이 depth map generation을 VLM pre-training에 통합해 semantic·physical gap 해소 — GEM-4M 데이터셋·다양 embodied benchmark SOTA·실세계 robot action에서 superior task execution.

🎯 핵심 기여도

Embodied VLM이 robotics에서 impressive 성능·일반화를 보이며 특히 Vision-Language-Action(VLA) framework 내에서 그러함을 배경 소개.
그러나 표준 text-guided pre-training paradigm의 high-level semantic focus와 embodied 환경 execution에 critical한 low-level spatial·physical knowledge 사이의 significant gap이 남아있음을 진단.
GEM(Generative-supervised Embodied vision-language Model) 도입 — 이 divide를 bridge.
Depth map generation task를 VLM pre-training phase에 직접 통합 — generative objective를 main model과 jointly 학습하면 embodied intelligence가 substantial 향상되어 semantic understanding과 physical operation capability를 모두 강화.

💡 핵심 아이디어

Text-guided pre-training의 high-level semantic focus는 embodied execution에 필요한 low-level spatial·physical 지식과 격차가 있으며, depth map generation을 generative supervision으로 VLM pre-training에 통합하면 동일 모델에서 semantic·physical 능력을 동시 강화할 수 있다.

🔬 기술적 접근법

**방법론**: GEM — depth map generation 통합 VLM pre-training.
**핵심 기법**: (1) Depth map generation task를 VLM pre-training phase에 직접 통합, (2) Generative objective와 main model jointly training, (3) GEM-4M — grounding·reasoning·planning 데이터·고품질 depth supervision 결합 대규모 데이터셋 curate·release, (4) GEM-VLA — embodied action model로 deployment, (5) Simulation·real-world 양쪽 환경 평가.

📊 주요 결과

다양 embodied benchmark에서 GEM이 state-of-the-art 달성.
GEM-VLA의 task execution 능력 simulation·real-world 평가 모두에서 vastly superior.
GEM-4M 데이터셋 — grounding·reasoning·planning · 고품질 depth 통합.
Code·model·dataset: zhaorw02.github.io/GEM/.

💭 의의 및 한계

**의의**: Embodied VLM의 semantic-physical gap을 generative supervision으로 동시 해소, 대규모 embodied 데이터셋 GEM-4M release로 후속 연구 가속, sim·real 양쪽에서 실용 검증. **한계**: Depth 외 modality(force·tactile) 미커버, GEM-4M curation 비용·재현성, real-world 평가의 환경 다양성 한계.

🚀 실용적 활용

가정·산업 로봇의 vision-language-action 정책.
Manipulation·navigation 임무 수행 VLM 학습.
Embodied AI 일반의 multimodal pre-training 레시피.