Toward Native Multimodal Modeling: A Roadmap

Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling.

📋 한 줄 요약

**[Native Multimodal Modeling]** NMM 로드맵 — late-fusion에서 mid/early-fusion 기반 native architecture로의 전환 형식화, Multi-to-Text·Multi-to-Target·Multi-to-Multi의 3 카테고리로 input-output duality 통합, end-to-end industrial 파이프라인 정리.

🎯 핵심 기여도

Multimodal modeling이 modality-agnostic reasoning에서 world modeling으로의 vital 단계임을 강조.
초기 접근이 주로 late-fusion(encoder + frozen language backbone + output head 조립)에 의존했지만, 최근 native multimodal modeling(NMM)으로의 patch shift 진행 — modality의 intrinsic integration으로 superior 성능 달성.
Native architecture의 design space가 insufficiently defined된 상황에서 community에 formalized roadmap 제시.
Architectural nativity를 formal하게 정의 — mid-fusion·early-fusion을 non-native paradigm과 구분; input-output duality lens로 기존 native model을 3 카테고리로 조직 — (i) Multi-to-Text(cross-modal comprehension with text-only output), (ii) Multi-to-Target(scenario-oriented generation; image/audio/video), (iii) Multi-to-Multi(symmetric input-output unified modeling).

💡 핵심 아이디어

NMM의 정의적 프레임워크는 architectural nativity의 형식 정의(mid/early-fusion vs non-native)와 input-output duality 기반 3 카테고리 분류이며, 산업 관점에서 architectural coordination부터 inference·deployment·evaluation까지 end-to-end 파이프라인을 통합해야 진정한 unified transformer paradigm으로의 전환이 가능하다.

🔬 기술적 접근법

**방법론**: NMM 로드맵 — architectural nativity formal 정의 + 3 카테고리 분류 + 산업 파이프라인.
**핵심 기법**: (1) Mid-fusion·early-fusion vs non-native(late-fusion) 구분 정의, (2) Multi-to-Text — cross-modal comprehension with text-only output, (3) Multi-to-Target — image/audio/video generation, (4) Multi-to-Multi — symmetric input-output unified modeling, (5) 산업 perspective의 end-to-end pipeline — architectural coordination·massive data curation·full-stack training recipe·inference & deployment·comprehensive evaluation.

📊 주요 결과

Native vs non-native paradigm의 명확 구분 제시.
3 카테고리로 기존 native model 체계 조직.
산업급 investigation으로 end-to-end pipeline 정리.
Understanding·generation이 unified transformer paradigm 안에 seamless 공존하는 definitive NMM framework 비전 제시.

💭 의의 및 한계

**의의**: NMM 분야의 formal 정의·분류 표준 제공, 산업 관점의 end-to-end 가이드, 학계·산업 양쪽에 reference framework로 기능. **한계**: Position·survey 성격으로 새 실험 결과는 제한적, taxonomy 분류 경계가 빠르게 진화하는 모델에서 흐려질 수 있음, definitive NMM framework 자체는 미래 작업.

🚀 실용적 활용

Multimodal model 아키텍처 설계 가이드.
NMM 연구의 분류·비교 표준.
산업 LLM·VLM·omni-model 개발 로드맵.