LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose LatentOmni, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct LatentOmni-Instruct-35K, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

📋 한 줄 요약

**[Omni-Modal Reasoning / Latent CoT]** LatentOmni가 textual reasoning과 audio-visual latent state를 interleave해 temporal grounding 보존, OSPE로 cross-modal 시간 일관성 유지·LatentOmni-Instruct-35K로 학습, open-source 최우수.

🎯 핵심 기여도

오디오-비주얼 joint reasoning이 omnimodal understanding의 핵심이지만 현 MLLM이 두 modality 모두로부터 fine-grained evidence를 요구하는 추론에서 struggle함을 진단.
핵심 한계 — explicit text-based chain-of-thought(CoT)가 continuous audio-visual signal을 discrete token으로 압축하면서 temporal grounding을 약화시키고 intermediate reasoning을 language prior로 shift함.
Unified latent space가 더 나은 reasoning medium임을 주장 — dense sensory information 보존하면서 autoregressive generation과 호환.
LatentOmni 제안 — cross-modal reasoning framework로 textual reasoning과 audio-visual latent state를 interleave, feature-level supervision으로 latent reasoning state를 task-relevant sensory feature와 align하고 Omni-Sync Position Embedding(OSPE)으로 latent audio·visual state 간 temporal consistency 유지.

💡 핵심 아이디어

Omni-modal reasoning은 discrete text token으로 압축하는 explicit CoT가 아니라 audio-visual latent state를 reasoning chain에 interleave해 dense sensory 신호를 보존하면서 자기회귀 생성과 호환시켜야 하며, OSPE의 temporal alignment·feature-level supervision이 이를 안정 학습 가능하게 한다.

🔬 기술적 접근법

**방법론**: LatentOmni — latent-state interleaved cross-modal reasoning.
**핵심 기법**: (1) Textual reasoning과 audio-visual latent state interleave, (2) Feature-level supervision으로 latent reasoning state를 task-relevant sensory feature와 align, (3) Omni-Sync Position Embedding(OSPE)로 latent audio·visual state 간 temporal consistency, (4) LatentOmni-Instruct-35K — audio-visual interleaved reasoning trajectory 데이터셋 구축, (5) Multiple audio-visual reasoning 벤치마크에서 explicit text CoT baseline 대비 평가.

📊 주요 결과

다수 audio-visual reasoning 벤치마크에서 평가된 open-source 모델 중 최우수.
Explicit Text CoT baseline을 consistently outperform.
Latent-space joint reasoning이 stronger omnimodal understanding의 promising path임을 검증.

💭 의의 및 한계

**의의**: Audio-visual reasoning의 representation 패러다임을 text CoT에서 latent CoT로 전환, temporal grounding 보존의 메커니즘적 해법(OSPE), latent reasoning trajectory dataset 공개로 후속 연구 가속. **한계**: 35K 데이터셋 규모의 일반화 추가 검증, latent state 해석성 trade-off, audio-visual 외 추가 modality 확장은 별도 작업.

🚀 실용적 활용

Video QA·multimodal assistant.
Audio-visual content understanding.
Multimodal CoT의 latent paradigm 일반 설계.