QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.

📋 한 줄 요약

**[Multimodal LLM Agent 평가 / Social Deduction]** QUACK이 social deduction game에서 agent 발화의 ground-truth grounding을 outcome·trajectory·utterance 3 수준으로 감사, frontier VLM도 verifiable spatial claim의 15.1% hallucinate·accusation 절반 이상 unsupported.

🎯 핵심 기여도

Social deduction game이 LLM agent의 추론·기만·조정·belief modeling 탐사 testbed로 인기지만, 대부분 환경이 win rate 같은 game outcome으로만 채점·text-only interaction에 머물러 agent 언어가 실제 perceive·act한 것에 grounded되었는지 또는 failure mode를 식별하기 어려움 진단.
QUACK 도입 — multimodal social reasoning에서 agent language grounding 감사를 위한 open-source 환경·평가 프레임워크.
세 수준 평가 — game outcome·behavioral trajectory·utterance-level consistency.
핵심 Statement Verification Pipeline이 engine log에서 각 agent의 ground-truth trajectory 재구성, 모든 discussion claim을 그에 대해 검증 — spatial hallucination·unsupported accusation·deception collapse·language-action inconsistency를 자동 flag.

💡 핵심 아이디어

Social deduction agent 평가는 game outcome 만으로는 reasoning grounding을 진단할 수 없으며, multimodal 환경에서 engine log 기반 ground-truth trajectory에 대해 모든 utterance를 자동 검증하는 statement verification pipeline으로 spatial hallucination·unsupported accusation 등 구체적 failure mode를 정량 노출할 수 있다.

🔬 기술적 접근법

**방법론**: QUACK — 3-level open-source 평가 프레임워크 + Statement Verification Pipeline.
**핵심 기법**: (1) Multimodal social deduction game environment, (2) Game outcome·behavioral trajectory·utterance-level consistency 3 수준 평가, (3) Engine log에서 각 agent의 ground-truth trajectory 재구성, (4) 모든 discussion claim을 ground-truth와 비교 검증, (5) Spatial hallucination·unsupported accusation·deception collapse·language-action inconsistency 자동 flag.

📊 주요 결과

3 frontier VLM을 homogeneous·cross-model adversarial 설정에서 평가.
최강 agent도 verifiable spatial claim의 15.1% hallucinate.
Accusation의 절반 이상이 grounded evidence 없이 이뤄짐.
Full engine·평가 프레임워크·toolkit·log 공개: github.com/AAAAA-Academia-Attractions/QUACK.

💭 의의 및 한계

**의의**: Social deduction agent 평가에 grounding 검증 축 도입, multimodal social reasoning의 첫 체계 감사 프레임워크, frontier VLM의 hallucination·unsupported accusation 정량 노출로 alignment 연구에 직접 기여. **한계**: Social deduction 단일 게임 장르 중심, ground-truth trajectory 재구성은 engine log 접근 가능성 전제, 3 VLM 평가의 모델 다양성 제한.

🚀 실용적 활용

Multi-agent LLM 시스템의 honesty·grounding 평가.
VLM의 multimodal hallucination 진단·완화 연구.
게임·시뮬레이션 환경의 agent benchmarking 표준.