VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions

Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.

📋 한 줄 요약

**[Personalized Agent 평가]** VitaBench 2.0이 temporally ordered 사용자 시퀀스로 fragmented preference·proactiveness 평가, frontier 모델에도 real-world personalization은 high challenge로 substantial gap 노출.

🎯 핵심 기여도

LLM이 실세계 task에서 사용자와 협업하는 interactive agent로 진화하면서 효과적 협업이 명시되지 않은 사용자 이해에 점차 의존 — user intent가 fragmented daily interaction에 반영되며 personalized modeling과 proactive interaction 모두 요구한다는 배경 진단.
기존 agent 벤치마크가 주로 reasoning·tool use를 평가, realistic scenario에서의 user preference 추론·활용 도전을 largely overlook함 지적.
VitaBench 2.0 도입 — long-term user interaction에서 personalized·proactive agent 행동 평가 벤치마크.
Task가 individual user 별 temporally ordered sequence로 조직, preference가 fragmented heterogeneous interaction에 embedded — 성공적 task 완료에 agent가 이들로부터 continuously preference 추출·활용·업데이트 필요.

💡 핵심 아이디어

Real-world personalized agent 평가는 fragmented temporal interaction에서 preference를 추출·업데이트하고, 누락 정보를 적극 acquire하는 proactiveness까지 함께 평가해야 하며, 이를 위한 temporally ordered user sequence·extensible memory interface가 시스템 진단의 핵심 도구다.

🔬 기술적 접근법

**방법론**: VitaBench 2.0 — long-term personalized·proactive agent 평가 벤치마크.
**핵심 기법**: (1) Task를 individual user 별 temporally ordered sequence로 조직, (2) Preference를 fragmented heterogeneous interaction에 embedding, (3) Agent가 continuously preference 추출·활용·업데이트 요구, (4) Proactiveness 평가 — 누락 정보 인지·user·environment에서 능동 acquire 필요 task, (5) Extensible memory interface로 다양 memory 아키텍처의 controlled comparison.

📊 주요 결과

다양 frontier proprietary·open-source LLM 벤치마크.
Real-world personalization이 SOTA 모델에도 high challenge임 입증.
Current capability와 practical requirement 간 substantial gap 노출.
Failure mode·capability bottleneck의 광범위 분석으로 향후 모델 개선 insight 제공.

💭 의의 및 한계

**의의**: Personalized·proactive agent 평가의 새 표준 정립, extensible memory interface로 memory architecture 비교 연구 가능, frontier LLM의 personalization gap 정량 노출로 alignment·memory 연구 가이드. **한계**: 시뮬레이션·합성 user 시퀀스의 실제 사용자 행동 격차, memory interface의 specific 디자인 종속성, proactiveness 평가의 metric 보편성 추가 검증.

🚀 실용적 활용

Personal assistant·CRM agent 평가·개선.
LLM agent의 memory architecture 설계 가이드.
Long-term interaction 시스템의 alignment 진단.