MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

Memory is essential for enabling large language models to support long-horizon reasoning, yet existing memory systems remain unreliable and difficult to debug. Tracing memory's dynamic evolution is crucial to understand how information is synthesized, propagated, or corrupted over time. In this work, we study the new problem of error tracing and attribution in LLM memory systems. We propose a novel framework that transforms memory pipelines into executable memory evolution graphs, enabling fine-grained tracing of operational information flow. We then construct MemTraceBench, a benchmark collected from representative memory systems such as Long-Context, RAG, Mem0, and EverMemOS, to systematically study memory failure modes. We further introduce an automatic attribution method that iteratively traces operation subgraphs to pinpoint the root cause of any failed case. Our analysis reveals that memory failures are systematic, stemming from operation-level issues like information loss and retrieval misalignment. Crucially, we leverage these fine-grained attribution signals to guide downstream prompt optimization, establishing a closed-loop system that automatically corrects faults and boosts end-task performance by up to 7.62%. Code will be released at https://github.com/zjunlp/MemTrace.

📋 한 줄 요약

**[LLM Memory / Error Attribution]** MemTrace가 memory pipeline을 executable memory evolution graph로 변환·error attribution 가능 — MemTraceBench로 Long-Context·RAG·Mem0·EverMemOS 실패 모드 분석, attribution 기반 prompt 최적화로 end-task 7.62% 향상.

🎯 핵심 기여도

LLM의 long-horizon reasoning에 memory가 essential하지만 기존 memory system은 unreliable·debug 어려움을 진단.
Memory의 dynamic evolution을 tracing해 정보가 시간에 걸쳐 어떻게 synthesized·propagated·corrupted되는지 이해 필요성 제기.
LLM memory system의 error tracing·attribution이라는 새로운 문제를 연구; memory pipeline을 executable memory evolution graph로 변환해 operational information flow의 fine-grained tracing 가능하게 하는 novel framework 제안.
MemTraceBench 구축 — Long-Context, RAG, Mem0, EverMemOS 같은 representative memory system에서 수집한 벤치마크로 memory failure mode를 systematically 연구; iteratively operation subgraph를 trace해 failed case의 root cause를 pinpoint하는 automatic attribution 방법 도입.

💡 핵심 아이디어

LLM memory system의 unreliable behavior는 operation 수준의 information loss·retrieval misalignment로부터 systematic하게 발생하며, memory pipeline을 evolution graph로 표현·operation subgraph attribution으로 root cause를 자동 추적하면 fine-grained 신호로 prompt를 최적화해 end-task 성능까지 closed-loop으로 향상시킬 수 있다.

🔬 기술적 접근법

**방법론**: MemTrace — memory evolution graph + automatic attribution + prompt optimization.
**핵심 기법**: (1) Memory pipeline을 executable memory evolution graph로 변환, (2) Operational information flow의 fine-grained tracing, (3) MemTraceBench — Long-Context·RAG·Mem0·EverMemOS 수집, (4) Iteratively operation subgraph trace해 failed case root cause pinpoint, (5) Attribution 신호로 downstream prompt 최적화·closed-loop fault correction.

📊 주요 결과

Memory failure가 information loss·retrieval misalignment 같은 operation-level 이슈에서 systematic하게 발생함을 분석.
Fine-grained attribution signal로 downstream prompt 최적화 유도.
Closed-loop 시스템이 자동 fault correction·end-task 성능 최대 7.62% 향상.
Code 공개 예정: github.com/zjunlp/MemTrace.

💭 의의 및 한계

**의의**: LLM memory의 debug·attribution이라는 새 문제 정식화, 4 대표 memory system 통합 벤치마크 제공, automatic attribution으로 end-task까지 closed-loop 향상의 실용성. **한계**: Memory evolution graph 표현의 매우 복잡한 시스템 확장성, attribution이 operation-level signal에 의존, 7.62% 향상이 task·system 의존적일 가능성.

🚀 실용적 활용

LLM agent의 memory system debug·개선.
RAG·Mem0·EverMemOS 등 memory framework 평가.
Long-horizon reasoning system의 reliability 향상.