To Diffuse or Not to Diffuse: Matching Graph Retrieval to Memory Structure in LLM Agents

Long-term memory is important for LLM agents that need to maintain personalization, answer questions about past interactions, and stay consistent across sessions. However, existing memory systems are difficult to compare because they often change the memory representation, indexing method, retrieval algorithm, and evaluation setup at the same time. In this work, we build a unified evaluation framework for long-term memory retrieval using a shared 5W-style graph representation based on Who, What, When, Where, Why, and Extra tags. Under this controlled setup, we compare several graph-based retrieval methods, including localized ontology traversal, tag-based retrieval with vocabulary normalization and TF-IDF weighting, and AdaptiveGraph, a Personalized PageRank-based diffusion method. Experiments on LongMemEval and ATANT show that no single retrieval method works best across all memory settings. On LongMemEval-S, AdaptiveGraph performs best among graph methods, reaching up to 0.844 MRR. On ATANT Core, localized ontology traversal consistently outperforms graph diffusion, suggesting a topic dilution effect where diffusion helps in long, diverse histories but hurts precision in compact, single-topic narratives. We also find that vocabulary normalization and extraction quality have a large impact on retrieval, and that many remaining errors come from memory ingestion rather than graph ranking. These results suggest that long-term memory retrieval should be chosen based on the structure of the memory collection instead of being treated as a one-size-fits-all component.