EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments
Jundong Xu, Qingchuan Li, Jiaying Wu, Yihuai Lan, Shuyue Stella Li, Huichi Zhou, Bowen Jiang, Lei Wang, Jun Wang, Anh Tuan Luu, Caiming Xiong, Hae Won Park, Bryan Hooi, Zhiyuan Hu
Current LLM agent benchmarks evaluate agents against fixed environments, which doesn't reflect deployment reality where terminal configs change, software versions update, and user preferences shift. EvoArena formalizes this as sequential progressive updates across three domains and tests whether agents can track those changes correctly — current frontier agents average only 39.6% accuracy on these evolving tasks. EvoMem addresses this by storing memory as structured patch histories rather than flat snapshots, letting agents reason about *what changed* rather than just *what is*. Gains are modest: 1.5% average on EvoArena itself, with larger improvements on GAIA (+6.1%) and the conversational benchmark LoCoMo (+4.8%).
No production traction yet — the GitHub references are all arxiv-tracking digest repos, not implementations or downstream integrations. Zero citations so far. The paper is a recent preprint and the benchmark and memory mechanism don't appear to have a dedicated code release linked. Worth watching if you're building agents that need to handle environment drift, but nothing is shipping from this work yet.
Large language model (LLM) agents have achieved strong performance on a wide range of benchmarks, yet most evaluations assume static environments. In contrast, real-world deployment is inherently dynamic, requiring agents to continually align their knowledge, skills, and behavior with changing environments and updated task conditions. To address this gap, we introduce EvoArena, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains. We further propose EvoMem, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory. Experiments show that current agents struggle on EvoArena, achieving an average accuracy of 39.6% across evolving terminal, software, and social-preference domains. EvoMem consistently improves performance, yielding an average gain of 1.5% on EvoArena and also improving standard benchmarks such as GAIA and LoCoMo by 6.1% and 4.8%. Beyond individual tasks, EvoMem further improves chain-level accuracy by 3.7% on EvoArena, where success requires completing a consecutive sequence of related evolutionary subtasks. Mechanistic analysis shows that EvoMem improves evidence capture in the memory, indicating better preservation of complete evolving environment states. Our results highlight the importance of modeling evolution in both evaluation and memory for reliable agent deployment.