VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents

Authors: Yuhao Chen, Yi Xu, Xinyun Ding, Xiang Fang, Shuochen Liu, Luxi Lin, Qingyu Zhang, Ya Li, Quan Liu, Tong Xu.

Overview

Long-term agents need to remember whose preferences apply, when those preferences have changed, and which contextual conditions matter. VehicleMemBench evaluates these capabilities through executable tool interactions rather than recall questions alone.

The benchmark contains 500 queries across 50 multi-user scenarios. It covers preference conflicts, conditional constraints, coreference resolution, state shifts, and error correction. A vehicle simulator exposes 23 device modules and 111 executable APIs.

Evaluation

We execute both agent-generated and reference actions from the same initial environment state. Exact State Match measures whether the final states agree, while field- and value-level scores distinguish selecting the right attributes from setting the right values.

Comparing autonomous memory with directly supplied gold memory reveals substantial performance gaps. Simple vehicle-focused summaries remain competitive with general-purpose memory systems, motivating better memory consolidation, retrieval, and handling of context and time.

Download poster