FutureSim: Replaying World Events to Evaluate Adaptive Agents

Problem
This paper addresses the gap in evaluating AI agents’ adaptability in dynamic environments, particularly in the context of real-world events. Existing benchmarks often fail to simulate the complexities of open-ended scenarios where agents must adapt to new information as it arrives. The authors propose FutureSim, a grounded simulation framework that replays actual world events in chronological order, allowing for a more realistic assessment of agents’ predictive capabilities. This work is presented as a preprint and has not yet undergone peer review.

Method
FutureSim is designed to evaluate agents’ performance in forecasting world events beyond their knowledge cutoff. The framework utilizes a chronological replay of real-world events, specifically news articles and question resolutions, over a three-month period (January to March 2026). The evaluation involves frontier AI agents tested in their native harness, focusing on their ability to predict future events based on past information. The authors employ a Brier score to quantify prediction accuracy, with a specific emphasis on long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. The paper includes careful ablation studies to validate the effectiveness of the FutureSim framework.

Results
The evaluation of various AI agents within FutureSim reveals significant disparities in predictive performance. The best-performing agent achieves an accuracy of only 25%, while many agents exhibit a Brier skill score worse than random guessing, indicating a failure to outperform a baseline of no prediction. These results highlight the challenges faced by current AI systems in adapting to new information over extended time horizons, underscoring the need for improved methodologies in this domain.

Limitations
The authors acknowledge several limitations, including the potential for overfitting to the specific events chosen for the simulation and the limited scope of the three-month evaluation period. They also note that the framework may not capture all aspects of real-world complexity, such as the influence of socio-political factors on event outcomes. Additionally, the reliance on historical data may not fully account for unforeseen future developments. An obvious limitation not discussed is the potential bias introduced by the selection of news articles, which may not represent a comprehensive view of all relevant events.

Why it matters
The implications of this work are significant for the field of AI, particularly in the development of agents capable of long-term adaptation in dynamic environments. FutureSim provides a novel benchmark that can facilitate the study of emerging research directions, such as improving agents’ reasoning under uncertainty and enhancing their memory and search capabilities. By establishing a realistic framework for evaluating adaptability, this work paves the way for future advancements in AI systems that can better navigate the complexities of real-world scenarios.

Authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping
Source: arXiv:2605.15188
URL: https://arxiv.org/abs/2605.15188v1