AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Problem
Current benchmarks for frontier models predominantly assess single-turn responses or short-horizon tasks, neglecting the complexities of long-horizon iterative processes essential in scientific and engineering domains. This paper addresses this gap by proposing AutoLab, a novel benchmark designed to evaluate ultra long-horizon closed-loop optimization tasks. The authors highlight that existing evaluations do not adequately reflect the sustained effort required for iterative improvement, which is crucial for real-world applications. This work is presented as a preprint, indicating it has not yet undergone peer review.

Method
AutoLab comprises 36 expert-curated tasks across four domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task initiates with a correct but suboptimal baseline, challenging agents to enhance it within a defined wall-clock budget. The evaluation involves 17 state-of-the-art models, including claude-opus-4.6, focusing on their ability to iteratively benchmark, edit, and incorporate feedback. The authors emphasize that the core technical contribution lies in the design of the benchmark itself, which is open-sourced, including the evaluation harness and task artifacts, to facilitate further research in long-horizon agent capabilities.

Results
The evaluation reveals that the primary determinant of success in long-horizon tasks is not the initial quality of the agent’s output but rather its persistence in iterative refinement. While claude-opus-4.6 demonstrates robust long-horizon optimization capabilities, many other frontier models, including proprietary ones, either terminate prematurely or fail to make significant progress within their budget constraints. The results indicate that models that effectively manage time and engage in persistent iteration outperform those that do not, highlighting a critical aspect of agent design for long-horizon tasks.

Limitations
The authors acknowledge that the benchmark may not encompass all possible real-world scenarios, potentially limiting its applicability. Additionally, the reliance on a strict wall-clock budget may not reflect the varying resource constraints in practical applications. The paper does not address the scalability of the benchmark tasks or the generalizability of the findings across different domains beyond those tested.

Why it matters
The introduction of AutoLab represents a significant advancement in the evaluation of autonomous agents, emphasizing the necessity of persistence and iterative learning in long-horizon tasks. This work lays the groundwork for future research aimed at developing more capable agents that can navigate complex, iterative processes in real-world applications. The open-source nature of the benchmark encourages collaboration and innovation in the field, as highlighted in the context of ongoing research efforts, as published in arXiv cs.AI.