Automated Benchmark Auditing for AI Agents and Large Language Models

Problem
This paper addresses the inadequacies in current AI benchmark evaluation methodologies, particularly for large language models (LLMs) and AI agents. Traditional verification methods struggle to keep pace with the complexity of modern benchmarks, which often contain implicit assumptions, incomplete specifications, and fragile evaluation logic. The authors present a preprint work, indicating that the findings have not yet undergone peer review.

Method
The core technical contribution is the development of the Auto Benchmark Audit (ABA) framework, which employs an agentic approach to systematically audit benchmark tasks. ABA identifies issues such as hidden dependencies, specification gaps, and grading logic flaws. The framework was applied to a dataset comprising 168 benchmarks across nine domains, including those from previous NeurIPS publications. The auditing process involves automated analysis validated by expert reviews and independent reports, ensuring high precision in identifying problematic tasks. The authors do not disclose specific details regarding the architecture or compute resources used for training the auditing agents.

Results
The application of ABA revealed critical issues in over 25.7% of the evaluated benchmarks, including ambiguous task designs, conflicts in execution environments, and incorrect ground truths. The impact of these identified issues on model performance was significant; filtering out problematic tasks led to an increase in average performance metrics on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively. These results highlight the extent to which flawed benchmarks can distort capability assessments for AI models.

Limitations
The authors acknowledge that while ABA effectively identifies numerous issues, it may not capture all potential flaws in benchmark tasks. The reliance on expert validation, while a strength, may also introduce subjectivity. Additionally, the framework’s performance across different types of benchmarks and its scalability to larger datasets remain untested. The paper does not address the potential for false positives in the auditing process or the implications of the identified issues on long-term benchmark evolution.

Why it matters
The implications of this work are significant for the future of AI benchmarking. By systematically identifying and addressing flaws in benchmark tasks, ABA can enhance the reliability of performance assessments for LLMs and AI agents. This framework not only aids in refining existing benchmarks but also sets a precedent for the development of more robust evaluation methodologies. The release of the ABA tool and task annotations will facilitate further research and development in the field, promoting higher standards in benchmark design and evaluation.

Authors: Junlin Wang, Federico Bianchi, Shang Zhu, Fan Nie, Yongchan Kwon, Bhuwan Dhingra, James Zou
Source: arXiv:2605.26079
URL: https://arxiv.org/abs/2605.26079v1