ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Problem
This preprint addresses the gap in the capability of current peer review processes to effectively assess reproducibility in scientific research, particularly given the increasing scale and complexity of modern studies. Traditional methods of evaluation often fall short due to the intricate dependencies and methodological choices involved in experimental workflows. The authors propose a systematic approach to automate reproducibility assessment, which is crucial for maintaining scientific integrity.

Method
The core technical contribution of this work is the Agentic Reproducibility Assessment (ARA) framework, which formalizes reproducibility assessment as a structured reasoning task. ARA operates by extracting a directed workflow graph from scientific documents, linking sources, methods, experiments, and outputs. The framework employs both structural and content-based scoring mechanisms to evaluate the reconstructability of these workflows. The authors conducted experiments on a dataset of 213 ReScience C articles, which represents the largest cross-domain benchmark of human-validated computational reproducibility studies to date. ARA’s implementation leverages large language models (LLMs) with varying model temperatures to assess reproducibility across different scientific domains.

Results
ARA achieved approximately 61% accuracy across three benchmarks, demonstrating its effectiveness in reconstructing workflows and assessing reproducibility. Notably, it outperformed existing baselines on the ReproBench dataset, achieving an accuracy of 60.71% compared to the previous best of 36.84%. Similarly, on the GoldStandardDB benchmark, ARA reached an accuracy of 61.68%, surpassing the prior benchmark of 43.56%. These results indicate ARA’s potential to significantly enhance the reproducibility assessment process in peer review.

Limitations
The authors acknowledge several limitations, including the reliance on the quality of the input scientific documents and the potential biases inherent in the training data for the LLMs. Additionally, while ARA demonstrates strong performance on the selected benchmarks, its generalizability to other domains or types of scientific literature remains to be fully validated. The framework’s effectiveness in real-world peer review scenarios, where documents may vary widely in structure and clarity, is also an area for further exploration.

Why it matters
The implications of ARA are significant for the future of scientific peer review. By automating the reproducibility assessment process, ARA can alleviate the burden on human reviewers, allowing them to focus on more nuanced aspects of manuscript evaluation. This could lead to a more rigorous and scalable peer review process, ultimately enhancing the reliability of published research. Furthermore, ARA’s structured approach to workflow reconstruction may pave the way for improved methodologies in assessing reproducibility across various scientific disciplines, fostering greater transparency and trust in scientific findings.

Authors: Kevin Riehl, Andres L. Marin, Nikofors Zacharof, Fan Wu, Patrick Langer, Robert Jakob, Anastasios Kouvelas, Georgios Fontaras et al.
Source: arXiv:2605.02651
URL: https://arxiv.org/abs/2605.02651v1