The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers

Problem — The paper addresses the gap in understanding how documentation practices in AI research have evolved over the past decade, particularly in light of the reproducibility crisis. It highlights the lack of comprehensive studies on the impact of reproducibility checklists and the overall trend toward open science. This work is a preprint and has not undergone peer review.

Method — The authors conducted a systematic analysis of 56,800 papers published at five leading AI conferences from 2014 to 2024. They identified seven reproducibility variables, which were quality-assured to evaluate documentation practices. The analysis focused on the sharing of code and data, as well as the inferred reproducibility rates based on these practices. The authors compared their findings to empirical reproducibility rates from previous studies to estimate trends over time.

Results — The study found a significant increase in documentation practices, with the percentage of papers sharing both code and data rising from 11% in 2014 to 64% in 2024. Inferred reproducibility rates also improved, estimated to have increased from 28% in 2014 to 64% in 2024. These results suggest a strong positive correlation between improved documentation and reproducibility, indicating a shift in the AI research community’s approach to open science.

Limitations — The authors acknowledge that their estimates of reproducibility are based on documentation practices rather than direct empirical testing, which may not fully capture the actual reproducibility of the research. Additionally, while the study covers a substantial dataset, it is limited to five specific conferences, which may not represent the entire AI research landscape. The authors do not address potential biases in the selection of papers or the reproducibility variables chosen for analysis.

Why it matters — This work provides valuable insights into the evolving landscape of AI research documentation and reproducibility, highlighting a significant shift toward open science practices. The findings suggest that the AI community is increasingly prioritizing transparency and reproducibility, which could enhance the credibility and reliability of AI research. This trend is crucial for future developments in the field, as it may lead to more robust and replicable findings, ultimately benefiting both researchers and practitioners. The implications of this study are particularly relevant for ongoing discussions about reproducibility in AI, as published in arXiv cs.AI.