What We Learned by Reproducing 2,200 papers from ICML

Problem — This work addresses the gap in understanding the reproducibility of machine learning research, specifically focusing on the challenges faced when attempting to replicate findings from 2,200 papers presented at ICML. The authors note that reproducibility is a critical issue in the field, yet there is limited empirical data on the success rates of reproducing published results. This paper is a preprint and has not undergone peer review, which may affect the reliability of its findings.

Method — The authors employed a systematic approach to reproduce the results of 2,200 papers from ICML. They categorized the papers based on various factors such as the type of experiments conducted, the datasets used, and the reported results. The methodology included analyzing the availability of code and data, as well as the clarity of the experimental setup described in the papers. Specific metrics for success in reproduction were defined, although detailed architectural or computational specifics were not disclosed in the available text.

Results — The available text does not report quantitative results regarding the success rates of reproducing the papers or comparisons against specific baselines. However, the authors provide qualitative insights into common barriers encountered during the reproduction process, such as insufficient documentation and lack of access to code or datasets.

Limitations — The authors acknowledge that their findings are limited by the scope of the papers selected for reproduction and the inherent challenges in reproducing complex machine learning experiments. They also note that the lack of standardized reporting practices in the original papers complicates the reproduction efforts. Additionally, the study’s preprint status means that the findings have not yet been validated through peer review.

Why it matters — This work underscores the importance of reproducibility in machine learning research and highlights the need for improved practices in reporting experimental results. The insights gained can inform future research directions and encourage the adoption of more rigorous standards for reproducibility in the field, as published in Hugging Face Blog.