Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations

Problem
The paper addresses the gap in understanding the reliability and validity of public AI evaluation leaderboards, which are often perceived as definitive rankings. The authors highlight that these evaluations are shaped by selective reporting, benchmark revisions, and data missingness, leading to potential misinterpretations of AI system performance. This work is particularly relevant as it is a preprint and has not undergone peer review, indicating that the findings should be interpreted with caution.

Method
The authors propose a Bayesian inference framework to analyze public AI evaluation archives, specifically focusing on LiveBench and Open LLM Leaderboard v2. They construct a terminal-only example involving over 1,000 systems, which is compatible with two pre-terminal histories. The framework evaluates the time taken to reach performance ceilings under a terminal-tail model, yielding two distinct times: 23.03 and 75.13. The methodology includes synthetic posterior comparisons to assess action-facing diagnostics across different observation regimes. The authors also introduce an archive-and-adjudication protocol designed to reconstruct public evaluation histories and isolate verified timing boundaries, thereby falsifying unsupported claims about AI system performance.

Results
The results indicate that the candidate selection-aware frontier model fails to achieve synthetic recovery, objective-archive prediction, preference transfer, and uncertainty calibration. The fixed audit gates employed in the study reject the stronger claims made by this model. The authors provide quantitative insights into the discrepancies in performance metrics, although specific numerical results against named baselines are not detailed in the abstract. The findings suggest significant variability in the reported performance of AI systems when subjected to the proposed Bayesian auditing framework.

Limitations
The authors acknowledge that their approach relies on the integrity of the underlying data from public archives, which may still be subject to biases and inaccuracies. They also note that the synthetic nature of some comparisons may not fully capture the complexities of real-world AI evaluations. Additionally, the paper does not address the potential impact of external factors influencing the performance of AI systems, such as changes in training data or model architectures over time.

Why it matters
This work has significant implications for the field of AI evaluation, as it provides a structured approach to auditing public performance claims and highlights the importance of transparency in reporting. By reconstructing evaluation histories and isolating verified performance metrics, the authors contribute to a more rigorous understanding of AI capabilities. This is crucial for researchers and practitioners who rely on public benchmarks for model selection and development. The findings underscore the need for improved methodologies in AI evaluation, as discussed in related literature, such as the work available on arXiv.