Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

Problem

The paper addresses the inadequacy of the SWE-bench leaderboard in effectively ranking coding agents, highlighting that the current metrics fail to differentiate between the top entries. The authors conduct an audit of 254 SWE-bench submissions across four splits, revealing that the leading entries have converged in performance, making it difficult to establish a clear order. This work is presented as a preprint and has not undergone peer review.

Method

The authors propose a five-step audit protocol for profiling shared outcomes among submissions, which includes testing paired differences, reporting grouping sensitivity, and estimating the instance budget for resolution. The evaluation is based on resolved instances, with the leading two entries resolving 396 out of 500 instances. The success and failure metrics indicate that the top ten submissions collectively achieved 285 successes and 51 failures, with 164 instances distinguishing outcomes. The median nesting score of the submissions is reported as 0.935, compared to a score-implied baseline of 0.774. The within-model scaffold ranges show a variation of 29.8 percentage points, while the spread among the top thirty submissions is only 8.8 points. Statistical tests were conducted, revealing that six out of nine cell-mean interaction tests were significant after Holm correction, although none of the 29 adjacent pairs were distinguishable at an alpha level of 0.05. In a larger test split, 14 out of 23 pairs were successfully separated.

Results

The results indicate that the leading two entries resolved 396 instances out of 500, while the top ten submissions achieved 285 successes and 51 failures. The median nesting score of 0.935 significantly outperformed the baseline of 0.774. The within-model scaffold range of 29.8 percentage points suggests substantial variability, contrasting with the smaller 8.8-point spread among the top thirty submissions. Statistical analysis showed that six of nine tests were significant post-correction, but no adjacent pairs were distinguishable at the specified alpha level. The larger test split provided clearer separations, with 14 out of 23 pairs being distinguishable.

Limitations

The authors note that the observational design of their study does not allow for the identification of causal effects related to scaffold performance. Additionally, the lack of rejection in statistical tests does not imply equivalence among the submissions, which could mislead interpretations of the results.

Why it matters

This work has significant implications for the evaluation of coding agents, suggesting that current leaderboard metrics may not accurately reflect the capabilities of the top submissions. By proposing a new audit protocol and evaluation metrics, the authors aim to enhance the robustness of performance assessments in coding tasks, which could influence future research directions and the development of more effective coding agents.