The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic

Problem
This preprint addresses the conclusions drawn by Mirzadeh et al. (2025) regarding the GSM-Symbolic benchmark, which reported significant performance drops in 25 Large Language Models (LLMs) when evaluated on template-generated variants of GSM8K problems. The original authors posited that these drops indicated a lack of genuine reasoning capabilities in LLMs. Długosz et al. argue that this conclusion is based on flawed statistical analysis, highlighting the need for a more rigorous evaluation of LLM performance on these tasks.

Method
The authors re-evaluate 20 open-weight models using Generalised Linear Mixed Models (GLMMs) with per-question random effects to account for variability in performance across different problem instances. This approach allows for a more nuanced understanding of model behavior by controlling for individual question characteristics. They also identify a significant distributional shift in the GSM-Symbolic dataset, specifically regarding the prevalence of larger integers in problem texts compared to the GSM-Base dataset, quantified using the Kolmogorov-Smirnov (K-S) statistic (K-S = 0.12, p < 0.001). By controlling for this “large number effect,” the authors find that it accounts for significance in approximately half of the remaining performance cases. The analysis reveals distinct failure profiles for models that exhibit statistically significant performance changes, including issues with variable binding, arithmetic capabilities, and dual-task interference.

Results
The re-evaluation indicates that only 50% of the models show statistically significant performance changes when the original prompt format is used, challenging the blanket assertions made by the original GSM-Symbolic authors. The identification of the large number effect as a confounding variable suggests that the performance drops may not be as severe as previously reported. The authors provide detailed performance metrics for the models, although specific numerical results against named baselines are not disclosed in the abstract. The findings suggest that the reasoning capabilities of LLMs are more nuanced than previously claimed, with model-specific failure modes that require targeted interventions.

Limitations
The authors acknowledge that their analysis is limited to 20 open-weight models and does not encompass the entire landscape of LLMs. Additionally, while they identify the large number effect as a significant factor, other potential confounding variables may still exist. The study’s reliance on GLMMs, while robust, may not capture all aspects of model performance variability. Furthermore, the implications of their findings on the broader understanding of LLM reasoning capabilities remain to be fully explored.

Why it matters
This work has significant implications for the evaluation of LLMs, particularly in the context of reasoning tasks. By challenging the conclusions drawn from the GSM-Symbolic benchmark, the authors advocate for more rigorous statistical methodologies in assessing model performance. Their findings suggest that previous claims regarding LLM reasoning capabilities may be overstated, emphasizing the need for a more granular understanding of model behavior. This re-evaluation could influence future research directions, prompting a reconsideration of how reasoning is measured and interpreted in LLMs.

Authors: Dominika Agnieszka Długosz, Arlindo Oliveira, Natalia Díaz Rodríguez
Source: arXiv:2605.28700
URL: https://arxiv.org/abs/2605.28700v1