GIM: Evaluating models via tasks that integrate multiple cognitive domains

Problem
This paper addresses the limitations of existing benchmarks for evaluating large language models (LLMs), particularly as current benchmarks become saturated. The authors argue that existing strategies either conflate memorization with capability (e.g., GPQA, HLE) or abstract reasoning from practical contexts (e.g., ARC-AGI). They propose a novel evaluation framework, the Grounded Integration Measure (GIM), which consists of 820 original problems designed to assess models’ abilities to integrate multiple cognitive operations in realistic scenarios. This work is presented as a preprint and has not yet undergone peer review.

Method
The GIM benchmark comprises 820 problems (615 public, 205 private) that require models to perform tasks involving constraint satisfaction, state tracking, epistemic vigilance, and audience calibration. Each problem is crafted by experts and scored using a rubric with a median of six independently judged criteria. The authors employ a continuous response 2-parameter logistic (2PL) Item Response Theory (IRT) model to analyze over 200,000 prompt-response pairs across 28 models, allowing for robust ability estimates that account for inaccuracies and missing data. The evaluation framework includes a comprehensive leaderboard that spans 22 models and 47 test configurations, enabling a detailed analysis of how test-time compute impacts model performance.

Results
The authors present a leaderboard that ranks 22 models across 47 unique test configurations, revealing that configuration choices (e.g., thinking budget and quantization) significantly influence performance, sometimes as much as the choice of model itself. The paper does not disclose specific numerical results or effect sizes against named baselines, focusing instead on the methodological contributions and the implications of their findings.

Limitations
The authors acknowledge that their benchmark may not capture all aspects of cognitive integration and that the reliance on expert-authored problems could introduce biases. Additionally, the public-private split, while designed to mitigate contamination, may still be susceptible to overfitting. The paper does not address potential limitations related to the generalizability of the benchmark across different domains or the scalability of the evaluation framework.

Why it matters
The GIM framework represents a significant advancement in the evaluation of LLMs by emphasizing the integration of cognitive operations in realistic contexts. This approach could lead to more nuanced assessments of model capabilities, moving beyond traditional benchmarks that may not accurately reflect practical reasoning skills. The release of the evaluation framework, calibrated IRT parameters, and public problems provides a valuable resource for future research, potentially influencing the design of subsequent benchmarks and the development of more capable AI systems.

Authors: Rohit Patel, Alexandre Rezende, Steven McClain
Source: arXiv:2605.18663
URL: https://arxiv.org/abs/2605.18663v1