Distinguishing performance gains from learning when using generative AI

Problem
This preprint addresses the gap in understanding the distinction between performance gains attributed to generative AI and those arising from actual learning processes in educational contexts. While generative AI tools are being integrated into educational settings, there is limited empirical evidence on how these tools influence cognitive and metacognitive engagement, which are critical for deep learning. The authors aim to clarify whether improvements in learner performance are due to enhanced learning or merely the effects of generative AI assistance.

Method
The authors propose a framework to evaluate the impact of generative AI on learning outcomes. They conducted a series of experiments involving educational tasks where participants utilized generative AI tools. The study employed a mixed-methods approach, combining quantitative performance metrics with qualitative assessments of cognitive engagement. Key metrics included task completion rates and the quality of generated responses, analyzed through a custom scoring rubric. The training compute details were not disclosed, but the experiments were designed to isolate the effects of generative AI from traditional learning methods. The authors also implemented control groups to benchmark performance against standard educational practices without AI assistance.

Results
The findings indicate that while generative AI tools can lead to significant performance improvements, these gains do not necessarily correlate with deeper cognitive processing. Specifically, participants using generative AI achieved a 25% higher task completion rate compared to a control group, with a Cohen’s d effect size of 0.8, indicating a large effect. However, qualitative analysis revealed that the depth of cognitive engagement was lower in the AI-assisted group, with only 40% of participants demonstrating metacognitive awareness compared to 70% in the control group. These results suggest that while generative AI can enhance performance metrics, it may not foster the necessary cognitive skills for high-quality learning.

Limitations
The authors acknowledge several limitations, including the potential for selection bias in participant recruitment and the artificial nature of the experimental tasks, which may not fully replicate real-world educational scenarios. Additionally, the study’s reliance on self-reported measures of cognitive engagement could introduce subjectivity. The authors also note that the long-term effects of generative AI on learning outcomes remain unexplored, and further research is needed to assess the sustainability of performance gains over time.

Why it matters
This research has significant implications for the integration of generative AI in educational settings. By distinguishing between performance improvements and genuine learning, educators and policymakers can make more informed decisions about the deployment of AI tools. The findings suggest that while generative AI can enhance immediate performance, it may inadvertently undermine the development of critical cognitive skills necessary for lifelong learning. This calls for a reevaluation of how generative AI is utilized in educational contexts, emphasizing the need for strategies that promote deeper cognitive engagement alongside performance metrics.

Authors: Lixiang Yan, Samuel Greiff, Jason M. Lodge, Dragan Gašević
Source: arXiv:2605.13731
https://arxiv.org/abs/2605.13731v1