EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading

Problem
The paper addresses the gap in reliable rubric grading for large language models (LLMs), emphasizing that accurate score prediction alone is insufficient. Existing methods for credit assignment and intervention are primarily tailored for self-contained reasoning tasks, such as mathematical problems, and fail to diagnose where grading reasoning is flawed or how the model’s belief about the final score evolves during the grading process. This work is presented as a preprint, indicating that it has not yet undergone peer review.

Method
The authors propose Evidence-Diagnosed Intervention Training (EDIT), which consists of two phases: EDIT-SFT and EDIT-RL. In the EDIT-SFT phase, the model identifies problematic reasoning steps by analyzing internal signals, specifically the posterior belief over the final mark and input-grounding scores. This phase revises only the identified local steps using a rubric checklist to ensure alignment with grading criteria. The second phase, EDIT-RL, employs reinforcement learning to calibrate the grader by implementing belief-guided reward shaping. This approach penalizes significant harmful drifts in belief while permitting beneficial exploration of grading strategies. The training process leverages real-world multi-subject grading benchmarks, although specific details on the architecture, loss functions, and training compute are not disclosed.

Results
EDIT demonstrates superior performance compared to strong baselines, including supervised fine-tuning and reinforcement learning methods, across both in-domain and out-of-domain splits on two real-world grading benchmarks. The paper reports significant effect sizes, although specific numerical results are not provided in the abstract. Ablation studies further confirm that the internal-state diagnostics employed in EDIT are critical drivers of the observed performance improvements.

Limitations
The authors acknowledge that their approach may be limited by the quality and comprehensiveness of the rubric checklists used during the EDIT-SFT phase. Additionally, the reliance on internal model signals may not generalize across all LLM architectures or grading contexts. The paper does not address potential scalability issues when applying EDIT to larger datasets or more complex grading scenarios.

Why it matters
The implications of this work are significant for the development of more reliable and rubric-faithful LLM grading systems, which can enhance automated assessment in educational settings. By providing a structured approach to diagnosing and intervening in grading reasoning, EDIT could lead to more accurate and fair evaluations of student work. This research contributes to the ongoing discourse on improving LLM capabilities in real-world applications, as published in arXiv.