MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following

Problem
This paper addresses the gap in evaluating large language model (LLM) judges specifically for multi-constraint instruction following tasks. Existing methodologies primarily focus on overall response judgments, neglecting the nuanced assessment of individual constraints. The authors introduce MCJudgeBench, a benchmark designed to evaluate LLM judges at the constraint level, which is crucial for understanding how well these models adhere to multiple requirements in a given instruction. This work is presented as a preprint and has not yet undergone peer review.

Method
MCJudgeBench consists of instances that include an instruction, a candidate response, an explicit list of constraints, and gold labels for each constraint categorized as {yes, partial, no}. The benchmark incorporates controlled perturbations on the response side to assess the robustness of the judges. The evaluation protocol features variants of evaluation prompts to measure judge stability. The authors employ both correctness and inconsistency metrics to evaluate proprietary and open-source LLM judges. They differentiate between intrinsic inconsistency, which arises from stochastic decoding, and procedural inconsistency, which is influenced by prompt and response perturbations. This dual-metric approach allows for a comprehensive analysis of judge performance across different dimensions.

Results
The evaluation reveals that judge reliability is multifaceted; high overall performance does not correlate with consistent detection across all label categories, particularly for the less frequent “partial” and “no” cases. For instance, while some judges exhibit high correctness rates, they do not necessarily demonstrate lower inconsistency rates. The findings indicate that reasoning-based evaluations can enhance correctness but do not consistently improve stability across all judges. The results underscore the necessity of constraint-level evaluation, as it exposes specific failure modes that are obscured in aggregate assessments.

Limitations
The authors acknowledge several limitations, including the potential biases introduced by the selection of constraints and the specific types of perturbations applied. They also note that the benchmark may not encompass all possible instruction types or response variations, which could limit generalizability. Additionally, the reliance on gold labels may not fully capture the subjective nature of some constraints, leading to discrepancies in evaluation. An obvious limitation not discussed by the authors is the potential impact of model size and architecture on performance, which could vary significantly across different LLMs.

Why it matters
The introduction of MCJudgeBench has significant implications for the evaluation of LLMs in multi-constraint scenarios. By emphasizing constraint-level assessment, this work encourages a more granular understanding of model performance, which is essential for developing robust AI systems capable of complex instruction following. The findings highlight the importance of evaluating not just the correctness of responses but also the consistency and reliability of judges across various constraints. This benchmark could serve as a foundation for future research aimed at improving LLMs’ adherence to multi-faceted instructions, ultimately enhancing their applicability in real-world tasks.

Authors: Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, Ronald Clark
Source: arXiv:2605.03858
URL: https://arxiv.org/abs/2605.03858v1