When Built-in Thinking Helps and Hurts: Constraint-Level Error Shifts in Instruction Following

Problem
This work addresses the unclear effects of built-in reasoning mechanisms in large reasoning models (LRMs) on instruction following performance. While prior research has established that LRMs enhance capabilities in math and coding tasks, the specific influence on instruction adherence remains underexplored. The authors present a preprint study using IFEval with Qwen3 models (1.7B-32B) and Hunyuan models, aiming to elucidate how reasoning affects error patterns in instruction following.

Method
The study employs Qwen3 models with sizes ranging from 1.7B to 32B, utilizing a Thinking ON/OFF control mechanism to analyze the impact of reasoning on instruction following. The authors categorize prompts into two constraint types: Planning (global tasks) and Precision (local tasks). They conduct a series of experiments to measure pass rates and error shifts across these categories. The analysis includes matched-length evaluations to assess the influence of answer length on performance. Additionally, a cross-encoder relevance metric is used to analyze thinking traces, revealing correlations between relevance and compliance for different constraint types. Activation patching is performed across four model sizes (1.7B-14B) to investigate restoration rates of Precision and Planning flip instances.

Results
The findings indicate that aggregate pass-rate changes are minimal, ranging from -0.55 to -3.52 percentage points. However, a significant observation is that 10-20% of prompts switch between pass and fail statuses when toggling Thinking ON/OFF, suggesting that reasoning alters the error distribution rather than uniformly degrading performance. Specifically, Planning tasks show improved performance under reasoning, while Precision tasks consistently decline. The correlation analysis reveals a positive relevance-compliance link for Neutral prompts (r ≈ 0.15), near-zero for Planning (r ≈ 0.02), and a small negative correlation for Precision (r ≈ -0.05). Activation patching results show that Precision flip instances are restored more frequently than Planning flips, with restoration rates of 32-58% for Precision compared to 14-40% for Planning, particularly pronounced at the 14B model size.

Limitations
The authors acknowledge that the study is limited by its preprint status, which may affect the robustness of the findings. Additionally, the reliance on specific model architectures (Qwen3 and Hunyuan) may limit generalizability to other LRM families. The analysis of error patterns is also constrained by the chosen metrics, which may not capture all dimensions of instruction following performance.

Why it matters
This research provides critical insights into the nuanced effects of reasoning mechanisms on instruction following in LRMs, highlighting the importance of understanding error patterns across different task types. The findings suggest that while reasoning can enhance performance in certain contexts, it may also introduce vulnerabilities in others, particularly in tasks requiring precision. This work lays the groundwork for future investigations into optimizing reasoning strategies in LRMs, as published in arXiv.