Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models

Problem
The paper addresses the limitations of frozen small code models (≤1.5B parameters) that are designed for offline and privacy-constrained applications but often generate plausible yet incorrect code. The authors investigate the efficacy of 26 post-hoc operators—methods that select, verify, repair, or reprocess model outputs without retraining—to enhance the accuracy of these models. This work is particularly relevant as it is a preprint and has not undergone peer review, highlighting the need for empirical validation in this area.

Method
The study employs a deterministic execution oracle and a matched-compute protocol to evaluate the performance of the post-hoc operators against a Best-of-N (BoN) baseline. The 26 operators include various strategies such as selection, verification, repair, elimination, portfolios, sound vetoes, and generation conditioning. The evaluation focuses on their ability to improve held-out accuracy on specific benchmarks, namely HumanEval+ and MBPP+. The authors introduce two notable operators: an expression-layer recovery method (M1) that enhances accuracy by recovering correct programs that the standard extractor discards, and an adaptive consensus early-stop (ACE) that provides a compute-saving control mechanism.

Results
The results indicate that none of the 26 semantic post-hoc operators outperformed the BoN baseline in terms of held-out accuracy. The study identifies three mechanistic barriers: a coverage wall, a capability scissors effect, and a near-empty consensus trap, which collectively hinder the effectiveness of these operators. However, the M1 operator demonstrated a significant improvement, achieving a +12 task increase on HumanEval+ with a p-value of 2.4e-4, while maintaining a do-no-harm guarantee (b10=0). The ACE operator provided a ~19% compute saving without introducing harm. These findings were replicated across three model cells, reinforcing the robustness of the results.

Limitations
The authors acknowledge that the lack of improvement from the majority of post-hoc operators may stem from fundamental limitations in the models themselves, rather than the operators. They also note that a distribution-free do-no-harm bound cannot certify a harm rate ≤ alpha at zero observed harm unless the sample size is sufficiently large (n ≥ 45). Additionally, the study does not explore the potential for combining multiple operators or the effects of different model architectures on the outcomes.

Why it matters
This research underscores the importance of rigorous measurement and coverage assessment in the development of post-hoc reasoning techniques for code generation models. The findings suggest that rather than relying on semantic post-hoc reasoning, efforts should focus on improving the foundational model architecture and harnessing effective evaluation strategies. This work contributes to the ongoing discourse on enhancing the reliability of AI-generated code, as discussed in related literature, and is available on arXiv.