FASE: Fast Adaptive Semantic Entropy for Code Quality

Problem
The paper addresses the limitations of existing methods for quantifying uncertainty in multi-agent code generation systems, particularly the reliance on costly LLM-driven equivalence checks. Current approaches struggle with the reliability of generated code due to hallucinations and error propagation among agents. The authors propose FASE as a solution to improve the assessment of functional correctness without the need for ground-truth answers, filling a gap in the literature regarding efficient uncertainty quantification in autonomous software development. This work is presented as a preprint and has not undergone peer review.

Method
FASE leverages a novel metric based on the minimum spanning tree of structural and semantic dissimilarity graphs to approximate functional correctness. The authors detail the construction of these graphs, which capture the relationships between code snippets generated by multiple agents. The metric is designed to be computationally efficient, requiring only about 0.3% of the runtime cost associated with traditional semantic entropy methods that rely on LLM entailment. The evaluation of FASE is conducted using the Qwen3-Embedding-8B model, which serves as the embedding mechanism for the code representations.

Results
FASE demonstrates significant improvements over existing semantic entropy methods. In evaluations on the HumanEval and BigCodeBench benchmarks, FASE achieves a 25% average improvement in Spearman correlation and a 19% increase in ROCAUC score compared to the Pass@1 metric derived from ground-truth test cases. These results indicate that FASE not only enhances the reliability of code quality assessments but also does so with minimal computational overhead, making it a practical alternative for real-world applications.

Limitations
The authors acknowledge that while FASE reduces computational costs significantly, it may still be sensitive to the quality of the underlying embeddings and the structure of the dissimilarity graphs. Additionally, the reliance on the Qwen3-Embedding-8B model may limit generalizability across different embedding architectures. The paper does not address potential biases in the training data used for the embeddings, which could affect the performance of FASE in diverse coding scenarios.

Why it matters
FASE represents a substantial advancement in the field of multi-agent code generation by providing a cost-effective and efficient method for uncertainty quantification. Its ability to improve the reliability of generated code can facilitate more robust autonomous software development processes, ultimately leading to higher quality software systems. The implications of this work extend to various applications in AI-driven software engineering, as it offers a scalable solution to a critical challenge in the field. This research is available on arXiv.