Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

Problem
This preprint addresses the inadequacy of existing benchmarks in evaluating Large Language Model (LLM) agents for backend code generation, particularly regarding their ability to adhere to strict structural constraints. While LLMs perform well under loose specifications, production-grade software necessitates compliance with architectural patterns, database schemas, and object-relational mappings. The authors highlight a gap in the literature where functional correctness is often prioritized over structural integrity, leading to a lack of understanding of how LLMs manage complex structural requirements in multi-file backend generation tasks.

Method
The authors conducted a systematic evaluation of LLM agents across 80 greenfield generation tasks and 20 feature-implementation tasks, utilizing a unified API contract to maintain consistency. They employed a dual evaluation strategy that included end-to-end behavioral tests and static verifiers to assess the agents’ performance in handling structural constraints. The study involved a sensitivity analysis of various web frameworks, such as Flask, FastAPI, and Django, to determine how framework characteristics influence agent performance. The evaluation metrics included assertion pass rates, with a focus on identifying specific structural defects, particularly in data-layer interactions.

Results
The findings reveal a significant phenomenon termed “constraint decay,” where agent performance deteriorates as structural requirements increase. On average, capable configurations experienced a 30-point drop in assertion pass rates when transitioning from baseline to fully specified tasks. In contrast, weaker configurations often approached zero performance. The analysis indicated that agents excelled in minimal frameworks like Flask but struggled in convention-heavy environments such as FastAPI and Django, where the average performance was substantially lower. The error analysis pinpointed data-layer defects, including incorrect query composition and ORM runtime violations, as the primary causes of failure.

Limitations
The authors acknowledge that their study is limited to specific web frameworks and may not generalize across all backend environments. They also note that the evaluation does not account for the dynamic nature of real-world software development, where requirements may evolve. Additionally, the reliance on static verifiers may not capture all potential runtime issues, and the study does not explore the impact of different LLM architectures or training methodologies on performance.

Why it matters
This work underscores the critical challenge of ensuring that LLM agents can satisfy both functional and structural requirements in backend code generation. The concept of constraint decay highlights the fragility of LLMs when faced with increasing complexity, suggesting that future research must focus on enhancing the robustness of these models in adhering to strict software engineering principles. The implications extend to the development of more reliable coding agents, which could significantly impact software development practices and the automation of backend systems.

Authors: Francesco Dente, Dario Satriani, Paolo Papotti
Source: arXiv:2605.06445
https://arxiv.org/abs/2605.06445