Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Problem

Non-deterministic responses of large language models (LLMs) complicate failure reproduction in LLM agents. This paper addresses the gap in regression testing methodologies for LLMs, particularly in ensuring reliable failure reproduction and testing of code changes. The work is presented as a preprint and has not undergone peer review.

Method

Chronicle employs an architecture that records agent runs at non-deterministic boundaries, encapsulating these runs as immutable envelopes. The core algorithm, cut-point replay, selectively serves a chosen subset of these recorded boundaries while executing the complementary subset live with new code. This approach allows for targeted testing of code changes against previously recorded failures. The authors utilized data from 6 recorded failures with simulated model boundaries to validate their method. Notably, the recording process incurs an overhead of 23 μs per crossing, which constitutes 0.008% of an assumed 300 ms model call. Importantly, the full replay mechanism issues zero model calls and demonstrates bit-stability across 20 repetitions, ensuring consistent results during testing.

Results

The results indicate that the recording overhead is 23 μs per crossing, with no baseline reported for comparison. The full replay mechanism achieves stability with zero model calls, again with no baseline provided. The cut-point tests successfully fail on faulty code and pass on guarded and benign changes for all 6 incidents recorded. In a mutation study, the cut-point tests effectively catch every mutant that allows the recorded unsafe action to pass, contrasting with a baseline approach that stubs every boundary and catches none.

Limitations

The authors do not report any limitations in their work. However, the absence of a comparative baseline for recording overhead and full replay stability may limit the contextual understanding of performance improvements.

Why it matters

The implications of this work are significant for downstream applications in LLM development and deployment. By providing a reliable mechanism for regression testing, Chronicle enhances the robustness of LLM agents against code changes, thereby improving the reliability of AI systems in production environments. This method could pave the way for more systematic testing frameworks that address the inherent non-determinism of LLMs, ultimately contributing to safer and more predictable AI applications.