Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

Problem

The paper addresses a gap in the evaluation of large language models (LLMs) regarding their ability to reason about code execution within repository-level contexts. Specifically, it highlights the lack of dynamic benchmarks that assess LLMs' performance on real-world code execution scenarios. This work is particularly relevant as it provides a structured approach to understanding how well LLMs can interpret and reason about the runtime behavior of code, which is crucial for applications in software engineering and automated code analysis. The work is presented as a preprint and has not yet undergone peer review.

Method

The authors propose a new benchmark called SWE-Flux, which consists of 480 execution-grounded instances derived from 12 real Python repositories. The instances are designed to evaluate LLMs on various aspects of code execution reasoning. The answers to the benchmark questions are automatically harvested from instrumented test executions, ensuring that they are grounded in actual runtime behavior. The benchmark includes two types of questions: single-test and multi-test questions, covering topics such as control flow, loops, program state, dataflow, exceptions, and program invariants. Five different LLMs were evaluated on their performance against this benchmark, providing a comparative analysis of their reasoning capabilities.

Results

The evaluation results indicate that the best-performing model achieved an accuracy of 37% on the SWE-Flux benchmark. The performance insights reveal that LLMs perform better on localized behavior aspects, such as invariants, intra-procedural control flow, exceptions, and simple loops. However, they struggle significantly with more complex reasoning tasks, particularly in areas like dataflow analysis, inter-procedural execution, precise state reasoning, and suite-level aggregation. Additionally, the authors implemented an oracle-harvesting pipeline that generates fresh benchmark variants for nearly 90% of the selected instances, resulting in more challenging questions for the models.

Limitations

The authors note that LLMs exhibit difficulties with complex reasoning tasks, particularly those involving dataflow and inter-procedural execution. These limitations suggest that while LLMs can handle certain aspects of code reasoning, their capabilities are not yet sufficient for more intricate scenarios that require a deeper understanding of program behavior. The paper does not discuss other potential limitations, such as the generalizability of the benchmark to other programming languages or the scalability of the evaluation framework.

Why it matters

This work has significant implications for downstream research in the field of software engineering and AI. By establishing a benchmark like SWE-Flux, it provides a foundation for future studies aimed at improving LLMs' reasoning capabilities regarding code execution. The insights gained from this benchmark can inform the development of more sophisticated models and techniques that enhance the ability of AI systems to understand and reason about complex software behaviors, ultimately contributing to advancements in automated code analysis, debugging, and software maintenance.