Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Problem
The paper addresses the gap in systematic diagnostics for failures in large language model (LLM) agents, which currently rely on manual inspection of execution traces. This ad-hoc approach is inefficient and fails to capture patterns that emerge across large populations of traces, particularly in production environments where traces can be extensive (tens of thousands of tokens). The authors formalize the problem of corpus-level trace diagnostics and propose a solution to automate the generation of insights that characterize systematic behavioral patterns across trace groups, supported by evidence.

Method
The core technical contribution is the Insights Generator (IG), a multi-agent system designed to automate the diagnostic process. IG operates by proposing and testing hypotheses across a corpus of execution traces, ultimately generating an evidence-backed insights report. The architecture consists of a scout-investigator model, where the scout identifies potential issues and the investigator validates these hypotheses against the corpus. The system leverages a combination of qualitative assessments and objective metrics to evaluate its performance. The training compute and specific datasets used for training IG are not disclosed in the paper.

Results
The evaluation of IG demonstrates significant improvements in performance metrics. Human experts utilizing IG reports achieved a 30.4 percentage point increase in scaffold performance compared to the unmodified baseline. Additionally, coding agents that incorporated insights derived from IG exhibited consistent and stable performance gains. The IG architecture’s detection coverage was found to be comparable to existing diagnostic approaches, while domain experts rated the depth and quality of evidence in IG reports as superior. These results suggest that IG not only enhances diagnostic capabilities but also translates into tangible performance improvements in downstream tasks.

Limitations
The authors acknowledge that the effectiveness of IG may be contingent on the quality and diversity of the execution traces in the corpus. They also note that while IG provides valuable insights, it may not fully replace the nuanced understanding that human experts bring to complex diagnostic scenarios. An obvious limitation not explicitly mentioned is the potential computational overhead associated with running the multi-agent system, which could impact scalability in very large corpora. Furthermore, the paper does not discuss the generalizability of IG across different LLM architectures or domains.

Why it matters
The Insights Generator represents a significant advancement in the automation of diagnostics for LLM agents, addressing a critical bottleneck in the deployment and maintenance of these systems. By providing systematic, evidence-backed insights, IG can enhance the efficiency of debugging and improve the overall reliability of LLM applications. This work has implications for future research in automated diagnostics, potentially influencing the design of more robust LLM architectures and the development of tools that facilitate better understanding of model behavior in production settings.

Authors: Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz et al.
Source: arXiv:2605.21347
URL: https://arxiv.org/abs/2605.21347v1