Monitoring Agentic Systems Before They're Reliable

Problem
The paper addresses the gap in monitoring methodologies for agentic systems that are not yet reliable, particularly in the context of structural defects overshadowing task-level errors. The authors highlight that existing literature lacks a comprehensive approach to evaluate these systems at early maturity stages, where traditional task-level error detection becomes infeasible due to the prevalence of structural failures. This work is presented as a preprint, indicating it has not yet undergone peer review.

Method
The authors propose a novel monitoring and triage methodology that decomposes the evaluation of agentic systems into three dimensions: quality, suitability, and efficiency. They implement this across three monitoring scopes: within-run, cross-run, and structural. The methodology utilizes variance as a characterization signal to identify failure types. A severity classification adapted from Failure Mode and Effects Analysis (FMEA) is employed to prioritize findings for human investigation. The evaluation is conducted on a synthetic testbed comprising 220 runs across 120 document bundles, with controlled error injections to simulate various failure scenarios.

Results
The findings reveal that the type of monitoring scope significantly influences the detection of failure types. Specifically, within-run monitors effectively surface deterministic stage defects with a coefficient of variation (CV) of 0.02. In contrast, cross-run monitors identify stochastic integration consequences with a CV of 1.25, where 24% of findings occur at Level 2 severity. The structural monitor demonstrates perfect consistency in identifying integration gaps, achieving a CV of 0.00. Notably, injected task-level errors are indistinguishable from clean baselines, underscoring the masking effect of structural defects on task-level signals. The deterministic triage process routes 97% of findings to automated tracking, reserving only 2% for human investigation, which reflects variable behavior.

Limitations
The authors acknowledge that their findings are based on a synthetic testbed, which may limit the generalizability of the results to real-world applications. Additionally, the proposed maturity-staging model is based on preliminary evidence from Stage 1, suggesting that further validation is necessary to confirm its efficacy across diverse agentic systems. The specific calibrations for different domains remain to be explored, which could affect the applicability of the methodology in regulated industries.

Why it matters
This work has significant implications for the development and deployment of agentic systems, particularly in regulated environments where reliability is critical. By emphasizing the importance of early monitoring, the authors advocate for a proactive approach to identifying and addressing structural defects before they lead to task-level failures. The proposed methodology and maturity-staging model can inform future research and practical applications in monitoring agentic workflows, as published in arXiv.