Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Problem
This paper addresses the limitations of existing evaluation frameworks for agentic systems, particularly in the context of large language model (LLM) agents. Current tools primarily focus on observability and employ static, hand-crafted error taxonomies that lack adaptability to diverse domains. The authors propose Agentic CLEAR, an automatic and dynamic evaluation framework designed to enhance the oversight and assessment of agent behavior across various environments. This work is presented as a preprint and has not yet undergone peer review.

Method
Agentic CLEAR introduces a multi-level evaluation framework that operates above the observability layer, providing insights into agent behavior at three levels of granularity: system, trace, and node. The architecture is designed for seamless integration with existing agentic systems and features an intuitive user interface to facilitate accessibility. The framework leverages a data-driven approach to generate textual insights, which are derived from analyzing agent interactions across multiple benchmarks. The authors conducted experiments involving four distinct benchmarks and seven agentic settings, processing tens of thousands of LLM calls to validate the effectiveness of Agentic CLEAR. The specific loss functions and training compute details are not disclosed in the paper.

Results
The evaluation of Agentic CLEAR demonstrates strong performance in producing high-quality feedback that aligns closely with human-annotated errors. The framework’s ability to predict task success rates was also highlighted, showcasing its effectiveness in assessing agent performance. While specific numerical results are not provided in the abstract, the authors claim that the insights generated by Agentic CLEAR significantly enhance the understanding of agent behavior compared to existing baselines.

Limitations
The authors acknowledge that while Agentic CLEAR provides valuable insights, it may still be limited by the quality of the underlying LLMs and the specific benchmarks used for evaluation. Additionally, the framework’s reliance on data-driven insights may not capture all nuances of agent behavior, particularly in highly dynamic or novel environments. The paper does not address potential scalability issues or the computational overhead associated with integrating Agentic CLEAR into existing systems.

Why it matters
The development of Agentic CLEAR has significant implications for the field of AI, particularly in the evaluation of autonomous agents. By providing a more adaptable and insightful framework for assessing agent behavior, this work paves the way for improved oversight in complex environments where LLMs operate. The ability to generate dynamic evaluations can enhance the reliability and safety of agentic systems, fostering greater trust in their deployment across various applications. Furthermore, the insights gained from Agentic CLEAR could inform future research on agent design and training methodologies, ultimately contributing to the advancement of more capable and responsible AI systems.

Authors: Asaf Yehudai, Lilach Eden, Michal Shmueli-Scheuer
Source: arXiv:2605.22608
URL: https://arxiv.org/abs/2605.22608v1