SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

Problem
This paper addresses the gap in evaluating synthetic datasets used for testing tool-calling agents, particularly in scenarios where real datasets are inadequate due to issues like sensitivity or sparsity. The authors highlight the lack of a standardized framework to quantify how well synthetic data replicates the characteristics of real execution traces, which is critical for ensuring the reliability of evaluations. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose SynAE, a comprehensive evaluation framework designed to measure the quality of synthetic data in the context of multi-turn, tool-calling agent evaluations. SynAE assesses synthetic data across four key metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. The framework employs a multi-axis evaluation approach to capture the validity, fidelity, and diversity of synthetic datasets. The authors conduct experiments using recent agent benchmarks and simulate common synthetic data failure modes through both realistic and controlled generation schemes. The architecture of SynAE is not explicitly detailed, but the methodology emphasizes the importance of a multi-faceted assessment rather than relying on a single metric.

Results
SynAE demonstrates its effectiveness by revealing fine-grained variations in the quality of synthetic data across the defined metrics. The authors report that no single metric suffices to fully characterize synthetic data quality, underscoring the necessity for a multi-dimensional evaluation approach. While specific numerical results or effect sizes against named baselines are not provided in the abstract, the framework’s ability to detect variations suggests significant improvements in synthetic data evaluation practices.

Limitations
The authors acknowledge that SynAE’s evaluation metrics may not cover all possible dimensions of synthetic data quality, indicating that further refinement and expansion of the framework could be necessary. Additionally, the reliance on specific benchmarks may limit the generalizability of the findings. The paper does not address potential computational overhead associated with the multi-axis evaluation, which could impact scalability in real-world applications.

Why it matters
The introduction of SynAE has significant implications for the development and evaluation of tool-calling agents, particularly in contexts where real data is scarce or sensitive. By providing a structured framework for assessing synthetic data quality, SynAE enables practitioners to make more informed decisions about the reliability of their evaluations. This work paves the way for future research to enhance synthetic data generation techniques and improve the robustness of agent evaluations, ultimately contributing to the advancement of AI systems that rely on tool-calling capabilities.

Authors: Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti
Source: arXiv:2605.22564
URL: https://arxiv.org/abs/2605.22564v1