Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Problem

This work addresses a significant gap in the evaluation of data agents specifically tailored for enterprise-scale workflows. The authors highlight the lack of comprehensive benchmarks that can assess the performance of these agents in realistic scenarios. The paper is a preprint and has not undergone peer review.

Method

The core contribution is the development of the Argo-Bench framework, which encompasses 210 data science and analytics tasks. The tasks are derived from a simulated food delivery platform set in New York City, utilizing a dataset that includes 81 million orders projected for 2024, organized into 235 tables with a total of 7.5 billion rows. The schema model is based on the Oracle E-Business Suite, ensuring relevance to enterprise applications.

The framework allows agents to perform various actions, such as banning accounts, allocating budgets, and issuing back pay. The grading mechanism evaluates agent performance based on the consequences of their actions within the simulation, providing a robust metric for assessing effectiveness. Each task is paired with reference solutions that are executable, facilitating direct comparison of agent performance against established benchmarks.

Results

The strongest model evaluated within the framework achieved scores of 95 or higher on 34.8% of the tasks, demonstrating a significant capability in handling the defined challenges. The average score across all tasks was reported at 59.5 points, indicating a baseline performance level against which future models can be compared. The evaluation included 14 frontier and open-weight models, providing a competitive landscape for the results.

Limitations

The authors note that real enterprise data warehouses are too sensitive to be publicly released, which necessitates reliance on public datasets for evaluation. Additionally, they point out that answer keys in established benchmarks are frequently incorrect, which could skew performance assessments and hinder the reliability of comparisons.

Why it matters

The introduction of Argo-Bench has significant implications for the development and evaluation of data agents in enterprise contexts. By providing a structured framework for assessment, it enables researchers and practitioners to benchmark their models against a standardized set of tasks, fostering advancements in the field. This work lays the groundwork for future research aimed at improving data agent performance in complex, real-world scenarios.