RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

Problem
Existing benchmarks for agent performance often fail to capture the complexities and realism of actual developer-agent interactions. This paper addresses the gap in the literature by presenting RealClawBench, a live benchmark framework constructed from real OpenClaw sessions. The authors highlight that traditional benchmarks overlook critical aspects such as local execution environments, implicit intents, and the need for verification, which are essential for accurately assessing agent capabilities. This work is a preprint and has not undergone peer review.

Method
RealClawBench employs two primary mechanisms to convert real developer-agent sessions into reproducible tasks: reconstructed execution environments and deterministic verifiable scorers. The reconstructed execution environments simulate the conditions under which real requests are made, while the verifiable scorers ensure that the tasks can be automatically evaluated for correctness. The benchmark consists of 281 executable tasks, sampled from a larger pool of real sessions, maintaining a maximum Jensen-Shannon divergence of 0.0448 between the final and source distributions. The authors do not disclose specific details regarding the architecture or training compute used for the models evaluated against this benchmark.

Results
The evaluation of 14 contemporary models on the RealClawBench reveals that the best-performing system achieves a task completion rate of only 65.8%. This performance indicates significant room for improvement in handling realistic developer-agent workloads, suggesting that current models may not be adequately equipped to address the complexities of real-world scenarios. The results underscore the need for more robust evaluation frameworks that reflect actual user interactions.

Limitations
The authors acknowledge that the benchmark’s reliance on real user sessions may introduce variability due to the diverse execution environments and implicit intents present in the data. Additionally, the limited number of tasks (281) may not fully represent the vast range of potential developer-agent interactions. The paper does not address potential biases in the sampled sessions or the generalizability of the benchmark across different domains or agent types.

Why it matters
RealClawBench provides a significant advancement in the evaluation of agent capabilities by aligning benchmarks with real-world usage scenarios. This framework enables researchers to better assess the performance of agent systems in practical applications, paving the way for improvements in agent design and functionality. The implications of this work extend to the development of more effective AI systems that can handle the complexities of real-world tasks, as published in arXiv cs.CL.