Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

Problem

Local serving stacks significantly influence tool call execution and the resulting evaluation metrics. This paper addresses the gap in understanding how these stacks can confound local tool-use evaluations, particularly in the context of different model behaviors and their interactions with tool calls. The work is presented as a preprint and has not undergone peer review.

Method

The authors propose a structured approach to evaluate tool call execution across various local serving stacks, including Ollama, llama.cpp, vLLM, and SGLang. Key components of their methodology include:

• Tool Call Protocol: A valid tool call must be parseable, ensuring that the model can effectively interact with the tools.
• Model Handling: Different models exhibit varied behaviors in returning tool calls, either as text or native tool calls, with some calls being outright rejected.
• Failure Metadata: The lack of preserved rejection and retry exhaustion data leads to misclassification of tool call outcomes, which the authors identify as a critical issue.
• Text Tool List: The introduction of a text tool list enhances fidelity for models that accept tool calls, improving the accuracy of evaluations.
• Uniform Text Protocol: This approach, while beneficial in some contexts, reduces fidelity for the Llama-3.2 model compared to its native tool-call support.
• Cross-Stack Probes: The authors conduct experiments across different stacks to assess the consistency and reliability of tool call evaluations.
• Constrained Decoding: This technique is employed to eliminate parse failures, although it may lead to non-termination issues in some cases.
• Estimation Methods: The paper discusses two estimation methods—turn-pooled and per-instance estimates—highlighting that they can differ by as much as 55 points.

Results

The results indicate that:

• Fidelity Recovery: The addition of a text tool list significantly improves fidelity for models that accept tool calls, compared to baseline fidelity metrics without this enhancement.
• Fidelity Reduction: The use of a uniform text protocol results in reduced fidelity for the Llama-3.2 model when compared to its performance with native tool-call support.
• Estimation Difference: The disparity between turn-pooled and per-instance estimates can be as high as 55 points, underscoring the importance of the chosen estimation method.
• The available text does not report quantitative results for specific baseline comparisons or overall performance metrics.

Limitations

The authors acknowledge several limitations in their study:

• The absence of structured failure metadata for rejection and retry exhaustion, which complicates the analysis of tool call outcomes.
• Variability in cross-stack handling may lead to inconsistencies in results, affecting the generalizability of their findings across different local serving stacks.

Why it matters

This work has significant implications for the evaluation of AI models that utilize tool calls, as it highlights the critical role of local serving stacks in shaping evaluation outcomes. By addressing hidden confounds in tool-use evaluations, the findings can inform future research on model performance and reliability, ultimately leading to more robust AI systems.