External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs

Problem
This work addresses the gap in understanding how external operational experience can be effectively utilized in production LLM systems, particularly focusing on the trade-offs between quality and cost during deployment. The authors highlight that while external experience can enhance task performance, it also introduces challenges such as increased prompt burden, latency, and serving pressure. This study is particularly relevant as it is a preprint and has not undergone peer review, indicating that the findings should be interpreted with caution.

Method
The authors propose a framework for evaluating external experience serving strategies in a production moderation context. They compare several approaches: a no-experience baseline, random experience controls, global prompt injection, and retrieval-based selective injection. The evaluation is conducted using real-world data, with a focus on two contrast tasks—tool-use and GPQA—that reveal different output-cost dynamics. The study emphasizes the importance of case-dependent experience, where selective retrieval is shown to outperform unconditional global injection. The authors analyze both task quality and serving costs, providing insights into how different strategies impact performance metrics and operational expenses.

Results
The findings indicate that selective retrieval significantly enhances task quality compared to global prompt injection, particularly in scenarios where the experience is case-dependent. The results demonstrate that retrieval quality is more critical than merely increasing the Top-K outputs, suggesting that the effectiveness of experience serving is contingent on the specific context of the task. The authors report that the same serving policy can yield varying cost-benefit profiles across different output regimes, such as short-output versus decode-heavy tasks. This nuanced understanding of quality-cost trade-offs is crucial for optimizing LLM deployment strategies.

Limitations
The authors acknowledge that their study is limited to specific production settings and may not generalize across all LLM applications. They also note that the effectiveness of external experience serving is highly dependent on the task-specific cost structure and the design of the serving interface. Additionally, the study does not explore the long-term implications of integrating external experience into LLM systems, nor does it consider the potential for diminishing returns as experience accumulates.

Why it matters
This research has significant implications for the deployment of LLM systems in production environments, particularly in contexts where operational efficiency and cost management are critical. By framing external experience as a selective, cost-aware decision rather than a universal enhancement, the authors provide a strategic perspective that can inform future work in LLM optimization. The insights gained from this study can guide engineers and researchers in developing more effective serving strategies that balance quality and cost, ultimately leading to improved performance in real-world applications. This work is available on arXiv.