TAHOE: Text-to-SQL with Automated Hint Optimization from Experience

Problem
This paper addresses the challenges of deploying Text-to-SQL systems in production environments, particularly the difficulties in handling strict SQL dialects, large schemas, and evolving user preferences. Existing methods, such as supervised fine-tuning, are often costly and inflexible, while agentic test-time scaling incurs significant expenses. The authors propose Tahoe as a solution to these issues, presenting it as a preprint work that has not yet undergone peer review.

Method
Tahoe employs a novel error-driven hint learning pipeline that operates across both development and deployment phases. It consolidates debugging traces into a structured Hint Bank, which includes two types of hints: Syntax Hints and Semantic Hints. Syntax Hints are derived from compiler feedback and are tailored to specific SQL dialects, while Semantic Hints are generated from execution and user feedback, focusing on schema and user-specific logic. The architecture features a Strategy Layer that models conflicting user intents as competing strategies, utilizing recency signals and post-learning attribution statistics to summarize various performance metrics (success, harm, inertness, and support). During inference, Tahoe retrieves relevant hints to guide the LLM through a two-step process: Logic Planning followed by SQL Synthesis. The implementation focuses on the development-phase workflow, with plans for future work to incorporate deployment-time human feedback updates.

Results
Tahoe demonstrates significant improvements in Text-to-SQL performance on the Spider 2.0-Snow benchmark without necessitating updates to model parameters. Specifically, it raises the pass rate from 61.95% to 79.42% and the pass-at-4 metric from 72.57% to 87.61% across 113 supervised examples using GPT-5.5. Additionally, Tahoe achieves a 100% pass rate for Snowflake syntax and reduces the average number of compiler-feedback critic rounds from 2.79 to 0.12 per sampled candidate. The Hint Bank’s effectiveness is further validated by its transferability to weaker models, yielding a 19.7 percentage-point increase in pass rate on Doubao-2.0-lite.

Limitations
The authors acknowledge that the current implementation of Tahoe focuses solely on the development-phase workflow, leaving the integration of human feedback during deployment as a future consideration. Additionally, the reliance on a specific LLM (GPT-5.5) for evaluation may limit the generalizability of the results across different architectures. The paper does not address potential scalability issues when applied to larger or more complex databases.

Why it matters
Tahoe’s approach to dynamic hint optimization represents a significant advancement in the field of Text-to-SQL systems, particularly in making them more adaptable and efficient for real-world applications. By decoupling performance improvements from model parameter updates, Tahoe paves the way for more flexible and cost-effective deployment strategies. This work has implications for future research in automated database management and user intent modeling, as it highlights the importance of structured hint management in enhancing LLM capabilities in SQL generation, as published in arXiv cs.AI.