VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

Problem
This paper addresses the inefficiencies in speculative decoding (SD) for large language models (LLMs), particularly the high inference costs associated with the binary accept/reject decisions of existing draft-verify methods. The authors identify a gap in the literature regarding the potential for intermediate verification strategies that can leverage a slim submodel for moderate verification tasks, rather than relying solely on the full verifier. This work is presented as a preprint, indicating that it has not yet undergone peer review.

Method
The proposed method, Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), introduces a multi-tier framework that incorporates a routed slim-verifier. The architecture processes draft tokens in a hierarchical manner: high-confidence tokens are directly accepted, medium-confidence tokens are regenerated using the slim-verifier, and uncertain tokens are subjected to full-model verification. The slim-verifier is derived from the full verifier through intra-model routing, allowing it to handle tokens that require moderate verification resources. The authors do not disclose specific details regarding the training compute or the exact architecture of the models used, but they demonstrate compatibility with existing SD frameworks without necessitating changes to their training procedures.

Results
VIA-SD achieves a reduction in rejection rates by 0.10 to 0.22 across four representative tasks and multiple model families. It also delivers speedups of 10-20% over strong SD baselines and achieves 2.5-3x acceleration compared to non-drafting decoding methods. These results indicate a significant improvement in inference efficiency while maintaining the integrity of the output, showcasing the effectiveness of the multi-tier approach in practical applications.

Limitations
The authors acknowledge that while VIA-SD improves efficiency, it may not eliminate all instances where full-model verification is necessary, particularly for low-confidence tokens. Additionally, the performance gains may vary depending on the specific model architecture and task complexity, which are not exhaustively explored in the paper. The lack of peer review also raises questions about the robustness of the findings and their reproducibility.

Why it matters
The implications of VIA-SD are significant for the future of LLM inference, as it presents a scalable and efficient paradigm that can be integrated into existing frameworks. By reducing the reliance on full model calls, VIA-SD can lower operational costs and improve response times in real-world applications. This work contributes to the ongoing discourse on optimizing LLM performance and efficiency, as discussed in related literature on speculative decoding and model verification techniques, and is available on arXiv.