Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing

Problem
This paper addresses the inefficiencies in speculative decoding for large language models (LLMs), particularly the limitations of existing drafter models that are optimized using token-level supervised objectives. The authors highlight that these models often fail in hard-to-draft positions, leading to early mismatches that truncate the accepted prefix and invalidate subsequent speculative windows. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce PPOW (Performance-Driven Policy Optimization with Adaptive Windowing), a novel reinforcement learning framework designed to optimize the drafter’s performance at the window level rather than the token level. PPOW employs three key components:

1. Cost-Aware Speedup Reward: This reward function incentivizes the drafter to produce outputs that maximize speedup while considering computational costs.
2. Distribution-Based Proximity Reward: This component encourages the drafter to generate candidate tokens that are close to the target distribution, enhancing the likelihood of acceptance.
3. Adaptive Divergence-Aware Windowing: This mechanism prioritizes windows that exhibit high confidence-weighted divergence between the draft and target models, focusing on informative windows that are more likely to yield valid outputs.
   The training process leverages a unified decoding protocol across multiple model families, although specific details on training compute are not disclosed.

Results
PPOW demonstrates significant improvements in speculative decoding efficiency, achieving average acceptance lengths ranging from 6.29 to 6.52 tokens. The framework also realizes speedups of 3.39 to 4.36 times compared to baseline models across various benchmarks. These results indicate a substantial enhancement in performance, showcasing the effectiveness of window-level optimization in speculative decoding scenarios.

Limitations
The authors acknowledge that their approach may still be sensitive to the choice of hyperparameters and the specific architecture of the underlying models. Additionally, they do not address potential scalability issues when applied to larger models or more complex tasks. The reliance on reinforcement learning may also introduce variability in training outcomes, which could affect reproducibility. Furthermore, the paper does not explore the impact of different reward structures on performance, which could be a valuable avenue for future research.

Why it matters
The implications of this work are significant for the field of LLM inference, particularly in enhancing the efficiency of speculative decoding methods. By shifting the optimization focus from token-level to window-level, PPOW provides a framework that can be adapted to various model architectures and tasks, potentially leading to faster inference times and improved performance in real-world applications. This research opens avenues for further exploration into adaptive optimization techniques in reinforcement learning, particularly in contexts where efficiency is critical.

Authors: Jie Jiang, Xing Sun
Source: arXiv:2605.14978
URL: https://arxiv.org/abs/2605.14978v1