CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

Problem
The paper addresses the limitations of existing multi-token prediction (MTP) methods in large language models (LLMs), which suffer from a fundamental architectural flaw: the competition between the MTP head and the backbone language model (LM) head during the generation of the first token. This competition leads to degraded output quality, characterized by repetitive and incoherent text. The authors highlight that prior MTP approaches, particularly those utilizing complex gate networks, fail to provide both speed and quality, indicating a significant gap in the literature regarding efficient and coherent multi-token inference. This work is presented as a preprint and has not undergone peer review.

Method
The authors propose a novel design principle termed “Backbone-as-Architect,” which ensures that the backbone LM head is solely responsible for generating the first token, while the MTP heads handle subsequent tokens. This separation mitigates the head-backbone competition that plagues previous methods. The core technical contribution is the introduction of the Collocation-Length Predictor (CLP), a lightweight span-level decision layer that predicts the number of additional tokens that can be accepted at each decoding step. CLP is implemented using a single linear layer with only 4.6K to 7.7K parameters, significantly reducing the complexity compared to the 1M-parameter gate networks used in prior work. The training and evaluation are conducted on Qwen2.5 models with varying sizes (0.5B, 1.5B, 7B).

Results
Experimental results demonstrate that CLP achieves a speedup of 1.20x to 1.29x on the 1.5B model and 1.14x to 1.20x on the 7B model, with a repetition ratio of less than 0.02, indicating zero quality degradation. In contrast, gate-based approaches only achieve a speedup of 1.07x and produce outputs with a repetition ratio exceeding 0.5%. Additionally, the authors find that using shorter prediction horizons (k=2) enhances MTP head accuracy by 24% on larger models, underscoring the importance of prediction accuracy in the acceleration process.

Limitations
The authors acknowledge that their approach may not generalize to all types of LLM architectures, as it is specifically designed around the Qwen2.5 models. They also note that while CLP significantly improves speed and maintains quality, the method’s performance on even larger models or different architectures remains to be evaluated. Furthermore, the reliance on a linear layer, while efficient, may limit the expressiveness of the model in more complex scenarios.

Why it matters
The implications of this work are significant for the field of natural language processing, particularly in enhancing the efficiency of LLMs during inference without compromising output quality. The clear roadmap established for future improvements in MTP head prediction accuracy could lead to further advancements in model design and application. This research contributes to the ongoing discourse on optimizing LLM performance, as published in arXiv.