Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

Problem

This paper addresses the gap in understanding the precision invariance of greedy decoding in large language models (LLMs). The authors highlight that greedy decoding can lead to significant output divergence based on the precision of the computations, which has not been thoroughly explored in existing literature. The work is presented as a preprint and has not undergone peer review.

Method

The authors evaluate six models ranging from 1.1B to 7B parameters across four families, with additional divergence characterization at 12B parameters. They conduct an empirical analysis of 22 layers to assess accumulated body error and identify that the outcome of greedy decoding is influenced by the top-two logit margin at the language model head in relation to directional perturbation. To mitigate divergence, they propose an intervention method that selectively recomputes the LM head in FP32 precision when the logit margin falls below a specified threshold. Performance metrics indicate that this intervention leads to an increase in exact agreement by 22-36 percentage points on the A10G benchmark and 12-21 percentage points on the L4 and A100 benchmarks. The latency overhead for this intervention is reported to be less than 4% during low-batch (batch size <= 4) single-stream inference. The applicability of the method is mapped across six models and four different batch sizes, with the hypothesis that training-time precision stability is a critical factor in performance.

Results

The study finds that 49-100% of prompts exhibit divergence under greedy decoding conditions. Furthermore, all five testable predictions regarding the outcomes of the proposed intervention were confirmed, demonstrating the effectiveness of the method in reducing output divergence.

Limitations

The authors acknowledge that their method serves as a partial mitigation strategy and does not guarantee universal determinism in output. They note that the benefits of their approach diminish when body-originated error becomes predominant, particularly at batch sizes of 8 or greater and under end-to-end FP8 precision settings.

Why it matters

This work has significant implications for the deployment of large language models in real-world applications, where precision and output consistency are critical. By demonstrating that greedy decoding is not precision-invariant and providing a method to address this issue, the authors pave the way for more robust LLM inference strategies. This could lead to improved performance in applications requiring high reliability and accuracy, influencing future research on model training and inference optimization.