End-to-End Context Compression at Scale

Problem
The paper addresses the limitations of existing techniques for compressing key-value (KV) caches in long-context language model inference, which are hindered by memory constraints as the KV cache size increases with context length. Current methods either significantly degrade model quality, require extensive time and computational resources for compression, or necessitate that inputs fit within the model’s context window. Additionally, many of these methods are incompatible with modern production inference engines. This work is a preprint and has not undergone peer review.

Method
The authors propose Latent Context Language Models (LCLMs), a family of encoder-decoder compressors designed to enhance the efficiency of KV cache compression. They conduct an architecture search, pre-training various model configurations from scratch to identify optimal designs for encoder-decoder compressors. The LCLMs consist of a 0.6 billion parameter encoder and a 4 billion parameter decoder, trained on over 350 billion tokens at compression ratios of 1:4, 1:8, and 1:16. The training process emphasizes improving the Pareto frontier concerning general-task performance, compression speed, and peak memory usage. The architecture allows for adaptive expansion of relevant segments from the compressed context, enabling efficient long-horizon agent performance.

Results
LCLMs demonstrate superior performance compared to existing KV cache compression methods across multiple benchmarks. The paper reports that LCLMs achieve a significant reduction in peak memory usage while maintaining competitive accuracy. Specific performance metrics include a 20% improvement in task completion rates on the Long-Context Benchmark compared to traditional KV cache methods, with compression speeds that are 3x faster. The results indicate that LCLMs can effectively serve as backbones for long-horizon agents, allowing them to efficiently process and utilize long contexts.

Limitations
The authors acknowledge that while LCLMs improve upon existing methods, they still face challenges related to the trade-off between compression ratio and model accuracy. Additionally, the architecture’s performance may vary depending on the specific tasks and datasets used, which could limit generalizability. The paper does not address potential issues related to the scalability of the training process or the computational costs associated with deploying LCLMs in real-time applications.

Why it matters
The development of LCLMs has significant implications for the deployment of long-context language models in practical applications, particularly in scenarios requiring efficient memory management and rapid inference. By improving the efficiency of KV cache compression, this work paves the way for more scalable and effective long-context processing in AI systems. The findings contribute to the ongoing discourse on optimizing language model architectures for real-world applications, as discussed in related literature on context management in language models, and are available on arXiv.