KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

Problem
This paper addresses the inefficiencies in communication during disaggregated large language model (LLM) serving, particularly focusing on key-value (KV) cache compression. Existing methods for KV compression are static and do not adapt to the dynamic nature of production workloads, which can lead to suboptimal performance and increased latency. The authors present KVServe, a novel framework that is service-aware and adaptive, aiming to optimize KV communication in real-time. This work is a preprint and has not yet undergone peer review.

Method
KVServe introduces a modular strategy space for KV compression, allowing for the integration of new components and cross-method recomposition. The framework consists of three main components: (1) a unified KV compression strategy space that accommodates various compression techniques; (2) a Bayesian Profiling Engine that efficiently explores this space, significantly reducing offline search overhead by a factor of 50, and distilling a 3D Pareto candidate set; and (3) a Service-Aware Online Controller that utilizes an analytical latency model combined with a lightweight bandit algorithm to select optimal profiles under operational constraints, addressing the mismatch between offline and online performance. The framework is integrated into vLLM and evaluated across multiple datasets, models, GPUs, and network configurations.

Results
KVServe demonstrates substantial performance improvements over baseline methods. In experiments, it achieves up to 9.13× speedup in job completion time (JCT) for PD-separated serving and up to 32.8× reduction in total time to first token (TTFT) for KV-disaggregated serving. These results indicate a significant enhancement in efficiency and responsiveness compared to existing static KV compression techniques, although specific baseline methods are not detailed in the abstract.

Limitations
The authors acknowledge that the performance gains are contingent on the accuracy of the analytical latency model and the effectiveness of the bandit algorithm in real-time scenarios. They also note that the framework’s performance may vary with different workload characteristics and system configurations, which could limit generalizability. Additionally, the reliance on offline profiling may introduce latency in dynamic environments where workload characteristics change rapidly. The paper does not address potential overheads associated with the Bayesian profiling process during initial deployment.

Why it matters
KVServe’s adaptive and service-aware approach to KV cache compression has significant implications for the deployment of LLMs in production environments. By optimizing communication efficiency, it can reduce operational costs and improve user experience through faster response times. This work lays the groundwork for future research into dynamic resource allocation and adaptive systems in AI serving architectures, potentially influencing the design of more responsive and efficient inference systems.

Authors: Zedong Liu, Xinyang Ma, Dejun Luo, Hairui Zhao, Bing Lu, Wenjing Huang, Yida Gu, Xingchen Liu et al.
Source: arXiv:2605.13734
URL: https://arxiv.org/abs/2605.13734v1