MiniPIC: Flexible Position-Independent Caching in <100LOC

Problem
This paper addresses the limitations of existing prefix caching mechanisms in retrieval-augmented workloads, particularly in the context of vLLM (variable-length Language Model) engines. Traditional prefix caching requires identical prefixes for cache reuse, which is inefficient for workloads with recurring structured inputs, termed “spans.” Additionally, existing Position-Independent Caching (PIC) implementations necessitate significant code modifications or external KV state management, leading to performance bottlenecks due to host-to-device transfer overhead. The authors present MiniPIC as a solution to these challenges, aiming to provide a minimalistic and flexible caching approach without extensive server changes.

Method
MiniPIC is designed with two core components: a positional-encoding-free KV cache and user-controlled cache-reuse primitives. The architecture allows for the storage of unrotated K vectors in the KV cache, while applying Rotary Positional Encoding (RoPE) to K tiles during attention computation using logical positions specific to each request. The system introduces three user-facing primitives: block-aligned padding, span separator (SSep), and prompt depend (PDep). These primitives enable modifications to the hashing behavior and the effective block-level causal attention structure, facilitating the realization of multiple PIC methods (e.g., Block-Attention, EPIC, and Prompt Cache) within a single vLLM instance. The implementation requires fewer than 100 lines of core-engine modifications and includes a custom attention backend, ensuring compatibility with existing KV cache CPU offload strategies.

Results
MiniPIC demonstrates significant performance improvements on the 2WikiMultihopQA benchmark. Specifically, it achieves a 49% increase in prefill throughput compared to the baseline vLLM. Additionally, it reduces the cached-span time-to-first-token by up to two orders of magnitude, while maintaining linear prefill scaling for uncached spans. The worst-case overhead incurred by MiniPIC is only 5.7%, indicating a highly efficient implementation that does not compromise performance.

Limitations
The authors acknowledge that while MiniPIC offers substantial improvements, it may still be limited by the underlying architecture of the vLLM engine and the specific workloads it is applied to. The paper does not extensively evaluate the performance across a diverse set of benchmarks or real-world applications, which could provide further insights into its generalizability. Additionally, the reliance on user-controlled primitives may introduce complexity for users unfamiliar with the underlying mechanisms.

Why it matters
The introduction of MiniPIC has significant implications for the efficiency of retrieval-augmented workloads, particularly in scenarios requiring rapid access to structured inputs. By minimizing the need for extensive code changes and optimizing cache reuse, MiniPIC can enhance the performance of various applications in natural language processing and beyond. This work contributes to the ongoing discourse on efficient caching strategies in large language models, as discussed in related literature, and is available on arXiv.