JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management

Problem

This work addresses the significant memory constraints faced when running open-weight models for coding and reasoning tasks on local devices, specifically laptops. The authors highlight the challenges of managing large context sizes while maintaining performance, particularly in environments with limited memory resources. The paper is a preprint and has not undergone peer review.

Method

The authors propose an MLX-based inference runtime that incorporates several key components to optimize memory usage and processing efficiency:

• KVExec: A mechanism for compressed key-value execution, which reduces memory overhead during inference.
• PhaseSwap: This component manages residency of model components, allowing for efficient transitions between different operational phases without excessive memory usage.
• StateTrans: Facilitates state-preserving transitions during serving, ensuring that the model can maintain context across requests.

The method employs just-in-time materialization and release of resources, which operates independently of model-weight quantization, allowing for flexible memory management. The specific training compute used for the model is not disclosed, but the data source is identified as Qwen3.8-27B MXFP4. The execution capacity was tested on a 24 GiB M4 Pro MacBook.

Results

The results demonstrate significant improvements in context handling and processing speed:

• Single-Request Context: The system successfully completed a single request with a context of 212,992 positions, outperforming the mlx-vlm baseline, which managed only 30,720 positions (a 6.93x improvement).
• Two-Request Run Aggregate Context: The system achieved a total context of 229,376 positions across two requests.
• Performance Test: In a performance evaluation with a 32K-input and 64-output probe, the system processed at a rate of 19.11 tokens per second.
• AIME 2026 Problems: The model correctly answered 29 out of 30 problems presented in the AIME 2026 benchmark.
• Median Peak Process Footprint: The median peak memory footprint during repeated workloads of 32K+6K was recorded at 16,374 MiB.

Limitations

The authors do not report any limitations in their work. However, the absence of disclosed training compute details may limit reproducibility and understanding of the model's performance characteristics.

Why it matters

The implications of this work are significant for the deployment of large language models in resource-constrained environments. By enabling the processing of larger contexts on standard laptops, this research paves the way for more accessible AI applications in coding and reasoning tasks, potentially democratizing access to advanced AI capabilities. The techniques developed could inspire further optimizations in model serving and memory management, influencing future research and practical implementations in the field.