ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

Problem
This paper addresses the gap in memory efficiency during full-parameter fine-tuning of large language models, specifically focusing on the limitations of existing methods that require substantial GPU memory and computational resources. The authors propose a novel framework, ChunkFT, which is particularly relevant for practitioners working with large models like Llama 3-8B and Llama 3-70B. Notably, this work is a preprint and has not undergone peer review.

Method
ChunkFT introduces a memory-efficient fine-tuning approach that reformulates the full-parameter fine-tuning process by utilizing a dynamically activated working set. This method allows for gradient computation on arbitrary sub-tensors without necessitating changes to the underlying network architecture. The authors provide a theoretical convergence analysis in a deterministic setting, ensuring that the proposed method is grounded in solid mathematical principles. The empirical evaluation involves fine-tuning Llama 3-8B on a single RTX 4090-24GB GPU and Llama 3-70B on two H800-80GB GPUs. The memory footprint for fine-tuning a 7B model with a 1K input length is reported to be only 13.72GB, showcasing the efficiency of the ChunkFT framework.

Results
ChunkFT demonstrates significant improvements in memory usage, running time, and optimization quality compared to existing memory-efficient baselines. The authors report that their method achieves performance levels comparable to, and in some cases exceeding, those obtained through full-parameter fine-tuning. Specific downstream evaluations on tasks such as language understanding, mathematical reasoning, and MT-Bench indicate that ChunkFT consistently outperforms traditional methods. While exact numerical results are not provided in the abstract, the implications suggest a substantial effect size in favor of ChunkFT over named baselines.

Limitations
The authors acknowledge that while ChunkFT is effective, it may not be universally applicable to all model architectures or tasks, particularly those that require extensive modifications to the network structure. Additionally, the theoretical convergence analysis is limited to deterministic settings, which may not fully capture the complexities of stochastic training environments. The paper does not address potential scalability issues when applied to even larger models or datasets beyond those tested.

Why it matters
The implications of ChunkFT are significant for the field of NLP and large-scale model fine-tuning. By enabling efficient full-parameter fine-tuning with reduced memory requirements, this framework opens avenues for researchers and engineers to work with larger models on less powerful hardware. This democratization of access to advanced model fine-tuning could accelerate research and application development in various domains, including language understanding and reasoning tasks. Furthermore, the theoretical foundation and empirical validation provided by the authors contribute to the ongoing discourse on optimizing large-scale models in a resource-constrained environment.

Authors: Yongkang Liu, Zijing Wang, Mengjie Zhao, Ercong Nie, Mingyang Wang, Qian Li, Feiliang Ren, Shi Feng et al.
Source: arXiv cs.CL
URL: https://arxiv.org/abs/2605.21177v1