SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning

Problem
This paper addresses the limitations of existing parameter-efficient fine-tuning (PEFT) methods, particularly Low-rank Adaptation (LoRA), which struggles with representational capacity when decreasing the rank of updates. The authors highlight that while LoRA effectively reduces resource requirements, it converges towards the top r singular values of the pre-trained weight matrix, leading to suboptimal performance when the rank is too low. This work is presented as a preprint and has not yet undergone peer review.

Method
The core contribution of this paper is the Spectrum Modulation Adapter (SMoA), which enhances the representational capacity of PEFT while maintaining a lower parameter budget. SMoA partitions each layer into multiple aligned spectral blocks and applies a Hadamard-modulated low-rank branch to each diagonal block. This approach allows for a broader coverage of the pretrained spectral directions without significantly increasing the number of trainable parameters. The authors provide a theoretical framework supporting the efficacy of this method, although specific details regarding the architecture, loss functions, and training compute are not disclosed.

Results
SMoA demonstrates superior performance compared to LoRA and other competitive LoRA-style baselines across multiple tasks. The paper reports an average performance improvement in lower-budget settings, although specific metrics and benchmark names are not detailed in the abstract. The authors emphasize that SMoA achieves this enhancement without a proportional increase in computational cost, making it a viable alternative for fine-tuning large language models.

Limitations
The authors acknowledge that while SMoA improves upon LoRA, it still operates within the constraints of parameter-efficient methods, which may not fully capture the complexity of certain tasks. Additionally, the theoretical analysis provided may not encompass all practical scenarios, and the empirical results are limited to the tasks evaluated in the study. The lack of detailed benchmarking metrics and comparisons to a wider array of baselines is another limitation that could affect the generalizability of the findings.

Why it matters
The implications of this work are significant for the field of NLP and model fine-tuning. By providing a method that allows for efficient fine-tuning of large language models with a focus on spectral updates, SMoA could enable researchers and practitioners to achieve better performance on resource-constrained tasks. This advancement may facilitate the deployment of large models in environments with limited computational resources, thereby broadening the accessibility and applicability of state-of-the-art NLP technologies.

Authors: Yongkang Liu, Xing Li, Mengjie Zhao, Shanru Zhang, Zijing Wang, Qian Li, Shi Feng, Feiliang Ren et al.
Source: arXiv:2605.21147
URL: https://arxiv.org/abs/2605.21147v1