SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

Problem
This paper addresses the challenge of parameter-efficient fine-tuning in text-to-image diffusion models, specifically focusing on the difficulties of composing multiple custom concepts without incurring representation interference. Existing methods either require expensive post-hoc fusion techniques or involve freezing adaptation subspaces, which can limit the expressiveness and fidelity of the generated concepts. The authors propose a novel approach, SeqLoRA, to mitigate these issues. This work is presented as a preprint and has not yet undergone peer review.

Method
SeqLoRA introduces a constrained continual learning framework that employs bilevel optimization to jointly optimize Low-Rank Adaptation (LoRA) factors. The architecture leverages a matrix sub-Gaussian process to model residual layer activations, allowing the authors to derive high-probability bounds on catastrophic forgetting. The method is designed to minimize residual interference energy more effectively than traditional frozen-basis approaches. The training process involves optimizing two levels: the upper level focuses on the overall performance of the model, while the lower level fine-tunes the LoRA parameters. The authors provide theoretical convergence guarantees for their algorithm, ensuring stability and reliability in the optimization process.

Results
The experimental evaluation of SeqLoRA demonstrates significant improvements in identity preservation and scalability when generating images across multiple concepts. Specifically, the method was tested on a dataset involving up to 101 distinct concepts. Compared to baseline methods, SeqLoRA achieved a reduction in attribute interference and eliminated the need for costly fusion techniques. The authors report that SeqLoRA outperforms existing modular methods, although specific quantitative metrics and comparisons to named baselines are not detailed in the abstract.

Limitations
The authors acknowledge that while SeqLoRA effectively reduces representation interference and improves scalability, it may still be susceptible to certain forms of catastrophic forgetting, particularly in highly complex scenarios with overlapping concepts. Additionally, the theoretical guarantees provided are based on specific assumptions about the data distribution, which may not hold in all practical applications. The paper does not address the computational overhead associated with the bilevel optimization process, which could be a concern for real-time applications.

Why it matters
The implications of SeqLoRA extend to various applications in generative modeling, particularly in scenarios requiring the integration of multiple concepts without loss of fidelity. By providing a more efficient framework for continual learning in text-to-image generation, this work paves the way for advancements in personalized content creation, interactive AI systems, and other domains where multi-concept representation is crucial. The theoretical foundations laid out in this paper also contribute to the broader understanding of optimization in neural networks, particularly in the context of minimizing interference in learned representations.

Authors: Javad Parsa, Enis Simsar, Amir Joudaki, Thomas Hofmann, André M. H. Teixeira
Source: arXiv:2605.22743
URL: https://arxiv.org/abs/2605.22743v1