Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline

Problem
This paper addresses the gap in effective fine-tuning techniques for flow-based generative models, specifically focusing on human preference alignment. The authors propose a novel deterministic adjoint matching framework that formulates this alignment as an optimal control problem over velocity fields. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution is the introduction of a deterministic adjoint matching framework that allows for direct regression of control towards a value-gradient-induced target under the current policy. This approach leads to a stable training objective that simplifies the fine-tuning process. The authors implement a truncated adjoint scheme that concentrates computational resources on the terminal portion of the trajectory, where reward-relevant signals are most concentrated. This results in significant computational savings while maintaining alignment quality. Additionally, the framework generalizes beyond traditional KL-based regularization, enabling more flexible trade-offs between alignment strength and distributional preservation. The training process leverages flow models SiT-XL/2 and FLUX.2-Klein-4B, although specific details on training compute are not disclosed.

Results
The proposed method demonstrates consistent improvements across multiple alignment metrics when evaluated on the SiT-XL/2 and FLUX.2-Klein-4B benchmarks. The authors report substantial gains in diversity and mode preservation compared to baseline models, although specific numerical results and effect sizes are not detailed in the abstract. The improvements suggest that the adjoint matching framework effectively enhances the performance of flow-based generative models in aligning with human preferences.

Limitations
The authors acknowledge that their approach may be limited by the assumptions inherent in the optimal control formulation and the reliance on the quality of the underlying flow model. They do not discuss potential challenges related to the scalability of the method or its applicability to other generative model architectures. Additionally, the computational savings achieved through the truncated adjoint scheme may vary depending on the specific characteristics of the training data and the complexity of the target distribution.

Why it matters
This work has significant implications for the field of generative modeling, particularly in enhancing the alignment of models with human preferences, which is crucial for applications in content generation, interactive systems, and AI ethics. By providing a more efficient and flexible framework for fine-tuning flow models, the authors pave the way for future research that can build upon their findings to develop more robust generative systems. The generalization of the framework beyond KL-based regularization also opens avenues for exploring new alignment strategies that could further improve model performance in diverse applications.

Authors: Zhengyi Guo, Jiayuan Sheng, David D. Yao, Wenpin Tang
Source: arXiv:2605.06583
URL: https://arxiv.org/abs/2605.06583v1