Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation

Problem
This paper addresses the limitations in existing subject-driven image generation methods, which typically encode text and reference images separately. This separation hinders effective cross-modal reasoning and leads to artifacts such as copy-paste issues. While recent frameworks have attempted to integrate multimodal models with diffusion models to enhance instruction adherence, they often neglect the critical aspect of identity preservation. The authors present a novel approach that leverages Multimodal Large Language Models (MLLMs) to jointly encode text and images, thereby filling this gap in the literature. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose a novel architecture that conditions diffusion models on MLLMs, which are designed to jointly process textual and visual inputs. A key innovation is the Dual Layer Aggregation (DLA) module, which aggregates features from multiple levels of the MLLM to optimize the conditioning process. Additionally, the model employs a Variational Autoencoder (VAE) for identity conditioning, ensuring that the generated images maintain the identity of the subject while adhering to textual instructions. The training process involves a multi-stage denoising strategy that progressively balances the semantic information derived from the MLLM with the fine-detail identity information provided by the VAE. The paper does not disclose specific training compute resources.

Results
The proposed method demonstrates significant improvements over baseline models in subject-driven image generation tasks. The authors report that their approach achieves superior performance in terms of human preference evaluations, although specific quantitative metrics (e.g., FID scores, accuracy) are not detailed in the abstract. Comparisons are made against existing multimodal and diffusion-based models, highlighting the effectiveness of the DLA module and the integrated conditioning strategy. The results indicate a marked reduction in copy-paste artifacts and improved identity preservation, although exact effect sizes are not provided.

Limitations
The authors acknowledge that their approach may still face challenges in extreme cases of identity preservation, particularly with highly complex subjects or ambiguous textual instructions. They also note that the reliance on MLLMs may introduce biases inherent to the training data of these models. Additionally, the paper does not discuss the computational efficiency of the proposed method, which could be a concern for real-time applications. The lack of extensive quantitative results in the abstract may also limit the ability to fully assess the model’s performance.

Why it matters
This work has significant implications for the field of multimodal AI, particularly in applications requiring high fidelity in image generation while adhering to specific textual instructions. By effectively integrating MLLMs with diffusion models, the proposed method enhances the capability of AI systems to generate contextually relevant and identity-preserving images. This advancement could facilitate further research into more sophisticated multimodal interactions and applications, such as personalized content creation, virtual reality, and interactive storytelling.

Authors: Shuhong Zheng, Aashish Kumar Misraa, Yu-Teng Li, Yu-Jhe Li, Igor Gilitschenski
Source: arXiv:2605.26111
URL: https://arxiv.org/abs/2605.26111v1