Where Should Knowledge Enter? A Layered Framework for Knowledge Infusion in Multimodal Iterative Generative Mo

Problem
The paper addresses the gap in existing multimodal generative models that struggle to reliably incorporate structured, domain-specific, or safety-critical knowledge during generation. Current methods, such as prompt augmentation and fine-tuning, lack a systematic categorization based on the generative process components they modify. This work is a preprint and has not undergone peer review.

Method
The authors propose a layered framework for knowledge infusion in iterative generative models, specifically targeting diffusion models. They identify four distinct intervention layers: surface infusion (input/output boundary), trajectory infusion (transition function), latent infusion (intermediate state), and parametric infusion (model parameters). The framework is instantiated using two diffusion backbones and a multimodal knowledge graph. The authors implement three of the four layers—surface (both input-side and output-side) and trajectory-latent (mid-generation)—to evaluate their effectiveness in reducing knowledge-violating outputs.

Results
In a controlled safety-alignment experiment, the proposed framework demonstrated a significant reduction in knowledge-violating outputs, achieving a 70.97% decrease compared to vanilla generation. Each additional layer of knowledge infusion was shown to address specific failure classes that prior layers could not, empirically confirming the framework’s complementarity prediction. The results indicate that the layered approach effectively enhances the reliability of generative outputs in knowledge-sensitive contexts.

Limitations
The authors acknowledge that the framework does not include parametric infusion, which may limit the overall effectiveness of knowledge integration. Additionally, the study is constrained by the specific diffusion models and knowledge graph used, which may not generalize to all multimodal generative scenarios. The lack of peer review also raises questions about the robustness of the findings.

Why it matters
This work has significant implications for the development of reliable multimodal generative models, particularly in applications requiring adherence to structured knowledge, such as healthcare and legal domains. By providing a systematic approach to knowledge infusion, the framework can guide future research in enhancing the safety and reliability of generative models. The findings contribute to the ongoing discourse on improving generative model performance in knowledge-sensitive applications, as published in arXiv.