Knowledge Editing in Masked Diffusion Language Models

Problem
Knowledge editing, the process of updating or correcting factual information in language models, has primarily been studied in autoregressive models (ARMs). The existing literature lacks exploration of knowledge editing in masked diffusion models (MDMs), which operate bidirectionally and generate text through iterative denoising. This paper addresses the gap by investigating whether the locate-then-edit approach, effective in ARMs, can be successfully applied to MDMs. The authors aim to determine if the assumptions underlying knowledge editing hold in this new paradigm, a question that remains unaddressed in prior research.

Method
The authors adapt the locate-then-edit framework for MDMs, specifically applying it to two MDMs: LLaDA and Dream, and comparing their performance against two ARMs: LLaMA and Qwen, all at matched scales. The core technical contribution involves analyzing the causal tracing of edits to identify effective layers for knowledge modification. The study reveals that the most effective editing occurs in the early-to-mid-layer MLP at the last subject token across both model types. However, the authors find that while single-token edits are successful in both paradigms, multi-token edits degrade in MDMs due to the necessity of passing through unmasked intermediate states that were not optimized for the edit. To address this, the authors propose a corrective method that optimizes the edit for these intermediate states, thereby enhancing multi-token performance.

Results
The findings indicate that while single-token edits achieve comparable success rates in both MDMs and ARMs, the performance of multi-token edits diverges significantly. Specifically, the MDMs exhibit systematic degradation in editing effectiveness as the target length increases, a trend not observed in the ARMs. The proposed correction restores multi-token performance in MDMs, although specific quantitative results and benchmarks are not detailed in the abstract. The paper emphasizes the importance of layer selection and optimization strategies in achieving effective knowledge editing across different model architectures.

Limitations
The authors acknowledge that their findings are limited to the specific MDMs and ARMs tested, and the generalizability of the results to other architectures or larger datasets remains uncertain. Additionally, the paper does not explore the long-term implications of knowledge editing on model performance or the potential for unintended consequences in knowledge representation. The study is also a preprint and has not undergone peer review, which may affect the robustness of the conclusions drawn.

Why it matters
This research has significant implications for the development of knowledge editing techniques in MDMs, highlighting the need for tailored optimization strategies when transitioning from ARMs. The findings suggest that while knowledge editing can be adapted across model types, the underlying mechanisms and performance characteristics differ, necessitating further investigation. This work contributes to the broader understanding of how different model architectures handle knowledge representation and modification, as published in arXiv.