GKnow: Measuring the Entanglement of Gender Bias and Factual Gender

Problem
This preprint addresses a significant gap in the literature regarding the mechanistic understanding of gender bias in neural networks. Previous studies have either concentrated on specific gender-related tasks, such as gendered pronoun prediction, or have not effectively differentiated between factually gendered outputs (correctly identifying gender based on semantic properties) and gender-biased outputs (which stem from stereotypes). The authors propose a new benchmark, GKnow, to systematically evaluate gender knowledge and bias in language models across various gender-related predictions.

Method
The core technical contribution is the curation of the GKnow benchmark, which facilitates the assessment of gender knowledge and bias in language models. The authors employ neuron ablation techniques to investigate the circuits and individual neurons responsible for gendered predictions. They utilize existing benchmarks, including DiFair and the test set of GKnow, alongside StereoSet, to evaluate the impact of neuron ablation on the disentanglement of factual gender knowledge from stereotypical gender bias. The methodology emphasizes the need for a nuanced approach to understanding how gender bias and factual gender knowledge are intertwined within neural architectures.

Results
The results indicate a strong entanglement between gender bias and factual gender knowledge at both the circuit and neuron levels. Specifically, the ablation of neurons intended to reduce gender bias led to a significant decrease in factual gender knowledge, suggesting that traditional debiasing methods may be ineffective. The authors report that benchmarks designed to evaluate gender bias can obscure the loss of factual gender knowledge that occurs with neuron ablation. Quantitative results demonstrate that the performance on DiFair and GKnow’s test set deteriorates significantly post-ablation, highlighting the limitations of current debiasing strategies.

Limitations
The authors acknowledge that their findings may not generalize across all language models, as the study primarily focuses on specific architectures. Additionally, the reliance on neuron ablation as a method for evaluating bias may not capture the full complexity of gender representation in language models. They also note that while GKnow provides a more comprehensive framework for assessing gender bias, it is still a work in progress and may require further refinement and validation across diverse datasets and tasks. An obvious limitation not discussed is the potential for overfitting to the specific benchmarks used, which may not represent real-world applications.

Why it matters
This work has significant implications for the ongoing development of robust benchmarks for evaluating gender bias in AI systems. By disentangling factual gender knowledge from gender bias, the authors provide a clearer framework for understanding how language models process gender-related information. This understanding is crucial for developing more equitable AI systems and for informing future research on bias mitigation strategies. The GKnow benchmark can serve as a foundational tool for researchers aiming to enhance the interpretability and fairness of language models in gender-related tasks.

Authors: Leonor Veloso, Hinrich Schütze
Source: arXiv:2605.12299
URL: https://arxiv.org/abs/2605.12299v1