Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection

Problem

The paper addresses the misclassification of harmful memes by vision-language models, specifically highlighting issues related to missing internal evidence and routing problems. This work is particularly relevant as it is a preprint and has not undergone peer review, indicating that the findings should be interpreted with caution.

Method

The authors evaluate two models, Gemma-3 and Qwen3.5, employing a variety of techniques including sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments. They benchmark their methods against six harmful content datasets, with additional evaluations in Spanish and Hindi-English code-mixed contexts. Performance metrics include macro-F1 scores and sensitivity metrics, with notable findings:

• Qwen macro-F1: 0.740 (sparse readouts) vs 0.432 (native)
• Gemma macro-F1: improved from 0.532 to 0.714
• Residual reconstruction: 0.486
• Sensitivity Metrics: Qwen silent-feature ablation shows 24-63 times more probe sensitivity, while routed-feature patching on yes/no tasks demonstrates 16-140 times more output sensitivity.
• Calibration-only routing recovery achieves 93.3% of the mean gap.
• Probe-distilled LoRA is introduced to enhance native predictions.
• A case study on Gemma-3-12B using the Facebook Hateful Memes dataset shows a macro-F1 score of 0.756 compared to 0.685 for native predictions.

Results

The results indicate significant improvements in harmful meme detection:

• Qwen macro-F1: 0.740 (sparse) vs 0.432 (native)
• Gemma macro-F1: 0.714 (sparse) vs 0.532 (native)
• Residual reconstruction: 0.486 (no baseline reported)
• Qwen silent-feature ablation sensitivity: 24-63 times more sensitive (no baseline reported)
• Routed-feature patching sensitivity: 16-140 times more sensitive (no baseline reported)
• Calibration-only routing recovery: 93.3% of the mean gap (no baseline reported)
• Gemma-3-12B on Facebook Hateful Memes: 0.756 vs 0.685 (native) The available text does not report quantitative results for other metrics.

Limitations

The authors note several limitations, including the negative transfer observed with shared multi-task adaptation and the models' dependence on paired visual evidence for robustness. These factors may hinder the generalizability of the findings across different contexts or datasets.

Why it matters

This research has significant implications for the development of more robust vision-language models, particularly in the context of harmful content detection. By addressing the identified readout gap, the proposed methods could enhance the accuracy and reliability of these models, which is crucial for applications in content moderation and safety in digital environments. The findings may also inform future work on model interpretability and the integration of multimodal data.