Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery

Problem
This work addresses a significant gap in the understanding of circuit discovery methods in neural networks, particularly the assumption that structural differences in discovered circuits imply distinct functional mechanisms. The authors highlight that existing literature often interprets these structural variations as evidence of specialization, which may not hold true. This paper is a preprint and has not undergone peer review, indicating that the findings should be interpreted with caution.

Method
The authors employ a systematic approach using Literal Sequence Copying across four token-frequency bands and a control condition, utilizing five Pythia models ranging from 70M to 1.4B parameters. They extract a total of 75 circuits to analyze the relationship between structure and function. The methodology includes varying input statistics while maintaining a fixed task to observe the resulting structural differences. The authors conduct causal interchange interventions to assess the interchangeability of internal representations across frequency bands. They also perform repeated extractions within the same frequency band to investigate the sampling behavior of discovery algorithms, suggesting that these algorithms may sample from an equivalence class of valid subgraphs rather than identifying unique mechanisms.

Results
The findings reveal that structurally distinct circuits, despite their apparent specialization, implement the same computation across different frequency bands. Specifically, the core shared structure across bands recovers at least 99% of circuit performance, indicating a high degree of functional overlap. The authors demonstrate that edge-level evaluation uncovers a many-to-one mapping from structure to function, contrasting with source-level evaluations that inflate the perceived faithfulness of the circuits. This nuanced analysis challenges the conventional interpretation of circuit discovery results, emphasizing the need for more granular evaluation methods.

Limitations
The authors acknowledge that their findings are contingent on the specific models and tasks used in their experiments, which may limit the generalizability of the results. They also note that the reliance on edge-level evaluation may not be feasible in all contexts, potentially complicating the application of their insights to other circuit discovery methodologies. Additionally, the study does not explore the implications of phantom specialization in other types of neural architectures beyond the Pythia models.

Why it matters
This research has significant implications for the field of interpretability in machine learning, particularly in understanding the relationship between model structure and function. By introducing the concept of phantom specialization, the authors encourage a reevaluation of standard practices in circuit discovery and evaluation. Their findings suggest that researchers should adopt edge-level evaluations and cross-condition transfer tests to gain a more accurate understanding of model behavior. This work contributes to the ongoing discourse on the interpretability of neural networks and the reliability of circuit discovery methods, as published in arXiv cs.CL.