IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck

• Significant concerns with the proposed method:
  • Circuit analysis does not always aim to maximize task performance: This paper claims (200-201) that "Circuit Identification aims to identify the most crucial components within a model, striking a balance between minimizing the number of components and maximizing their ability to perform tasks". This is false. The papers and methods it cites seek to do something (perhaps subtly) different: find all components (and edges) that are important to task performance, whether their contribution is positive or negative. See for example Wang et al.'s (2022) discovery of negative name mover heads, and Syed et al.'s (2023) use of absolute attribution scores. That is, we care about components that directly harm task performance as well. As the authors themselves state, IBCircuit is interested in maximizing model performance, and will likely miss these components.
  • The role of counterfactual examples: It is nice that this method requires no counterfactual examples, and the gaussian noise injection is carefully done so as to avoid out-of-distribution issues. However, I wonder whether this method isn't computing something different than what other circuit-finding methods find. They use counterfactual patching, using examples drawn from the same task; therefore, when they perform patching, they bake in the assumption that the model is performing a given task, and find only the circuit responsible for processing task examples. This method does not bake in that assumption, and will presumably also find components necessary for identifying the task as well. This is worth thinking about and discussing.
• Large differences with other papers' conceptions of circuits: The circuit papers that this paper compares to (Wang et al., 2023; Hanna et al. (2023)), as well as much of the related work (ACDC, edge attribution patching) conceive of the model's computational graph as a graph $G=(V,E)$, and a circuit as a subgraph of that. Notably, a graph includes both nodes and edges, but IBCircuit works only at the node level. I guess the authors used a node-level adaptation of attribution patching (and ACDC?) when evaluating, as is necessary for a fair comparison. But regardless of whether they did, IBCircuit, working on nodes alone, seems much less powerful than the other methods which can find edges as well.
• Limited / flawed evaluation:
  • Flawed evaluation methodology: The evaluations of IBCircuit are pretty flawed and mixed in results:
    • Due to the aforementioned node vs. edge issue, IBCircuit (a node based method) compares against edge-based methods, and it's not clear if they (especially ACDC) have been adapted into node-based variants. Moreover, the evaluation performed (node-based ablations) are thus different from how those original papers evaluated their results. We can see this discrepancy clearly in Figure 2(b), where attribution patching has very poor performance, while in the original paper that introduced it, the same curve (for the same task) looks much better, for both attribution patching and ACDC. I'm not sure if this is due to the authors' implementation of attribution patching, because of the node-based ablations, or something else, but something is wrong here.
    • The authors report logit and probability difference for IOI and greater-than, but don't report the base model's performance on each. It would be good to do so, because oftentimes circuit work aims to find a circuit that replicates the original model's performance; thus, as discussed above, higher performance isn't always better.
    • The authors invent a new metric (frac_outperformance) in Figure 3, but it's not clear why it's valuable. As discussed earlier, we don't really want the circuit to outperform the whole model. Indeed when ablation is performed using corrupted examples + patching, as this paper does, it's easy to get high performance at low node counts by ablating nodes (like negative name mover heads) that are important, as this will force those negative heads to behave as positive heads. But this isn't really desirable; it actually means you have missed many important components.
    • IBCircuit's performance across Greater-Than tasks seems pretty middling, even by this paper's flawed metrics. Evaluating on other tasks (see Hanna et al. (2024) for other circuit-related tasks) would more convincingly prove IBCircuit's value.
  • No discussion of or comparison to recent circuit-finding methods: Beyond the circuit-finding methods (ACDC and attribution patching) currently discussed, there are many new ones that go unmentioned. Bhaskar et al. (2024), for example, use a learnable mask over model edges to find circuits, which also answers the objection about methods that consider nodes/edges independently; see similar ideas from Chintam et al. (2023). Hanna et al. (2024) and Marks et al. (2024) introduce new versions of (edge) attribution patching that improve its accuracy as well. While Bhaskar et al. is somewhat new (3 months old), the edge attribution papers have been out for half a year, and are definitely worth discussing, and even comparing against.
• Unsupported claims about factual recall: In the abstract (23-34), the authors claim "the results from IBCircuit suggest that the earlier layers in Transformer-based models are crucial in capturing factual information"; a similar claim appears in the conclusion. But no evidence of this appears in the text; no factual recall tasks are even studied.
• Proofreading: In general, the manuscript could use some proofreading; see the list of typos below.

Overall, this paper overlooks many theoretical concerns about circuit finding, and performs insufficient evaluation to prove that its proposed method is actually better / somehow more useful than prior methods.

• The authors should quantify the efficiency gain of IBCircuit as it is one of the main claims in the paper. For example, the authors should provide runtime comparison between different circuit discovery methods to support the claim.
• The paper is missing a highly relevant work: Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models by O'Neill and Bui, where they use SAE to identify holistic circuits in transformers efficiently. Due to the high relevance, the authors should specifically compare the key technical differences between IBCircuit and the SAE approach, and discuss potential advantages or disadvantages of each method, and incorporate the prior work as one of the baselines.
• Since IBCircuit involves model training and hyperparameter tuning (i.e., the $\alpha$ in the objective function), the authors should conduct an ablation study showing how different values of $\alpha$ affect the performance and characteristics of the identified circuits.
• The paper didn't discuss the reproducibility of the method or plans to release. It's hard to evaluate the quality of their code for reproducibility purpose either since it's not provided as one of the supplementary files.
• Writing skill could be improved: a lot of grammatical/formatting errors found and the flow is not fluent. (See minor issues in the questions section). Readers might find certain parts hard to understand as a result.
• The paper is missing one of the standard tasks often evaluated in circuit discovery work, the docstring task. The authors should consider including the docstring task to be more comparable to prior work.
• In addition to comparisons between the identified circuits and the original models, the authors should also consider including another important effectiveness experiment: comparisons between the identified circuits with random circuits as random circuits are often a strong baseline.

As mentioned in the "Summary" section, the claim about the internal organisation of the model is not substaintated in the main content section. The authors claimed that "the earlier layers in Transformer-based models are crucial in capturing factual information," but this is not discussed anywhere in the remainder of the paper.

Furthermore, this claim directly contradcits Meng et al.'s (2022) hypothesis, which has been cited by the authors. They claimed that factual information is located in the middle layers. Let alone the discussion on "BERT rediscovers the classical NLP pipeline" (Tenney et al., 2019) This layer view of the model is still a highly controversial topic.

W1. The innovation in this work is limited. The method of using Gaussian noise for perturbation has been applied in prior work, such as the referenced paper by Yu et al. (2022), particularly in sections 4.2.1 and 4.2.2.Although the authors mention being inspired by prior work, the core contribution is too similar to it.

W2. I believe that circuit discovery shares similarities with several existing tasks, including explainability in graph networks. Integrating key ideas from leading methods in related tasks into circuit discovery is valuable; however, I would have liked to see more substantial contributions beyond adapting these existing approaches.

W3. The paper claims the method can scale to larger models, but lacks experiments to support this.