Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs

Thank you for your detailed comments and feedback!

We are glad that the novelty of our analysis and methodology came through clearly, and we appreciate the recognition of the strengths in our circuit comparison approach, the normalized IoU analysis, and the effectiveness of back-patching. Below, we address your concerns point by point.

1. "I wonder if the same technique (back-patching) can be used for textual inputs and we can still see performance improvement"

This is a great suggestion, which we indeed executed and reported in Appendix E.3 (referencing it in line 255). Since textual tokens are already “aligned with themselves”, gains here would reflect added compute. We find that visual back-patching yields larger improvements than textual back-patching in most settings, supporting our hypothesis that gains stem from improved alignment rather than additional depth alone. We will highlight this more clearly.

2. "I don't really immediately see why the proposed back-patching method is the logical solution to the circuit difference problem"

We appreciate your question, and want to clarify – the difference between circuits employed in the model to solve visual / textual tasks isn't necessarily a problem – it’s just a finding. This, in addition with our additional findings:

(1) the difference lies mainly in the data positions; and

(2) visual embeddings only converge towards the higher-performing textually-aligned embedding in later layers;

leads us to utilize the aligned embeddings from later layers earlier in the model, to allow for higher performance with a simple intervention. Per your comment, we will better clarify this motivation in the introduction section.

3. "I wonder if any of the non-adaptor architectures still have this property."

Most open-weight SOTA VLMs use adapter-based architectures, including those we study. Whether our findings extend to non-adapter architectures is an important direction for future work, and we’ll highlight this in the limitations section.

4. Regarding comments on presentation:

• "patching effects are not defined, I have to assume that it refers to the "attribution patching" in section 2.2": This is correct! We will point to section 2.2 in this figure to make this definition more clear.

• "in line 229 the authors referred to another work without clearly explaining "unembedding matrix": The unembedding matrix is another name for the language modeling head (or vocabulary projection), the last linear layer in the model that projects the embedding to the vocabulary space. Thank you for pointing this out - we will rephrase this line to avoid confusion.

• "the authors mentioned that the special visual tokens lead to the non-trivial positional mapping, but from my understanding, in the previous paragraph the authors already mentioned that they will not consider the data positions": This non-trivial mapping happens in few cases where a single token is mapped to multiple tokens. One such example is a visual prompt that has a "<vision_end>" token and a "\n" following token and the analog textual prompt containing a single '"""\n' token which we map to both visual tokens. In this case, we consider these tokens to be query tokens (as they don't contain any visual information), and thus the mapping between query tokens isn't always one-to-one. We will make this part more clear in the revision.

We hope our answers are satisfactory. Please let us know if you have any other questions or concerns.

Thank you for your valuable comments and feedback!

We appreciate you recognizing the novelty, elegance and generality of our interpretability-guided improvement to VLM performance, and for finding our paper clear and well-presented. We address your comments one by one:

1. "No Statistical Significance results…To address this concern: Show that the performance improvements using back-patching are statistically significant"

Thank you for bringing this up. Per your suggestion, we show that back-patching leads to statistically significant performance improvement by running a bootstrapping experiment (1000 iterations, full sample size). The results, across all but one combination of model+task, show high significance confidence. The full bootstrapping results are in the following table (showing the clean accuracies, taken from Appendix C, and the bootstrapped mean and std of the post back-patching accuracy):

To measure significance, we use a high confidence threshold of 0.99, and validate that the lower bound of the mean (which is the mean across 1000 iterations minus the error margin) is higher than the clean baseline accuracy. This shows all performance improvements are significant except for the Qwen+factural recall case, where back-patching leads to only a minor improvement. We will add these results to Table 1 (back-patching results), describe the bootstrapping procedure in the revision's appendix, and add the relevant code.

2. "Evaluation does not incorporate scaling of models…To address this concern: Perform an evaluation where the same model family with different sizes is compared against back-patching, comparing the performance improvements"

To incorporate evaluation across scales, as well as answer your later question on the back-patching effect on PaliGemma, we perform the back-patching experiment on PaliGemma2 models, at 3 model sizes - 3B, 10B and 28B. The results, presented in the following table, show that across model sizes, back-patching is a valid fix to the "lack of visual alignment" problem, showing relative performance improvements.

Across most tasks and scales, back-patching improves performance by a significant margin, across scales. The bootstrapping settings are identical to the ones in the previous table.

While back-patching isn't a perfect solution across settings, we don't view it as such. To quote the reviewer, we view it as "a very interesting finding that requires more research before application" -- it's a PoC of a potential way to bridge a limitation of VLMs, based on the mechanistic insights which are the main focus of our paper.

Nevertheless, we will add this evaluation of back-patching across scales and its implications to the revised paper.

3. “their patching approach increases the computational cost of inference, so this needs to be weighed against just using a larger model.”

We treat back-patching as a PoC based on mechanistic insights (which are the main focus of our paper) rather than a ready solution. With that said, the additional experiments on the PaliGemma family show that this method can be applied on models of different scales to achieve performance improvements. Thus, we argue that such interpretability-based insights can provide improvements orthogonally to using larger models, which can be limited by availability.

4. "Would the authors expect the back-patching to lead to a higher focus of VLMs on image modality than on text modality as reported in, e.g., [3,4,5]?"

This is an interesting question. We speculate back-patching doesn't affect the amount of attention given to image positions, but mainly allows the propagation of more "concise" and textually-aligned information from image tokens from early layers. However, further analysis into the effects and limitations of back-patching on visual token processing is an interesting direction. As we view the main focus of our work in the mechanistic analysis, and view back-patching as a finding rather than a perfect solution, we leave these investigations to future work.

5. We will fix the two additional comments (reference to Appendix C from Table 1, PDF formatting issues) in the revision of the paper.

To summarize, we wish to re-iterate that we don’t view back-patching as a limitation-free solution to the performance gap, but rather a fix showing that interpretability-guided insights can help bridge modality performance gaps.

We hope our updated results and answers are satisfactory. Please let us know if you have any other questions or concerns.

Thank you for the feedback!

We appreciate you finding our findings interesting and insightful, as well as mentioning the thoroughness of our ablations. We address your comments and questions one by one:

1. Regarding your comments and questions on evaluation beyond toy settings:

"Can authors evaluate the effectiveness of back-patching on a couple of common VQA benchmarks that have paired realistic image-texts (e.g., object counting can also be done based on images with dense captions such as [1])?"

Thank you for this suggestion. We have performed an evaluation of back-patching on general vision-question answering using the VQAv2 dataset [9] and the more up-to-date RealWorldQA [10] dataset, two common visual-question answering benchmarks. In the following table, we report the results, showing that back-patching textually-aligned visual embeddings is effective even in general VQA prompts:

The RealWorldQA results are calculated on the entire dataset. The VQAv2 results are calculated as the average of 3000 randomly sampled visual prompts (without image repetition). We limit the analysis to 3000 prompts due to computational requirements of exploring different back-patching settings. Bootstrapping was performed (similarly to the settings in the response to reviewer 1QgH) for 1000 iterations with full sample size to validate statistical significance.

We will add these results to the appendix and update the code in the revision.

"Restricting prompts to single‑word answers and short captions departs from real VQA workloads",
"Circuit discovery is performed only on prompts whose ground‑truth answer is a single word"

While this limits analysis approach, this stems from an inherent limitation of circuit discovery methods (as we mention in lines 121-122). Such methods (as in [1,2,3,4,5], and more) are all limited to identifying model components crucial to the completion of a single token. Extending these methods to analyze the completion of a complex, multi-token completion isn't trivial and is an active research question in mechanistic interpretability. Additionally, our circuit intersection analysis relies on the positional alignment of query tokens, making it less feasible to be applied on prompts with different-length questions.

While this limits the capabilities of all circuit discovery research, we follow standard procedures ([3,6,7,8] and more) to isolate the behavior we are localizing, such that it could be expressed in a single token containing the answer, and will allow us to identify the key components for each task, which we further analyze.

We will add a discussion on this point to a revised limitations section.

2. Rationale for circuit‑level analysis: "Could you clarify why this circuit‑based lens is preferable to more common attribution techniques—e.g., gradient‑ or perturbation‑based importance scores—when diagnosing modality gaps?"

This is an interesting question. Could you provide specific references or methods? As “gradient- or perturbation-based importance scores” could refer to different approaches.

If you mean input attribution techniques (identifying influential input features), these are valid and provide a different perspective for investigating modality performance gaps. We chose circuit extraction because it reveals the computational pathways that process different modalities, allowing us to characterize the role of different model components and how they interact. This structural lens enabled our key analyses: examining circuit intersections (Section 4.1) and testing component interchangeability between modalities (Section 4.2) to identify similar or different functionalities. By isolating causally sufficient subgraphs, we can understand how different modalities utilize shared versus distinct computational mechanisms.

If you mean component attribution techniques (scoring individual model components), we should clarify that our approach actually incorporates these methods. We use attribution patching with integrated gradients [1,4], which is activation-based and gradient-based, to score component importance and build sufficient circuits from high-importance components.

We view our circuit-based analysis as one rigorous lens among several valid methodological approaches. If you have any further questions in this manner, we would love to elaborate.

3. Writing comments:

"Formula (1) appears without motivation (e.g., why k=5?) or derivation, and obscures rather than illuminates the Hanna et al. (2024) method."

Thank you for pointing this out. We wish to clarify: Formula (1), showing the importance metric we measure per model component, was taken almost as-is from Hanna et al. [4], and is a standard circuit discovery metric. The number of integrated gradient steps (k) is chosen to be 5 in the original work [4] to be a good tradeoff between accuracy and runtime. Per your comment, we will add a relevant section in the appendix to expand on this method instead of merely citing it.

"Key terms such as “textual task,"visual task,” and how prompts are split into “discovery” and “evaluation” subsets are introduced without definition or concrete examples. It would be better to expand this paragraph."

Regarding textual vs visual tasks - we present a quick example of textual and visual tasks in the abstract (e.g., an analog pair of a visual and textual tasks is "counting a specific object in an image" versus "counting a specific word in a word sequence"), but we realize this should be re-iterated. We will add such an example to the intro and connect to it in the caption of Figure 2.

Regarding discovery vs evaluation subsets - we explain the division to discovery and evaluation datasets in Appendix B.3. We will add a reference to it to Section 2.2., where these split is first presented.

Per your request, we will polish the preliminaries section to improve its presentation and remove unnecessary formalism.

.

We hope our updated results, answers and fixes are satisfactory. Please let us know if you have any other questions or concerns.

References:

[1] N Nanda, "Attribution Patching: Activation Patching At Industrial Scale", 2022.

[2] F Zhang, et al., "Towards Best Practices of Activation Patching in Language Models: Metrics and Methods", 2023

[3] K Wang, et al., "Interpretability in the wild: a circuit for indirect object identification in gpt-2 small", 2022

[4] M Hanna, et al., "Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms", 2024

[5] E Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", 2025

[6] M Hanna, et al., "How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model", 2023.

[7] N. Prakash et al., "Language models use lookbacks to track beliefs", 2025.

[8] A. Stolfo et al., "A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis", 2023.

[9] Y. Goyal et al., "Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering", 2017.

[10] X.AI. Grok-1.5 vision preview. https://x.ai/blog/grok-1.5v, 2024a. 9, 15

[11] A. Mueller et al., "MIB: A Mechanistic Interpretability Benchmark", 2025

Thank you for your thoughtful and constructive feedback!

We're glad to hear that you found our paper to be very strong, with the motivation, methodology and impact of our work coming through clearly.

To address your comments and questions:

1. "I would strongly shy away from framing the analysis around Marr's three levels"

In mentioning Marr’s levels of analysis, our intent was to loosely connect structural and functional circuit analysis with the implementational and algorithmic levels, respectively (as you correctly assumed). However, we acknowledge that this mapping is ambiguous and not essential for understanding our findings. We will remove this reference entirely in the revision to maintain focus.

2. “The authors may want to check out recent work by Campbell (2024)”

We appreciate the reference to Campbell et al. 2024 [1], which we agree seems very relevant to our findings. Our work highlights that visual data representations align with their textual counterparts too late, which can be seen as a form of representational interference. Back-patching effectively injects aligned representations back into early layers of the model, functionally reducing this interference, similar in spirit to introducing more serial processing.

3. "It would be very interesting to see the results of an experiment where the order of the query and the data in the prompt is swapped."

We agree that reversing the order of the query and data might affect how information is integrated and may impact interchangeability and back-patching performance. However, some current VLMs are typically trained with the query following the image-text input, so reversing the order constitutes a distributional shift. While Qwen and Gemma 3 models do allow this reversal with some accuracy, we chose to focus on the more prevalent setting to match that of Kaduri et al., 2024 [2]. We will add this point to our limitations section.

4. "different pattern of activation similarity for Gemma…Could this be due to Gemma being trained from scratch on multimodal data?"

This is a great question, which we discuss in Appendix E.2. While Figure 7 shows that Gemma’s activation similarities increase in earlier layers than other models, the best back-patching layers (presented in Appendix E.2) match this pattern, showing that in most tasks, back-patching achieves the best performance when patching embeddings from higher-similarity layers back into lower-similarity layers (where in Gemma3, these layers indeed come earlier in the model, perhaps due to it being trained from scratch on multimodal data, as you suggest). We will add an additional reference to this discussion appendix around lines 233-234 to make it more clear.

5. We appreciate the detailed suggestions for improving the clarity of the paper, and we will address these issues in the revision of the paper. Specifically - we will improve the definition of logit difference in line 72, fix quotation mark bugs, and make the order of bars in figure 6 more intuitive by showing a minimal bar when the faithfulness is near-zero so the order will be clear.

We hope our answers are satisfactory. Please let us know if you have any other questions or concerns.

References:

[1] D. Campbell, Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem, 2024

[2] O. Kaduri et al., What's in the Image? A Deep-Dive into the Vision of Vision Language Models, 2024