We are grateful to the Reviewer for their thoughtful feedback. We respond below to the weaknesses and questions that were presented in the review.

Comparison with previous literature [W1, Q1]: Interpretability methods derived from the Logit Lens (LL) [a] are the ones that most closely resemble our proposal, as they produce explanations for internal representations of transformers by projecting them to the output space. In fact, as we state in Section 3, LL is akin to performing one step of our iterative method on each individual sample. The Attention Lens [b] (referenced in the paper as [13]) has extended the standard LL to operate on individual attention heads instead of entire layers, but it requires training a linear probe for each head, closely following the idea of the Tuned Lens [c]. This makes it computationally expensive and hard to scale to modern, billion-parameters models like the ones we analyze in this paper, as each probe would have a number of parameters in the order of tens to hundreds of millions and models have a number of heads in the order of thousands. However, inspired by this comment, we provide a new baseline for our head selection method: we apply LL to head representations (as in Table 4 in the supplementary material, this method consists in a version of Attention Lens that relies on standard LL instead of Tuned Lens), and then score heads according to the average logit produced on tokens belonging to the target property. Then, we select top-scoring heads as in our method, and evaluate their impact on image classification and captioning tasks. We obtain the following results, reported for k=16 and 32 heads and normalized to the base model performance:

The results show that our method based on SOMP identifies more functionally relevant heads with respect to LL, as reflected both by stronger decay in performance on image classification tasks and by the stronger effect of concept control interventions (inhibitory and enhancing) in image captioning.

Regarding the work by Dar et al. [d], the core point of difference with respect to our paper is that it investigates head roles by only looking at weights of an attention block, while we work in the representation space. While attention weights reveal how information is routed within a model, the actual representations produced by attention heads reflect the interplay between the model’s learned parameters and the input data distribution. As such, these representations provide a more complete and behaviorally relevant view of what a head is doing. Moreover, the method of [d] is based on the fundamental assumption of having discrete tokens, embedded independently using a shared projection matrix. While this assumption is met in standard LLMs, this is not the case for VLMs such as LLaVA, in which image tokens are embedded continuously using pre-trained vision encoders (e.g., CLIP’s ViT). We thank the Reviewer for highlighting this relevant line of work, and we will include a brief discussion of these points in the Related Work section of the final version.

Semantic generalization [W2, Q2]: We thank the Reviewer for commenting on the impact of our interventions on ‘broader semantic areas’, as this made us realize that, indeed, this point is ambiguous in the current draft. We will rephrase L209-210 as: “Notably, we showed that our approach can extrapolate a broad and consistent semantic area from a restricted list of keywords”. In this context, generalization within a well defined semantic area is a desiderata rather than a weakness.

Polysemanticity of heads [W2, Q3]: We do not claim that selected heads are monosemantic. Instead, we posit and empirically verify that properties can be controlled by sets of carefully selected heads, rather than single heads. As the Reviewer correctly points out, head redundancy in large models allows interventions targeting spuriously correlated concepts to be compensated by other heads. This redundancy, combined with polysemanticity, helps mitigate conflicts and side effects between properties, as illustrated in Figure 4, where partial head overlaps do not affect unrelated tasks.

Scaling to more complex concepts [W3, Q4]: To address the Reviewer’s concern regarding scalability to more complex concepts, we conducted an experiment to verify that our method is still effective when we consider the composition of two simple concepts. This is grounded in the compositionality assumption, recently discussed in the neural network literature [e,f], which posits that complex concepts often emerge from structured compositions of simpler ones. If our method can manipulate compositions of attributes, this offers evidence that it could generalize to more complex and contextual concepts.

We applied inhibitory and enhancing interventions (as in Section 5.2 of the paper) to composite concepts by targeting the union of top-8 heads selected for each of the two base properties. We analyze two property combinations: ‘colors’+‘sentiments’ and ‘colors’+‘quantity’. The two cases differ on the entanglement between the two properties: in the former case, the two sets of heads intersect on one head, while in the latter they are completely disjoint. We report the results of the inhibitory and enhancing interventions in the two settings in the table below:

These results show that interventions targeting composite head sets yield significant and targeted effects on both underlying properties. This suggests that our method retains functional control even when concepts are more entangled or contextually layered, supporting its potential to scale to more complex scenarios.

Additional models [W4, Q5]: Following the Reviewers’ suggestions, we have expanded our model selection, which now includes Gemma3-12B [g], LLaVA-NeXT-13B [h] and Qwen2.5-VL-7B [i]. These include both larger-capacity models and diverse vision-language architectures with strong performance. Results on these models largely confirm trends observed in LLaVA-NeXT-7B, and are reported in detail in the tables below for a selection of image classification and captioning tasks. For each model-task pair we report the results of the most effective intervention among $k={8,16,32}$, normalized with respect to the original model performance, with random baselines (mean over 5 runs) within parentheses. For captioning tasks, we omit CIDEr scores for brevity, as all results exceed 0.91, indicating high-quality generation.

We will include complete results for all models in the final version of our paper.

[a] nostalgebraist, interpreting gpt: the logit lens, LessWrong, 2020

[b] Sakarvadia et al., Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism, arXiv, 2023

[c] Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, arXiv, 2023

[d] Dar et al., Analyzing transformers in embedding space, ACL, 2023

[e] Lepori et al., Break It Down: Evidence for Structural Compositionality in Neural Networks, NeurIPS, 2023

[f] Lee et al., Geometric Signatures of Compositionality Across a Language Model's Lifetime, ACL, 2025

[g] Gemma Team, Gemma 3 Technical Report, 2025

[h] Liu et al., LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024

[i] Bai et al., Qwen2.5-VL Technical Report, 2025

We thank the Reviewer for carefully evaluating our work and for appreciating its methodological foundations. We respond below to the weak points identified in the review.

W1: While information flow in transformers is highly nonlinear, it has been shown that large language and vision-language models tend to represent high-level concepts in linear structures in the residual stream [a, b, c, d], especially in late layers close to the output [e]. This emergent property is the key for the success of our method: even if a concept is expressed in a non-linear structure in some early head, at some stage in the residual stream it will be transformed into a linearly decodable one. We show the validity of this assumption with the empirical evidence provided by our experiments, where our method results in a principled control over the target concept expression at output level. We will further clarify this in the Introduction section (L45-L47).

W2: Following the Reviewers’ suggestions, we have expanded our model selection, which now includes Gemma3-12B [f], LLaVA-NeXT-13B [g] and Qwen2.5-VL-7B [h]. These include both larger-capacity models and diverse vision-language architectures with strong performance. Results on these models largely confirm trends observed in LLaVA-NeXT-7B, and are reported in detail in the tables below for a selection of image classification and captioning tasks. For each model-task pair we report the results of the most effective intervention among $k={8,16,32}$, normalized with respect to the original model performance, with random baselines (mean over 5 runs) within parentheses. For captioning tasks, we omit CIDEr scores for brevity, as all results exceed 0.91, indicating high-quality generation.

We will include complete results for all models in the final version of our paper. We note that we did not include proprietary models such as GPT-4, PaLM, or Flamingo in our experiments, as their internal activations and attention head representations are not publicly accessible. Nonetheless, the inclusion of recent, high-performing open-source models at larger scales strengthens our claim that the observed head specialization effects and controllability generalize across architectures.

References:

[a] Park et al., The linear representation hypothesis and the geometry of large language models, ICML, 2024

[b] Jiang et al., On the Origins of Linear Representations in Large Language Models, ICML, 2024

[c] Rajaram et al., Line of Sight: On Linear Representations in VLLMs, arXiv, 2025

[d] Papadimitriou et al., Interpreting the linear structure of vision-language model embedding spaces, arXiv, 2025

[e] Gandelsman et al., Interpreting clip's image representation via text-based decomposition, ICLR, 2024

[f] Gemma Team, Gemma 3 Technical Report, 2025

[g] Liu et al., LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024

[h] Bai et al., Qwen2.5-VL Technical Report, 2025

We thank the Reviewer for their insightful assessment of our work and for suggesting several potential improvements to it. We try to address here the main weaknesses and questions that emerged in the review.

Linear decomposition [W1, Q1]: While information flow in transformers is highly nonlinear, it has been shown that large language and vision-language models tend to represent high-level concepts in linear structures in the residual stream [a, b, c, d], especially in late layers close to the output [e]. This emergent property is the key for the success of our method: even if a concept is expressed in a nonlinear structure in some early head, at some stage in the residual stream it will be transformed into a linearly decodable one. We show the validity of this assumption with the empirical evidence provided by our experiments, where our method results in a principled control over the target concept expression at output level. We will further clarify this in the Introduction Section (L45-L47).

Scaling to more complex concepts [W2]: To address the Reviewer’s concern regarding scalability to more complex concepts, we conducted an experiment to verify that our method is still effective when we consider the composition of two simple concepts. This is grounded in the compositionality assumption, recently discussed in the neural network literature [f, g], which posits that complex concepts often emerge from structured compositions of simpler ones. If our method can manipulate compositions of attributes, this offers evidence that it could generalize to more complex and contextual concepts.

We applied inhibitory and enhancing interventions (as in Section 5.2 of the paper) to composite concepts by targeting the union of top-8 heads selected for each of the two base properties. We analyze two property combinations: ‘colors’+‘sentiments’ and ‘colors’+‘quantity’. The two cases differ in the entanglement between the two properties: in the former case, the two sets of heads intersect on one head, while in the latter they are completely disjoint. We report the results of the inhibitory and enhancing interventions in the two settings in the table below, normalized to the original model performance:

These results show that interventions targeting composite head sets yield significant and targeted effects on both underlying properties. This suggests that our method retains functional control even when concepts are more entangled or contextually layered, supporting its potential to scale to more complex scenarios.

Keyword list quality [W3 and Q3]: We acknowledge that properties are defined by lists of keywords that may be incomplete or biased. However, as we showed in our experiment on toxicity mitigation (Section 4.2 of the manuscript), our method is able to identify relevant heads over a subset of keywords and then generalize to a disjoint (but semantically related) subset. This suggests that our approach can generalize from a partial or noisy concept definition, as long as the keywords are semantically related to the underlying property. To strengthen this point, in the case of VLMs, we perform a new experiment: we apply our method to identify heads related to 'colors' on Flickr30k (as in Section 5.2), but we replace a fraction of our color-related keywords with random tokens from the model vocabulary. We report in the table below the Jaccard similarity between the original set of top-k heads identified by our method for this property and the top-k heads identified on noisy lists of keywords, while varying the fraction $\rho$ of random keywords:

These results confirm that head selection is reasonably stable even when up to 50% of the keyword list is replaced with unrelated terms. This robustness suggests that the method is not overly sensitive to noise or incompleteness in the keyword list, and that it can tolerate imperfections in concept specification.

Hallucination and factual consistency [W4 and Q4]: We thank the Reviewer for highlighting the importance of evaluating hallucination risks and factual consistency. While we do not explicitly annotate hallucinated content, we address this concern using several well-established automatic metrics that serve as proxies for semantic and factual consistency in our image captioning experiments.

In addition to the CIDEr scores already reported in the submission (which assess how well generated captions match reference captions in content and structure), we now also report BERTScore [h], METEOR [i], and CLIPScore [j]. BERTScore captures fine-grained semantic similarity to reference captions, METEOR reflects syntactic and lexical quality, and CLIPScore directly measures alignment between the generated caption and the image content. We report results in the table below for the strongest intervention setting (32 heads).

Across both inhibitory and enhancing interventions, all metrics remain stable, indicating that the generated captions preserve semantic coherence, linguistic fluency, and visual factuality. These results suggest that our method does not introduce hallucinations or factual inconsistencies, even when intervening strongly on targeted attributes.

Concept embeddings [Q2]: Our work focuses on identifying functionally relevant heads that are responsible for the generation process of specific concepts in VLMs. This motivates the choice of decomposing head representations directly onto the unembedding matrix to identify head properties which are mostly adopted by the model during the generation process. This approach ensures consistency with the model’s decoding process avoiding the introduction of additional alignment overhead.

As correctly pointed out by the Reviewer, there are other approaches for concept discovery. One direction is to use models designed to build representations at a higher semantic abstraction level, such as Large Concept Models [k]. Differently from the unembedding matrix, their representations are not aligned with the token-level representations of LLMs and VLMs, and aligning them would require an additional and possibly complex procedure, especially in the multimodal setting involving text and image tokens. Another approach would be to learn the concept dictionary directly from the head representations, for example extending methods like [l] that analyze representation shifts under fine-tuning. This is a complementary approach, which we acknowledge in the manuscript. However, also these procedures ultimately require projecting onto the unembedding matrix for interpreting and grounding the concepts.

Compensating effects [Q5]: Yes, it is indeed possible that suppressing a single attention head has little or no observable effect due to compensation by other heads. This is a known phenomenon in transformer models, where redundancy across heads can mask the impact of individual interventions. To address this, our method always considers sets of heads rather than individual ones. In practice, intervening on 16 heads is typically sufficient to produce significant effects in a 7B model, as demonstrated across multiple tasks in our experiments. By targeting multiple heads associated with a given property, we mitigate the risk of compensation and capture a broader share of the relevant signal. As a potential extension, one could set a threshold on the cumulative explained variance (rather than fixing the number of heads) to dynamically determine how many and which heads to intervene on. This would allow the method to adapt to the redundancy structure in different models or tasks.

Intuitive method explanation: We thank the Reviewer for the suggestion and agree that a more intuitive explanation would enhance accessibility. We will revise Section 3 in the final version to improve clarity and provide additional intuition for our method.

References:

[a] Park et al., The linear representation hypothesis and the geometry of large language models, ICML, 2024

[b] Jiang et al., On the Origins of Linear Representations in Large Language Models, ICML, 2024

[c] Rajaram et al., Line of Sight: On Linear Representations in VLLMs, arXiv, 2025

[d] Papadimitriou et al., Interpreting the linear structure of vision-language model embedding spaces, arXiv, 2025

[e] Gandelsman et al., Interpreting clip's image representation via text-based decomposition, ICLR, 2024

[f] Lepori et al., Break It Down: Evidence for Structural Compositionality in Neural Networks, NeurIPS, 2023

[g] Lee et al., Geometric Signatures of Compositionality Across a Language Model's Lifetime, ACL, 2025

[h] Banerjee and Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, IEEvaluation@ACL, 2005

[i] Zhang et al., BERTScore: Evaluating Text Generation with BERT, ICLR, 2020

[j] Hessel et al., CLIPScore: A Reference-free Evaluation Metric for Image Captioning, EMNLP, 2021

[k] Barrault et al., Large Concept Models: Language Modeling in a Sentence Representation Space, arXiv, 2024

[l] Khayatan et al., Analyzing Fine-tuning Representation Shift for Multimodal LLMs Steering, ICCV, 2025

We thank the Reviewer for their feedback on our work, and we respond here to the weaknesses and questions outlined in the review.

Comparison with baselines: We agree that including a comparison with prior interpretability methods is a valuable addition to our work. In the absence of any specific request/suggestion from the Reviewer, we adopt the Logit Lens (LL) method [a], which has been vastly employed in interpreting internal representations of LLMs (e.g., [b,c]) and more recently VLMs [d].

In Section 3 of our submission, we already provided a qualitative comparison of our method with LL, showing that Simultaneous Orthogonal Matching Pursuit (SOMP) returns consistently less redundant explanations, due to its inherent orthogonality constraint. Motivated by the Reviewers’ suggestions, to make the comparison quantitative we propose a simple adaptation of LL to head interpretability in VLMs. Specifically, given a keyword-defined property (for example, 'colors'), we assign to each head a score corresponding to the mean logit assigned to property-related tokens after applying LL. The top-k scoring heads are selected, as in our method, and subjected to interventions to evaluate their functional importance.

We report in the table below the results of this analysis on image classification and captioning tasks on LLaVA-NeXT-7B applying the intervention on attention heads. We evaluate performance for k=16 and k=32 attention heads.

The results show that our method based on SOMP identifies more functionally relevant heads with respect to LL, as reflected both by stronger decay in performance on image classification tasks and by the stronger effect of concept control interventions (inhibitory and enhancing) in image captioning.

Q1: We thank the Reviewer for raising this point. SOMP was chosen specifically because it identifies a shared, sparse support across all examples. This results in a column-sparse matrix $\mathbf{W^*}$, where the same few dictionary atoms (i.e., semantic directions) are active across all samples, and other columns are entirely zero.

In contrast, Lasso would yield a fully sparse matrix with non-zero coefficients scattered across different atoms for different samples, complicating interpretability and head ranking. In the final version of our paper, we will clarify this distinction by explicitly describing $\mathbf{W^*}$ as column-sparse.

Q2: We are grateful to the Reviewer for pointing to this related work that we were not aware of. It is true that both our method and [e] rely on the unembedding matrix to interpret intermediate representations. However, the purposes and mechanisms differ, as in [e] the unembedding matrix is used in a gradient-based saliency framework to determine which input tokens are most responsible for classification decisions. The analysis depends on the choice of a specific downstream task and applies techniques inspired by Grad-CAM. In contrast, we use the unembedding matrix as a semantic dictionary to identify which attention heads contribute most to specific latent concepts. Our method is gradient-free, task-agnostic, and operates on head outputs, not input token gradients. In the revised manuscript, we will include a brief discussion of these points, and note that their method could potentially complement ours in refining our analysis on a few, salient tokens.

References:

[a] nostalgebraist, interpreting gpt: the logit lens, LessWrong, 2020

[b] Halawi et al., Overthinking the truth: Understanding how language models process false demonstrations, ICLR, 2024

[c] Yu et al., Characterizing Mechanisms for Factual Recall in Language Models, EMNLP, 2023

[d] Neo et al., Towards interpreting visual information processing in vision-language models, ICLR, 2025

[e] Hou and Castanon, Decoding Layer Saliency in Language Transformers, ICML, 2023

Thank you for answering my questions. I have updated my score.