Interpreting the linear structure of vision-language model embedding spaces

Thank you for your positive review and constructive (and actionable) feedback! We have addressed some of the weaknesses/reasons to reject (W) and your question (Q) below:

W1: Dataset specification for Figure 12: You're absolutely right—we used the COCO dataset for all quantitative analyses including Figure 12. We'll clarify this in the revision and ensure all figures clearly specify their data sources.

W2: SAE terminology: Fair point about terminology. While we use "autoencoder" following standard SAE literature, you're correct that these are more accurately described as sparse dictionary learning methods. We'll add a clarifying note about this distinction in our methods section.

W3 & Q1: Encoder-based model focus: We focused on encoder-based models (CLIP, SigLIP variants) plus one autoregressive model (AIMv2) because they produce explicit joint embeddings optimized for cross-modal similarity. Decoder-based models like LLaVA process modalities differently and would require adapted analysis methods. However, you're right that we should better justify this scope and acknowledge it as a limitation for future work. We did not see the scope of this paper as providing a thorough experimental account of how architecture and learning objectives influence multimodal representation, but instead to introduce methods, metrics, resources, and interesting outlooks in order to advance interpretability in VLMs and better understand how the cross-modality correspondence can work geometrically. Hopefully, using the methods introduced in this paper, follow-up work on architecture can yield important insights about how to better get deep multimodal representation, which would be extremely interesting!

Q2: Training procedures: Absolutely, as asked by other reviewers too, we’ve added a dedicated section in the appendix detailing hyperparameter, convergence, training procedure to ensure full reproducibility.

Q3: Qualitative analysis: Great suggestion! We're adding concrete examples of discovered concepts with visualizations in our supplementary material to make the abstract "concept" notion more tangible. We’ve included some examples here

https://vlm-concept-visualization.com/demo-multimodal-concepts/COLM_2025_qualitative_examples.pdf

showing concepts that vary in abstractness from surface-level visual features to more abstract meaning and linguistic features.

Dear Reviewer Hk33,

The authors have responded to your review. Does their answer address your questions and reasons to reject?

Thanks for your positive and constructive review! We have addressed some of the weaknesses/reasons to reject (W) and your question (Q) below:

W1 Missing non-linear dynamics: Spot on! We believe this to be a really important question the authors debated extensively during the writing process! We justify our linear analysis for three reasons: (i) VLM training objectives explicitly enforce linear separability between aligned/misaligned pairs, (ii) downstream tasks predominantly use linear probes, making linear structure directly relevant to understand what downstream task can capture and (iii) the Linear Representation Hypothesis posits that neural networks can encode up to exp(n) nearly orthogonal feature in space of dimension n, extractable linearly.

Our high reconstruction scores (Figure 1) suggest the LRH is at least partially true here: we can reconstruct the large majority of variance using relatively few linear concepts. While SAEs certainly dictate what we can and cannot see, they capture as much as any linear probe method. Importantly, understanding what linear probes can capture is itself valuable, since this determines what downstream applications can access.

However, you're absolutely right that non-linear features exist and that SAEs can’t extract them (but neither do linear probes). We deliberately title our work "linear structure" to be clear about this scope. This tension between linear vs. non-linear feature extraction in multimodal spaces is something we plan to continue exploring in future work.

W2 Limited to seeing artifacts of the SAE methodology This is an excellent and fundamental question. You're right that any interpretability method introduces its own lens, this is true for PCA, NMF, SAEs, or any non-trivial analysis. However, our findings reveal something meaningful about VLM structure regardless of methodology: BatchTopK SAEs achieve the best reconstruction (Figure 1) precisely because VLM spaces have a specific geometric organization. The fact that the optimal reconstruction strategy uses concepts that are almost modality-orthogonal concepts selected in a modality aware way tells us something important about how these spaces are structured.

Our key insight is that VLM spaces exhibit a paradox: they're geometrically organized to separate modalities (shown in our anisotropy experiments, Figure 5), yet the most effective way to reconstruct activations is through a nearly modality agnostic basis. This suggests that semantic meaning in VLMs is encoded orthogonally to modality separation, a finding that emerges from the reconstruction objective, not the SAE methodology per se.

Q1 Generalization to other models and layers Our analysis spans contrastive (CLIP, SigLIP, SigLIP2) and autoregressive (AIMv2) models, showing consistent patterns despite different training objectives. For other architectures, we expect similar linear structure given the linear representation hypothesis, though specific organization might differ.

Regarding layers: we focused on final embeddings where the joint multimodal space is established. Intermediate layers in most VLMs are unimodal (separate text/image processing), so they wouldn't exhibit the cross-modal structure we analyze. However, studying how unimodal representations transition to multimodal ones could reveal how this joint organization emerges. Furthermore, expanding these analyses to single-backbone architectures could let us see how multimodality emerges throughout layers in those models, and understand if it arises with similar dynamics to other complex processing like shape perception or syntax processing. We think that these are very interesting avenues to future work!

In case it is interesting, we have also added some extra qualitative analyses of our features here: https://vlm-concept-visualization.com/demo-multimodal-concepts/COLM_2025_qualitative_examples.pdf

Thanks!

Thank you for the clarification, the rating is appropriate.

W1 Using shared-backbone VLMs. Thanks for this suggestion, we think that testing these methods and results on a diverse set of models is definitely a very necessary direction for future work. For this paper, we see our main contribution as being that we introduce methods, metrics, resources, and interesting outlooks in order to advance interpretability in VLMs and better understand how the cross-modality correspondence can work geometrically, rather than give a thorough experimental account of how architecture and learning objectives influence multimodal representation

We run all of our analyses on four models: three models trained with contrastive objectives (CLIP, SigLIP, SigLIP2), and one autoregressive model (AIMv2). This lets us demonstrate the kind of generally consistent insights that we can gain when we use our methods, but of course such work cannot claim to be uncovering some truth that holds for all VLMs. As you say, for that, we would need to test much more exhaustively

You are definitely right, though that we should talk about the whole space of VLMs, and we will take care and set out and describe the diversity of VLMs better in the introduction and background, in order to make it clear that we are only testing on a specific subset of four. We’ll be sure to do this in revisions, and we thank you very much for the citation pointers!

W2 Are we just seeing artifacts of the SAE methodology? This is an excellent and fundamental question. You're right that any interpretability method introduces its own lens, this is true for PCA, NMF, SAEs, or any non-trivial analysis.

However, our findings reveal something meaningful about VLM structure regardless of methodology: BatchTopK SAEs achieve the best reconstruction (Figure 1) precisely because VLM spaces have a specific geometric organization. The fact that the optimal reconstruction strategy uses concepts that are almost modality-orthogonal concepts selected in a modality aware way tells us something important about how these spaces are structured.

Our key insight is that VLM spaces exhibit a paradox: they're geometrically organized to separate modalities (shown in our anisotropy experiments, Figure 5), yet the most effective way to reconstruct activations is through a nearly modality agnostic basis. This suggests that semantic meaning in VLMs is encoded orthogonally to modality separation, a finding that emerges from the reconstruction objective, not the SAE methodology per se.

The "SAE projection effect" we identify (Figure 8) could actually explain this structure: concepts can be geometrically neutral yet functionally unimodal due to distributional differences between modalities. This is a property of the VLM space itself, revealed by but not created by our analysis method.

We agree with you on the fact that this is indeed a rich direction for future SAE interpretability work. Thanks for the thought-provoking discussion, and we will be sure to include these insights in the framing and discussion of the paper in the rewrite!

W3 Scalability/Generalization of Findings. While we used COCO for training SAEs, our stability analyses across different data mixtures (Section 5) suggest that high-energy concepts remain consistent even when input distributions change significantly. However, testing on more diverse datasets would strengthen generalizability claims. We plan to extend this analysis to multiple datasets in future work.

Q1 hyperparameters: We’ve now added a dedicated appendix section detailing our full training setup to ensure reproducibility. All SAEs were trained for 30 epochs (usually plateauing around 10-15 epochs) on 600k COCO embeddings (18M total samples), with a batch size of 1024 and unless specified, k=5. We used a dictionary size with expansion factor of 5 for the demo and 10 for the results, optimized with AdamW (weight decay 1e-5, gradient clipping 1.0), and a cosine learning rate schedule: warmup from 1e-6 to a peak of 5e-4, decaying back to 1e-6.

Q2 SAE training convergence We determined convergence by monitoring reconstruction loss plateaus over 30k steps (10 epochs). We also verified that learned dictionaries remained stable and found that a plateauing in the loss also induce the stability of the dictionary.

Q3 Effect of number of concepts on stability The stability score (Equation 4) averages over all c concepts, so larger dictionaries don't artificially inflate scores. We verified this by computing stability at different dictionary sizes (1k, 2k, 4k concepts) and found consistent patterns in the energy-weighted results.

Q4 Diverse datasets and task-specific VLMs This is excellent future work. Task-specific fine-tuning or domain-specific data could indeed alter the linear structure we observe, particularly the modality bias patterns. Understanding this adaptation would be a nice future work that you potentially yield practical insights.

Reviewer WVqz,

The authors have responded to your review. Does their answer address your questions and reasons to reject?

Hello, and thanks for your review, it’s really helpful to read your constructive comments on the paper, and we’re really glad you found our findings insightful. We’ve included some answers to your questions (Q) and to some of your comments in the weaknesses section (W) below.

Q2 Qualitative analyses (& W2): Thanks for this suggestion, we include a qualitative analysis, where we showcase concepts of different abstractnesses, from surface-level visual features to higher-level concepts in vision and linguistics https://vlm-concept-visualization.com/demo-multimodal-concepts/COLM_2025_qualitative_examples.pdf. We conducted thorough qualitative analyses when we first trained our SAEs, in order to initially trust the SAE concepts and get our bearings for writing this paper, but we did not include them in the draft in favor of more quantitative analyses. You are definitely right though, some of the qualitative examples that motivated us would make the whole paper stronger and more convincing! We will add this qualitative analysis and some extra discussion to the final version.

Q1 & Q3: Applicability of concepts and metrics, and the usefulness of VLM-Explore (& W1, W3): Thank you for this important question! While we were motivated by downstream applicability, we should have made these applications more explicit, and we will do so in our final draft. Our analyses and released resources offer several concrete benefits for interpretability and model debugging such as:

(i) Cross-modal alignment assessment: A fundamental question in VLM evaluation is whether models truly learn joint representations or merely statistical correlations. Our Bridge Score provides a principled way to identify which concept pairs actually contribute to semantic alignment, enabling researchers to diagnose when models fail at cross-modal understanding.

(ii) Robustness analysis: By examining which concepts are stable across training runs (high-energy) vs. unstable (low-energy), practitioners can identify which model behaviors are reliable and which might be artifacts of specific training conditions.

(iii) Data bias detection: The modality score can reveal when models over-rely on modality-specific features rather than semantic content, helping identify problematic biases in training data or model architecture.

(iv) Failure mode analysis: When VLMs make errors, our concept decomposition can identify which semantic components were incorrectly activated, providing more granular explanations than standard attribution methods.

(v) Model comparison: The metrics we introduce enable systematic comparison of how different VLM architectures organize semantic information, informing architectural choices.

(vi) Finally, a more ambitious use case could be a fine-tuning guidance: if we understand which concepts are geometrically aligned vs. functionally unimodal can then guide targeted interventions during model fine-tuning.

Concerning the practical value of VLM-Explore: the interactive tool enables researchers to quickly identify problematic concept clusters, validate model understanding on specific domains, and generate hypotheses about model behavior that can be tested more rigorously. We envision it being particularly useful for rapid model auditing and educational purposes.

VLM-Explore offers an intuitive visualization that combines into one interactive figure four important aspects that help us interpret and debug VLMs. These are: 1) how the linear directions of the latent space relate to each other (through the UMAP) 2) what the linear features actually represent (through the maximally activating examples panel) 3) how the two modalities share the space (through the Modality Score color) and 4) what semantics connect the two modalities (through the BridgeScore edges).

To finish, why are we just looking at linear features? This is an excellent point raised that we considered extensively. Here is our reasoning: while joint spaces need not be linear, linear analysis is particularly valuable here because: (1) VLM training objectives (contrastive learning) explicitly encourage linear separability of aligned vs. misaligned pairs, (2) downstream tasks typically use linear probes on these embeddings, making linear structure directly relevant to practical performance, and (3) our approach aligns with the Linear Representation Hypothesis, the idea that neural networks encode semantic concepts along linear directions in their representation spaces, which has strong empirical support across language models, vision models, and multimodal systems.

W4 organization: Lastly, we will definitely take your reorganization suggestions into account when rewriting, they’re useful and make sense, thanks!