We appreciate the considerate reviews and raising important questions. We hope our responses address the reviewer's concerns.

h3c2-W1

The authors should revise the manuscript to consolidate the most important findings in the main body and present them in a clearer and more structured format.

We revised the overall structure of the paper. We added separate subsections for key findings in each section in $\S$ 3 and $\S$ 4. Please see GR1.

h3c2-W2

The paper lacks sufficient discussion on the broader implications and significance of its findings.

We clarified contributions and added broader implications in $\S$ 5 and in overall sections. Please see GR3.

Important findings and their practical value are as follows:

1. SAE identifies diverse interpretable concepts which inform spatial locality ($\S$ 3.3).
   • Providing which group of patches (i.e., segmentation mask) activates certain SAE latent enhances the interpretability of our SAE.
2. There are SAE latents that have a crucial impact on class discrimination ($\S$ 4.1.2).
   • Findings of the strong connection between interpretable concepts and model behavior "opens up more avenues on mechanistic interpretability research" (quoting GMQj Strength 2).
3. While CLIP and MaPLe activate SAE latents in similar patterns, the impact of the same latent in two models is different ($\S$ 4.1.2).
   • Our analysis provides initial results for an important problem -- understanding the adaptation mechanism of the vision encoder which is a core visual component in large vision-language models, such as LLaVA-1.5.

h3c2-Q1

Can you provide a more concise and coherent explanation of the logic behind the analysis, supported by a smaller set of essential figures for "Adaptation methods and SAE"?

We revised $\S$ 4.2 (previous $\S$ 3.3) to clarify the terms used for explaining the adaptation mechanism. We added Fig. 6 which illustrates the analysis.

In short, we find that both non-adapted and adapted models recognize similar concepts (supported by the high correlation in the SAE latent activation scatter plot). However, the model predictions vary although they use the same concepts (supported by different patterns in the prediction heatmap).

Please see GR1 for overall changes made to address the ambiguities.

h3c2-Q2

What does “suddenly capture new concepts” mean in this context? In human cognition, new concepts are usually built upon existing knowledge. Given this, how does this study distinguish between the reuse of old concepts and the acquisition of new concepts?

We consider the top activating latents as recognized (captured) concepts from an input. To analyze the model "using" the concept, we conduct the top-k masking experiment and observe the model outputs (Fig. 6). In the experiment, we control the active concepts used for SAE reconstructing the model representation and replace the reconstructed output with the original model representation.

We compare the top activating latents in before (CLIP) and after (MaPLe) adaptation from the same input to analyze the acquisition of new concepts ($\S$ 4.2.2). We force two models to use the same concept in the SAE decoder and compare the model predictions (i.e., the "use" of the concepts) ($\S$ 4.2.2).

We revised regarding expressions and added explanations in the paper $\S$ 4.2.2 to convey clearer messages.

Thank you for your favorable evaluation of our revised manuscript. If you have any further questions or points requiring clarification, please let us know.

We appreciate the constructive feedback and positive support. We hope our responses address the reviewer's concerns.

qx77-W1

It would make results stronger if the authors could provide more quantitative metrics to measure the sophistication of detected concepts.

We use the label entropy as a proxy for the "abstractive level" of SAE latents. We add Fig. 3 which shows SAE latents with different statistics. We also use the label standard deviation to sort the latents and see relative abstractiveness.

Although the label entropy informs if one latent represents a specific class or not, these statistics are incomplete to rigorously define the granularity of SAE concepts as the reviewer pointed out. We revised the expression "CLIP understanding sophisticated concepts" to "SAE latents represent a wide range of interpretable concepts".

To address the concerns regarding cherrypicking of the results, we provide an interactive demonstration which presents more qualitative results. We plan to add more examples.

qx77-W2

It would make the paper more readable if the authors could provide a brief overview of MaPLe on the manuscript.

We revised the overall manuscript to improve readability and clearly provide the necessary details. We add a separate section for the analysis method and experiment setup before the key findings of each section, including an explanation of MaPLe ($\S$ 4.2.1. and Fig. 5(a)). Please see GR1 for the summary of the overall paper revision.

qx77-W3 & Q1

Could the authors provide some justification for design choices (including ViT layer, feature statistics, and SAE dimensions)?

We conducted the ablation studies to set hyperparameters, but as the reviewer pointed out, we have reported a limited part of the results in the previous manuscript. We added quantitative results used for hyperparameter tunings (including SAE dimensions) in $\S$ A.1. Please see GR2.2. We state the justification for ViT layer in GR2.3.

We utilize the activated frequency, the mean activation value, and the label entropy as summary statistics following previous work (Pricken et al., Fry et al.). We additionally use label standard deviation, which is tailored for special cases like ImageNet where close labels are semantically close to each other. We added the references for choosing the statistics in the manuscript ($\S$ 3.2).

qx77-Q2

How would the proposed approach perform on other language-vision models besides CLIP? Would the selected hyperparameters need to be different in this case?

We believe our proposed PatchSAE and analysis methods are applicable to other vision encoders with tailored modifications and optimal hyperparameter tuning. In our view, the current set of training configurations would provide a good starting point for training SAEs on different vision models. Since SAEs on vision models are relatively under-explored, we answer based on SAEs for various LLMs. We plan to examine this experimentally in future studies.

We appreciate the constructive and incisive feedback. We hope our responses address the reviewer's concerns.

67Ef-W1

Lacks quantitative evaluation of SAE's reliability.

We added evaluation setup and quantitative results to validate SAE's reliability. Please see GR2.1.

As the reviewer pointed out, training SAEs on vision models is under-explored. Following previous work (Templeton et al., Bricken et al.), we evaluate the reconstruction ability and the sparsity using training metrics such as MSE loss and L1 loss. We set the training configurations through ablation studies (Table 2). When no significant difference is observed, we follow the selection in existing work (Fry et al.).

The SAE dimension (the expansion factor) does not show a significant difference in terms of training metrics. Although the sparsity (L1 and L0) metrics can be used as proxies of monosemanticity, the metrics are insufficient to clearly evaluate if the SAE achieves monosemanticity. We thereby stick to using the value selected by the baseline. We lower the tone that emphasizes the absolute value of the SAE dimension size in Abstract.

67Ef-W2

Potential distribution shift between base and fine-tuned CLIP.

We appreciate the reviewer pointing out an important point to strengthen the reliability of our analysis. To minimize the risk of distribution shift, we choose prompt-based adaptation methods that do not update the model parameters but append learnable tokens as additional inputs. We demonstrate the transferability of SAE in both methods ($\S$ 4.1.1, $\S$ A.4, and Fig. 11). Please see GR2.4.

67Ef-W3

Regarding "re-mapping" or "re-using" visual concepts.

We revised $\S$ 4.2 and added Fig. 6 to clarify explaining adaptation mechanisms. Please see GR1 for overall changes made to address the ambiguities.

The key logic behind $\S$ 4.2 is that: We find that both non-adapted and adapted models recognize similar concepts (supported by the high correlation in the SAE latent activation scatter plot). However, the model predictions vary although they use the same concepts (supported by different patterns in the prediction heatmap). Therefore, we conclude that the adaptation adds new mappings (re-map) between the commonly fired concepts and downstream task classes.

67Ef-W4

Significance of contribution.

We clearly state the distinctive contribution in $\S$ 2 and $\S$ 5. Please see GR3.

In short, the main contributions of this work are as follows:

1. We share the full framework of training SAE and using it as an interpretability on vision transformer. Distinct from previous works, we propose to use patch-level image tokens for SAE latent analysis which provides spatially localized attributions of SAE latents and is easily transformable to a higher (image- / class- / dataset-) level of analysis.
2. Using SAE as an interpretability tool, we examine the relationship between interpretable concepts and downstream task-solving ability in the CLIP vision encoder. This analysis addresses an important problem--understanding the adaptation mechanisms of a core vision component of large vision-language foundation models.

We appreciate the insightful comments and positive support with constructive feedback. We hope our responses address the reviewer's concerns.

GMQj-W1

The organization of the paper makes it hard to follow the flow of ideas.

We deeply appreciate for providing detailed feedback regarding the structure. We revised the overall structure of the paper to make it easy to follow. We put related work before the main parts ($\S$ 2) and add clear explanations for experiment setups as a separate subsection before key findings in each section. Please refer to GR1.

GMQj-W2

The scope of experiments and observations is somewhat limited.

We clarify our contributions in the paper. Please see GR3. Key differences from existing work are:

1. Fry et al. showed the presence of interpretable SAE latents in CLIP and Daujotas et al. demonstrated the impact of SAE latents in controllable text-to-image generation. We propose to use image tokens for SAE latent analysis which allows spatially localized understanding of SAE latents and is easily transformable to a higher (image- / class- / dataset-) level of analysis. Fig. 4 and Fig. 9 highlight the effectiveness of localizing SAE latents.
2. Replacing the model output with SAE reconstructed one is inspired by previous works. We adopt this method to analyze the relationship between interpretable concepts and downstream task-solving ability, which addresses an important problem--understanding the adaptation mechanisms of foundation models.

GMQj-Q1

Can the authors explain why they chose this particular layer for their analysis?

We chose layer 11 as a representative case which we assume to contain rich information covering overall regions. We added experiments for SAEs trained on different layers. Please see GR2.2.

As Fig. 9 shows, we find SAE latents show different patterns in different layers. The lower layer shows local attention, while the deeper layer shows global attention results. We believe the deeper layer is a reasonable choice for analyzing its influence to classification predictions. Moreover, to minimize the reconstruction error caused by replacing the model output with SAE reconstructed one in top-k masking experiments, we choose the second last layer instead of the very last layer. We identify more analysis on different layers as a promising direction for future work.

GMQj-Q2

I would be interested in seeing the results of experiment 3 (adaptation methods and SAE) with fine-tuning.

We chose prompt-based adaptation methods for controlled comparisons of zero-shot and adapted methods in the shared SAE latent space. After confirming the transferability of SAE in both methods (GR2.4), we share the SAE.

To expand the experiments for fine-tuning method that directly updates the model parameters, we need to validate the transferability of the SAE. If SAEs are not sharable, we can either train separate SAEs and come up with methods for matching two latent spaces or jointly train sharable SAEs on two models for the top-k masking experiments conducted in our work. We consider examining the differences between different types of adaptation as a promising future research direction.

GMQj-Q3

Typo

Thank you for pointing out the details. We revised the sentence in the manuscript.

We sincerely appreciate your positive assessment of our revised manuscript. If you have any additional questions or require further clarification, please let us know.

We thank reviewer LbiH for the valuable time and constructive feedback. Here are our responses to weaknesses and comments.

Q1：Lack of description of the optimization of t0 which is the most significant part of the framework.

A1: Thank you for the question. We have now provided a more detailed description of the threshold optimization process:

As the knowledge distribution evolves during training, the threshold $t$ needs to be adjusted (Lines 310-315). Specifically, given a calibration set $S_{c}=\\{(x_i,y_i)\\}_{i=1}^N$ where $x_i$ represents the input and $y_i$ is the label, we perform the following steps at regular intervals (e.g., after each epoch):

1. The model performs inference on $S_c$ to obtain tuples $(c_i, n_i)$ for each sample, where $c_i$ indicates whether the model correctly predicts sample $i$, and $n_i$ is the negative log-likelihood.

2. Then we employ grid search to calculate threshold $t^*$: a set of thresholds $\mathcal{T} =\\{t _ j\\} _ {j=1}^{M}$ is uniformly sampled from $[min(\\{n _ i\\} _ {i=1}^N),max(\\{n _ i\\} _ {i=1}^N)]$. The optimal threshold for the next round is selected by maximizing:

where $TPR_t$ and $TNR_t$ denote the true positive and true negative rates, respectively, in classifying predictions as correct or incorrect based on the threshold $t$, evaluated on the calibration set $S_c$. The size of calibration set $S_c$ is detailed in Appendix A.

We originally planned to include the detailed optimization for the threshold $t$ in the appendix. However, for the sake of clarity, we will instead incorporate these implementation details into the main text in Section 4.1.

Suggestion 1：Providing more detailed explanation of the t0 optimization process, including mathematical formulation, algorithmic steps, and practical guidelines for implementation across different models and tasks.

R1： The detailed explanation of the $t$ optimization process is provided in A1.

**Sugge