AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space

• The overall concept is not made clear in the Introduction. The method is relatively simple conceptually, consisting of a step in which multiple models' features are aligned to the brain to provide a common reference frame, followed by clustering of the features within this common space. The Introduction does not make this pipeline clear. Specifically, while the alignment into a common space is clear, the clustering procedure is not explained: first, the authors evoke neuroscientific ideas on lines 39-41, without discussing how these relate to the problem of clustering features, nor even that the goal of the current work is to find clusters of features. Subsequently, on lines 42-47, they discuss the problem in terms of "graph edges incidents on each pixel", and "channel grouping hypothes[e]s". While this is partly a matter of subjective taste, I believe that discussing the problem at hand in terms of reducing the dimensionality of features at each location (image patch), thus finding a small number of dimensions (combinations of features) which can explain the affinity structure between patches, would be more easily understood by most readers.

• Several key details about the methods are left out: for example, almost no information about training (learning rate, optimizer used, batch size, number of epochs) is provided.

• Related to the previous point, as the Appendix contains several important methodological details (such as the use of additional regularization losses) but is never referred to in the main text, the authors should add references to it where relevant.

• A key component of the proposed method is the learning of a mapping of different models' features into a common space, and as shown in Figure 3, this does indeed result in the cosine similarities of different channels' activations becoming more similar across models. At the same time, the goal of the method is to uncover differences between models. The authors should include an explicit discussion of what kind of model differences are likely to be preserved, and which are likely to be destroyed in the alignment process. As a possible suggestion in this direction, the paper makes several references to the models' features before alignment (for example comparing their segmentations' evaluations with the aligned features in Figure 5), but these features are never visualized. A direct comparison of each model's (clustered) features before and after alignment would be very informative.

• The feature clusters are visualized by reducing their dimensionality to 3D using t-SNE, and visualizing the resulting 3D features as RGB colors. This visualization, however, is not easily interpretable, as different channels are conflated together by the additional dimensionality reduction. Visualizing single channels separately might be more useful to understand the nature of the discovered clusters. Was there a specific reason for choosing the 3D t-SNE visualization rather than showing single channels?

• Overall, it is not clear what the discovered channels can tell us about the models. The single interpretable channel that is discussed in depth in the paper is figure/ground. While this provides a good sanity check on the meaningfulness of some of the discovered features, the ability of vision transformers to segment objects has been observed in several papers (e.g. Melas-Kyriazi et al. 2022, Xu et al. 2023), and the different responsiveness of different regions in the visual cortex to figure and background is well established. Other concepts revealed by the method (the category concepts in Figure 8) are shown on the surface of the brain, but it is hard to interpret what these brain maps mean. The authors should make the questions that can be answered using the proposed method clear and explicit.

• The paper fails to cite closely related work in the Related Works section. Particularly, it cites mechanistic interpretability work in other fields, such as language, but not the recent rich line of work that has specifically looked at vision transformers' ability to perform specific visual tasks. The two papers cited above (Melas-Kyriazi et al. 2022, Xu et al. 2023) are a good example, the former in particular as it proposes a segmentation method based on spectral clustering which is very close to the present one. Another relevant paper is El Banani et al. (2024), which looks at 3D-related tasks. As this is not my field of expertise, I am not aware of papers that look for meaningful directions in the space of vision transformers' channels, but I would be surprised if this didn't exist. I would recommend the authors to do a more exhaustive literature search to find papers that more closely relate to the method proposed here.

• In Figure 7, the figure-ground visual concepts discovered by different models are plotted on brain maps. In the text, the authors write that "the foreground or background pixels activates similar brain ROIs across the three models". However, a glance at the brain maps reveals similarities, but also differences. A statistical evaluation of the similarity between different models' brain maps would be recommended.

References

El Banani, M., Raj, A., Maninis, K. K., Kar, A., Li, Y., Rubinstein, M., ... & Jampani, V. (2024). Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21795-21806).

Melas-Kyriazi, L., Rupprecht, C., Laina, I., & Vedaldi, A. (2022). Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8364-8375).

Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., & De Mello, S. (2023). Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2955-2966).

As I wrote in the "weaknesses" section, I believe the precise scope of the method (what kinds of questions can and cannot be answered with it) has not been properly acknowledged and discussed by the authors.

Nevertheless, this paper has problems, some more important than others.

So I will separate them into major problems (M) and minor problems (m). I want to make it clear that for me, all these problems are solvable and do not detract from the quality of the paper.

Let's start with what I think are the Major problems (M):

M1. Related Work Quality (Page 9, Section 4):

• The related work section is critically weak, with only 24 references and lacks depth in discussing relevant literature. The paper misses an entire set of works on (1) concepts xai, (2) alignment of brain and activations (3) study of representations and (4) attributions methods, which are either crucial or should be mentioned for this study. A significant rewrite is necessary to properly position the paper within the existing body of work.

M2. Validity of t-SNE for Distance Measures (Figure 9):

• The paper uses t-SNE for analyzing bifurcation in feature space, but t-SNE is known to distort distances. This raises concerns about the validity of conclusions drawn from t-SNE plots regarding feature bifurcation.

Now for the minors problems:

m1. Redundancy of Discovery Claim (Page 2, Line 25):

• The claim that channel feature correspondence exists across networks is not new, as it has been extensively studied in major works (eg using CKA, RSA...), please update and compare your work to this litterature.

m2. Reliance on Channels (General):

• Channels are not necessarily the best basis for analysis, as recent researchs suggests there are better ways to represent features, directions (neuron is not a great basis). The paper should address why it continues to rely on channels, considering the limitations.

m3. Orthogonality Assumption (General):

• The paper's assumption of orthogonality in feature decomposition does not align with current understanding, especially regarding the neural collapse phenomenon in late layers. Say it otherwise, all point for the class tench are nearly collapse in the latest layer, relaxing othogonality (e.g dict learning) may be a good idea (althought if i am correct, the nystrom approx should not yield perfectly orthogonal vectors). This should be discussed.

m4. Parameter Sensitivity (Appendix):

• As always when we have hyperparameter, i expect a small discussion discussing the effect of changing the parameters (λ eigen, λ zero, and λ cov). This would help understand the robustness of the method to these hyperparameters.

m5. Direct t-SNE Application (General):

• The paper uses eigenvectors for t-SNE. It would be more straightforward to apply t-SNE directly to the data, and the paper should justify the chosen approach.

Yes, the limitations identified by the authors are accurate and well-documented. Regarding the weakness I mentioned, I reserve the right to increase the score if the authors adequately address my major concerns.

As I said in the previous section, I was not able to understand the methodological details of this paper. A lot of terms and concepts are used throughout the paper without a proper explanation, and as a consequence I cannot honestly understand what is going on with the methods of this paper to properly evaluate it. This paper seems to have a lot of work in it, so I want to believe that what's happening here is that (1) this is one of the first - if not the first - paper from the authors, and thus they lack the experience to explain what they did in a way that their peers can understand, (2) a lot of the concepts used are seen by the authors as very obvious jargon from the subfield, and thus the problem is that I'm not familiar with this subfield, or (3) both. I'm leaving my doubts in the next section, in the hope that they will allow us to understand better whether my understanding difficulties are related to points (1), (2), or (3). As a consequence, I'm rating this paper as a borderline reject, hoping that during the rebuttal period the authors will have time to tackle this readability issues, and as a result I'll be able to properly reassess this work.

Limitations of this work are mentioned in the Conclusion section, but no discussion about the potential negative impact of this work is presented.

I see three major weaknesses:

1. The necessity of the brain is unclear to me. Instead of aligning features to the brain, you could've aligned the features of the different networks to each other - creating an "emergent" universal feature space. Would your results, e.g. w.r.t. the figure-gound segmentation, change much if you do so? If not, what does bringing the brain into play buy us here in terms of network response interpretability? This is unclear to me.

2. Most of the results need robustness checks. For e.g., in Fig. 6 you show foreground vs background difference in neural response associations. Presumably, that's an average across a lot (all?) of images. Could you indicate some sign of robustness, for e.g, running a permutation test to assess how likely the differences you see would've been expected given the data statistics alone? Same holds for Figs. 7 and 8. We need to know if these differences are flukes or not.

3. Reliance solely on ViTs. To make your point more general, showing that a high-performing CNN shows the same results would be very informative. ViTs have more expressivity in terms of patches interacting with each other - perhaps figure-ground segmentation isn't as strong in CNNs (although if previous research is to be trusted, CNNs should have some notion of figure-ground segmentation; see Hong et al. NatNeuro 2016 and Thorat et al. SVRHM 2021).

Refs:

• Hong, Ha, et al. "Explicit information for category-orthogonal object properties increases along the ventral stream." Nature Neuroscience 19.4 (2016): 613-622.
• Thorat, Sushrut, Giacomo Aldegheri, and Tim C. Kietzmann. "Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization." SVRHM 2021 Workshop @ NeurIPS.

The authors mentioned methodological limitations. It is sufficient.