Understanding Visual Feature Reliance through the Lens of Complexity

"The proposed measure is closely related to the accuracy with which a feature can be linearly decoded from the layers of a network. I believe the paper would benefit from an explicit discussion of what, exactly, is the additional information provided by the proposed method which cannot be gained from directly looking at the "raw" decoding accuracy."

We are not sure which accuracy you refer to.

The classification accuracy is not smooth, nor continuous, in network predictions. This typically makes its interpretation more noisy. Indeed, for hard tasks, the accuracy may be near the one of a random predictor, while the loss itself might be significantly lower than the one of a random predictor. Furthermore, accuracy only makes sense for classification tasks over discrete sets. But predicting a concept coefficient is not as simple as predicting if it is “present” or “absent”. Instead, it is a continuum where a concept can be more or less influential. In this case, we face a regression task, as explained in Appendix C. This smoothness allows better handling of inaccuracies in the concept extraction stage.

If you refer to MSE, it is indeed closely related to $\nu$-information in the Gaussian posterior case (appendix C), but thanks to the “optional ignorance” property of $\nu$-information, we are certain that small MSE can be attributed to knowledge of the input, and not to simplicity of the task.

"I am not sure about what the implications are for models which do not include residual connections. The residual stream is found to be central in "teleporting" simple features to later layers. The authors should explicitly discuss what they predict the pattern of results would be for networks without residual connections. For example, would simple features be overall less relevant to the network's responses? Or would the network implicitly implement a residual-like stream? The limitations of using this architecture exclusively are briefly acknowledged in the Limitations section, but the specific role of residual connections is not discussed there."

Thanks for this suggestion. We added a paragraph in the limitations section to discuss the significance of residual connections, and the questions you raise.

"The proposed measure is based on a loose assumption that the decodability of features tends to increase across layers. However, it is possible that certain features are only available in early layers (for example, certain low-level image features might be discarded). These features would have a low cumulative -information, and might thus be erroneously classified as high-complexity. The authors should explicitly discuss whether this is a concern at all, and for what reasons."

This is a good question. To clarify, we do not have this problem because we only seek to decode features available at the last layer. We create the dictionary at the penultimate layer, ensuring each feature exists at the end. We then calculate their complexity, avoiding issues with features that don't exist at the penultimate layer. However, you highlight an interesting point: what about features present at layer 2 but not at layer 5 for example? This is outside the scope of our current study, but an interesting question for future work.

"The plot in Figure 5B shows that more important features tend to become less complex as training progresses. As the authors have access to the features which are subject to this process, and those which are not, it would be extremely interesting to visualize them, showing what exactly is happening - are the same features being extracted at earlier layers? Or are they actually changing, and starting to resemble simpler features such as colors or edges? The answer to this question would clarify more precisely what the proposed complexity measure is capturing. Is it possible that features we visually evaluate as "complex" can be extracted at early layers if they are important to the network's task? Or is layer depth always correlated with visual complexity?"

Concerning the tracking of type of features, we indeed have some figures available showing this, but the problem is that tracking concepts (i.e., not just averaging complexities but genuinely tracking and linking features across epochs) requires an algorithm with hyperparameters. Some features may disappear at certain points. We chose to show only the average and top and bottom complexity curves to avoid relying too much on the linking method for our analysis. However, this is an excellent remark, and we would like to investigate this further in future work.

We greatly appreciate the time and effort you invested in reviewing our work; thank you.

"Regarding the qualitative exploration of dictionary vectors, and their grouping into clusters (for “what”): is there a good reason to think relative positions mean anything in feature space, and therefore that clustering is meaningful? Were the dictionary vectors normalized beforehand, or else did the variable magnitude affect the UMAP and clustering? Why were 150 clusters used? Did a sweep over k or any other analysis indicate that the vectors are indeed clustered to some degree ? "

Clustering was used to qualitatively inspect what complex and simple features look like, and it represents a qualitative part of our study. Clustering in embedding space is a common practice, and normalization does not change the cluster membership. We found that meta-clusters were robust to choice of cosine or L2 normalization.

Regarding the choice of 150 clusters, we aimed to avoid overcrowding figures and selected 30 meta-features that span different degrees of complexity, sampling randomly at each level. Each cluster was qualitatively interpreted, but the labeling of clusters is subjective. We didn't use external models for labeling, and while these labels provide useful insights, they are inherently qualitative and should be interpreted as such. We have clarified this aspect in the manuscript to emphasize the qualitative nature of this part of the analysis and to acknowledge the limitations of the clustering and labeling process.

"Regarding the use of CKA for “where”: The proposed complexity measure is already based on how far back in the processing the feature is still recognizable to a linear probe. Why show CKA instead of the usable information as a function of depth for simple vs complex features? It’s not intuitive to me to infer anything about the similarity of the “residual” branch aka simply the feature vec one layer deeper, compared to that of the delta (“main branch”), which is not used as a feature by the network."

Good question as you are indeed correct. We acknowledge that $\nu$-information could provide a valuable perspective, and could be used for that. However, to avoid circular reasoning and to employ a well-established metric in the literature, we chose CKA. Moreover, our results with $\nu$-information are consistent with those obtained using CKA, and we have included these additional findings in the appendix as suggested. The choice of CKA was also driven by its established use in analyzing neural network activations, ensuring our findings are interpretable and comparable within the broader research context.

"In Appendix A: “The dictionary D was designed to encapsulate 10 concepts per class”. Does this mean the NMF was run separately for each class, and then the dictionaries were all combined? If that’s the case, are there multiple copies of some features (e.g. “wheel” being found for every vehicle class)?"

The dictionary design process utilized the published Craft method, ensuring 10 concepts per class to achieve balanced reconstruction across classes. While this approach can result in duplicated concepts, it ensures comprehensive coverage of class-specific features. We believe this redundancy does not detract from the overall utility of the dictionary; rather, it enhances the robustness of feature representation by capturing subtle variations within shared concepts across different classes.

"Fig 6: What are we supposed to infer from the epoch 1 results? If everything is still essentially random at this stage of training, then we're just seeing meaningless values plotted against each other, right?"

In this figure, we contrast features at the beginning versus end of training, to get a picture of what has shifted. The end of epoch 1 represents the point where the dataset has been seen once, and several gradient steps have been taken. The features used at the end of epoch 1 have equal chances of being simple or complex (decodable at layer 1 or layer 10, for example). In contrast, by the end of training, we observe a simplicity bias, where important features are more likely to be decodable early. This demonstrates how the network evolves to prioritize simpler features over time, highlighting the dynamic nature of feature learning and the emergence of important features throughout the training process.

"The inhibitory/excitatory results (Fig 12) are striking -- every single inhibitory feature has complexity less than ~0.5?"

Thank you for this remark. We also find this observation intriguing. To be honest, we are not certain whether this is a specific property of ResNet50, ImageNet (because of 1k classes), or a more general phenomenon. This warrants further investigation and could be a fruitful avenue for future research.

"Regarding the higher importance of simpler features: are they simply present more often, or are they more influential? I wonder if there's a frequency-complexity relationship that would be insightful."

This is an excellent question. Previous work has already pointed to these issues [1], and concurrent research has posed similar questions with controlled toy examples [2]. It appears that the relationship between frequency and complexity is indeed complex, involving both factors. Simpler features may be more frequently present and also more influential in driving the network's decisions. We believe this is a fundamental question for deep learning.

[1] Hermann, K., Mobahi, H., Fel, T. Mozer, M. (2023). On the Foundations of Shortcut Learning.
[2] Lampinen, A. K., Chan, S. C., and Hermann, K. (2024). Learned feature representations are biased by complexity, learning order, position, and more.

Thank you for your feedback! We're glad our responses helped clarify the analysis. We'll make sure the comparison with CKA is included in the appendix as suggested. We appreciate your support and the score increase. Thanks again, and we hope the final version meets your expectations!

"Not Easy To Follow On Math: The paper's mathematical sections are difficult to follow. Clearer and intuitive explanations would improve accessibility."

We thank the reviewer for this feedback. We strove to keep the math portions of the paper as accessible as possible, but welcome feedback if there were particular points of confusion.

"The paper states that both and are positive due to the use of Non-Negative Matrix Factorization, which aligns with the nature of ReLUs. Could you elaborate on why negative values for wouldn't work, despite the positive nature of ReLU? It's not clear why allowing and to be negative would be problematic, especially considering that."

We are applying a published method widely recognized to generate concepts that are interpretable, sparse, and compositional; see [1, 2] for an overview of the large literature on the subject.
Theoretically speaking, the importance of using a consistent semi-ring on both sides (non-negative semi-ring or negative semi-ring, as you pointed out) ensures compositionality and interpretability, preventing the cancellation of concepts (we only add value on top of each other; they don't cancel).
Practically speaking, NMF (i) aligns well with the properties of ReLU activations, which are sparse and positive. NMF (ii) allows us to decompose activations without orthogonality constraints, which is crucial since model activations can collapse, making concepts non-orthogonal. Moreover, NMF (iii) supports compositionality, enabling multiple concepts to coexist within an activation.

[1] Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature.
[2] Gillis, N. (2014). Nonnegative Matrix Factorization.

"In line 126, is defined as. Given that is described as positive elsewhere, wouldn't be more precise? Or are you intentionally allowing negative values for?"

This is a good remark. We have corrected this inconsistency to ensure clarity.

"The nature of, , and in Figure 1 is unclear. If these represent the -th element of, they would be scalars, which doesn't seem to align with what's depicted. Could you clarify what these variables represent?"

You are correct, and we apologize for the lack of clarity. $z_1, z_2$, and $z_3$ ​ are indeed scalar values representing directions in the activation space (also called “concepts”). The images shown are feature visualizations, which are optimized to maximize the activation of these concepts (i.e., finding $x$ such that $x = \arg\max z_i(x)$. We have clarified this in the manuscript to avoid confusion.

"Figure 1 shows feature visualization across different layers, and the paper mentions earlier layers elsewhere. However, line 622 states that feature extraction was done only for the penultimate layer. Could you explain this and clarify how earlier layer information was obtained if extraction was limited to the penultimate layer?"

Thank you for this remark. We have corrected this inconsistency in the manuscript. Feature extraction was indeed performed at the penultimate layer. The visualizations in Figure 1 aim to decode features $z_1, z_2$, and $z_3$ that exist at the penultimate layer at lower layers, such as blocks 1 and 3 of the ResNet50. We use linear probing at these earlier layers to visualize how features develop and are transformed across the network. This method clarification has been added to the manuscript.

"While the findings are insightful, they are based on a single architecture (ResNet50) and dataset (ImageNet). The results might not generalize to other models or tasks without further validation." "The assumption that each layer optimally represents features for downstream linear probes may not hold universally, potentially leading to overestimated complexity scores in some cases. More empirical validation across different models and tasks could strengthen the proposed metric's robustness."

This is a valid point, which we discuss in lines 325-331 and in Appendix I. We agree and would like to extend our study to include diverse models in the future. However, our perspective with this work is that understanding why ResNet50 generalizes on ImageNet is already a significant challenge.

"The focus on specific epochs (e.g., epoch 90) may overlook important dynamics occurring at intermediate stages of training. A more granular analysis could provide a clearer picture of feature evolution, i.e., Can you analyze the changes in importance during the training process?"

Thank you for your comment. However, there might be a misunderstanding. Sections 5 and 6 of our paper directly address this issue. We indeed perform a dynamical study of features across epochs, and we show (i) in Section 5 that the most complex features emerge later in terms of epochs, while section 6 demonstrates both (ii) the simplicity bias over epochs and (iii) the dynamic compression of important features throughout the epochs.

"How would the proposed V-information complexity metric perform on different neural network architectures, such as Transformers or GANs?"

We're not entirely sure if we understand the question. What do you mean by 'perform'? The metric, as introduced in the seminal paper [1], measures mutual information under a complexity constraint. We use it to determine the simplicity or complexity of features. How would you suggest evaluating the performance of our metric? If you're asking about applicability, our metric can be easily applied to transformers or GANs without any issues, as it only requires having access to intermediate activations.

[1] A Theory of Usable Information under Computational Constraints, Xu et al., 2020

"Can the authors provide more details on how the complexity metric correlates with other existing complexity measures in the literature? Which one provides better insights?"

You raise a valid point, as complexity can be defined in various ways. Recent research employs category theory to introduce a redundancy-based metric, which merges neurons until a distance gap surpasses a threshold, using this gap as a hyperparameter [1]. In Appendix E, we demonstrate a correlation between our metric and the general redundancy score presented in [2]. However, this method has two limitations: (i) it requires a hyperparameter, and (ii) features considered redundant by this metric may still be complex according to our measure. Our complexity metric is more focused on the computational operations needed to decode a feature optimally. In this regard, our complexity metric can be viewed as a relaxation of computational complexity metrics, such as Kolmogorov complexity.

[1] Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. Chen et al. (2023).

[2] Diffused redundancy in pre-trained representations. Nanda et al. (2024)

"How does the model's simplicity bias, as observed in the study, impact its generalization performance on unseen data?"

This phenomenon has been extensively studied in previous research, which tends to show that a simplicity bias can negatively impact model performance on unseen data. Models that overly rely on simpler features may fail to capture the more complex patterns necessary for robust generalization. We have incorporated references to these relevant studies [1,2] in our Related Work section to provide additional context and support for this observation.

[1] The Pitfalls of Simplicity Bias in Neural Networks. Shah, H., Tamuly, K., Raghunathan, A., Jain, P., Netrapalli, P.

[2] Overcoming Simplicity Bias in Deep Networks using a Feature Sieve. Tiwari R. ,Shenoy, P. (2023).

"Are there specific strategies that could be employed to mitigate the simplicity bias and encourage the learning of more complex, yet important, features?"

One potential strategy we could imagine to mitigate simplicity bias using our metric is to backpropagate through the score of the $\nu$-information. This score is differentiable, allowing for backpropagation (backprop through single matrix inversion). While this method can theoretically prioritize the learning of more complex features, it presents significant challenges in terms of memory and scalability. We could imagine addressing this with an online score (or batch estimation) instead of a real Complexity score.

"The paper lacks coherence because multiple components (e.g., metrics, algorithms) introduced in this paper do not come from the same theoretical framework. [...] These metrics/algorithms are borrowed from different theoretical frameworks and contexts, so it is in doubt whether they can be used together. I would appreciate it if the authors are able to re-organize all components of the paper under a coherent framework, e.g., the information theoretic perspective (if at all possible)."

Regarding the different theoretical frameworks for the metrics and algorithms introduced, we believe that (1) these components can indeed be integrated together as is usually done and (2) we could if we wanted to write the entire framework from an information theoretic perspective. For instance, dictionary learning methods have a strong connection with information theory [1,2]. Additionally, CKA is based on HSIC, which is closely related to mutual information ($HSIC = MMD(P, \prod P_i$) and mutual information $KL(P, \prod P_i))$. The framework of $\nu$-Informations is not tied-up to information theory alone, quite to the contrary. It inherits most of the theoretical properties of Shannon’s entropy (e.g …) but the possibility to specify any function class for the predictive family is what makes it powerful and applicable to a broad range of domains (including deep learning, but not only).

It would be possible to reorganize the components under a unified framework, even if it is not the traditional approach for each component. Thank you for your suggestion; we are adding these insights in the appendix.

[1]: B. Dumitrescu and P. Irofti, Dictionary Learning Algorithms and Applications

[2]: A Personal Introduction to Theoretical Dictionary Learning, Karin Schnass

"My second concern is about the paper’s significance, since many claims have been discussed in previous works and appears non-novel. The conclusion that simple features are usually color/edge detectors that are located in earlier layers is not surprising, nor do the conjecture that simple features tend to propagate through the residual connection [...]Overall speaking, I do not learn new insights from the paper."

You are correct that Section 3 qualitatively presents a range of simple, medium, and complex features. While some of these are new, some align with features we have learned about from the prior literature. Still, it is worthwhile to see that our method discovers them. Regarding Section 4, we do not claim novelty for the late appearance phenomenon, although this is the first explanation from a complexity perspective. As another reviewer noted, this provides a fresh viewpoint on the phenomenon. We believe that Section 6 presents two new contributions: the emergence of simplicity bias using $\nu$-Information during training and the observation that neural networks generalize by compressing important features.

"Using K-means clustering to aggregate features into meta-features may lead to incorrect clustering results. K-means clustering is known for its poor performance for data clusters that are not circular-shaped and are of discrepant sizes. I am not sure how features are distributed in this paper’s experiments, but more advanced clustering strategies such as the spectral clustering or hierarchical clustering can be applied to mitigate this issue."

Thank you for your suggestion. We chose to use K-means based on the rationale that, in the collapsed space of a neural network, the choice of clustering algorithm has minimal impact on the results: studies have shown that feature distributions in this space are generally well-suited for K-means clustering [1].

[1] "Neural Collapse: A Phenomenon in the Terminal Phase of Deep Learning" by Papyan et al., 2020

"The paper claims that simple features tend to propagate through the residual connection, while complex features tend to propagate through the main branch of the network. Then a natural question arises: how are simple and complex features propagate in CNNs without residual connections (e.g., AlexNet, VGG)?"

This is indeed a relevant and intriguing question. Investigating whether identity functions or orthogonal transformations occur channel-wise in models like AlexNet and VGG could provide valuable insights. This topic warrants further exploration in future research, and has been noted in future directions.

"The feature fences is among the most simple features, while the features whiskers and dotted texture are among the most complex features. However, I do not see an essential difference between the features fences, whiskers, and dotted texture from the sample images in Figure 7 and Figure 8. For example, fences and dotted texture both seem to consist of lines that cross over each other to form holes. Similarly, the whiskers feature is also composed of lines, although the lines do not cross over each other but appear parallel. Why is fences a simple feature, but whiskers and dotted texture complex features?"

We argue that fences represent repetitive local patterns, in contrast to whiskers or insect legs, which are unique, finely structured motifs. This distinction ties into the texture versus shape debate: texture pertains to repetitive patterns, while shape involves unique, widely distributed elements across an image, necessitating the extraction of more structured motifs.

"Another question is that previous studies show that CNNs are usually biased towards textures, i.e., they tend to learn textures as a shortcut solution, but why is the dotted texture here measured to be a complex feature?"

Firstly, the dotted pattern falls within the medium complexity category, ranking 14th out of 30 meta-features. Although not highly complex, the appendix reveals that images responding to this pattern exhibit different orientations. This indicates that the pattern detects texture while also capturing it from multiple angles and viewpoints.

the main point [...] is to justify that all these algorithms and metrics can used together and do not conflict with each other Is Craft truly compatible with the proposed v-information metric?

Can you clarify what you mean by "conflicting with each other" ? None of these frameworks operate with mutually exclusive hypotheses.

These components are expected to be organized in a coherent framework

This is rather ambitious, as it would require a comprehensive understanding of all phenomena arising in deep learning training. We believe our work provides a first step into the direction of identifying promising components of a "coherent framework".

why choose the Craft method to extract features but not other methods? The same question can be asked for other components: the feature visualization method, the CKA metric, the importance metric , etc

All these methods are published, and have demonstrated their effectiveness in deep learning benchmarks, including human experiments.

We observe consistent results using different tools and theories. This diversity is a strength, not a weakness: this is precisely what makes us confident in the robustness of our observations. As all results and metrics point toward the same direction, we are confident to not overfit a single method.