Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory

We thank reviewer Po5f for the valuable feedback and suggestions for improvement. We address each point in the comment below.

Q1: The authors may move some empirical results in plot to Section 1, to make the readers more easily to jump into the main idea and the discoveries of the work.

Thanks for the suggestion, we have added an illustration in the Fig.1 in the updated paper to help readers understand the main idea more easily.

Q2: The related work part can be extendeded. Specifically, please discuss the difference and relation to the recent work in ICML 2023: On the power of foundation models. As far as I know, the paper also intensively uses the category theory tools for interpreting neural networks especially foundations models. Also, as the paper proposes a new pruning mehtod, the related work part need also discuess peer methods.

We have extended the related work section in the updated paper, where we discuss and introduce previous works using category theory in machine learning including "On the power of foundation models". Due to the space limitation, we add the discussion about pruning methods in Appendix F in the updated paper.

Q3: For experiment part, the authors are required to compare peer pruning methods. Also the authors shall more comprehensively discuss the pros and cons of their pruning method in the context of network pruning.

We have added the discussion about the difference between our proposed method and other pruning methods as well as the pros and cons in Appendix F in the updated paper. The main difference is that IFM has a different purpose compared with channel pruning methods. The proposed IFM aims at finding the most compact representation without significantly affect the performance while the purpose of channel pruning is better preserving the performance under certain computation budget. The main advantage of IFM is that it does not require access to data while the disadvantage of IFM is that the compression ratio could not be precisely controlled. In the experiment part, we compare our method with another SOTA pruning method INN [1] that also does not require finetuning. Since our proposed method does not require finetuning while most other pruning methods do, it might be unfair to directly compare our results with other pruning methods.

[1] Kirill Solodskikh, Azim Kurbanov, Ruslan Aydarkhanov, Irina Zhelavskaya, Yury Parfenov, Dehua Song, and Stamatios Lefkimmiatis. Integral neural networks. In CVPR, pp. 16113–16122. IEEE, 2023.

Q4: How sensitive of the proposed IFM method to the hyperparameter \beta? The authors shall discuss it and give some ablation studies if possible.

As introduced in the Appendix D, for all the experiments, we apply grid search on $\beta$, from 0.01 to 0.2. The empirical results reported in the experiment part (Fig.3 and Fig.4) reports different results with varing $\beta$. Generally, the \beta set as 0.05 could achieve good balance between the performance and the compression ratio across the experiments we conducted in this paper. We will add a more comprehensive ablation studies in our paper.

We thank reviewer R8Xc for the time and effort devoted to the reviewing process. We address each of the questions in the comments below.

Q1: The proposed method is straight forward, what is the point of introducing category theory? The theoretical part of the paper has very little to do with the practical part and doesn't seem to provide any extra insights.

In the theoretical part of this paper, we define functionally equivalent feature and feature complexity (layer-wise). To facilitate the understanding, we empirically describe them here. Functionally equivalent features mean that, given a feature map of one model, under certain transformations, we could get the output of the model by inferring with another different model that has learned functionally equivalent feature. Take a step further, if we could swap two elements in the feature map while keeping it to be functionally equivalent to the original feature map, it means the two elements could transform to each other, which indicates redundancy in the feature map. Feature complexity is defined as the dimensionality of the most compact representation of the feature without redundancy.

There are strong connections between the theoretical part and the practical part:

• With the definition of functionally equivalent features, we theoretically prove that the well-known empirical phenomenon Linear Mode Connectivity (LMC) indicates a special case of two models learning functionally equivalent features.

• We extend the weight matching methods in LMC literature and propose the method IFM to measure feature complexity. Empirical results show that the defined redundancy widely exists in the features learned by neural networks.

In fact, pruning effect is not the main purpose of the proposed method, but a byproduct resulting from the redundancy in features.

The reasons for introducing category theory: To formalize the empirical description above in the language of mathematics, we find category theory an appropriate tool.

• The definition using the language of category theory is concise and elegant in its expression. Two networks learn functionally equivalent features iff there exist a natural transformation between the corresponding functors (formal definition is in Def. 3.1).
• With our definition of categories, functors and natural transformations in the deep learning context, we hope the category theory perspective of our definition could inspire future works in more theoretical analyses.
• Despite the use of the terms in category theory, each of them has a corresponding specific meaning in the training or testing of neural networks. Please refer to Table 1 in the updated paper for details.

To facilitate the understanding of the theoretical part of the paper, revisions have been made. We add an overview figure (Fig. 1) to empirically illustrate the concepts we introduce in this paper. We also add a table (Table 1) listing the corresponding specific meaning of each used category theory terms in our definition. The revisions in the updated paper are indicated in blue.

Q2: It seems that any pruning technique that sacrifices even a fraction of accuracy for some gains in computational/training time costs, is not going to be of practical use.

While reducing computational cost could be important for scenarios such as edge devices. We would like to clarify that the main purpose of this paper is not neural network pruning. As indicated in the title, we focus on the problem that feature similarity methods sometimes fail to provide sufficient insights (e.g. similar features could also yield different outputs [1,2]). In this paper, we extend the concept of equivalent feature and propose functionally equivalent feature to mitigate the gap between feature similarity and functional similarity (the similarity between outputs)[3]. We further define layer-wise feature complexity corresponding to the dimensionality of the most compact representation of the feature and propose algorithm IFM to measure it. Various insights have been provided (Sections 5.2 and 5.3). Here we list a few insights:

• Redundancy widely exist in features learned by neural networks.
• Larger networks have more redundant parameters.
• Across the layers of a neural network, the feature complexity first increases then decreases.

[1] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity through statistical testing. Advances in Neural Information Processing Systems, 34:1556–1568,2021.

[2] Lucas Hayne, Heejung Jung, Abhijit Suresh, and R McKell Carter. Grounding high dimensional representation similarity by comparing decodability and network performance. 2022.

[3] Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint arXiv:2305.06329, 2023.

Q3: I don't understand what the concept of shape of the features relates to. Is it the shape of the matrix/tensor representing the features?

Yes, it is the shape of the matrix/tensor representing the features. We have added an description in Sec.2.2 to make it more clear.

Q4: What is actually feature complexity? Is it the total count of neurons in a network after pruning? How do we know it means something? Does it correlate with generalisation?

Ideally speaking, feature complexity is the smallest number of neurons in a network (at a certain layer) that one can achieve without affecting the functionality of the network (loss, accuracy, etc.)

We propose IFM to approximately measure the feature complexity. As we apply grid search on the hyper parameter $\beta$ in Eq. 9, empirical results (Fig. 3(a) and Fig. 4(a) in the updated paper) approximately show the fraction of redundant parameters. These results suggest that redundancy widely exists in features learned by neural networks.

We are not sure if the defined feature complexity is correlated with generalization and we leave it for future works. However, empirical results (Fig. 4(b) in the updated paper) indicate that for two networks of the same structure, the performance drops faster for the network with better performance as more features are merged. We conjecture that networks with higher feature complexity would have better generalization performance.

Q5: The statement "feature complexity corresponds to the most compact representation of a neural network" sound like something related to the minimum description length (MDL) or minimum message length (MML) principles. Any comment how your work relates to those?

The feature complexity we defined in this paper aims to reflect the inherent structure of the neural network, which is the result of training data, network structure and training algorithm. Therefore it does not have direct relationship with MDL or MML principle [1] that guides the model selection or model training. However, the proposed feature complexity could be used as a tool to help the application or interpretation of the minimum message length principle. It could be an interesting topic and we leave it for future works.

[1] Wallace, C. S. (Christopher S.), -2004. (2005). Statistical and inductive inference by minimum message length. New York: Springer. ISBN 9780387237954. OCLC 62889003

We thank reviewer QeEM for the valuable feedback and suggestions for improvement. According to the suggestions, we have added an overview figure in the updated paper and fixed the mentioned minor typos. We address each point in the comment below.

Q1: In the abstract, the authors are suggested to emphasize the feature complexity is layer-wise to make it more clear as this is a new concept to my knowledge; they may use the term layer-wise ASAP in the abstract.

Thanks for the suggestion, we have revised the introduction (Sec.1) and emphasized that the feature complexity is layer-wise.

Q2: it is vague to use the saying: merge of features in the first paragraph. Its exact and clear meaning need be better explained for self-containess.

We have revised the introduction section and make it more clear that we merge the weight to merge the feature.

Q3: I am a bit confused with Eq. 4, how $\tau_z^{l+1}$ is imposed on $Z^{l+1}$? Is it a dot operation or we shall write it as a function of $\tau_z^{l+1}$?

Sorry for the confusion. In Eq. 4, $\tau_z^{l+1}$ is a function. We followed the category theory literature and omitted parentheses for simplicity. We have added parentheses to avoid confusion.

Q4: It would also be good to clarify what are the Functionally Equivalent Features in Definitions 3.1, I think they are layer-wise $f^{l}(\theta_a^{l})$ vs. $f^{l}(\theta_b^{l})$?

Thanks for the suggestion. We have added an illustration figure in the updated paper (Fig. 1). However, since the input of $f^{l}$ is determined by the previous $l-1$ layers, the Functionally Equivalent Features in Def. 3.1 is not defined layer-wise but "model-wise".

Q5: Is it a bit abuses the notations? does Z() also equals to f()? Please clarify it.

In our notation, $Z^{l}$ represent the composition of the first $l$ layers of the network. We use $Z^{l}$ instead of $f^{l}(f^{l-1}(\cdots f^{1}()))$ for simplicity.

Q6: In the introduction part, the authors may mention the partial order idea to make the complexity concept more tangible to readers, rather than until give the specific definition in Section 3.

Thanks for the suggestion. We have added an illustration and intuitive description of the feature complexity in Fig. 1 in the updated paper to make the complexity concept more tangible.

Q7: How the proposed pruning comapre with other (SOTA) pruning method? Additional experiments will further enhance the paper.

We have added an detailed discussion of differences between the proposed IFM and other pruning methods in Appendix F in the updated paper. The comparision with another SOTA pruning method INN [1] that also does not require finetuning was provided in the experiment section. Since our proposed method does not require finetuning while most other pruning methods do, it might be unfair to directly compare our results with other pruning methods.

[1] Kirill Solodskikh, Azim Kurbanov, Ruslan Aydarkhanov, Irina Zhelavskaya, Yury Parfenov, Dehua Song, and Stamatios Lefkimmiatis. Integral neural networks. In CVPR, pp. 16113–16122. IEEE, 2023.

We thank reviewer QeEM for the valuable feedback and suggestions for improvement. We address each point in the comment below.

Q1: The term "functionally equivalent features" is central to the paper. It would be beneficial to the reader if a simple illustrative example or intuitive explanation accompanied its introduction.

Thanks for the suggestion. We have added an intuitive illustration (Fig.1) with intuitive explanation of "functionally equivalent feature" and "feature complexity" in the updated paper.

Q2: The paper would benefit from a detailed walkthrough of the mathematical formulations and proofs to ensure the robustness of the claims made.

Thanks for the suggestion. In our main submission, we provided detailed proofs in Appendix C. We also add a more detailed walkthrough of our formulation with category theory in Appendix C.3 of the updated paper.

Q3: It is vital to contextualize this work concerning existing literature. How does this work differ or extend previous work on the topic? A comprehensive comparison is essential.

There are two perspectives in the literature investigating similarity between neural networks, feature similarity [1,2,3,4] and functional similarity [5,6,7]. Feature similarity (representational similarity) focuses on the similarity between intermediate features while functional similarity focuses on the similarity between outputs [8]. However, evidences have shown that similar features could yield different output [9].

In this paper, we unify the two perspectives and propose functionally equivalent feature. Based on functionally equivalent feature, we further define feature complexity and propose an algorithm IFM to measure the feature complexity. The proposed algorithm is an extension of weight matching methods [10] in linear mode connectivity literature from between two neural networks to within one neural network.

In response to the suggestion, a more detailed introduction and comparison have been added in Sec. 5 in the updated paper.

[1] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NIPS, pp. 6076–6085, 2017

[2] Alex H. Williams, Erin Kunz, Simon Kornblith, and Scott W. Linderman. Generalized shape metrics on neural representations. In NeurIPS, pp. 4738–4750, 2021

[3] Shuai Tang, Wesley J. Maddox, Charlie Dickens, Tom Diethe, and Andreas C. Damianou. Similarity of neural networks with gradients. CoRR, abs/2003.11498, 2020.

[4] Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence: A method for comparing neural network representations. In ICML, volume 162 pp. 1607–1626. PMLR, 2022.

[5] Omid Madani, David Pennock, and Gary Flake. Co-validation: Using model disagreement on unlabeled data to validate classification algorithms. Advances in neural information processing systems, 17, 2004.

[6] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225–236, 2021.

[7] Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit, Ankit Singh Rawat, Seungyeon Kim, Aditya Menon, and Sanjiv Kumar. On the reproducibility of neural network predictions. arXiv preprint arXiv:2102.03349, 2021.

[8] Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint arXiv:2305.06329, 2023.

[9] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity through statistical testing. Advances in Neural Information Processing Systems, 34:1556-1568,2021.

[10] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.

Q4: Why was the scope of experiments limited to image classification tasks, and how would the approach perform on other tasks?

We conduct experiments on image classification tasks following previous feature similarity literature and pruning literature. In those literature, image classification task is the most widely used and investigated. To align with previous works, our experiments are conducted on most widely used network structures (VGG and ResNet) and benchmarks (CIFAR10 and ImageNet-1K).

Our definition does not have any assumption about the neural network structure and the proposed method does not require access to data. While our focus on image classification tasks aligns with the prevalent literature, the task-agnostic nature of our method suggests its potential applicability to various tasks. Considering the limited time and space, we leave the approach on other tasks for future works.

Q5: Can the authors provide a more intuitive or illustrative example of "functionally equivalent features" to aid understanding?

Yes, an illustrative figure has been added in the updated paper (Fig. 1). More intuitively, if two models learn functionally equivalent features, it means that, under certain transformations, we can replace any layer of one model with the corresponding layer of the other model and get the same output as the original model.

Q6: How does the Iterative Feature Merging (IFM) algorithm work in detail, and what makes it efficient? A clear and detailed algorithmic procedure, possibly with pseudocode, should be provided for a more comprehensive understanding.

In the main submission, we provided detailed algorithm in Appendix. A. To facilitate the understanding of the proposed method, we have added a flowchart in Fig. 1 in the updated paper which is indicated in blue. In general, the proposed method is efficient because it does not require the access to data and only matches the weights of neural networks.