Detecting and Approximating Redundant Computational Blocks in Neural Networks

We thank the reviewer for their insightful comments and constructive suggestions. In response, we have updated the manuscript (Table 1) to include the requested clarifications and added results using a larger dataset (ImageNet-1K), demonstrating the method's applicability to large-scale datasets. In Table 3, we also conducted new experiments to showcase the method's generalization across different datasets within the same architecture. Below, we address each point raised:

• W1: We appreciate the references provided by the reviewer, and we updated the related section with missing citations.
• W2: We understand the reviewer's observation regarding the lack of comparison with other techniques. However, our proposed method is entirely training-free, fine-tuning-free, and architecture-agnostic. In contrast, existing methods in the literature rely on at least some degree of fine-tuning or additional training or are tailored to specific architectures (as [1,2]). Consequently, a direct comparison may not provide a fair or meaningful evaluation. That said, we would be glad to include comparisons with analogous approaches if they exist. If the reviewer is aware of specific works that align with our method's constraints, we would greatly appreciate their suggestions for incorporating them into our analysis.
• W3: We have updated Table 1 to include results on the ImageNet-1K dataset. By incorporating these results, we hope to strengthen the experimental evaluation and address the reviewer's concern regarding the generalization of RBA to large-scale datasets. The updated results demonstrate the effectiveness of our method in achieving significant computational savings while maintaining strong performance, further validating its applicability to complex, real-world scenarios.

We thank the reviewer for their constructive feedback. In response, we have updated the manuscript with the requested clarifications and added results using a larger dataset (ImageNet1k), demonstrating the method's applicability to larger datasets. Additionally, we conducted new experiments to demonstrate the method's generalization across different datasets using the same architecture. Finally, we would like to clarify that the proposed method is not restricted to approximating two consecutive blocks; it can also approximate multiple, non-consecutive blocks. The number of approximations is limited only by the total number of available blocks. For instance, one could approximate $b_i$ to $b_{i+3}$ with $i=1$, and later $b_i$ to $b_{i+1}$ with $i=8$. Below, we address each identified weakness and question in detail:

Originality:

1. We appreciate the reviewer’s observation regarding the similarity between the two approaches studied. However, we would like to emphasize that our work extends the findings of Nguyen et al. (2020) to pre-trained foundation models, such as transformers, rather than being limited to convolutional architectures trained from scratch. Furthermore, the first part of the paper serves to contextualize the motivation behind the proposed framework, demonstrating how similarities across layers can be effectively leveraged to enhance the efficiency of large-scale models. This contextualization is integral to framing the broader applicability and impact of our approach.
2. We evaluated the representations using alternative metrics, finding that the results remain consistent across them. However, we note that: (a) CKA is computationally more expensive, especially when applied systematically across all blocks of a large model. (b) Cosine similarity measures the alignment of vectors but does not account for the magnitude of changes in representations. In contrast, BR uses the negative Mean Squared Error (MSE), which considers both the direction and magnitude of changes in representations. This aspect is particularly important in our context, as we aim to quantify the extent of transformation within the model. To summarize, the BR was chosen for its simplicity and computational efficiency. We have updated Figure 2 to display the BR matrix instead of the cosine matri,x and the cosine matrices have been moved to the appendix for completeness.

Quality:

• We acknowledge the reviewer’s concern regarding the limitations of the BR metric. In the additional experiments we are running, that adopt models with different layer sizes, we addressed this by utilizing the SVCCA metric [1], which provides a complementary perspective. This can be easily done since the proposed framework allows the integration of alternative metrics and transformations based on specific task requirements.
• To improve clarity, we have included Figure 6 in the manuscript, which visually illustrates the connection between the BR metric and the RBA approach. The results presented in Figure 6 were obtained using the ImageNet-1K dataset with the ViT small model. We believe this addition helps to better contextualize the relationship and enhances the overall understanding of the methodology.

Thank You for your detailed response to my review, while I am happy with the response for questions 1 and 4. I need to clarify my points 2,3 and 5 for completing the novel research conducted.

Clarification: The most significant drawback I noticed in this approach is stated in my weakness point 2, the results are not consistently maintaining or improving the performance. While I dont expect the design choices (BR and RBA) to reflect ideal results, to ensure the proposed method is best choice currently possible, I expect the paper to perform ablations comparing BR and RBA with other possible approaches. For BR, as reviewer tLds has suggested, I recommend comparing with other distance metrics like cosine similarity. For RBA I understand the core motive of the paper is to state "even a simple linear transformation is enough to approximate redundant blocks" the proof shown to back that statement are tables 1,2 and 3 primarily. While this proof is partially acceptable, it is important to estimate the loss in performance by comparing with say, nonlinear approximation for instance. For simplicity, I asked the authors to compare the "approximation" technique used by venkatraman et al. approach. To again clarify, since I want the comparison to be done only with BR and RBA i still find venkatraman et al's approach eligible to be compared. Moreover, I dont support the idea that the proposed approach is optimization free, RBA requires the latent space to be estimated, is the estimation performed on random data? if not, I consider this method to involve training. Moreover, if its random, I am curious to know the reason why that "design choice" was made.

Point by Point clarification:

2. While I appreciate the proposal made by the paper, I don't agree with the point "can be easily replaced with more advanced options, allowing users to adapt it to their specific needs." The paper must present why they think RBA and BR is suitable as an initial approach for approximating redundant blocks, not leave that to the users.
3. I hope I have made a detailed suggestion for this point in the above clarification, I am available to discuss this further for the betterment of the work.

5.) The design choices are BR,RBA primarily, I want some ablations to compare these proposals with other simple replacements, as the authors themselves accept the flexibility of the approach to switch BR and RBA approximations with more advanced choices.

I am still open to updating my score, but the rebuttal doesnt satisfy me enough to raise my score to a 6 or 8, therefore I await more actions to improvise and strengthen the proposed approach.

General Suggestion to paper authors: It would be simple for reviewers to view the changes made to the paper if they are highlighted red. This highlights can be removed during camera ready.

We thank the reviewer for their insightful comments and constructive suggestions. Below, we address each point raised:

1. W1 and Q1 (BR metric): We appreciate the reviewer’s concern regarding the use of the BR metric nomenclature. We have revised our contribution to focus not on proposing a new metric but on demonstrating that a simple metric is sufficient to capture the similarities. However, we believe that this terminology improves the readability of the manuscript and supports our goal of presenting the work in a clear and accessible manner. Thus, we maintained the acronym. We hope this explanation addresses the concern raised.
2. W2 (consistent results across different settings): We agree with the reviewer that the linear approximation used in our approach may not be optimal in all scenarios. However, the primary objective of this paper is to introduce a flexible framework that can be seamlessly integrated into a variety of pipelines. The transformation used in the framework is modular and can be easily replaced with more advanced options, allowing users to adapt it to their specific needs.
3. W3 (comparison with Venkataramanan et al [1]): We thank the reviewer for pointing out this related work. After a detailed analysis, we have identified several key differences between our approach and the method proposed in [1]. The primary distinction lies in the fact that, as stated in the appendix of [1] (Section "Architecture/Image Classification"), the authors re-train the networks from scratch after computing the SKIPAT. Additionally, [1] introduces a method for skipping MSA computation in one or more layers of a transformer, while our approach provides a general method to approximate any layers (not limited to the attention heads) in any architecture (not limited to transformers). Also, the SKIPAT parametric function in [1] employs two linear layers and a depth-wise convolution (DwC) in between, while our method involves only a linear transformation. We believe that comparing a method that requires training and is architecture-specific with one that does not and is architecture-agnostic is not fair. Nevertheless, we remain open to adding comparisons with methods that do not require further training and are architecture-agnostic if the reviewer can provide those references since, to the best of our abilities, we could not find any.
4. W4 and Q2 (other downstream tasks): As mentioned in the limitations section, we agree with the reviewer that including results on other downstream tasks could enhance the significance of the paper. Since our method can be applied to other tasks without any inherent limitations, we are currently conducting additional experiments using AutoEncoders for image reconstruction and ResNets, which will be eventually included in the camera-ready version.
5. Q3 (ablations): We kindly ask the reviewer to clarify Question 3, as we did not fully understand the specific ablation being requested. However, we are happy to provide any additional results as needed.

We thank again the reviewer for their thoughtful and constructive feedback. We hope our responses have adequately addressed your concerns, and we welcome any further feedback or suggestions to strengthen the work further.

While I still standby my argument that the approach requires a NAS, I am also interested in knowing the time taken to each removal (ex. 1 ->5), I don't want any new experiments to be added, but I may reassess my review if the time required for NAS is tolerable. Also, how would a NAS look like in this case, for example we have 1 -> 5 , 6->8 as two set of similar blocks, from your experience of conducting experiments presented in tables 1 to 3, will we need to search the subset combinations (i.e 1 -> 5 = {1->5, 2->4,3->5, etc}) or searching a combination between the two set of similar block (1->5 or 6->8) enough.

We thank the reviewer for their insightful comments and constructive suggestions. Below, we provide detailed responses to address each of the raised points and clarify aspects of our approach. Additionally, we updated the PDF submission with the corresponding corrections.

• Clarification on Equation 1: Our method focuses on approximating the latent space using internal representations. Specifically, in our setup, we start from block $b=1$. For models such as ViT, which have 12 internal blocks, the indexing runs from 0 to 11. As a result, block 0 is not subject to approximation in our approach; the approximation process begins from block 1. Therefore, when $b=1, h^{(0)}(x)$ refers to the output of the initial (zero-th) block of the transformer model. We hope this clarifies our notation.
• Addressing Block Redundancy: In this paper, we define redundancy in the context of a sequence of blocks rather than individual blocks. Specifically, a sequence of blocks $b_i,…,b_{i+n​}$ is considered redundant if the representation produced by the block $b_{i-1}$ is similar to that produced by the block $b_{i+n}$​. This indicates that the sequence as a whole contributes to redundancy, as the collective sequence leads to a similar final representation. Importantly, redundancy is a property of the sequence as an entirety, not of any single block in isolation. A single block $b_i$​ cannot be definitively labeled as redundant merely because it is part of a redundant sequence. For instance, to better illustrate the reasoning, consider a block followed by its inverse. Their composition is redundant since it is the identity, however the individual blocks are not.
• Clarification on ‘Skip' Mode in Table 2: We apologize for any ambiguity regarding retraining. In Table 2, we present the accuracy results for two distinct inference modes: (1) the “skip” mode, a naive baseline, where entire blocks are bypassed without any approximations, and (2) the “approximate” mode, where blocks are not fully skipped but are instead approximated (i.e., the approach we are proposing). These results are obtained without any additional retraining; the accuracies reflect the performance of the model under these two inference strategies as-is. To avoid any confusion, we have further clarified this point in the text.
• Comparison with Model Pruning Methods: We understand the reviewer’s observation regarding the lack of comparison with pruning techniques. However, our proposed method is training-free, fine-tuning-free, and architecture-agnostic, whereas most methods in the literature rely on at least some degree of fine-tuning, additional training, or tailored to specific architectures. Consequently, a direct comparison may not provide a fair or meaningful evaluation. That said, we would be glad to include comparisons with analogous approaches if they exist. If the reviewer is aware of specific works that align with our method’s constraints, we would greatly appreciate their suggestions to incorporate into our analysis.
• Experiments on Complex Datasets and Other Tasks: We appreciate the reviewer’s valuable suggestion to include additional experiments. To address this, we have extended our evaluation to assess the robustness and adaptability of our approach in more challenging settings. Specifically, we have updated Table 1 in the manuscript to present results on the ImageNet-1K dataset, which provides a more comprehensive benchmark for large-scale image classification tasks. Additionally, we also add in Figure 6, the correlation between results on ImageNet-1K and the BR metric. Regarding other tasks, we agree that results on other downstream tasks could improve the significance of the paper, as we expressed in the limitation section. Since there are no limitations in applying the method to other tasks, we are currently performing additional experiments using AEs for image reconstruction and ResNet-based architectures that will be eventually added to the camera-ready version.

We sincerely thank the reviewer for their thoughtful and constructive feedback, which has been instrumental in refining our approach and enhancing the clarity and quality of our manuscript. We hope our detailed responses have adequately addressed your concerns, and we welcome any further feedback or suggestions to strengthen the work further.

I appreciate the authors for their thoughtful and detailed responses. The clarifications for Q1, Q2, and Q3 have addressed my concerns. Additionally, the important experiments on ImageNet (Q5) has been added to the revised manuscript (in Table 1).

However, the comparison with other methods is still missing, which undermines the persuasiveness of the proposed method. This issue is also raised by reviewer tLds.

In conclusion, I will keep my original rating.

We sincerely thank all the reviewers for their constructive feedback on our work, suggesting actionable modifications to improve the paper further.

Following their advice, we list here the main changes we adopted in the manuscript:

• Evaluation of the proposed method on ImageNet1k (Table 1)
• Experiment to show that a simple linear transformation is shared across different datasets within the same architecture (Table 3)
• Ablation on the translation complexity, showing that a linear transformation is enough (Table 4)
• Further demonstration that the proposed metric (BR) correctly identifies redundant blocks (Figure 6 and Figure 7)
• Further clarified that our method is training-free, fine-tuning free, and architecture agnostic.
• Further clarified that our method can be applied between an arbitrary number of blocks.
• Clarified the BR contribution.
• Updated the related work section adding [1,2,3].
• Updated Figure 2 with the BR metric and moved the cosine similarity block-by-block similarities to the appendix.

[1] Shashanka Venkataramanan, Amir Ghodrati, Yuki M Asano, Fatih Porikli, & Amir Habibian (2024). Skip-Attention: Improving Vision Transformers by Paying Less Attention. In The Twelfth International Conference on Learning Representations.

[2] Zhang, Hanxiao, Yifan Zhou, and Guo-Hua Wang. "Dense Vision Transformer Compression with Few Samples." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Bai, Shipeng, et al. "Unified data-free compression: Pruning and quantization without fine-tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.