Towards Meta-Pruning via Optimal Transport

Thank you for taking the time to review our paper and for providing valuable feedback. We are pleased to hear that you find the idea of unifying pruning and fusion into a principled approach interesting and sensible. Your positive comments on the novelty of restoring information from all neurons to create more accurate compressed networks are greatly appreciated. We also appreciate your acknowledgment of the comprehensive and interesting evaluation, as well as the noteworthy findings presented in Section 5.

We would like to address the specific points you raised as weaknesses:

---

Robustness of uniform distributions:

The choice of uniform probabilities is rooted in considerations of entropy, which as considered in information theory, is a measure of uncertainty or disorder in a system. In the context of Optimal Transport, using uniform probabilities for both source and target distributions maximizes entropy and, consequently, represents a conservative approach that minimizes assumptions about the underlying structures of the distributions amidst uncertainty about true but unknown importance distributions.

In hindsight, we do agree that the formulation in the paper is a bit vague, and have thus reformulated it in the revised version (see Section 3.2.3, page 5). Furthermore, to add some further exploration with regards to the varying performance for different choices of target and source distributions, we added a corresponding ablation study in Appendix C.4. Thank you for bringing this to our attention!

---

Intra-Fusion & Pruning-at-Initialization metrics:

It would indeed be fascinating to explore the integration of Intra-Fusion within a data-free pruning-at-initialization (PAI) framework, such as with Synflow. Regrettably, PAI has predominantly focused on unstructured pruning rather than the structured pruning paradigm that Intra-Fusion addresses. For example, Synflow is designed for unstructured pruning, and the problem it intends to alleviate (layer collapse) does not have a direct parallel in structured pruning.

Nevertheless, as pointed out before, it would indeed be very interesting to combine these two lines of work. Most PAI approaches derive “winning tickets” by relying on the assumption that weights/neurons are discrete and separate entities, whereas Intra-Fusion posits that akin neurons can be merged. This could introduce a novel dimension to PAI wherein one could try to fuse the lost information from the dense ticket back into the chosen sparse mask — something which, to the best of our knowledge, remains unexplored and would form an interesting direction for future work.

---

Experimental Results on Speedups of the Split-Data Approach

Additional figure with timeline comparison:

We want to thank the reviewer for bringing to our attention that adding a concrete run-time comparison of split-data approaches and the whole data approach is of interest and will again clarify the chosen PaF/FaP methodology. Hence, we have added an additional figure in the Appendix D.1 giving a concrete side-by-side comparison of the timelines of the approaches and providing concrete speedup figures.

Practical speedup measures on VGG11-BN (training:1.81, PaF:1.42, FaP:1.31):

In the added figure we observe a practical speedup for a VGG11-BN (CIFAR10) of the original model training time of 1.81. Here the overall speedups of the split-data approaches are 1.42 (PaF), 1.31 (FaP). The increase in speedup of PaF over FaP is due to the fine-tuning being done on an already pruned network, thus making use of the improved inference time that comes with structurally pruned networks.

Training factorization yields higher speedups when train time >> fine-tune time: It is important to note that the split-data approaches presented are proof of the concept that model training can be effectively factorized using model fusion and pruning. By optimizing hyperparameters like “number of fine-tuning epochs” at different stages of the approaches further speedups are possible. For models and datasets where the time for original training $T_1$ is much greater than time for fine-tuning $T_2$ $(T_1 >> T_2)$, speedups would show to be much more significant.

---

Thank you once more for investing your time and effort in reviewing our paper! We hope that we could provide further clarity on the choice of source and target distributions, and the distinctions between Intra-Fusion and PAI.

We would like to express our gratitude for your thoughtful and constructive review of our paper. Your positive feedback regarding the data-free setup and the novelty of the split-data training approach is truly encouraging, and we appreciate your acknowledgment of the potential impact our method may have on the broader research community.

Your recognition of our strategy involving the integration of discarded neurons using a modified version of OTFusion is particularly satisfying. We aimed to provide a well-reasoned and thoughtful approach to structured pruning, and we are pleased to see that our efforts in this regard have resonated with you.

---

Extension of prior work:

We also appreciate your insightful comments on the comparison of our method with existing literature. Your suggestion to extend our comparisons to include a wider range of prior works is well-taken. To address it, we added experimental results of deploying the importance metric in [1] (see Appendix E.2, Table 18). To further highlight that the improvements of Intra-Fusion are agnostic to the choice of importance metric, we also include results of using random scores drawn from a uniform distribution as an importance metric (see Appendix C.3).

---

Writing enhancements:

Regarding your note on the typographical errors in references to Figure 6 on pages 3 and 4, we sincerely apologize for any confusion caused by this oversight. We have since corrected these errors and ensured that the references accurately point to Figure 1.

---

Once again, we appreciate the time and effort you have dedicated to reviewing our paper, and we hope to have addressed your suggestions adequately.

[1] CHIP: CHannel independence-based pruning for compact neural networks. NeurIPS, 2021, Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, Bo Yuan.

Thank you for the thorough reading and the feedback on our work.

---

Comparison to performance of Lottery Ticket Hypothesis (LTH):

We would like to highlight that LTH and Intra-Fusion address distinct aspects of neural network optimization.

Our work deals with structured pruning, which involves removing entire neurons based on an importance metric to compress already trained large neural networks. On the other hand, the Lottery Ticket Hypothesis posits that there exist sparse unstructured subnetworks within larger neural networks that, when trained in isolation, can achieve comparable performance. Moreover, LTH is strictly speaking not a deployable pruning methodology, for it offers a ‘retrospective’ method for which the entire network training has to be, repeatedly, carried from scratch until the pruning mask of desired sparsity is obtained. This is because,

• it requires the availability of the network checkpoint at an earlier epoch, if not the initialization, to apply the intermediate pruning masks. This is rarely the case in practice.
• Besides, it also assumes the original training dataset (or some additional dataset) is available (not data-free). Compared to Intra-Fusion, this adds to computational (due to repeated training) and data procurement costs.

As evident from the above discussion, LTH is concerned with unstructured pruning, barring odd exceptions, as opposed to the structured pruning paradigm which Intra-Fusion addresses. A focus on structured pruning lets us deliver immediate speedup in inference and storage costs, while unstructured pruning is not directly amenable to hardware acceleration.

Nevertheless, we do agree that our paper can only be enriched by an even broader comparison. For this reason, in the new version of the paper, we also compare Intra-Fusion with an additional recent top-performing importance metric CHIP in [1] (see Appendix E.2, Table 18). Moreover, to highlight the point that Intra-Fusion is agnostic to the importance criterion, we include results with random pruning (see Appendix C.3).

---

Adaptivity of the FaP approach:

Thank you for bringing up this interesting point. Research on model fusion is still in a relatively nascent stage, and fusion techniques such as Vanilla Averaging (which is a key ingredient of Federated Averaging) require identical model architectures. However, the model fusion technique we use in our split-data method, namely OTFusion [2], is able to fuse models with different widths. Hence, fusing model architectures that exhibit some degree of heterogeneity is still possible.

However, fusing models of radically heterogeneous architectures, such as CNNs with ViTs, as you point out, is still an unexplored area, and conceptually very complex. However, one could potentially identify similar layers and fuse them within two models of highly heterogeneous architectures, thereby yielding a child network that is separate in some layers, and conjoined in others.

---

In conclusion, we appreciate the valuable feedback from the reviewer, which has enhanced the clarity of our paper. We've clarified the distinctions between Intra-Fusion and the Lottery Ticket Hypothesis, and provided a broader comparison in the revised version (Table 18, and Appendix C.3).

Regarding the adaptivity of our FaP approach, we acknowledge the complexity of fusing models with radically different architectures. We've discussed the challenges and potential directions for future exploration. We kindly request the reviewer to reconsider their evaluation in light of the addressed concerns and the enriched comparison.

[1] CHIP: CHannel independence-based pruning for compact neural networks. NeurIPS, 2021, Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, Bo Yuan.

[2] Model Fusion via Optimal Transport . NeurIPS, 2020, Sidak Pal Singh, Martin Jaggi.

We thank the reviewer for the detailed feedback and for recognizing the benefits of combining model pruning and model fusion.

---

How is the method compared to other recent top-performing structured pruning methods

Intra-Fusion challenges the default meta-pruning approach:

It is important to note that we are not competing with any specific pruning methodology per se, but with the overarching “meta-pruning” approach, i.e. how a given importance metric is used.

Validation of Intra-Fusion on CHIP and random importance metric:

However, we are happy to further validate that Intra-Fusion manages to leverage a given importance metric more effectively than the traditional approach of simply discarding the least important neuron pairings. Hence, we added experimental results of deploying the importance metric in [1] (see Appendix E.2, Table 18). To further highlight that the improvements of Intra-Fusion are agnostic to the choice of importance metric, we include results of choosing importance at random (see Appendix C.3).

---

The advantage of the proposed method seems quite marginal

The significance of the proposed data-free pruning method lies in its ability to alleviate the resource-intensive nature of traditional model pruning. Nowadays pre-training large models on extensive data can be extremely laborious and confined to a selected few that have access to extensive resources. In many cases, the models are open-sourced, but the data used for pre-training are not. Providing a data-free way to compact these models without the need for extensive re-training can have large implications, particularly where data privacy concerns, resource constraints, or domain-specific limitations may hinder the availability of training data.

---

The paper (Frantar & Alistarh, 2023) [2] seems not to be structured pruning paper. It is unstructured.

Thank you for bringing this to our attention! In the revised version we have replaced the citation with [3], which unequivocally concerns structured pruning.

---

Question: Training cost of the method compared to the default pruning scheme

Algorithmic complexity as in OTFusion:

Regarding a concrete analysis of our Intra-Fusion approach, we would like to kindly refer to the analysis done in the original OTFusion paper [4] as our Intra-Fusion approach exhibits the same algorithmic complexity.

Temporal cost of Intra-Fusion seems negligible in face of training times:

In case you are interested in concrete runtime figures from practice, we would like to also provide these. Deploying pruning via Intra-Fusion (without iterative retraining) in our pipeline takes about 3.78s. The default pruning scheme in contrast runs in about 2.45s. In the context of training, fine-tuning and iterative pruning (with fine-tuning) we find this increased temporal cost to be negligible.

---

In conclusion, we sincerely appreciate the reviewer's insightful feedback and thoughtful recognition of the strengths of our work. We have taken great care to address the concerns raised, providing additional experimental results and clarifications. We kindly request the reviewer to reconsider their evaluation in light of these improvements.

[1] CHIP: CHannel independence-based pruning for compact neural networks. NeurIPS, 2021, Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, Bo Yuan.

[2] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ArXiv, 2023, Elias Frantar, Dan Alistarh.

[3] Towards Any Structural Pruning. CVPR, 2023, Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang.

[4] Model Fusion via Optimal Transport . NeurIPS, 2020, Sidak Pal Singh, Martin Jaggi.

We thank the reviewer for the detailed feedback and for recognizing the benefits of combining model pruning and model fusion.

---

Writing Improvements:

Specifically, we would like to thank the reviewer for directing our attention towards the listed writing issues. We have gladly taken this as an opportunity to improve the quality and readability of our paper. In particular, to further the clarity of our approach, we have corrected the mentioned issues and rewritten the cause-laden sentences that we could identify. Please inform us if there are any remaining sections that are still difficult to understand so that we can remove any possible ambiguity.

---

What does "meta" manifest in?

The “meta-level” of deploying importance metrics:

By “meta” in the context of “meta-pruning”, we refer to the process of taking a step back and considering what happens on a “higher/meta level” when doing pruning. Specifically, in the context of pruning the current paradigm is: keep the most important neuron pairings in a group and discard the rest. While most research has gone into developing more sophisticated and effective importance metrics, the overlying “meta-level” of leveraging these importance metrics has stayed the same. Intra-Fusion tries to challenge this and proposes an alternative way of utilizing an agnostically given importance metric.

Intra-Fusion leverages an agnostic importance metric more effectively:

Namely, we propose a new “meta-pruning approach” that instead of discarding the less important neuron pairings, integrates their learned features into the more important neuron pairings. To further highlight that Intra-Fusion is a more effective way of leveraging any agnostically given importance metric, we additionally include experimental results of using random scores drawn from a uniform distribution as an importance metric (see Section C.3).

---

So, is 'cardinality' defined in the context of layers or groups?

We want to thank the reviewer for this question. The word “(group) cardinality” stands for the number of neuron pairings in a group, where all neurons contained in a neuron pairing have to be handled in unison. We realize that mentioning the concept of cardinality in the context of layers and in the context of groups might be confusing. In the updated version we now solely refer to “group cardinality”.

---

Please, let us know in case there are any further questions or comments. We will be more than happy to answer them or add further clarifications. Otherwise, in light of the above, we hope you will give your evaluation of the paper a second thought. Thank you once again for taking the time and we remain at your disposal.