CoNNect: A Swiss-Army-Knife Regularizer for Pruning of Neural Networks

[RW1]: We understand the reviewer's concern about the complexity of the implementation of CoNNect. The computational overhead is in fact very moderate given that CoNNect can be computed in a time that is bounded by the time it takes for a single forward pass the neural network. More formally, for a batch size of $M$, the time used for computing CoNNect is proportional to $1/M$. We kindly refer the reviewer to our response [RQ1] of reviewer SZg4 and [RQ1] of eRZy. We have emphasized this point in the revised manuscript in lines 310-314.

[RW2]: We understand the reviewer's concern. However, it is not our goal to compare CoNNect with the suggested pruning frameworks, such as Lottery Ticket Hypothesis-based pruning (Frankle & Carbin, 2018). Instead, we want to demonstrate how CoNNect can be integrated into such pruning frameworks. We believe CoNNect is a conceptual contribution that, without much effort, can be used to enhance many hard pruning approaches (as we show for magnitude pruning and LLM-pruner). The fact that we can show good results argues for the importance of connectivity in neural network performance (Axiom 2). In the paper, we show how magnitude can be improved when training a regularized model using CoNNect, rather than training it via standard L1 regularization. On a similar note, one could study if integrating CoNNect in established pruning approaches, such as Lottery Ticket Hypothesis-based pruning, can lead to better results. To better emphasize the positioning of our paper in terms of its contribution, we have revised the manuscript accordingly.

[RW3]: We have included an example of CoNNect on GNNs in Appendix D.2 in the revised manuscript. Moreover, we have transformers covered through our implementation in LLM-pruner in Section 4.3. Also, we have implemented modern CV models, please see response [RW4] of reviewer SZg4.

Due to limited time, we have not implemented CoNNect in RNNs, but one would have to account for the following. Observe that $W$ becomes a cyclic graph (through the recurrent connections), with infinitely long paths. It follows that $\varphi(W) = \sum_{k=1}^\infty (\theta(W))^k = (I - \theta(W))^{-1} - I,$ and we must show that this exists. Note that it is sufficient to ensure that the largest eigenvalue of $\theta(W)$ is strictly less than 1, which follows by assuming that the output layer does not have recurrent connections, as is common for RNNs.

[RW4]: We agree that hyperparameters can have effects on regularizer behavior. Please note we already included several ablation results for the experiment in Section 4.1, which can now be found in Appendix D.1. Moreover, we have added several more by changing the regularizer coefficients. CoNNect appears to be very robust in terms of its hyperparameter and consistently outperforms L1 and L2 regularization. In fact, the weakest performance of CoNNect, see Figure 9(b), still manages to beat almost all cases of L1 and No regularization.

[RQ1]: We appreciate the references provided by the reviewer and are willing to elaborate on their relationship in our paper. But the comparison with these two methods may not be entirely "apple-to-apple." SparseGPT is a complex iterative algorithm that always requires parameter updates during pruning. And Wanda focuses on unstructured pruning, where the pruned model may encounter issues such as discontinuous memory access during inference or require specialized hardware to address it. In contrast, in our experiments in Section 4.3, we only extend the loss function in the LLM-pruner to include CoNNect. Only such a small change already results in improved results, which argues for connectivity-based pruning. This is what makes CoNNect a conceptual contribution since it can be applied in pre-established pruning frameworks such as LLM-pruner. In fact, we believe integrating CoNNect in SparseGPT and Wanda can be an interesting direction for future work.

We further provide results under more aggressive sparsities for the reviewer’s reference, where our method generally demonstrates an advantage. Experiments were set to prune 60% and 90% of the structures, ultimately resulting in sparsities of 47% and 70%, respectively. To align with SparsGPT/Wanda, we present the following accuracy results without normalization.

Table.1 Llama-7B Accuracy Results.

[RQ2]: We thank the reviewer for these suggestions. We test our method on other series and scales of models, and the results show that our method still demonstrates positive effects. More results on CV models can be found in our response [RW4] to reviewer SZg4.

Table.2 Vicuna-7B Accuracy Results

Table.3 Llama-13B Accuracy Results

References: Frankle, Jonathan, and Michael Carbin. "The lottery ticket hypothesis: Finding sparse, trainable neural networks." arXiv preprint arXiv:1803.03635 (2018).

[RW1]: We thank the reviewer (and reviewer SZg4) for this comment on residual neural networks. We mainly omitted the residual connection in the analysis because it makes the presentation of the theoretical results easier.

Simply connecting input and output can only ensure that the network avoids layer collapse, but it does not further aid in model compression. While residual connections can ensure a direct link between input and output features, CoNNect can be used to further enhance the connectivity rather than replace these connections. Despite the existence of residual connections, the backbone's processing of input information remains indispensable; otherwise, the entire network would degrade. CoNNect enhances the overall connectivity of the network by optimizing the parameters of the backbone, thereby facilitating the compression of the backbone modules without hindering the performance too much.

We have included a detailed implementation of CoNNect in Appendix B, also highlighting the case of residual connections. Regularizing with CoNNect is feasible because residual connections are typically non-parameterized and therefore won't bother the optimization process when enhancing the network's connectivity.

To further demonstrate CoNNect on residual neural networks, we apply CoNNect on ResNet-56, please see our response [RW4] to reviewer SZg4 for these results.

[RW2]: We thank the reviewer for pointing out that we should compare CoNNect with the newest L1 regularization methods and have added these references in our revision. We believe we have in fact implemented the loss of shrinkage operator in our current numerical results. To study the competitiveness of CoNNect against Spred-L1 regularization, we have applied Spred-L1 in the numerical example in Section 4.1 and see that its performance indeed improves upon standard L1 regularization and matches CoNNect's performance. However, this comes at the cost of doubling the number of trainable weights, which increases the time and space complexity, whereas CoNNect does not suffer from such an increase in complexity. We also have compared Spred-L1 in the same manner of Section 4.2 by following the official implementation and still highlight our superiority over it in the table below.

Table.1 Results of CoNNect and Spred-L1 for training full-scale VGG-11 on CIFAR-10.

[RW3]: We chose the CoNNect hyperparameter $\lambda_2$, see (7) in the revision, such that we can show we obtain good results at high pruning ratios. To show better results in lower pruning ratios, one can simply decrease the value of our $\lambda_2$ (where for $\lambda_2 \to 0$, we approach the performance non-regularized model).

[RW4]: Thank you for pointing out both typos. We improved our manuscript accordingly.

[RQ1]: In the manuscript, we present CoNNect via the adjacency matrix because that is how the connectivity measurement in Katz centrality is initially defined. However, in our implementation of CoNNect we do not use the adjacency matrix. Instead, it suffices to define a slightly adapted forward function of the neural network. We (for more implementation details, see Appendix B in the revised manuscript). The time complexity of the adjusted forward function is bounded by the original forward function, see also our response [RQ1] for reviewer SZg4. That means that for a batch size of $M$, the time used for computing CoNNect is proportional to $1/M$, which is for modern batch sizes a very moderate computational cost.

[RQ2]: We evaluate the computational overhead of LLM from Section 4.3 in terms of MACs. Since we share a similar grouping strategy, we can achieve inference acceleration comparable to LLMPruner. More results on CV models can be found in our response [RW4] to reviewer SZg4.

Table.2 Statistics of the base model and the compressed model.

Dear authors,

Thanks for your time and effort in providing detailed response and extra experiments. As my W2, W3, W4, Q1, and Q2 are addressed I raise my score from (rating = 5, conf = 3) to (rating = 6, conf = 4).

I am not considering a higher score, because the current analysis does not apply to residual structures (which is ubiquitous in modern deep neural networks). This limitation weakens the contribution of this paper.

Best,

Reviewer eRzy

Dear authors,

I would like to clarify my previous comment: my initially opinion is that, the current 'connectivity modeling' did not explicitly take the skip connection into account, and it just treat the residual shortcut as a constant and omit its effect on connectivity.

From my understanding, the residual shortcut is indeed an unprunable weight that significantly affects the global connectivity. Let's consider a residual MLP given by the code attached below, if we prune all the neurons in self.hidden_layer1, the connectivity edge between neurons from self.input_layer and self.hidden_layer2 is still nonzero, as they are connected by the shortcut. This fundamentally change the adjacency matrix induced by the neurons / weights. I guess this can also change the property of the stationary points of the penalty.

Code attachment:

import torch
import torch.nn as nn
import torch.nn.functional as F

class ResidualMLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(ResidualMLP, self).__init__()
        self.input_layer = nn.Linear(input_dim, hidden_dim)
        self.hidden_layer1 = nn.Linear(hidden_dim, hidden_dim)
        self.hidden_layer2 = nn.Linear(hidden_dim, hidden_dim)
        self.output_layer = nn.Linear(hidden_dim, output_dim)
        self.activation = nn.ReLU()

    def forward(self, x):
        # Input layer
        x = self.activation(self.input_layer(x))
        
        # Residual block 1
        residual = x
        x = self.activation(self.hidden_layer1(x))
        x += residual  # Add skip connection
        
        # Residual block 2
        residual = x
        x = self.activation(self.hidden_layer2(x))
        x += residual  # Add skip connection
        
        # Output layer
        x = self.output_layer(x)
        return x

[RW1]: We agree with the reviewer that Axiom 2 can use improved motivation. During pruning, it can happen that many or all connections between two layers are pruned, rendering the pruned model to be useless. We have illustrated this using a figure in the introduction. We also slightly revised the text to emphasize this point. Moreover, through a series of numerical experiments, we demonstrate how incorporating CoNNect into existing pruning methods enhances neural network connectivity and yields good results.

[RW2]: Please note it isn't Katz centrality but the connectivity measurement in Katz centrality that we are using. This form of connectivity measurement allows us to favor a few direct paths over many parallel paths. We do not see how other connectivity metrics in graph theory can achieve the same result.

[RW3]: The goal of the small MLP model in section 4.1 is to provide a small example that illustrates the working of CoNNect. We believe that it nicely illustrates how typical regularization approaches fail in finding the sparsest network representation, whereas CoNNect much more often succeeds. Moreover, to strengthen our results, we performed many ablation tests in Appendix D.1 and see that CoNNect is much stronger than L1 and No regularization.

[RW4]: We thank the reviewer for pointing out this typo. We have adjusted the manuscript accordingly.

[RW1]: We thank the reviewer for this comment. Neural network connectivity is crucial, as without it, the input signal cannot reach the output, typically resulting in a non-functional neural network. We have illustrated this by slightly revising the introduction and inserting Figure 1 after Axiom 2 in the revised manuscript. Moreover, we show through various numerical experiments how integrating CoNNect in existing pruning methods improves neural network connectivity. The good results emphasize the role of Axiom 2 in ensuring the network’s functionality.

[RW2]: The goal of this analysis is to show that all stationary points induced by the CoNNect regularizer are unstable, except for the global minimizers. What our result shows is that CoNNect itself does not contain any stable stationary points that can serve as an attraction region. In case the algorithm gets stuck in a stationary point of the regularizer, the loss function will always push the solution to leave the stationary point unless the loss function itself is stationary at that point.

[RW3]: Applying CoNNect to residual neural networks is in fact quite straightforward. The identity connection itself does not contain any trainable parameters and thus can be ignored when regularizing with CoNNect. Based on the reviewer's suggestion, we now provide a detailed implementation of CoNNect in Appendix B. Moreover, please see our response at [RW4] for an application on modern CV models.

[RW4]: Similarly to Section 4.3, we integrate CoNNect into the preceding work of LLMPruner, DepGraph (Fang et al., 2023), and obtain the following results. DepGraph is also a structured pruning method that can be based on Taylor's importance. We iteratively prune until the predefined speed-up targets, such as 2.5$\times$, 8$\times$, or 16$\times$, are achieved, which is calculated as the ratio of MACs before and after pruning.

Table.1 Results on CV models.

[RQ1]: Fortunately, the time complexity of CoNNect is bounded by a single forward pass of the NN. Thus, for a batch size of $M$, the additional time used for computing the CoNNect regularizer is proportional to $1/M$. Moreover, the space complexity is negligible since we only have to define a slightly adapted forward pass function for the same neural network and no storage of additional parameters is required. We have emphasized this in the revised manuscripts on lines 310-314.

[RQ2]: We thank the reviewer for finding our illustration helpful. We have added a figure showcasing layer collapse in the introduction.

References: Fang, Gongfan, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023): 16091-16101

[RW1]: We appreciate the reviewer's comment and would benefit from specific guidance on this point. Are there any specific works that you would like us to discuss in the manuscript?

[RW2]: We understand the reviewer's concern. However, even after an extensive literature search, we do not see to what references the reviewer refers to.

In terms of the paper's contribution, we think that CoNNect provides a conceptual contribution to neural network pruning and can enhance existing pruning methods by improving neural network connectivity (Axiom 2). CoNNect's versatility is demonstrated via a soft pruning approach (e.g. regularization), and, as we show in Section 4.3, it is easily integrated into hard pruning approaches, such as one-shot pruning via LLM-pruner (Ma et al., 2023). Our positive numerical results highlight the importance of connectivity in neural network performance, stressing the relevance of Axiom 2.

[RW3]: Following up upon our previous comment [RW2], we merely want to demonstrate how CoNNect can complement current neural network pruning approaches. Moreover, it is not that we are conducting a comparison between CoNNect and methods such as magnitude pruning. Instead, we show how a magnitude pruning can benefit from using CoNNect regularization as opposed to traditional methods such as L1 (see Section 4.1). We have emphasized this in the revision to avoid such confusion.

In terms of integration of CoNNect within various pruning methods, we would like to highlight that CoNNect shows good results versus DepGraph (see our reply [RW4] on reviewer SZg4 for these results) and LLM-pruner, both of which have been compared with a large variety of methods and have outperformed these. This implies that CoNNect is also competitive against these methods.

[RW4]: We believe the reviewer may have misread Table 2, as it does show significant differences. We improve the results of LLM-pruner (Ma et al., 2023), a highly cited paper in the field, in almost all cases via integrating CoNNect. Moreover, the differences are comparable with those achieved in the LLM-pruner paper. We have simplified the presentation in Table 2 in the revision and have stressed these results in the text. We hope any remaining confusion is resolved.

[RQ1]: We can best highlight theoretical results in terms of the numerical example in Section 4.1. What we show is that CoNNect outperforms benchmark regularization methods when performing magnitude and SynFlow pruning. In this example, it becomes clear that traditional regularization methods suffer from layer collapse at high sparsity levels when pruning, or even already during training. Instead, CoNNect manages to train the NN in such a way that the weights can be pruned according to a simple magnitude-based pruning strategy. In other words, it is trained to prevent layer collapse and so it is guaranteed to satisfy Axiom 2.

References: Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.

Dear Reviewer JAt6,

As I have also been working on connectivity enhancing pruning. I am also curious about, what are the latest works that perform the best in accomplishing connectivity enhancement in pruning? Would you please provide some reference?

Best,

Reviewer eRZy

The reviewer would like to thank the authors for the clarifications.

However, a fundamental issue still remains and that is the performance of the proposed approach when compared with other recently proposed approaches in pruning (such as expander based 'Pal et al. Revisiting the Lottery Ticket Hypothesis: A Ramanujan Graph Perspective ICLR 2022', 'Hoang et al. Revisiting Pruning at Initialization through the Lens of Ramanujan Graph ICLR 2023 or weights and activation based 'Sun et al. A Simple and Effective Pruning Approach for Large Language Models ICLR 2024').

Even though the approach based on the proposed regularizer may be a conceptual contribution, the end results are not convincing yet. Also, it turns out that the concept of Katz centrality is not entirely novel as similar concept as in the paper has been available in the existing literature.

We appreciate the reviewer's acknowledgment of our clarifications. We hope to address the remaining fundamental issue below.

We have consulted the suggested papers and compared their performance with CoNNect results. Pal et al. (2022) mostly shows results on older architectures, such as Lenet. They provide a single result for VGG-19 on CIFAR-10 (see their appendix A.5). For comparison, we apply our CoNNect implementation via DepGraph (Fang et al., 2023) and show how we consistently outperform in Table 1. Please note that the final density is not exactly controllable and one must aim for a target MAC speed-up (the ratio of results before and after pruning). We refer to Fang et al. (2023) for further reference.

Table.1 Results of VGG-19 on CIFAR-10 dataset by CoNNect.

Comparing us to Hoang et al. (2023), we show how CoNNect outperforms all of their ResNet-34 implementations. Even while considering that the approaches used in Hoang et al. (2023) are unstructured pruning, which are known to perform better in accuracy than structured approaches, see Table 2.

Table.2 Results of ResNet-34 on CIFAR-10 dataset by CoNNect.

Considering the Wanda paper (Sun et al., 2023), we want to highlight that Wanda's performance is inferior to ours in terms of inference speed-up. Given that Wanda utilizes an $n:m$ semi-structural pruning, they achieve a wall time speed-up of around $1.6 \times$ at 50% sparsity level, where CoNNect + LLM-pruner performs a structural pruning, achieving a speed-up of around $1.9\times$ at the same 50% sparsity level. For more details, we would like to refer the reviewer to our response [R2] to Reviewer DVNq (at 29 Nov 2024, 12:09 CET).

Finally, we would like to explain that CoNNect does not utilize Katz centrality for pruning, but merely the connectivity measurement employed in Katz centrality, which is a fundamental difference. Assuming the reviewer refers to Li et al. (2020), who use Katz centrality in NN pruning, we would like to refer the reviewer to our response [R1] to Reviewer DVNq (at 29 Nov 2024, 12:09 CET) to highlight our differences with this work.

We hope these comments made our results more convincing and will lead to a favorable score. Please do not hesitate to reach out if further clarification is needed.

References:

Fang, Gongfan, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023): 16091-16101

Sun, Mingjie, Zhuang Liu, Anna Bair, and J. Zico Kolter. "A simple and effective pruning approach for large language models." arXiv preprint arXiv:2306.11695 (2023).

Li, Wenjing, Minghui Chu, and Junfei Qiao. "A pruning feedforward small-world neural network based on Katz centrality for nonlinear system modeling." Neural Networks 130 (2020): 269-285.