Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once

Dear reviewer,

Thank you for the time and effort you have invested in reviewing our manuscript. We appreciate your valuable comments and suggestions. Below, we address each of your points in detail.

Q1: The whole pipeline looks like a huge ensemble of existing techniques, such as the "Dense Pivot Co-training" stage from USNet and BigNAS, the "Sparse Network Interpolation" stage from AutoSlim and LotteryPool … However, the author did not make meaningful discussions in each stage, on their differences from prior arts. I would like to hear the authors clarify.

A1: We would like to argue that one of our major novelty is to expand the scope of sparse co-training to cover diverse sparsity patterns and multiple sparsity ratios at once. This research goal stands out as novel because previous works have not addressed such a wide range of sparse patterns and ratios. The harmony among different sparsity patterns is made possible by UMG and Dense Pivot Co-training. By using UMG as a universal pruning criterion and producing closely aligned sparse masks, we relieve the gradient conflicts of different sparsity patterns during training. Then by Dense Pivot Co-training that inserts a dense mini-batch step at every alternative sparse mini-batch step, we further enforce the optimizing directions of subnetworks of different sparsity patterns to be aligned with the same dense network. Meanwhile, the dense network at each IMP iteration has the same initialization as in the Lottery Ticket Hypothesis, thus the optimization directions from different IMP iteration are aligned because of the same dense network initialization. Thus, the optimization directions from different sparsity ratios and patterns are all regularized to be aligned together. We are also the first one to apply LTH for sparse co-training to amortize the sparse co-training pressure (we only co-trains subnetworks of a single sparsity ratio from different sparsity patterns at the same time in each IMP iteration, while finally it produces a lot more subnetworks in total without the need to co-train them together.), while related work such as Lottery Pool only considers aiming to produce a single stronger subnetwork.

We also do not simply reuse existing methods but develop novel adaptations for the sparse co-training circumstance. Specifically, (1) we adapt the refilling method in Chen et al[1] as UMG by incorporating N:M sparsity and letting both unstructured and N:M sparsity decide which channels to refill; (2) we adapt the AST[2](instead of USNet or BigNAS) as Dense Pivot Co-training by not just alternating mini-batches among sparse networks of the same sparsity ratio and pattern but inserting a dense mini-batch step and combine it with IMP to achieve optimization alignments across different sparsity ratios and patterns. (3) we adapt the network interpolation method in Lottery Pool[2] as our network interpolation for sparse co-training and the core difference is that they only aim to produce a single sparse network with an interpolated sparsity ratio, while we need to use the interpolation to produce multiple subnetworks with a shared parameter set.

Q2: I would like to see some more relevant metrics such as training time, memory savings, or inference speed ups if any. Without those, it is hard or meaningless to fetch any real benefit of training with sparsity.

A2: Thanks for the question. we want to emphasize that the motivation of our sparse co-training is parameter efficiency instead of training or inference speed up. Since for each kind of sparsity pattern, the Sparse Cocktail produces multiple subnetworks with exactly the same sparsity ratios w.r.t. other corresponding methods, we thus have almost the same memory savings, and inference speed up and omit the comparison. We also empirically find that Sparse Cocktail has only around 1/8 extra total wall-clock training time primarily due to 1 extra distillation step every 2 mini-batches.

Q3: Is Dense Pivot Co-training just weight rewinding (which is a pretty standard trick), or are they different (in which way)?

A3: The dense pivot co-training is an orthogonal technique w.r.t. the weight-rewinding. The weight-rewinding means rewinding the weights to initialization weights(or very early epochs) at the start of each IMP iteration, which is operated on the epoch level. For example, if the IMP has 10 iterations and each iteration trains the network for 100 epochs, then the weight rewinding will be performed for 9 times at the start of each iteration, from the 2nd to the last iteration. In contrast, the Dense Pivot Co-training inserts a dense network training step before each mini-batch of the sparse co-training, where parameters are shared and continuously updated without rewinding and it is operated on a mini-batch level. For example, at the i-th iteration of Sparse Cocktail, the mini-batch orders during training are arranged like this: dense —> unstructured subnetwork —> dense —> channel-wise subnetwork —> dense —> N:M subnetwork —> dense —> unstructured subnetwork ... To summarize, weight rewinding is an epoch-level operation and dense-pivot co-training is a mini-batch-level operation, and they are thus orthogonal.

Q4: Why the three mask generations in Section 3.4. are called “unified”?

A4: The mask generation is called "unified" primarily because now the selection of 3 masks are all based on individual weight magnitudes by changing the pruning criterion of channel-wise pruning. In traditional channel-wise pruning, the pruning criterion is usually based on the batch norm scale factor, which is different from individual weight magnitudes. If we combine this traditional channel-wise pruning criterion with weight magnitude-based unstructured and N:M pruning, there could be conflicts for sparse co-training with different sparsity patterns. Now in our proposed unified mask generation, this is changed by introducing the refilling criterion, which decides which channels to prune based on the magnitude sum of all individual weights in each channel. In this way, the sparse co-training can better orchestrate their shared parameters so that they will not produce conflicts in pruning criteria and cancel each other's performance.

Dear reviewer,

Thank you for the time and effort you have invested in reviewing our manuscript. We appreciate your valuable comments and suggestions. Below, we address each of your points in detail.

Q1: No discussion of training time cost. The proposed joint/switchable training appears to take much longer time than any single sparse training method. Please report the details and provide a fair discussion on training cost.

A1: Thanks for your question. We have discussed the training cost in Appendix D and will make this point more clear in the revised version. Overall, we keep the batch size and total training epochs of all the methods, including Sparse Cocktail, AST, AC/DC, and MutualNet, as 1500 epochs to have a fair comparison. Specifically, (1) for Sparse Cocktail, we use 10 iterations for IMP and each iteration contains 150 epochs. (2) for AST and MutualNet, we directly perform the co-training for 1500 epochs. (3) for AC/DC we co-train each of the 10 dense-sparse network pairs for 150 epochs. In this way, all the methods have the same number of training iterations and batch size (regardless of which subnetwork will be trained at each mini-batch), and thus the same training cost. We also empirically find that Sparse Cocktail has only around 1/8 extra total wall-clock training time primarily due to 1 extra distillation step every 2 mini-batches.

Q2: Hyperparameter setting was missed in Appendix C (empty - though mentioned multiple times in the main paper)!! This paper has so many moving widgets and it seems challenging to get all the hyper-parameters and settings right in practice.

A2: We apologize for this problem, we have presented the hyper-parameters in Table 5 and Table 6 in the Appendix, but this is misplaced away from the Appendix C section title due to the typesetting problem. We will fix this issue and add textual references in Appendix C in the revised version.

Q3: The algorithm is not entirely clear from the Figure and methodology section. For example, how many sparsities is each sparse pattern trained for, what are the performances of each sparsity pattern and how does a subnetwork’s performance improve after merging (if it does).

A3: Thanks for your question. The number of sparsity ratios can be a variable hyper-parameter while in our setting, we use 10 sparsity ratios for both unstructured and channel-wise sparsity, and 3 sparsity ratios for N:M sparsity (1:2, 2:4, 4:8) following prior sparsity work as we discussed in the start of Section 4. The performance of each sparsity pattern are shown in Table 1 and Table 2, generally speaking, unstructured and N:M subnetworks have comparable performance to the dense network and channel-wise subnetworks have degraded performance due to the structural sparsity nature. Regarding the performance influence of network interpolation, we list the average performance before and after Interpolation of Sparse Cocktail on image classification datasets. The results show that there will be slight performance degradation after interpolation, but we consider this a necessary sacrifice since we aim to produce a single network with shared parameters of multiple subnetworks.

On CIFAR10 dataset:

On ImageNet dataset:

Q4: The author’s don’t comment on the loss landscape of each of the subnetworks obtained during training. From previous work by Paul et al [1] I would expect each of the obtained subnetworks to lie in the same loss basin. In order to assess the effectiveness of the dynamic distillation step I would expect to look at the Hessian or the linear mode connectivity between the subnetworks obtained.

A4: Thanks for your comment. We empirically find that different subnetworks of the same sparsity pattern obtained by IMP with weight rewinding are located in the same loss basin, i.e. there are no significant error barriers for interpolations similar to Figure 2 in [3], while subnetworks from different sparsity patterns have at most ~3.2% error barriers on average due to the divergence in sparse masks. We show the latter phenomenon in Figure 7 in appendix, by plotting the performance change using different interpolation factors as in Algorithm 1. However, by finding proper interpolations the error barrier problem can be mitigated or avoided.

Dear reviewer,

Thank you for the time and effort you have invested in reviewing our manuscript. We appreciate your valuable comments and suggestions. Below, we address each of your points in detail.

Q1: I am concerned about the novel contributions of this paper, and the results presented in this paper are a combination of existing works with little novelty of its own.

A1: We would like to argue that one of our major novelty is to expand the scope of sparse co-training to cover diverse sparsity patterns and multiple sparsity ratios at once. This research goal stands out as novel because previous works have not addressed such a wide range of sparse patterns and ratios. The harmony among different sparsity patterns is made possible by UMG and Dense Pivot Co-training. By using UMG as a universal pruning criterion and producing closely aligned sparse masks, we relieve the gradient conflicts of different sparsity patterns during training. Then by Dense Pivot Co-training that inserts a dense mini-batch step at every alternative sparse mini-batch step, we further enforce the optimizing directions of subnetworks of different sparsity patterns to be aligned with the same dense network. Meanwhile, the dense network at each IMP iteration has the same initialization as in the Lottery Ticket Hypothesis, thus the optimization directions from different IMP iteration are aligned because of the same dense network initialization. Thus, the optimization directions from different sparsity ratios and patterns are all regularized to be aligned together. We are also the first one to apply LTH for sparse co-training to amortize the sparse co-training pressure (we only co-trains subnetworks of a single sparsity ratio from different sparsity patterns at the same time in each IMP iteration, while finally it produces a lot more subnetworks in total without the need to co-train them together), while related work such as Lottery Pool only considers aiming to produce a single stronger subnetwork.

We also do not simply reuse existing methods but develop novel adaptations for the sparse co-training circumstance. Specifically, (1) we adapt the refilling method in Chen et al[1] as UMG by incorporating N:M sparsity and letting both unstructured and N:M sparsity decide which channels to refill; (2) we adapt the AST[2] as Dense Pivot Co-training by not just alternating mini-batches among sparse networks of the same sparsity ratio and pattern but inserting a dense mini-batch step and combine it with IMP to achieve optimization alignments across different sparsity ratios and patterns. (3) we adapt the network interpolation method in Lottery Pool[3] as we state in A6 below.

Q2: The results are shown on different sparse subnetworks obtained from multiple sparse masks. However, it is likely that the performance of these sparse subnetworks is stable merely because of the relatively low sparsity reported in the paper. In order to see the effectiveness of the method, I would like to see the performance of the subnetworks with higher sparsity (> 90%) especially for unstructured sparsity patterns.

A2: We would like to argue the compared sparse co-training methods all have performance degradation when the sparsity gets very high. In the two compared unstructured sparse co-training methods, AC/DC [4] has ~3.5% performance degradation (compared to vanilla dense network) at 95% sparsity and ~8.5% degradation at 98% sparsity; AST [2] has less performance degradation at high sparsity primarily likely because it doesn't involve co-training with different sparsity ratios and only focus on single sparsity ratio but different masks.

Dear Reviewer,

Thanks for your further questions. We can address your concerns as follows:

Q9: However, I am still unconvinced of the novel contributions of the paper. It seems that the main goal still is to combine multiple existing methods by extensive hyperparameter tuning and engineering like Dense Co-pivot training.

A9: As we have discussed in Section 1, our main goal is to train multiple sparsity ratios and patterns all at once, in order to achieve real-time switching with the need to store only one set of shared parameters, based on the inference platforms that may have different supports on accelerable sparsity patterns and their available resources may vary over time which needs switchable sparsity ratios.

We do not include extensive hyperparameter tuning. As we show in Table 5 and 6 in the Appendix, we just follow the default hyperparameters as in the original LTH paper [5] for producing unstructured lottery tickets, and keep the element-wise sparsity roughly the same for channel-wise sparsity pattern by setting channel-wise pruning ratio as 0.1. The candidate pool is also not a hyperparameter but a search space of our greedy network interpolation algorithm. Other optimization hyperparameters are common default choices and we empirically find that they are not very sensitive to the final performance. The Dense Pivot Co-training does not include heavy engineering, as it can be simply implemented by inserting a dense training step and a distillation step every 2 mini-batches, and does not include any hyperparameter tuning.

Q10:Moreover, only by training IMP, I could potentially obtain multiple sparse networks...

A10: As we have shown in our ablation study in Section 4.2, every part of Sparse Cocktail has an important contribution to the final performance. In essence, sparse co-training with IMP and weight-rewinding is necessary for a normal performance and every other part contributes to the performance boosts. We do not think one can transfer from unstructured sparsity to N:M sparsity during IMP following the mentioned approach in Q10. Notably, we do not increase the training time or iterations compared to vanilla IMP which solely trains unstructured sparsity, as Dense Pivot co-training utilizes the graident alignment effect as in the AST paper[2] using the alternative sparse training. For example, in the original LTH paper, they used 182 epochs at each IMP iteration while in Sparse Cocktail we use 150 epochs with the same mini-batch size on the CIFAR10 dataset. Compared to other sparse co-training methods that can co-train multiple sparsity ratios (i.e. Mutual Net and AST), we also find they require a nearly full training schedule of 1500 epochs when co-training only 10 subnetworks.

Hope our response can address your concerns!

[5] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, Jonathan Frankle, et al 2019.