Learning Structured Sparse Neural Networks Using Group Envelope Regularization

We are pleased to address the points raised by reviewer kcJf.

1. We would like to draw the attention of the reviewer that $\theta^T A_{s_j}$ (as in equation) is not the L2 norm of the theta on index set $s_j$, but rather a vector where every entry which its index belongs to $s_j$ equals to its corresponding in $\theta$, and otherwise zero.
2. We would like to draw the attention of the reviwer that $s_k$ was defined in Remark 1s_k(\theta) = \frac{1}{2}|{\theta}|{2}^{2}$if$|{\theta}|{0}\leq k$and$s_k(\theta)=\infty$, otherwise.
3. Thanks for the author for his comment, we give a description of AdaHSPG+ algorithm and cition in section 5.
4. We have added additional results for cifar100.
5. Lasso works best when the underlying true model is sparse. In many real-world neural network applications, especially in complex datasets, this assumption of sparsity might not hold true. Hence, applying Lasso could lead to an oversimplified model that fails to capture the complexity of the data. On the one hand, we know that methods of this type create a built-in bias, but on the other hand, they reduce the model variance, and thus the performance increases. This has been proven in linear models (linear regression) and shown empirically in deep learning models. Since the regularizer we propose embodies prior knowledge about the model suitable for the problem, i.e. it is over-parameterized and therefore has redundant components, then we can assume that the limited search space, the one that will lead to a bias in the model, is also the one that will necessarily reduce the variance of the model.
6. There is interchangeability between K and Lambda, but an advantage of WGSEF is that we only really need to control Lambda. K should be set so it represents the desired sparsity level, and for reasonable values of Lambda it will be a lower bound on the sparsity level. The user only needs to adjust Lambda, which is no different than Lasso / Group Lasso.

We are pleased to address the points raised by reviewer FGZh.

1. We have re-edited the related work, sections 5 and 6.
2. We have added additional results for cifar100.
3. An advantage of our framework over others is that it doesn’t require us/the user to “discover” groups. Groups naturally arise from the NN architecture, where in CNNs these are the filters, in a dense layer these are the columns, and in a transformer can be the attention heads. We are focused on the hardware, and the network weights can be represented as sparse matrices (e.g. SciPy BSR, CSC, CSR matrices), thus reducing the required disk space and RAM, and allowing faster inference. Since during inference, a block that represents a neuron is all zeros, there is no need to multiply and sum any incoming elements. Chen, Tianyi, et al. "OTOv2: Automatic, Generic, User-Friendly.", on the other hand, is a method that discovers Zero-Invariant Groups (ZIGs), which are groups that can be removed from the network while keeping the inter and intra-dimension “correct”, which is far less flexible than our method. OTO considers shrinking the network, we consider compression via sparsification, but we still consider advances in GPU hardware that allow speed-ups using sparse matrices. We gain flexibility, without compromising on the accuracy. Please see announcements from Nvidia in cuSPARSE:

a. Basic Linear Algebra for Sparse Matrices on NVIDIA GPUs

b. Accelerating Matrix Multiplication with Block Sparse Format and NVIDIA Tensor Cores

We are pleased to address the points raised by reviewer xL1V.

1. First, we needed to find a way to extend SEF to a group structure. After-the-fact, imposing the L2 norm on the groups followed by the L0 norm on these groups norms may look natural, but this wasn’t the case when we started. We have considered different norms, and squared norms (which we realized creates a degenerated SEF instance). After reaching the final setup of the L2 norm on the groups, we still needed to develop an optimization algorithm. Adding group weights was not simple and for that reason, we began with fixed-size groups, however, since this was limiting we extended it again to allow arbitrary group sizes.
2. The results for the method solely were presented in each of the tables except for the first one, where the purpose of using HSPG was to make a similar comparison in terms of the optimization algorithm that incorporates the regularize.
3. We have added additional results for cifar100. We viewed ResNet / VGG / MobileNet as the main architectures, which in particular are being used in applications on edge devices where the HW resources are scarce. As for transformers, our method is very suitable for it, specifically for the attention blocks, and we would add these experiments in future works. However, the literature on NN sparsity specifically in structured sparsity is focused more on CNNs (and previously on RNNs). As we wanted to benchmark our work to SOTA structured sparsity algorithms, we focused on CNNs. Moreover, transformers are relatively new architecture, and only recently (~2 years) have reached their massive sizes of billions of parameters, so sparsity is getting more attention there only now, and thus there is less prior work to compare with.

a. We have considered this option (variants of Adam optimizer), but unfortunately, our projection algorithm is not currently suitable to be used with Adam. Proximal-type algorithms solve the problem: $\min_v \{f(v) + \frac{1}{2}\|v - x\|\}$. With “standard” proximal methods such as proximal gradient or SGD-prox, we solve with regards to the L-2 norm, and our proof heavily relies on properties of the L-2 norm. In order to use Adam, we will need to solve the prox problem with regards to some D-norm, with D being a diagonali8 PD matrix. For details on a “prox-Adam” algorithm, please see Yun et al. “Adaptive proximal gradient methods for structured neural networks.”, section 2. b. It’s important to note that we are still able to run the algorithm is less epochs than other benchmark methods, and achieve SOTA results. While it would be nice to use Adam, the optimization procedure itself is a means to an end, not the end itself. 5. a. Thanks for your comment we extended the survey of the previous works in the revised version. b. Our main contributions are stated in the abstract and intro and are as follows: We suggest an efficient algorithm to impose group sparsity, which is flexible to be used with any NN architecture. We allow to define groups by hardware properties so that sparsification yields significant inference speed-ups while keeping accuracy as-is. Moreover, we consider the simplicity of the algorithm as an advantage. Experimental results are good compared to SOTA. 6. We believe that a good sparsification algorithm / framework needs to also consider the optimization algorithm, otherwise it might be useless (for example, L0 norm which is intractable). Finding the right algorithm wasn’t as straightforward as anticipated, therefore we provide details on this subject. 7. Group Lasso imposes L2 regularization on each group separately, without tying groups together. Therefore it imposes inter-group weight sharing, but not intra-group regularization. Our method, on the other hand, provides both types of regularizations in a holistic manner. Moreover, we believe the L0 formulation pushes the entire group values to be set to zero where this is the underlying real structure, while keeping positive groups almost unchanged (this is an intuition and not formally analyzed or proved). 8. The whole inverse problem simply means the whole optimization problem (that is basically an inverse problem - finding model parameters). 9. Thanks to the reviewer for the correct comment. Most deep learning applications on edge devices can be considered as in the interpolation threshold regime (high amount of parameters but not huge + the models were trained on a sufficient amount of training set), over-parameterization results in redundant variables without improving network results / accuracy. It requires more disk-space to store weights, more vRAM to load weights, and results in more FLOPs during inference, requiring more expensive hardware and more energy consumption, as well as environmental implications. This happens in main real-world deep learning applications, especially in end devices. 10. See bullet (6a) on this. 11. The datasets used are written inside the table - table 1 column 2.