End-to-End Neural Network Compression via $\frac{\ell_1}{\ell_2}$ Regularized Latency Surrogates

We thank the reviewer for the valuable feedback. Below we address some of the key concerns raised.

$**Limited Novelty**$: We would like to emphasize that introducing masks in the objective is not our contribution. Our main contribution is to address the challenges that arise in optimization of the resulting objective. Below we highlight our key contributions

Our first contribution is to design a novel $\ell_1/\ell_2$ regularizer for optimizing the FLOP regularized objective. This helps us effectively compress models with batchnorm and layernorms. This is in contrast to prior works which use $\ell_1$ based FLOPs regularizer and as a result fail to work well in the presence of layer/batchnorms. Please see Figure 2 in our paper for a comparison of our technique with $\ell_1$ norm based methods.

The second contribution is to provide a simple algorithm for solving this objective. Past works have mostly relied on SGD/Adam to solve the FLOPs regularized objective. However, these algorithms are extremely slow to converge to sparse solutions and often require post processing steps. In our work, we overcome this problem by using proximal/projected techniques that are proven to converge to sparse solutions [1].

$**Experiments on diverse models**$: We note that the experiments we presented in the paper already involve diverse models (BERT, MobileNet, EfficientNet, ResNet) and diverse layers (Transformer, Linear layers in BERT, Convolutions in MobileNet, EfficientNet). We also experimented with various compression building blocks including low-rank factorization, channel/neuron pruning, quantization, as shown in figures 1,3,4 and 5.

$**Transfer learning and compression**$: We perform transfer learning and compression at the same time in our experiments, and add the FLOPs regularizer term to the cross-entropy loss during transfer learning.

[1] Boyd, S.P. and Vandenberghe, L., 2004. Convex optimization. Cambridge university press.

We thank the reviewer for the valuable feedback.

$**Quantization**$: We would like to note that we do have quantization results in the appendix (please see Figure 5 in the appendix). Due to space constraints, we didn’t place these experiments in the main paper.

$**On device performance**$: In Appendix B.1 we describe how to take the device into account for model compression using our framework. We develop a L1/L2 latency surrogate which takes the actual on-device latency into account. This helps directly optimize for on-device latency. In figure 6, the left plot presents the latency vs accuracy curves obtained using this technique for mobilenetv3 compression. The results show that working with latencies indeed helps us find models with better latencies than FLOPs regulrizer.

We thank the reviewer for the valuable feedback. Below we address some of the key concerns raised.

$**Limited Novelty**$: We would like to emphasize that introducing masks in the objective is not our contribution. Our main contribution is to address the challenges that arise in optimization of the resulting objective. Below we highlight our key contributions

Our first contribution is to design a novel $\frac{\ell_1}{\ell_2}$ regularizer for optimizing the FLOP regularized objective. This helps us effectively compress models with batchnorm and layernorms. This is in contrast to prior works which use $\ell_1$ based FLOPs regularizer and as a result fail to work well in the presence of layer/batchnorms. Please see Figure 2 in our paper for a comparison of our technique with $\ell_1$ norm based methods. We also compare against such techniques on a larger scale in figure 3(c), and empirically demonstrate our technique's better performance.

The second contribution is to provide a simple algorithm for solving this objective. Past works have mostly relied on SGD/Adam to solve the FLOPs regularized objective. However, these algorithms are extremely slow to converge to sparse solutions and often require post processing steps, which include extra hyper-parameters for thresholding the sparse values.This limits their ease-of-use for practical deployment. In our work, we overcome this problem by using proximal/projected techniques that are proven to converge to sparse solutions [1].

$**Multiple evaluations of our method**$: We would like to note that the experiments we performed on BERT, MobilenetV3, EfficientNet are large scale and it is computationally expensive to repeat these experiments multiple times. That being said, we note that we tried our technique on diverse tasks (please refer to experiments section in the paper) and noticed stable performance/loss curves across all the tasks.

[1] Boyd, S.P. and Vandenberghe, L., 2004. Convex optimization. Cambridge university press.

We thank the reviewer for their response. We would like to follow up on the novelty aspect. In this work, we are addressing a practically relevant optimization aspect that is often neglected by the model compression/NAS community. We believe this is very important for making these algorithms more applicable in practice. If the reviewer believes these ideas have been explored in prior works, we would be very grateful if the reviewer can point us to some papers that have systematically tackled these issues.