Meta Compression: Learning to compress Deep Neural Networks

We thank the reviewer for their valuable feedback. Our responses follow.

“The novelty of this paper is limited. The proposed method is a simple machine learning problem, which is easy to prove in a certain hypothesis.”

We address a fundamental question in this paper - can we predict the compression accuracy tradeoff of a novel pre-trained model without actually compressing the model? We formulate this problem and devise a framework based on meta-learning that solves this problem, building upon strong theoretical foundations. Furthermore, we demonstrate that diffusion models can be leveraged to generate additional evaluation data resembling test distribution in a meta-learning context.

“In the actual scenarios, the performance of a certain compression method is highly related to multiple hyper-parameters. This paper does not consider these factors.”

We consider two key hyperparameters for compression methods - target sparsity level and target quantization level. Fine-tuning hyperparameters such as learning rate, number of epochs, etc. were kept fixed and decided based on the values used for pretraining.

“The pruning and quantization are usually implemented serially in practice. However, the proposed method considers them separately and then merges them together using a simple criterion.”

All our experiments involve applying pruning followed by quantization serially. There is no merging criterion. For instance, if the optimal strategy is to use only pruning to 90% sparsity, the meta predictor will suggest pruning to 90% sparsity and quantizing to 32bit precision, which is equivalent to no quantization. In addition, we would like to highlight that the proposed framework can easily be extended to any joint pruning+quantization specification, for e.g. Slim pruning + LSQ quantization, as already discussed in the paper.

“The quantization and pruning methods tested in this paper are out-of-date. Also, other compression categories like low-rank matrix and tensor decomposition are not considered.”

The choice of compression methods selected in our experiments is based on a practical constraint - the availability of stable implementations that can compress a wide array of DNN architectures. Keeping this in mind, we considered all compression methods that could reliably compress all considered architectures with available implementations in the popular open-source model compression library - Neural Network Intelligence (NNI). We will consider additional compression methods in future work.

“According to Fig. 7, the proposed method does not work well on large-scale datasets.”

We agree that the performance gap between meta compression and exhaustive search is slightly higher in the case of ImageNet setup experiment, but this is mainly because fewer data points were used in the ImageNet setup experiment due to limited available compute in academic settings. However, we would like to highlight here that even in this case, the performance is significantly better than any static recommendation strategy using a single pruning and quantization method across all constraints.

“The training cost is still very high, i.e., 8 days. I don't think the proposed framework can save time a lot as compared to existing SOTA compression methods.”

While the cost of training the meta-prediction model is comparatively high, the main benefit of using the proposed approach is that obtaining recommendations for a new deployment scenario, consisting of previously unseen pretrained architectures and compression level constraints, is extremely cheap (order of a few milliseconds). Following this, the model can be compressed using the recommended compression method specification. Meta compression outperforms network architecture search approaches here in terms of compute as the latter require significantly more compute for every new deployment scenario.

We thank the reviewer for their valuable suggestions. We will take them into account for improving the introduction and the illustration in Figure 2a. Please find below the answers to your questions.

“The construction process involves numerous architectural considerations and requires training the compressed model as well. While the training cost remains manageable for smaller datasets like CIFAR-10, it's essential to investigate whether this approach introduces excessive overhead when applied to larger datasets such as ImageNet-1000.”

While one key benefit of the proposed framework remains that the computation cost of obtaining compression method recommendations for a new deployment scenario is very small (order of a few milliseconds), the scalability of the training time of the meta-learning model is also important, as suggested by the reviewer. The training time of the meta-learning model scales linearly with the time required for 1 epoch training of the base architectures. We have provided results for the ImageNet-1k dataset in the appendix, which were obtained by training the meta-learning model on fewer data points.

“Could you please offer further elaboration on the "META FEATURES"? Additional details about the specific features used would be beneficial for a clearer understanding.”

Additional details about meta-features and their preprocessing have been provided in Appendix B.1.

We thank the reviewer for their valuable feedback. Our responses follow.

Choice of accuracy predictor

Two architectures were considered for the accuracy prediction task - multi layer perceptron (MLP) and gradient boosted decision trees (GBDT), with hyperparameters such as depth of MLP and number of decision trees selected using cross validation. GBDT outperformed MLP in our experiments, which is also consistent with the findings of [1] where multiple performance prediction architectures for DNNs are considered. [1] White et al, “How Powerful are Performance Predictors in Neural Architecture Search?”, NeurIPS’21

Choice of compression stragies and generalizability

The choice of compression methods selected in our experiments is based on a practical constraint - the availability of stable implementations that can compress a wide array of DNN architectures. We found the open source library “neural network intelligence (NNI)” to align the closest with our requirements and used the compression methods implemented there for our experiments. Some methods such as movement pruning had to be discarded as their fine tuning diverged for several architectures leading to unreliable data. In summary, our experiments so far suggest that the proposed meta compression approach works well across all compression algorithms that can reliably compress a variety of DNN architectures.

“Applicability of the results as they are compared against only one compression constraint.”

While we evaluate and provide results for two key constraints in our analysis - compression constraint and accuracy constraint, additional constraints can also be easily incorporated into the proposed framework. For instance, a constraint on FLOPs/MACs can be implemented by limiting the search space to compression methods that reduce FLOPs(eg. Channel pruning), and predicting the performance of several pretrained models after compression using channel pruning methods to a level that satisfies the constraint.

“It is not clear how the research generalized with more recent modeling techniques specifically around transformers, which would help to strengthen the paper and understand the generalizability of the provided approach.”

We fully agree that using more recent models and compression methods would be beneficial. However, many of these (including transformers) require significant computing resources and are tailored for specific architectures, making the evaluation very challenging. We will consider such an evaluation as future work.

Thanks a lot to the authors for providing detailed answers and clarification to the questions asked. I still feel that the baseline comparison against more recent architectures plus at least another compression strategy evaluation would help to strengthen the light on the generalization of the approach, in the current format I would keep my initial assessment.

Thank you for your valuable comments; our responses follow.

“It was hard to contextualize this work for me. For example, how does this work compare to [1], which uses multi-objective Bayesian optimization to yield an entire Pareto-frontier for architecture + compression algorithm hyperparameters?”

Thank you for suggesting additional references. [1] is closer to network architecture search (NAS) methods cited in our paper, where Bayesian optimization is used to navigate the search space of multiple architectures and pruning levels, in contrast to reinforcement learning which is commonly used in NAS papers. It also has the same key drawback as other NAS methods - that several search iterations are needed for any new deployment scenario that corresponds to a new accuracy, compression level constraint. Note that each step of this search involves training a DNN architecture and evaluating its performance which is quite expensive. We take a different approach which involves learning from multiple data points across DNN architectures, compression methods, and compression levels, which requires more compute in the beginning to generate these data points, but makes the compression method recommendation for any new deployment scenario very cheap (few milliseconds).

“What is the novelty of this work? Is neither a new meta learning algorithm nor compression algorithm, as far as I can tell. To me, it seems the main novelty is the application of meta learning to model compression. In this case, I would need to see experimental evidence that meta learning for model compression is better than existing approaches using Bayesian optimization [1], DNAS, OFA”

As correctly pointed out by the reviewer, the novelty of our approach lies in devising a framework based on meta-learning to provide recommendations tailored for specific accuracy-resource constraints. Another novel insight is the successful demonstration of the feasibility of using diffusion models to generate additional data that resembles test distribution in a meta-learning context. We agree with the reviewer that providing additional results for comparison against the stated references will be useful, and we will consider it in the future.

“I was confused what exactly is the static erm strategy?”

The static ERM strategy involves selecting the single best pruning and quantization method combination that yields the highest performance on average across several accuracy/compression level constraints.