Efficient Bayesian DNN Compression through Sparse Quantized Sub-distributions

9. Exploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.

Thank you for your insightful suggestion! To address this, we have included an ablation study in Section 5.3 to analyze the impact of the spike-and-slab prior. Our results demonstrate that, compared to a Gaussian prior, the spike-and-slab distribution is more effective in achieving high sparsity. This advantage stems from the term $\sum_{i=1}^T \left(\tilde{\lambda}_i \log\frac{\tilde{\lambda}_i}{\lambda} + (1-\tilde{\lambda}_i) \log\frac{1-\tilde{\lambda}_i}{1-\lambda} \right)$ in the objective, which encourages the weights to align with the desired sparsity level. We hope this addresses your concern and provides clarity on the impact of different prior choices. We have also shown in Figure 2, that Bayesian averaging is robust to quantization thus further enhancing our methods.

10. Provide a more thorough discussion of the generalization ability, robustness, and potential applications of the proposed approach.

Thank you for highlighting this important aspect. We agree that discussing the generalization ability, robustness, and potential applications of our proposed approach would enhance the paper. Generalization Ability: Our method is designed to generalize across different architectures and tasks due to its unified framework for pruning and quantization. By leveraging variational inference and spike-and-slab priors, the approach is adaptable to diverse model types, from convolutional networks to transformer-based architectures, as demonstrated in our experiments. Robustness: The Bayesian model averaging component of our method plays a critical role in enhancing robustness. By mitigating the impact of quantization-induced noise, the approach maintains performance across varying sparsity and bit-width levels. This property makes the method reliable even in extreme compression scenarios. Potential Applications: Our framework has broad applicability in scenarios requiring efficient deployment of deep learning models. For instance, it is well-suited for edge devices with limited computational and storage resources. We will expand on these aspects in the revised version to provide a more thorough discussion and better highlight the practical and scientific contributions of our approach. Thank you again for your valuable feedback.

11. In line 176, is $T$ used before definition?

Thank you for pointing out. $T$ is defined as the total number of weights in the later context of the paper. We have changed it to the following for a better understanding.
"Let $\theta$ denote the weights vector in $R^T$ indicating the set of weights, with $T$ being the total number of weights in the DNN. Rather than storing $T$ full-precision weights, the weights are quantized into a few discrete values (i.e., shared full precision value), and only a small index indicating which shared value in $\mathcal{Q}_A$ is assigned is stored for each weight, where the $T$ is the total number of weights."

Thank you for your valuable feedback. We collect the response to address your concerns as follows.

1. Small Dataset, Small Model

Thank you for pointing this out. We acknowledge that our experiments are currently limited to smaller models and datasets due to constraints in computational resources. However, our proposed method is designed to be scalable, and we believe it can be applied to larger models and datasets as well. We plan to conduct experiments on larger-scale models and datasets in future work to further validate the scalability and effectiveness of our approach.

2. The idea is not novel.

Thank you for your comment. While we acknowledge that the idea of combining pruning and quantization is not novel, our approach introduces a significant advancement. Specifically, we unify pruning and quantization through variational inference, which allows us to explore a global optimization space rather than applying these techniques sequentially in a greedy manner, as done in previous works. This unified optimization framework is novel and enables more effective compression of deep neural networks. In summary, we propose a novel Bayesian variational learning approach that integrates pruning and quantization into a single framework, achieving a unified solution for efficient DNN compression. By leveraging spike-and-slab priors and Gaussian Mixture Models, we identify optimal sparse quantized sub-distributions, enhancing both compression efficiency and robustness to quantization-induced noise through Bayesian model averaging. We demonstrate the effectiveness of our method on multiple datasets, achieving state-of-the-art compression rates with minimal performance degradation.

3. More experimental results & Compared with more competitors

Thank you for pointing out the relevant references and methods for comparison. We agree that including comparisons with state-of-the-art techniques would significantly enhance the evaluation of our method. Due to time and computational limitations, we were unable to conduct these additional comparisons in this version. However, we are committed to incorporating them in future work to provide a more thorough assessment of our approach against recent advancements. We appreciate your suggestion, as it will help improve the overall quality and impact of our research.

4. The statement "HAQ is suboptimal" is not accurate

Thank you for pointing out the inaccurate wording. We have replaced it as: "[1] proposed a model compression pipeline that sequentially applies pruning and weight quantization, achieving significant compression rates without sacrificing much accuracy, however, the sequential application fails to explore the complementarity of pruning and quantization~[2]."

[1]: Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.

[2]: Shipeng Bai, Jun Chen, Xintian Shen, Yixuan Qian, and Yong Liu. Unified data-free compression: Pruning and quantization without fine-tuning.

5. Format of references

Thank you for pointing this out. We appreciate your attention to detail and will carefully review and correct the format of all references, including "Deep learning with limited numerical precision," to ensure they adhere to the required standards in the revised version of the paper.

6. Typos/grammatical errors

Thank you for pointing this out. We appreciate your careful review and will thoroughly proofread the paper to address any typos and grammatical errors in the revised version. Ensuring clarity and correctness is a priority for us.

7. The presentation of the paper is bad.

Thank you for your feedback. We will thoroughly proofread and polish the paper to improve its presentation.

8. Further Validation of Superior Performance

Thank you for the valuable feedback. We agree that a more comprehensive comparative analysis would further strengthen our argument. Our method introduces a unified optimization process that, unlike sequential or greedy compression approaches, provides greater flexibility in exploring the complementarity between pruning and quantization. Additionally, we propose a Bayesian model averaging approach to enhance robustness against quantization noise. The rationale is that averaging can mitigate random noise, making the model more resilient to the effects of quantization. We recognize the importance of providing detailed comparisons and in-depth discussions about the performance differences with other approaches. Thank you again for pointing this out.

Thank you for your valuable review and feedback, which have provided us with an opportunity to clarify and strengthen our work.

1. The experiments are not sufficient to demonstrate the effectiveness of the LLM model. The Bert model and GPT-2 model are not widely used LLM models.

Thank you for your feedback. We agree that including more widely-used LLMs could further strengthen our results. We selected BERT and GPT-2 as representative models due to their accessibility and the computational resources available to us. However, our method maintains the same computational complexity in terms of the number of operations, indicating that it should scale efficiently to larger models, and we believe it can generalize well to larger and more recent LLMs.

2. The baselines are not the SOTA work.

Thank you for highlighting this point. We acknowledge that including comparisons to state-of-the-art methods like AWQ would further strengthen our evaluation. Due to computational and time constraints, we were unable to run comparisons with all SOTA methods. However, we will certainly aim to include such comparisons in future work to provide a more comprehensive evaluation of our approach against the latest advancements. Your suggestion is greatly appreciated, and we agree that it will enhance the impact of our findings.

3. Can you provide the evaluation in Table 5 with some baselines to show the proposed framework outperforms the existing works?

Thank you for your insightful suggestion. We agree that adding more baselines in Table 5 would provide a clearer demonstration of how our proposed framework compares to existing works. Due to time constraints, we were unable to include all relevant baselines in the current submission. However, we plan to address this in future revisions to better highlight our method's performance relative to other approaches.

We deeply appreciate your detailed review. Your observations are invaluable, and we address your concerns as follows:

1 Some hyperparameter choices could be better justified

Thank you for your feedback! We have provided detailed hyperparameter settings in Appendix A and included explanations for their selection. We hope this addresses your concern, and we are happy to elaborate further if needed.

2 Similar results across other computer vision models

Thank you for your valuable feedback! Due to limitations in computational resources and time, we were unable to include a broader range of models and larger datasets in this study. We appreciate your suggestion and plan to extend our experiments in future work to include a wider variety of models, such as ViT, and evaluate on larger datasets like ImageNet1k to provide a more comprehensive analysis.

3 Other tasks like GLUE/SuperGLUE/MMLU etc with a wider variety of models and not just BERT

Thank you for the suggestion! Similar to vision tasks, due to limitations in computational resources and time, we were unable to include a broader range of models and larger datasets in this study. We also plan to explore other tasks such as GLUE, SuperGLUE, and MMLU in future work, using a wider variety of models beyond BERT (larger models) to provide a more comprehensive evaluation.

4 Ablation study

Thank you for your suggestion! We have included an ablation study to examine the effect of the spike-and-slab prior, which can be found in Section 5.3. In summary, our results indicate that compared to a Gaussian prior, the spike-and-slab distribution is better suited for achieving high sparsity. This advantage arises from the term $\sum_{i=1}^T \left(\tilde{\lambda}_i \log\frac{\tilde{\lambda}_i}{\lambda} + (1-\tilde{\lambda}_i) \log\frac{1-\tilde{\lambda}_i}{1-\lambda} \right)$ in the objective, which encourages the weights to align with the desired sparsity level. We hope this addresses your concern and provides clarity on the impact of different prior choices.

5 Analysis of scalability to larger models

Thank you for your feedback regarding scalability analysis. Our method does not increase the computational complexity in terms of the order of operations required, which suggests that it is scalable to larger models. The reason we did not include larger models in our experiments is due to limited computational resources. However, we are confident that our approach can be extended to larger architectures, and we plan to explore this in future work.

6 Table Caption:

Thank you for pointing this out! We have updated the captions for Table 1 and Table 2 to include the type of test/dataset used for evaluation, following your suggestion.

Thank you for your valuable feedback. We collect the response to address your concerns as follows.

1 Computational expenses and data requirement not well discussed.

We appreciate your valuable feedback regarding the discussion of data and computation requirements. Indeed, our method requires retraining the model on the full training dataset. However, the compression process can be finished within relatively small epochs. We added the following discussion in the revised PDF. "When compressing ResNet models, our method requires fine-tuning over the training dataset, completing the compression process within $10$ epochs." "During the compression process, the BERT model is fine-tuned on the training dataset, with the entire procedure completed within $3$ epochs."

2 Existence of extra hyperparameters requires further tuning, tradeoff between Pruning and Quantization.

Thank you for your thoughtful feedback regarding the effort required for hyperparameter tuning. We understand that this can be a concern, and we truly appreciate the opportunity to clarify. Hyperparameter tuning is indeed a fundamental part of modern machine learning workflows, and we have designed our method to align with this standard practice. By integrating pruning and quantization into a unified optimization process, our approach helps reduce the manual effort typically needed to balance these two operations, as shown in our experiments. We hope this addresses your concern, and we look forward to continuing to refine our approach to make it even more user-friendly in the future.

3 How to tune

Thank you for highlighting this important point about the practical usability of our method. We recognize the importance of providing clear guidance for practitioners on tuning hyperparameters and considering hardware-specific constraints. Regarding the inclusion of hardware-specific factors, we agree that this is a critical consideration for real-world deployment. While our current work focuses on general compression methods, extending it to account for target hardware characteristics is an exciting direction for future research. We plan to conduct additional experiments to analyze the interplay between pruning and quantization in the future.