ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs

We thank the reviewer for insightful and constructive feedback. We respond to each of these below.

Weak baseline: Beyond robustness aware mixed precision quantization, there are several methods [1,2,3] that show much less accuracy degradation on both ResNets and Mobilenets in Imagenet scale. What is the comparative advantage of ARQ compared to these methods even with accuracy degradation? For thorough comparison, shouldn't the authors include comparisons with various types of mixed precision quantization method?

There are no publicly available implementations of [1] and [2]. Although [1] provides a GitHub repository link in their paper, it is currently empty. Therefore, we could only perform comparison experiments with NIPQ [3]. Our experiments on MobileNet-V2 and ImageNet show that ARQ outperforms NIPQ[3] in both clean accuracy and average certified radius. We also conducted additional experiments on ResNet-20 and Cifar-10, which are shown in Figure 1 of our updated paper.

Comparison with existing robustness aware MPQ: Within robustness aware quantization, the current metrics cannot justify the benefit of ARQ in the aspect of robustness compared to existing robustness aware MPQ (in Table5).

We have performed additional experiments to show that our work outperforms existing works listed in Table 5.

MPQ+Empirical Robustness: None of the existing works presented in Table 5 focus on optimizing certified robustness with mixed-precision quantization. Typically, techniques that optimize empirical robustness do not achieve high certified accuracy. We performed additional experiments on an 8-bit quantized ATMC[4] ResNet-20 model. It’s shown that empirical robust quantization methods can hardly gain certified robustness.

Quantization+Certified Robustness: We performed additional experiments on ICR[5]. Our experiments show that ARQ 3-bit equivalent model can outperform ICR[5]’s 8-bit equivalent model(which is their lowest bit implementation) in both clean accuracy and robust accuracy for certified radii below 1.0, and on radius r=1.25 and 1.50, the performance loss of ARQ is minimal, especially considering the 7.3x BOPs difference and the improvement gained on smaller radii

Algorithm: Lack of explanation of several terms in Algorithm1 makes bad readability, i.e., FullRobustCertify, IncrementalRobustCertify.

Thanks for pointing this out. We have added explanation for the terms used in the Algorithm in the updated version of the paper.

[1] Huang, Xijie, et al. "SDQ: Stochastic differentiable quantization with mixed precision." International Conference on Machine Learning. PMLR, 2022. Empty github repo

[2] Kim, Han-Byul, et al. "MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 12. 2024.

[3] Shin, Juncheol, et al. "NIPQ: Noise proxy-based integrated pseudo-quantization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Gui, Shupeng, et al. "Model Compression with Adversarial Robustness: A Unified Optimization Framework." Advances in Neural Information Processing Systems. NeurIPS, 2019.

[5] Lin, Haowen, et al. "Integer-arithmetic-only Certified Robustness for Quantized Neural Networks." Proceedings of the IEEE/CVF International Conference on Computer Vision. ICCV, 2021.

We thank the reviewer for insightful and constructive feedback. We respond to each of these below.

The authors didn’t compare ARQ with any robustness-aware quantization methods as listed in Table5.

We have performed additional experiments to show that our work outperforms existing works listed in Table 5.

MPQ+Empirical Robustness: None of the existing works presented in Table 5 focus on optimizing certified robustness with mixed-precision quantization. Typically, techniques that optimize empirical robustness do not achieve high certified accuracy. We performed additional experiments on an 8-bit quantized ATMC[5] ResNet-20 model. It’s shown that empirical robust quantization methods can hardly gain certified robustness.

Quantization+Certified Robustness: We performed additional experiments on ICR[6]. Our experiments show that ARQ 3-bit equivalent model can outperform ICR[6]’s 8-bit equivalent model(which is their lowest bit implementation) in both clean accuracy and robust accuracy for certified radii below 1.0, and on radius r=1.25 and 1.50, the performance loss of ARQ is minimal, especially considering the 7.3x BOPs difference and the improvement gained on smaller radii.

The authors didn’t compare with other advanced fixed-precision or mixed-precision quantization algorithms, such as LSQ[1], LSQ+[2], or HAWQ[3][4].

We have performed additional experiments to make comparisons against HAWQ-V3[4] on ResNet-20. The detailed results are presented at Figure 1 in the updated version of paper. Our 4-bit equivalent models outperformed 4-bit and even 8-bit models of HAWQ-V3[4] in both clean accuracy and certifiable robustness.

Although ARQ has better accuracy and robustness, it consumes much longer time than other methods as shown in Table2. What is the impact of search time on the final accuracy.

We performed additional ablation studies to show the effect of search time on final accuracy and robustness on ResNet-20.

The authors only use BitOps as the constraints. What about the model size?

It is possible that our RL objective can be changed from BitOps to model size. However, in this work we focus on using BitOPs as the constraint. We have included model size measurements in our paper Table 1 and Appendix A.4 to provide a comprehensive view.

The techniques proposed in the paper seem incremental. The authors need to provide more explanation on the novelty of the method.

We are the first to introduce a mixed-precision quantization framework that optimizes both DNN's accuracy and certified robustness while limiting computational resources. Our approach uniquely uses direct feedback from the ACR of the quantized smoothed classifier. Our experiments demonstrate that ARQ consistently outperforms state-of-the-art quantization methods, often matching or surpassing FP32 networks' accuracy and robustness with just 0.84% of operations. And we show that reinforcement learning can be effectively utilized for such optimization problems.

[1] Esser S K, McKinstry J L, Bablani D, et al. Learned step size quantization[J]. arXiv preprint arXiv:1902.08153, 2019.

[2] Bhalgat Y, Lee J, Nagel M, et al. Lsq+: Improving low-bit quantization through learnable offsets and better initialization[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops.

[3] Dong Z, Yao Z, Arfeen D, et al. Hawq-v2: Hessian aware trace-weighted quantization of neural networks[J]. Advances in neural information processing systems.

[4] Yao Z, Dong Z, Zheng Z, et al. Hawq-v3: Dyadic neural network quantization[C]//International Conference on Machine Learning. PMLR.

[5] Gui, Shupeng, et al. "Model Compression with Adversarial Robustness: A Unified Optimization Framework." Advances in Neural Information Processing Systems. NeurIPS, 2019.

[6] Lin, Haowen, et al. "Integer-arithmetic-only Certified Robustness for Quantized Neural Networks." Proceedings of the IEEE/CVF International Conference on Computer Vision. ICCV, 2021.

Why use reinforcement learning?

Previous works [4, 6] have demonstrated the ability of reinforcement learning to efficiently explore and exploit the vast search space on mixed-precision quantization optimization tasks. Our experiments further demonstrate its effectiveness in handling more complex tasks, especially when certified robustness is taken into consideration.

In your paper, you argue that traditional quantization methods have primarily concentrated on maintaining neural network accuracy, this is not accurate. The goal of MPQ is to study how to reduce the network's parameters, namely, obtaining the trade-off between accuracy and complexity. They have at least two objectives to learn.

In Section 2.1, we have clearly stated the goal of quantization method in the first sentence:

Neural network quantization is a model compression technique that can reduce a network’s size and compute cost.

We also elaborate on this in Formula 1:

$Cost(f_P)<C_0$

Additionally, we explain what constitutes the costs:

$Cost(f_P)$ is the resource usage of the network, such as model size, the number of compute bit operations, or energy consumption, and C0 is a user-specified bound on the resource.

In your paper, you state two objectives, i.e., accuracy, and robustness. Why not consider complexity? Maybe you don't understand the MPQ.

In our paper, we frame the problem as a multiobjective optimization task. We treat complexity, specifically in terms of bit operations (BOPs), as a constraint. This approach is consistent with the epsilon-method for solving multiobjective problems, where one objective is converted into a constraint to explore solutions across multiple values.

Figure 1 clearly illustrates the tradeoff curve between BOPs and ACR/Accuracy, demonstrating how our framework effectively balances these factors. Additionally, Figure 3 in Appendix A.4 provides further insights into the tradeoff between model size and ACR/Accuracy, as requested by Reviewer dzkR.

[1] Huang, Xijie, et al. "SDQ: Stochastic Differentiable Quantization with Mixed Precision." International Conference on Machine Learning (ICML), 2022.

[2] Dong, Peijie, et al. "EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization." International Conference on Computer Vision (ICCV), 2023.

[3] Ma, Y., et al. "OMPQ: Orthogonal Mixed Precision Quantization." Proceedings of the AAAI Conference on Artificial Intelligence, 37(7), 2023, pp. 9029-9037.

[4] Wang, Kuan, et al. "HAQ: Hardware-Aware Automated Quantization With Mixed Precision." IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[5] Tang C, Ouyang K, Wang Z, Zhu Y, Wang Y, Ji W, Zhu W. Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance. arXiv preprint arXiv:2203.08368, 2023.

[6] Lou, Qian, et al. "AutoQ: Automated Kernel-Wise Neural Network Quantization." 8th International Conference on Learning Representations (ICLR), 2020.

The performance of ARQ is heavily reliant on the quality of the training of the original DNN. If the original network is not properly trained or lacks Gaussian augmentation, the quantized network may not meet expected performance.

ARQ has only been evaluated on image classification tasks. Its performance and effectiveness on other types of tasks have not been validated.

These limitations were acknowledged in our paper. We plan to explore methods to improve ARQ's robustness under different initial conditions and extend our research to other tasks beyond image classification in future work.

While ARQ performs well in the experiments, its generalizability across different network architectures and datasets remains an open question and requires further research for validation. The main limitations are experiments, i.e., more datasets (COCO2017, VOC), more larger networks (e.g., VIT, GPT-2, Efficientnet), more tasks (detection, segmentation, VQA), and more validation for robustness.

The concerns about generalizability are noted, but it's important to highlight that previous works the reviewer mentioned [1, 2, 3] and the baselines we compared against [4, 5] also primarily focus on ImageNet and ResNet/MobileNet architectures. Our work focuses on the certified robustness, which provides formal guarantees and eliminates necessity for empirical validation.

The performance of ARQ may be sensitive to hyperparameter choices, which could require additional tuning efforts to ensure optimal performance across different networks and datasets.

Section 4 of our paper thoroughly details the hyperparameters used and provides clear instructions for reproducing our results. While hyperparameter sensitivity is inherent in many machine learning methods, we have ensured that our methodology is transparent and reproducible.

The presentation of the paper is poor. For example, the statement of robustness is not clear, adversarial inputs or what? If it is adversarial inputs, why not provide the details for adversarial methods? PGD or FGSM? What is certified robustness?

While other reviewers have noted strengths in our presentation, we are committed to addressing all concerns to improve the clarity and quality of our paper. The whole Section 2.2 defines and describes certified robustness. Certified robustness refers to a formal guarantee that a neural network will perform consistently and accurately under perturbations, such as adversarial attacks or input noise.

The method is not novel, and too complex. The results are also not convincing.

We are the first to introduce a mixed-precision quantization framework that optimizes both DNN's accuracy and certified robustness while limiting computational resources. Our approach uniquely uses direct feedback from the ACR of the quantized smoothed classifier. Our experiments demonstrate that ARQ consistently outperforms state-of-the-art quantization methods, often matching or surpassing FP32 networks' accuracy and robustness with just 0.84% of operations. And we show that reinforcement learning can be effectively utilized for such optimization problems.

Why not compare with more competitors[1-3]? [1] SDQ: Stochastic Differentiable Quantization with Mixed Precision [2] EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization [3] OMPQ: Orthogonal Mixed Precision Quantization

Firstly, it's important to clarify that the mentioned mixed-precision quantization works [1, 2, 3] do not achieve reasonable robustness without modifications, making them unsuitable as direct competitors. Secondly, we have conducted additional experiments on [7, 8] as requested by reviewers L6Pe and dzkR. ARQ consistently outperforms these baselines, and the detailed results are included in the updated version of our paper.

What is the definition of robustness for MPQ?

In the context of MPQ, robustness is the ability of a quantized neural network to maintain consistent and correct classifications in the presence of adversarial input perturbations. Our approach specifically focuses on certified robustness.

We thank reviewer 32H8 for recognizing the importance of our work and for the insightful suggestions. We respond to each of these below.

The paper would benefit from more detailed analysis of memory usage patterns during training, though the runtime analysis provided in Table 2 is helpful.

Thank you for acknowledging our runtime analysis! Below is a table that analyzes the memory usage of different models during training:

The memory usage is closely tied to the number of parameters in the models and the input batch size. Training or fine-tuning models typically requires a similar amount of memory usage, so the observed memory consumption aligns with standard expectations for such tasks. Users can optimize memory consumption by adjusting the batch size during fine-tuning and robustness certification.

More discussion of practical hardware implementation considerations for mixed-precision inference would strengthen the work's immediate applicability.

How does the framework handle the transition between different precision levels during inference, and what are the implications for hardware implementation?

Previous work Bit Fushion [5] introduced bit-level composable architectures that can dynamically adjust to the required bit-widths. This approach allows the architecture to dynamically adjust the bit-widths of operations to match the specific precision requirements of each layer in a neural network.

The architecture in [5] can reconfigure itself at inference to accommodate varying bit-widths required by different layers. [5] uses an array of bit-level processing elements that can fuse or decompose to match the bit-width of individual DNN layers. The ARQ framework can integrate well into this architecture because ARQ assigns different bit-widths on a per-layer basis and employs linear quantization to simplify deployment. This compatibility allows ARQ to utilize Bit Fusion [5]'s dynamic bit-level composability, ensuring that each layer operates at its optimal precision.

How might the ARQ framework be adapted for sequence models or transformers, particularly regarding the interaction between attention mechanisms and mixed-precision quantization?

This is a good question! Previous works [1, 2] have explored the application of randomized smoothing to sequence models and transformers. [3, 4] have provided solutions of mixed-precision quantization on sequence models and transformers. These works can be helpful for extending the ARQ framework to NLP tasks. We are willing to explore these further in our future work.

What modifications to the incremental randomized smoothing approach might be necessary to support other certification methods while maintaining computational efficiency?

Incremental randomized smoothing is designed for randomized smoothing’s verification process by estimating the disparity ζx– the upper bound on the probability that outputs of smooth classifier f and quantized smooth classifier f^p are distinct. To broaden its applicability, the core algorithm should be restructured to align with the principles of new certification methods. This might involve redefining the disparity estimation process and adjusting algorithmic steps to accommodate different verification methods.

Could the reinforcement learning approach be extended to dynamically adjust precision levels during inference based on input characteristics?

That is a great idea! It's indeed possible to extend the reinforcement learning approach to dynamically adjust precision levels during inference based on input characteristics. To achieve this, we could consider increasing the quantization bit-level to a range that does not require additional fine-tuning. Given the high demand for certification (e.g., 100,000 noisy samples for a single image), it’s possible to use a subset of the noisy samples to determine optimal quantization bit-levels and then use them for further certification.

[1] Moon, Han Cheol, et al. "Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications." In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.

[2] von Oswald, Johannes, et al.. "Learning Randomized Algorithms with Transformers." arXiv preprint arXiv:2408.10818, 2024.

[3] Liu, Zhenhua, et al. "Post-training quantization for vision transformer." Advances in Neural Information Processing Systems 34 (2021): 28092-28103.

[4] Yao, Zhewei, et al. "Zeroquant: Efficient and affordable post-training quantization for large-scale transformers." Advances in Neural Information Processing Systems 35 (2022): 27168-27183.

[5] H. Sharma et al., "Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network," 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)