Nearly Lossless Adaptive Bit Switching

We would like to thank you for your careful reading, helpful comments, and constructive suggestions, which have significantly improved the presentation of our manuscript. We are delighted with the identification of the novelty and effectiveness of the proposed method. We have carefully addressed all comments, and below are our responses:

W1: Whether the second rounding lead to more quantization errors?
A: Thank you for your valuable comment. Firstly, we acknowledge that twice rounding introduces more quantization error compared to once rounding. However, compared to previous methods, the advantage of our method is that only saves the quantization parameters (i.e., scales) for the highest bit(e.g., int8). This allows us to save only the highest bit weights and switch to lower bits while eliminating the multiplication and division operations brought by multiple scales during adaptive bit-switching to accelerate the model's inference. Additionally, model training can compensate for this loss, achieving nearly lossless bit-switching. The ultimate goal of this approach is to achieve faster inference with adaptive bit-switching on hardware.

W2: Why ALRS is applied only for the scaling factors and not used in weights of small bit-width?
A: Thank you for your kind comments. We apologize for the lack of clarification in that statement. Firstly, ALRS is only applicable to the quantization scale factor because it is quite sensitive during quantization training and determines the final model's convergence performance. In our multi-precision or mixed-precision scenarios, weights are shared, and directly scaling their values would cause other precisions to fail to converge due to severe competition between different precisions. Additionally, applying the ALRS strategy to the quantization scales of different precisions (including small bit-widths) is indirectly equivalent to scaling the weights of that bit-width, because
$weight_i = Round(weight_{i-1}) / scale_i) * scale_i + delta_w(x) * lr$,
$scale_i = scale_{i-1} + delta_s(x) * ALRS$,
where the $delta_w$ and $delta_s$ denote the gradients of the $weight_{i-1}$ and $scale_{i-1}$ respectively, and $ALRS$ denotes the adaptive learning rate of the quantization scaling factors and $lr$ denotes the learning rate of the weights. Even so, we also tried applying ALRS to the weights but did not achieve better performance. We hope this explanation clarifies the rationale behind the argument.

W3: Fig. 1 is a bit confusing, some colored arrows are not well explained.
A: Thank you for your kind feedback. In Figure 1: (1) The red arrows indicate the expression of different precisions obtained through double rounding, then the green arrows indicate the combined optimization using the ALRS technique to achieve multi-precision, followed by the HASB technique to further achieve a mixed-precision model. (2) The black arrows in the network structure indicate the data flow between networks. (3) The gray dashed lines represent the techniques required at different stages. We will further refine Figure 1 in the revised manuscript.

W4: It is better to compare more MPQ (e.g., HAQ, DNAS, LIMPQ, etc) research and could include the [1] (PTQ-based) and [2][3] (QAT-based) into the Related Work.
A: Thank you for your suggestions. We have carefully read the literature you provided: HAQ [4], DNAS [5], and LIMPQ [6]. We find that DNAS mainly designs a differentiable NAS to search for a hardware-aware efficient network without considering quantization. However, we will incorporate DNAS's efficient NAS techniques to further enhance the performance of our mixed-precision quantization. Regarding HAQ, it uses reinforcement learning to integrate the hardware accelerator’s feedback into the design loop for the quantization strategy. However, the networks learned are hardware-customized, and the learned networks do not have specific bit-width standards, so a direct fair comparison with our method may not be possible. Nevertheless, we will attempt to combine HAQ's techniques in the future to further improve the performance of our method. For LIMPQ, please refer to our response to reviewer bT7w W4. Lastly, we promise to include the literature you suggested [1] (PTQ-based) and [2][3] (QAT-based) in the related work section of the revised manuscript.

[4]HAQ: Hardware-Aware Automated Quantization with Mixed Precision
[5]FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
[6]Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance

Dear reviewer T2Zq,

Thanks for your positive comments. We will add the additional results, discussions, and references in the final revision. We are pleased to answer your questions and concerns. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards

Authors

Thanks for the response. I have carefully read the rebuttal and other reviews, the response addressed my concerns. I will increase my rating while I hope the authors can incorporate these additional results, discussions, and references into the revision. BTW, DNAS refers to the paper entitled "Mixed precision quantization of convnets via differentiable neural architecture search".

We would like to thank you for your careful reading, helpful comments, and constructive suggestions, which have significantly improved the presentation of our manuscript. We have carefully addressed all comments, and below are our responses:

W1: The study should be done on larger models (LLMs), and the FP32 master copy is not a big problem in moderate GPUs.
A: Thank you for your kind comments. Sorry for the confusion. Firstly, the advantage of saving the highest integer bit-width (int8) we proposed may lie in small terminal devices (e.g., smartphones, small drones, etc.). Hardware implementation is not limited to GPUs but may include CPUs or FPGAs with limited resources and communication bandwidth, as well as Arm processors. Secondly, we acknowledge your point that the technique we proposed may have greater advantages in the field of LLMs. Due to limited time and resources, we conduct Multi-precision experiments on small LLMs[1] without using ALRS and distillation, please refer to Table 1 of the attached PDF in the global response. Note, except for not quantizing the embedding layer and head layer, due to the sensitivity of SiLU activation causing non-convergence, we don’t quantize the SiLU activation in the MLP and set batch size=16. The training process is shown in Figure 1(TinyLLaMA-1.1B) of the attached PDF.

W2: Couldn't find a source-code to reproduce the results of the paper.
A: Thank you for your kind feedback. Our open-source code can be accessed by clicking the last word "here" in the abstract, which is a hyperlink. As reviewer rFvS also points out, one of our advantages is that "Code is given."

Q1: Have any theoretical intuition for assuming a power-of-two relation between different precision or this is just because of fast HW implementation?
A: Thanks for the nice question. We apologize for the lack of clarification in that statement. The “power-of-two relation” arises because we use the same quantization parameters, i.e., scale, for models of different precisions when learning the weights. Switching from a higher bit-width to a lower bit-width is equivalent to clipping the lower bits of the weight values, retaining the higher bits. For example, the first four bits of the 8-bit weight are the same as the 4-bit weight. This is also equivalent to multiplying the scale value by 2 to the power of k, where k is the difference between the two bit-widths. Of course, in conventional multi-precision training where quantization parameters are not shared among different precisions, there is no “power-of-two relation” in their scales. The ultimate goal of this approach is to achieve faster inference with adaptive bit-switching on hardware. We hope this explanation clarifies the rationale behind the argument.

Q2: What is the cost HMT in different networks? How can we compare it against the e2e runtime?
A: Thank you for your kind question. The cost of computing the HMT (Hessian Matrix Trace) in different networks takes a few minutes (e.g., approximately 2 minutes for ResNet18 and 5 minutes for ResNet50). Since this computation is done offline and then directly used for end-to-end inference, this part of the computation can be considered negligible.

[1]TinyLlama: An Open-Source Small Language Model.

Thank you so much for answering my questions and providing more results. For some reason, I couldn't find the code :-)

I will increase my score.

Dear reviewer Aa14,

Thanks for your positive comments. We are sorry that you can't find the code we provided for some reason. We will forward an anonymous code link to you via AC, and we promise to make the code publicly available on GitHub later. We are pleased to answer your questions and concerns. We will add the relevant discussions in the final version. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards

Authors

We would like to thank you for your careful reading, helpful comments, and constructive suggestions, which have significantly improved the presentation of our manuscript. We are delighted with the identification of the novelty and effectiveness of the proposed method. We have carefully addressed all comments, and below are our responses:

W1: Using Hessian information seems a bit trivial, and the sampling probability is modified with a simple ascending heuristic, which is not Hessian-aware.
A: Thank you for your valuable comment. Firstly, we acknowledge that this method still has room for improvement, and we will refine it in the future by incorporating zero-cost NAS research or more advanced algorithms. However, the sampling probability is indeed modified using a simple heuristic approach based on the sensitivity (Hessian trace) of different layers during the training phase. During the inference phase, when using ILP for inference, the Hessian information is also used as a constraint factor for aligning the training phase, thereby avoiding the need for retraining, which maybe can be considered Hessian-aware. We hope this explanation clarifies the rationale behind the argument.

W2: The design of the double-rounding quantizer is similar to Bit-Mixer, Adabits, and ABN. Specifically, ABN also uses.
A: Thank you for your thoughtful comment. Similar to Bit-Mixer, Adabits, and ABN, our method learns different bit quantization parameters through shared weights, but the specific implementations are different. Firstly, Adabits and Bit-Mixer primarily achieve bit-switching through the Floor operation, while our method switches using twice Rounding operation. Secondly, ABN's formula appears similar to our doubling rounding, but the key difference is that different precisions use different quantization parameters. In contrast, our double rounding only updates the quantization parameters of the highest INT weight during training. Finally, in bit-switching scenarios, storing only one shared quantization parameter of our double-rounding is hardware-friendly and reduces the computation overhead of scale (floating-point value).

W3: The scaling factors of ALRS need further ablations.
A: Thank you for your valuable suggestion. We have conducted more ablation experiments on scaling factors, and the results can be seen in Table 4 of the attached PDF in the global response. It can be observed that the performance differences of the model under different scaling factor settings are not significant, further proving the effectiveness of ALRS.

W4: It is better to compare with these ILP-based mixed-precision quantization papers, e.g., [2] and [3].
A: Thanks for your kind suggestion. We have carefully read the literature [2] and [3] you provided. However, we find that only the results in Table 2 of [2] regarding ResNet18's 3MP (Top-1: 69.7) can be fairly compared with ResNet18's 3MP (Top-1: 69.92) in Table 3 of our main text. The other configurations either refer to ResNet50's w-bits=3MP, a-bits=4MP in literature [2] or ResNet50's w-bits=2MP, a-bits=4MP in literature [3]. We will further attempt similar bit-width configurations for a fair comparison in the revised version of the paper.

Q1: How do perform KD for the proposed method?
A: Thank you for your valuable question. We drew inspiration from the progressive in-place distillation of [4], i.e., self-distillation. However, we apply it to the multi-precision quantization method in this paper. Specifically, higher bit-widths distill to their neighboring lower bit-widths, such as 8-bit distills to 6-bit, 6-bit distills to 4-bit, and 4-bit distills to 2-bit.

[4] self-knowledge distillation with progressive refinement of targets.

Dear Reviewer bT7w,

Thanks for your positive comments. We are pleased to address your questions and concerns. We will add the relevant discussions in the final version. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards,

Authors

We would like to thank you for your careful reading, helpful comments, and constructive suggestions, which have significantly improved the presentation of our manuscript. We have carefully addressed all comments, and below are our responses:

W1: using a pretty straightforward idea of double rounding during training.
A: Thank you for your comment. Although the concept of double rounding appears simple, learning shared quantization parameters that maintain the characteristics of low integer precision makes it possible to switch to other lower bits almost losslessly. Our study verifies both the feasibility and efficiency of this method.

W2: Is ALRS important for methods that don’t use double rounding?
A: Thanks for your constructive comment. In fact, ALRS is a general technology for multi-precision. We have conducted ablation experiments on other methods, please refer to Table 2 of the attached PDF in the global response. The results further validate the effectiveness of our proposed ALRS.

W3: Do have an edge device for switching daily model precision, why and where is multi-precision important?
A: Thank you for your valuable comment. Unfortunately, it seems that there are no specific edge devices that implement adaptive bit-switching during the inference phase currently. However, existing hardware already has the capability for such bit-switching (for example, the Nvidia A100 supports INT8, INT4, and Binary [1], and small AI chips support INT4 and INT2 [2]). This potential has not yet been fully developed, but we believe this technology will become widespread in the future for the following reasons:

• Model Compression: For different terminal devices, the storage and computational resources provided vary. To reduce the cost of repeated quantization training and simultaneously provide multiple versions of models with different sizes [3], multi-precision is an effective means to solve this problem.
• Scenario-Based Precision Switching: Depending on different scenarios, real-time switching of model precision can be implemented. For instance, in the field of autonomous driving, complex road scenarios require high precision to ensure real-time accuracy, while simple road conditions can switch to lower precision to save energy.
• Large Language Models (LLMs): For complex logical tasks, a high-precision model may be needed to provide more reliable answers (similar to GPT4-pro), whereas for simple conversational tasks, a low-precision model can provide reliable answers (similar to GPT4o or GPT4o-mini) to accelerate inference.

W4: No results on more recent models (LLMs)?
A: Thank you for your nice comment. Due to limited time and resources, we conduct Multi-precision experiments on small LLMs[4] without using ALRS and distillation, please refer to Table 1 of the attached PDF. Note, except for not quantizing the embedding layer and head layer, due to the sensitivity of SiLU activation causing non-convergence, we don’t quantize the SiLU activation in the MLP and set batch size=16. The training process is shown in Figure 1(TinyLLaMA-1.1B) of the attached PDF.

Q1: Explain the connection between the ALRS method inspired by LARS?
A: Thank you for your comment. LARS first uses a separate learning rate for each layer instead of each weight. Secondly, the magnitude of updates is controlled according to the weight norm to better manage the training speed. Our ALRS refers to LARS's weight norm update strategy and modifies it to a gradient norm update strategy to adapt the learning rate of different precision. We hope this explanation clarifies the rationale behind the argument.

Q2: provide a scenario where your method is particularly important.
A: Please refer to Answers of W3.

Q3: Simply increasing the learning rate with the conventional multi-precision training approach?
A: Thanks for your kind comment. Initially, we attempted conventional multi-precision training by simply increasing the learning rate, but this led to non-convergence and even training collapse. We analyze that this is due to the excessive sensitivity of the quantization scale and the severe competition between different precisions.

Q4: Need to run 4/3 more iterations to ensure the total number of updates.
A: Thanks for the nice suggestion. We have conducted experiments on {8, 6, 4, 2}-bit and {4, 3, 2}-bit using more epochs. Please refer to Table 3 of the attached PDF, where we find a slight improvement.

[1]Once-for-all: Train one network and specialize it for efficient deployment.
[2]NVIDIA A100 Tensor Core GPU Architecture.
[3]A 7-nm Four-Core Mixed-Precision AI Chip With 26.2-TFLOPS Hybrid-FP8 Training, 104.9-TOPS INT4 Inference, and Workload-Aware Throttling.
[4]TinyLlama: An Open-Source Small Language Model.

Dear reviewer rFvS,

Thanks for your positive comments. We are pleased to answer your questions and concerns. We will add the relevant discussions in the final version. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards

Authors