2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution

Reviewer4 EXmU

Q1:The proposed DOB and DQC are much similar with DBDC and PaC, but only change weight compression to symmetric quantization, could you give more details about the differences?

A1: In fact, our DOBI and DQC are quite different from DBDC and Pac. The specific differences are as follows:

• DOBI vs. DODB:

  1. We perform DOBI on both weights and activations, while DODB only on weights.
  2. We use a one-sided search for activations, while DODB does not involve this aspect.
  3. Our search objective is to find the minimum MSE, which can be accelerated by GPU, whereas DODB aims to find the minimum interval containing the T-th parameter size, where T is a hyper-parameter.

• DQC vs. Pac:

  1. Granularity is different. DQC is performed on the four Transformer layers in the SwinIR architecture, which is the largest substructure. We have tried finer granularity, such as layer-wise (24 in total) or linear-wise (96 in total), and the results indicate that this granularity is the most suitable. Limited by the number of pages, this part of the content was not included in the paper. In contrast, DODB is performed on each residual block, totaling 32 blocks, slower and worse than DQC.

Q2:The motivation in the introduction are confusing. ...

A2:We assume that you are referring to lines 60-62 in the original text. Our statement is closer to the surface, while your statement is indeed the essence. We will modify this part to be "Secondly, most of these methods can not adapt well to Transformer-based models because of the deterioration of self-attention in quantized transformer.".

Q3:The selected baselines in this paper are rough and the experimental results are not convinced. This paper does not compare with transformer-based post-training quantization methods, such as PTQ4ViT[1], FQ-ViT[2], NoiseQuant[3] and RepQ-ViT[4]...

A3: Thank you for your suggestions! For the four articles you mentioned, we have completed comparative experiments with them. The results of the experiments are shown below and more results can be found in the attachment file in Author Rebuttal.

Thank you for the response and additional experiments. The specific differences about 2DQuant and PTQ4SR are clearly stated, which further shows that the differences between these two methods are marginal. DOBI on both weights and activations, one-side searching，new search object and different granularity are just engineering optimisations, the core distributions are still based on PTQ4SR. Although 2DQuant gets much better performance than baselines, the academic contributions of this paper are negligible. And I agree with Reviewer DoGX that this paper use the A+B approach to exaggerate the contribution. So I decided to keep my score.

Thanks for the detailed explanation, I think I misunderstood a bit about the contributions of this paper. 2DQuant is much useful for accelerating the deployment of transformer-based image super resolution. This method is particularly beneficial to reduce the quantization error of post-softmax and post-GELU feature maps in transformer architectures. After reviewing the latest response, I decided to raise my score to 6 (weak accept).

Q1a: The contribution lacks novelty... For example, the distribution of weights and activations is well studied by previous works.

A1a: First, the study of distribution is not our core contribution or innovation, it is one of our experimental observations. While our main contributions are 1) the proposed 2DQuant, which utilizes DOBI and DQC to quantize Transformer-based model. 2) 2DQuant achieves 3.60x compression ration, 5.08x speedup ratio, and 4.52dB increasement in Set5 compared with SOTA[6]. Second, [1][2][3] focus on CNN-based models, such as ResNet, EDSR, and RDN networks, but our paper targets SwinIR, a Transformer-based model. There are significant differences between CNN networks and Transformer-based networks in terms of both architecture and model performance, so the direct application of these CNN-based insights/techniques on Transformer-based networks is not straightforward and even impossible.

Q1b: Some papers have used searchable upper/lower bounds for quantization (although these methods are based on QAT, the essence of QAT and PTQ is no different, with only slight differences in training strategies) and NO references were given in this manuscript.

A1b:First, PTQ and QAT are two different concepts. PTQ stands for Post-Training Quantization, while QAT stands for Quantization-Aware Training. The former uses a pre-trained neural network without needing to update its parameters, only updating the quantizer parameters. In contrast, the latter typically uses an untrained neural network and updates both the neural network and quantizer parameters. From the perspective of memory usage, PTQ mainly requires storing the parameter values and their gradients (for backpropagation). The space required to store gradients is consistent with the space required for parameters. In addition to this, QAT needs to store the optimizer states for all parameters due to the necessity of updating the neural network's parameters. For example, Adam (used in [2]) requires saving momentum and squared gradients for each parameter, which has the same shape as the parameters. Therefore, for the same model, the memory requirement for QAT is typically twice that of PTQ, which is not a slight difference.

Second, we are not a survey paper, so we do not need to cite all relevant articles. We have already cited similar works (like [6]), and not citing [2] is not a serious issue. To further enhance the quality of the paper, we will include a discussion of this article in the Related Work section.

Q1c: The proposed distillation approach is a widely used routine operation in quantization tasks.

A1c: Although we both use a distillation method, there are significant differences in our specific implementations.

1. We only distill the quantizer parameters and do not need to distill the model parameters, which greatly reduces memory requirements and speeds up the algorithm.
2. The granularity of distillation is different. We target the largest substructure of the Transformer, which is the Transformer Layer. In SwinIR-light, there are a total of four Transformer layers, each containing six Transformer blocks. In contrast, [1-6] distills the residual blocks, of which there are 32 in EDSR—eight times more than ours. Therefore, our loss function computation will be faster and the distillation approach is different.

Q2:Experiments are insufficient...

A2: Both [4] and [5] are QAT methods, training the model from scratch, while our proposed algorithm is a PTQ method. Comparing between PTQ method with QAT method is unfair as QAT often performs better than PTQ because of the fine-tuning of model parameters. A fairer comparison between [4][5] and our paper is applying their methods to optimize the quantizer parameters but freeze the model parameters. The result is shown below and more is in Author Rebuttal's attachment file. The additional experiments show our robustness and advantages.

Q3:Experimental settings are not consistent. Only 100 images are used in [7], which is inconsistent with this paper.

A3: Our paper mentions using 32 images in line 228. First, we and [6] are two independent papers, so there is no need to adopt the same settings. Second, we achieved better performance with fewer images, which is our advantage instead of weakness. Finally, the results of 100 images are shown below, more can be seen in Author Rebuttal.

References: [1] Li, Huixia et al. “PAMS: Quantized Super-Resolution via Parameterized Max Scale.” ECCV (2020).

[2] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” ECCV (2022).

[3] Hong, Chee and Kyoung Mu Lee. “Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks.” ECCV (2024).

[4] Li, Yanjing et al. “Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer.” NIPS (2022)

[5] Liu, Shi et al. “Oscillation-free Quantization for Low-bit Vision Transformers.” ICML (2023).

[6] Tu, Zhaopeng et al. “Toward Accurate Post-Training Quantization for Image Super Resolution.” CVPR (2023).

Dear Authors,

I hope this message finds you well.

My concerns still revolve around the similarity between the core idea of this article and [1]. Although you mentioned the differences in training details between PTQ and QAT (such as the parameters updating during backward pass) and the contribution of your method to SwinIR (which I have never denied), the academic contribution of this paper is still not convincing. The difference between SwinIR and CNN models lies in the model structure, rather than the constituent modules of the model. Therefore, applying existing quantization methods to new model architectures in this paper has a certain amount of engineering contribution rather than academic contribution. If the paper considers contribution from the perspective of engineering quantity, should deployment on practical reasoning frameworks become a necessary factor? In conclusion, I choose to maintain my score.

Sincerely,

Reviewer DoGX

References: [1] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” ECCV (2022).

Q1:As presented in section 3, the authors use fake quantization during the design of the method, but didn’t mention if it can be replaced by the real quant, which can be implemented and bring real acceleration in hardware. The author should discuss if the proposed method can achieve real quantized inference when deployment.

A1: Fake quantization is a widely used technique in model quantization. Its purpose is to simulate the precision loss caused by quantization. Using fake quantization, we can obtain the model parameters of the quantizer. With the quantizer's parameters, we can deploy and test the model using common quantization techniques. We omitted this part in the article because the quantization method we adopted is already very mature in this regard and our speedup ratios are obtained from real deployment.

Besides, the speedup ratio reported in lines 20-22 of our paper is the actual （real quant） deployment speedup ratio——2DQuant achieves an increase in PSNR of up to 4.52 dB on Set5 (×2) compared to SOTA when quantized to 2 bits, along with a 3.60× compression ratio and a 5.08× speedup ratio.

Q2a:The algorithm of DQC is missed and make the description not very clear, I suggest to add it or merge it to the algorithm of DOBI.

A2a: Thank you for your suggestion. The nature of DQC is distillation between the FP model and the quantized model, which optimizes the parameters of the quantizers. We will merge this part of the algorithm into the DOBI algorithm.

Q2b: And Figure 3 not highlight the application of the proposed techniques, which is also suggested to revise.

A2b: You are correct.

Figure 3 is not closely related to the proposed techniques. However, it is still very necessary to present Figure 3. We included Figure 3 to show that in the Transformer block, we have quantized all the computationally intensive parts, such as BMM, FC, etc, some of which are often forgotten to be quantized. Other parts, such as Softmax and Layernorm, have relatively small computational loads, but quantizing these parts would have a significant negative impact on the model.

Q3: Some details of writing should be improved and polished carefully, e.g., the subtitles of section 2 and 3, some explain of the proposed equations are missing.

A3: Thank you very much for your suggestions on writing details. We have already made the changes you mentioned in the arXiv version.

Q1: ... how about the acceleration when deploying the quantized models into device?

A1: In the field of quantization research, when we apply the same quantization method to the same neural network module, the speedup ratio of the model does not change due to variations in the quantizer parameters. Therefore, the algorithm typically focuses on how to better optimize the quantizer parameters to improve model performance after applying such quantization methods.

Besides, the speedup ratio reported in lines 20-22 of our paper is the actual deployment speedup ratio——2DQuant achieves an increase in PSNR of up to 4.52 dB on Set5 (×2) compared to SOTA when quantized to 2 bits, along with a 3.60× compression ratio and a 5.08× speedup ratio.

Q2:.... I am not very sure how the EDSR results are obtained. The baseline in Table 3 is not very clear. Please clarify them.

A2: We clarify that the EDSR in Table 3 does not refer to the original FP model. Its specific meaning is explained in lines 237-239 – it is the test result of the previous SOTA method DBDC+Pac on EDSR. It should be noted that the parameter size of EDSR is 172.36MB, while the parameter size of SwinIR-light is 3.42MB, which is about 2% of the former. As for the baseline in Table 3, it is the full-precision version of SwinIR-light.

Our comparison with EDSR aims to illustrate that:

1. The best result in [1] was achieved on EDSR, and applying our quantization method on SwinIR-light performs better than their method on EDSR;
2. The choice of the quantized model itself is crucial. To achieve better quantization results, we cannot only experiment on classic models (as done in the [1]), because in terms of parameter size, cutting-edge models before quantization already outperform quantized classic models.

Q3:In Table 4 (b), when the batch size becomes larger, the performance changes slightly. This is not very consistent with the common observation, that larger batch size usually improves performance.

A3:Thank you for your keen observation. The main reason for this issue lies in the fact that DOBI provides excellent quantizer parameters. In the SR field, the numerical evaluation metric usually adopted is PSNR. If we only look at PSNR, with the increase of batch size, our performance shows a slight improvement. The reason for this slight improvement is that our one-stage search algorithm, DOBI, has already provided quite optimal quantizer parameters, resulting in a very high starting performance for the model. At the same time, the initialization parameters provided by DOBI ensure that the model can achieve almost the same excellent results regardless of changes in batch size during subsequent training.

I believe this phenomenon is beneficial because, even when GPU memory is limited, we can still achieve results comparable to those obtained with a large batch size by reducing the batch size and extending the training time for DQC.

Q4: The writing can be further improved. Some details should be given attention and revised. For example, Section 3.2, the first letter should be capital.

A4: Thank you for your suggestions on writing details. We have revised in the Arxiv version.

Q5:In Figure 1, the proposed method performs even better than the full-precision (FP) version. What are the potential reasons behind this observation?

A5:We clarify that the main reason is that FP models tend to overfit the training set, leading to weaker generalization performance on some test data. Model quantization might alleviate this overfitting. Although there might be an overall performance drop, the performance on certain data can be better than that of the FP model.

Can this method be applied to diffusion based image SR models? If so, please give some key ideas. It will be much better if the authors can provide some preliminary results.

A6:Yes! The core components of diffusion are the denoising network and the sampler, with UNet being a common form of the denoising network. The computationally intensive parts of the UNet are the convolutional layers. We can use the DOBI and DQC methods mentioned in the article to perform two-stage quantization of the convolutional layer parameters.

However, it is important to note that the generation process of diffusion models typically involves several steps, and the distribution of activations is usually associated with the time steps. Therefore, designing algorithms tailored to each time step might yield better results. For example, we can divide the time steps into different segments and apply different quantizers to the activations in different segments.

Q7:In Table 3, what is the model size for the original EDSR? I do not which version of EDSR did the authors use in the paper.

A7: As mentioned in A2, the EDSR in Table 3 refers to the data in Table 3 of [1]. Details and parameters of the model used can be found in the following link at line 11 in the blob/master/src/model/edsr.py file in the github repo "sanghyun-son/EDSR-PyTorch". To our knowledge, this is the largest version of EDSR.

Q8: What is the motivation by providing two versions, i.e., DOBI (ours) and 2DQuant (ours), in Table 3?

A8: DOBI is our one-stage algorithm while 2DQuant represents the final results of both the one-stage and two-stage processes. The reason we included both in Table 3 is to demonstrate that even just using DOBI, the model can exceed the SOTA performance on most datasets, and the two-stage algorithm can further enhance the model's performance.

[1] Tu, Zhaopeng et al. “Toward Accurate Post-Training Quantization for Image Super Resolution.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 5856-5865.