Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks

We thank reviewer p2GR for spending time reviewing and providing detailed feedback. We are encouraged that the reviewer recognized our state-of-the-art performance with similar or less computations. We address the reviewer's concerns and questions below in detail:

Q1-1: Does the distribution mismatch occur only in SR or other networks?
A1-1: The distribution mismatch problem is especially severe for SR networks. For example, a classification network (ResNet20) shows a much more minor image-wise and channel-wise distribution mismatch compared to SR networks (EDSR, RDN). We measure the average variance of features on DIV2K validation set for SR networks and ImageNet validation set for the classification network. Thanks for your comments, we added such analyses to the supplementary material.

Table R2: Average feature mismatch.

Q1-2: The distribution after quantization
A1-2: Thanks for your suggestion. Based on your comments, we added new figures to the supplementary material of distribution after quantization and the distribution after using our framework, ODM. The distribution of the ODM-applied network shows that ODM is effective in reducing the distribution mismatch.

Q2: Comparison with more recent models (e.g., SwinIR)
A2: Please kindly refer to our supplementary material (Table S1), in which we reported results on more recent models SwinIR [A] and CARN [B]. According to the results, our method achieves consistent gain over existing methods also on recent models.

Q3: EDSR-DAQ does not correspond to the original results of the DAQ paper
A3: We note that the result reported in the DAQ paper uses the EDSR backbone of 32 residual blocks (of 256 channel dimensions). In comparison, we use the EDSR-baseline backbone that consists of 16 residual blocks (of 64 channel dimensions), following PAMS and DDTB. We have reproduced EDSR-DAQ directly by the official codebase, which also matches the accuracy reported for EDSR-baseline in the official GitHub repository.

Q4: Experiments only on a scale of 4
A4: Also, kindly refer to our supplementary material (Table S2) for the results on scale 2 SR models. The results show that our method is also beneficial for scale 2 SR models.

Q5-1: Ablation only using the cooperative variance regularization
A5-1: If we understood correctly, Table 5 (b) presents the result of only using cooperative variance regularization.

Q5-2: Ablation indicates that the coop. var. reg is not important
A5-2: We would like to emphasize that, compared to the baseline, using cooperative variance regularization (Coop. Var. Reg.) brings 0.8 dB gain in PSNR, which demonstrates that Coop. Var. Reg is important. The two main components we propose (Coop. Var. Reg. and Sel. Off.) both serve to reduce the distribution mismatch in SR; using two components together gives a relatively minor gain compared to the gain from each component (0.09 dB / 0.18 dB). However, this does not mean that each component is useless; each brings 0.8 dB / 0.9 dB gain over the baseline. It rather hints that the two attributes can have overlapping effects for reducing distribution mismatch.

[A] SwinIR: Image Restoration Using Swin Transformer, CVPR2021.
[B] Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network, ECCV2018.

We thank Reviewer v9HR for spending time reviewing and providing insightful feedback. We are encouraged by the positive comments on the novel method, the simple yet effective idea, and easy-to-follow writing. Below is the detailed response to each question.

Q1: More details about the gradient conflict ratio
A1: Thanks for your helpful suggestion. Upon your advice, we updated the manuscript to add more details about the gradient conflict ratio. In the manuscript, we detailed how the gradient conflict ratio is measured: the ratio of parameters that the sign of gradient from reconstruction loss and that of regularization is different. Also, we additionally emphasized that, according to Figure 2 of the main paper, the conflict ratio decreases minimally during training, which indicates that variance regularization can hinder the reconstruction loss throughout training. Moreover, instead of disregarding the gradient of the regularization when the two gradient conflicts, we added analysis for weighting the regularization loss by the degree of conflict between two losses (measured by the cosine similarity). According to the results, it was slightly better to disregard the variance regularization loss when it is not cooperative.

Table R3: Analysis on the degree of gradient conflict on EDSR x4 (2-bit). cos() measures cosine similarity.

Q2: More discussion about the relation between variance regularization and selective distribution offsets
A2: Thank you for your advice. As the reviewer mentioned, variance regularization and selective distribution offsets both function to reduce the distribution mismatch (feature-wise std). Each component largely reduces the mismatch and results in higher accuracy over the baseline. When the two components are jointly used, the accuracy increases compared to using each component separately, although the increasing amount is relatively minor. This indicates that there is, to some extent, an overlapping effect of selective offsets and variance regularization. Nevertheless, both components contribute to reducing the mismatch, resulting in a further accurate quantized SR network. We added the discussion to the manuscript.

Q3: Do the selective distribution offsets learned on the features not processed in cooperative variance regularization?
A3: We would like to clarify that selective distribution offsets are also processed through cooperative variance regularization. The parameters of the quantized network (including the selective distribution offsets) are end-to-end trained. We updated the manuscript followingly.

We thank Reviewer Yy9i for the time spent reviewing and providing constructive feedback. We are encouraged by the positive comments on competitive performance and easy-to-follow writing. We address all raised comments and questions below in detail.

Q1-1: Fair comparison regarding training epochs
A1-1: We follow DDTB [A] to use 60 epochs, and for a fair comparison, we reproduce PAMS [B] and DAQ [C] for 60 epochs using the official codebase.

Q1-2: Influence on the number of epochs
A1-2: If we set the training epochs to 300 (the epochs reported in DAQ), the accuracies of the overall methods increase. However, we note that the order is preserved, and ours outperforms other methods.

Table R1: Different training epochs on 2-bit EDSR for Urban100 PSNR.

Q2: Comparison of training time
A2: Using a single RTX 2080Ti GPU, the calibration for the selective offsets takes $\sim$30 seconds. The overall training time for ODM is $\sim$4.0 hours for quantizing EDSR, which is no less than that of DDTB ($\sim$4.0 hours).

Q3: What makes ODM need higher storage space than DAQ?
A3: The additional storage space of ODM originates from the selective offset parameters. However, the storage overhead is relatively minimal compared to the significant bitOPs overhead of DAQ: as DAQ adaptively adjusts the quantization range parameters by calculating mean and variance at test-time, the bitOPs overhead is substantial.

Q4: Typo after Eq. (4)
A4: Thank you for pointing out. We have revised our manuscript.

Q5: ODM does not reduce BitOPs compared to DDTB
A5: We would like to note that, although the bitOPs of ODM is similar to that of DDTB, ODM occupies a smaller storage size, which is also an important computational cost benefit.

[A] Dynamic Dual Trainable Bounds for Ultra-Low Precision Super-Resolution Networks, ECCV 2022.
[B] PAMS: Quantized Super-Resolution via Parameterized Max Scale, ECCV 2020.
[C] DAQ: Channel-Wise Distribution-Aware Quantization for Deep Image Super-Resolution Networks, WACV2022.

We thank Reviewer Bh7W for contributing time reviewing and providing valuable feedback. We are encouraged that the reviewer found our work strongly motivated, easy to follow, and comprehensively experimented with. More importantly, we address the reviewer's concerns and questions below in detail:

Q1-1: Novelty in channel-wise quantization factor
A1-1: As the reviewer also pointed out, there are works on SR quantization that adopt channel-wise quantization functions (e.g., DAQ [A]). However, we would like to note that simply adopting channel-wise scaling and offsets incur substantial computational overhead. Instead, our method adopts only scaling for particular layers or shifting for certain layers and even none for some layers, which effectively reduces the computation overhead (e.g., the bitOPs overhead is $\sim\times 100$ smaller compared to DAQ [A]). We find our selective approach for offsets to be helpful for achieving accurate quantization accuracy without incurring significant overhead.

Q1-2: Channel-wise quantization will not be affected by the value differences between channels
A1-2: We would like to clarify that we utilize layer-wise quantization factor for each layer. For a few layers, we shift or scale the input of the convolutional layer with channel-wise offsets, which are then quantized with a layer-wise quantization factor. Thus, the quantization accuracy is affected by value differences in channels.

Q2: Repeated strategies for one problem
A2: As the reviewer mentioned, both variance regularization and selective offset strategy alleviate the distribution mismatch problem in SR. Using two components together gives a relatively minor gain (0.09 dB / 0.18 dB) compared to the gain from each component (0.8 dB / 0.9 dB). This indicates that there is, to some extent, an overlapping effect of the two strategies. Nevertheless, both components reduce the mismatch, resulting in a further accurate quantized SR network. We will add further discussion of the overlap to our supplementary material.

Q3: More insightful analysis and discoveries
A3: Thanks for your suggestion, we added deeper analyses on the distribution mismatch problem in Section S5.1 of the supplementary material: how solving such a problem is especially crucial for SR networks and how our method effectively reduces the mismatch. Also, as suggested by another reviewer, we added analysis on the gradient conflict problem in Section S5.2. Please kindly refer to the supplementary material for more details.

Q4: Will variance regularization lead to homogenization of features?
A4: Directly using variance regularization can lead to the homogenization of features, which can trigger a PSNR loss, as shown in our ablation study. Since we use variance regularization loss cooperatively with the reconstruction loss, the features are not fully homogenized, as an example is visualized in Figure S2 of the supplementary material. Moreover, we tried regularizing the difference in the mean of each channel, which gave a slight increase in accuracy for a few settings. Thank you for your constructive suggestions, we will integrate such an analysis into our manuscript.

Table R4: Variance regularization on EDSR x4 (2-bit).

[A] DAQ: Channel-Wise Distribution-Aware Quantization for Deep Image Super-Resolution Networks, WACV2022.