Revisiting Vector-Quantization for Blind Image Restoration

Thank you for your detailed review and the helpful feedback to our work. We are grateful for your recognition of our deep analysis of VQ-based BIR methods, the proposed SinCA as a compelling solution to overcome the limitations of VQ, as well as the broad applicability and effectiveness of our SinCA.

W1: Increased computational costs.

As detailed in Tab. 1, we compare the computational costs and inference time of the traditional VQ process with our SinCA across different methods. It can be observed that SinCA generally consumes fewer computational resources compared to the transformers used for code index prediction in models like CodeFormer and DAEFR. This is because SinCA does not increase the dimension of the LQ feature, which is used for index prediction. While SinCA introduces additional computational costs compared to the negligible overhead of the nearest-neighbor feature matching employed in FeMaSR and AdaCode, this increase is justified by the substantial improvements in image restoration performance that SinCA provides.

Furthermore, to address concerns regarding computational efficiency, we introduce four lightweight variants of SinCA: CodeFormer+SinCA (Light) with a single SinCA layer, DAEFR+SinCA (Light) with one SinCA layer, FeMaSR+SinCA (Light) with a single SinCA layer, and AdaCode+SinCA (Light) with five parallel SinCA layers, one for each of the five codebooks. We summarize the quantitative results in Tabs. 2-3 and provide their computational costs and inference times in Tab. 1. These results demonstrate that even minimal implementations of SinCA can yield substantial performance improvements with minimal impact on computational resources. As a result, SinCA proves to be a viable and effective solution, even in resource-constrained scenarios.

Tab. 1 Computational cost summary

For CodeFormer and DAEFR, FLOPs is calculated by input of size $512\times 512$, while for FeMaSR and AdaCode, FLOPs is obtained by input of size $128\times 128$ on $\times2$ task. The inference time is calculated as the average over 100 forward passes.

Tab. 2 Results of blind image super-resolution

Tab. 3 Results of blind face restoration

W2: insufficient experiments.

1.Whether SinCA preserve quality while enhancing fidelity without feature fusion . To study this, we train FeMaSR and CodeFormer using SinCA by removing the corresponding feature fusion module. The quantitative results are summarized in Tabs. 3-4. Qualitative comparisons are provided in Figs. 1-5 of our provided anonymous link (File name: 'Visual_Results_1.pdf'). One can see that without feature fusion, FeMaSR and CodeFormer using SinCA preserve image quality while enhancing fidelity on BIR tasks.

We agree that human subjects are the ultimate judge of image quality, and did subjective user study to evaluate the restoration quality and fidelity of "CodeFormer w/o fusion'' and ''CodeFormer+SinCA w/o fusion''. This study involves 32 participants, and each participant is asked to do preference evaluation on restored images of 25 LQ images randomly selected from Wider, WebPhoto, and LFW. In each trial, each participant was presented with a set of three images: one LQ input for the reference of restoration fidelity, the image restored by ''CodeFormer w/o fusion'', and the image restored by ''CodeFormer+SinCA w/o fusion''. In each set of presentation, the first image is LQ input while the order of two restored images is randomly shuffled. Results of user preference voting are listed in Tab. 5, which show that''CodeFormer+SinCA w/o fusion'' got 73.75% of total 800 votes.

The results validate that SinCA with adaptive quality-fidelity balance indeed preserves image quality while enhancing the fidelity of VQ-based BIR methods like FeMaSR and CodeFormer.

2.Overlooking the fidelity-quality balance in BFR. As illustrated in the Sec 5.3 of our paper, for different LQ images, SinCA adaptively balances the fidelity and quality by the self-part and cross-part in attention map. The adaptive balance in SinCA ensures implicit yet effective trade-off between using HQ codebook and LQ features for BIR. This is different from CodeFormer that uses an external parameter w for fidelity-quality balance. The capability of SinCA on adaptive quality-fidelity trade-off has been validated in Sec. 6.2 (Tabs.1-2) in our main paper.

To further support SinCA on BFR, we do more experiments of CodeFormer on real-world face datasets LFW, Wider, and WebPhoto. As noted by the reviewer, CodeFormer achieves enhanced quality when w=0. Thus, we compare "CodeFormer+SinCA'' with CodeFormer when w=0 (w/o fusion). In Tab.4, the results on the fidelity metrics of PSNR and SSIM computed on input LQ image and the quality metrics of NIQE and Perceptual Index (PI) show that ``CodeFormer+SinCA w/o fusion'' achieves adaptive balance on quality and fidelity. The visual comparisons are provided in Figs. 3-4 of the anonymous link (File name: 'Visual_Results_1.pdf'). We recommend the reviewer to click "view raw'' for a better view. Notably, the visual results of "CodeFormer+SinCA w/o fusion'' are competitive with or even surpass those of "CodeFormer w/o fusion'' (w=0) and "CodeFormer w fusion'' (vanilla CodeFormer).

We agree that the explicit quality-fidelity balance in CodeFormer is meaningful for BFR. However, our work aims for general replacement of VQ by SinCA for VQ-based BIR methods, not only for methods like CodeFormer. Many VQ-based BIR methods like FeMaSR, AdaCode, DAEFR, and RestorFormer do not have explicit quality-fidelity balance. To accomodate SinCA for general VQ-based BIR methods, we do not provide explicit balance between quality and fidelity in SinCA. In fact, it is not difficult to provide explicit quality-fidelity balance in SinCA. The key is how to strengthen the information from the self-part or the cross-part of the attention map in SinCA. This can be achieved by multiplying a weight hyper-parameter with the key matrix $\mathbf{K}_\mathbf{X}$ in Eqn. (4) of our main paper. We will consider it as a future work. Thank you very much for your constructive suggestion.

Tab 3 Results of FeMaSR w/o fusion

Tab 4 Results of CodeFormer w/o fusion

Tab 5 User study results

Thank you for your detailed review and constructive feedback on our work. We are grateful for your acknowledgment of the sufficient demonstration on the shortcomings of discrete VQ, proposed SinCA achieving better quantitative and qualitative performance on both BSR and BFR tasks, as well as your positive remarks on our simple but effective idea.

W1: Originality is somewhat lacking.

Low codebook usage and encoding prediction accuracy are already described in previous work. Briefly, these are not key observations of our work. The low codebook usage rate is not what we observe in our work. Though mentioned in previous works, the issue of low codebook usage rate is not always right. In fact, the issue of low codebook usage rate does not exist in CodeFormer and DAEFR for BFR task. This contradiction indicates that usage rate of codebook in VQ-based BIR methods is still an open problem that need experimental supports, which makes it meaningful to performexperiments to illustrate principles behind our observations. We will explain each point in details as follows.

(a) On codebook usage. Our first observation is that discrete VQ confines representation capability of HQ codebook to a finite set of code items, which is seldom mentioned in previous works. To study benefits of extending the discrete codebook to continuous space, inspired by Reviewer JLcW, we perform experiments to compare the performance of FeMaSR/CodeFormer using VQ or cross-attention (CA) on BIR. We summarize the results of FeMaSR and CodeFormer by replacing the discrete VQ with the continuous CA on both the image reconstruction task in the Prior Learning stage in Tab. 1 (with fixed encoder and decoder) and the image restoration task in the Image Restoration stage in Tab. 2 (with fixed decoder). It can be seen that CA improves both PSNR and SSIM results with 0.17$\sim$5.58dB and 0.0062$\sim$0.2075, respectively. This validates that replacing the discrete VQ with the continuous CA for feature learning significantly enhances the reconstruction and restoration performance of VQ-based BIR methods. We mentioned in our paper that this issue ''will be amplified by low codebook usage rates mentioned in previous works''. But this issue does not occur in CodeFormer and DAEFR developed for BFR. As illustrated in Fig. 2 (a) in our Observation 1 (Section 4.1 of our main paper), the rates of codebook usage in CodeFormer and DAEFR are 98.73% and 100%, respectively.

(b) Our second observation is that VQ is error-prone on LQ features. Even though mentioned in CodeFormer, it is still not an obvious conclusion on other VQ-based BIR methods. We thoroughly analyze the key reasons for this issue and find that the low accuracy of index prediction degrades the performance of VQ-based BIR methods. Besides, we also point out that higher prediction accuracy brings better image restoration performance (Fig. 3 (c) in our main paper). From these insights, we understand that it is challenging to enhance prediction accuracy given the inherent degradations. Consequently, we have opted to replace the error-prone index prediction with our SinCA, which significantly improves restoration performance.

These two observations demonstrate that VQ is a two-sided coin with clear rewards and punishments for VQ-based BIR methods. This motivate us to replace the discrete VQ operation by a continuous feature transformation module for BIR. Besides, the third observation showing the importance of LQ feature for BIR motivates us to integrate the self-attention of LQ feature in the cross-attention between the LQ feature and the HQ codebook, and develop our Self-in-Cross attention (SinCA) module to fully exploit the useful information in LQ feature and HQ codebook. The objective of our work is not to address the issue of ''codebook collapse'', where only a small part of HQ codebook is used during the VQ operation.

Our work provides the necessity to replace VQ by continuous feature learning in VQ-based BIR methods. Our SinCA is ''simple but effective'' for VQ-based BIR and a meaningful originality for low-level vision.

Tab.1 Reconstruction Results

Tab.2 Restoration Results

Thanks for your thorough review and the valuable feedback on our work. We sincerely appreciate your recognition of the novelty of the proposed SinCA, the reasonableness and sufficiency of our observations, as well as the presentation of our manuscript.

W1: More study of reconstruction loss.

In Sec. 3, page 4 of our main paper, we describe the total loss function employed in VQVAE and VQGAN for image generation. As the VQ-based BIR methods integrate the VQ process into the restoration framework, they incorporate modifications to the loss function to better align with restoration objectives. Since we mainly focus on replacing the discrete vector-quantization process with a more effective alternative, we maintain the original training strategy and follow the loss functions employed in these VQ-based BIR methods. Specifically, for CodeFormer and DAEFR, which employ cross-entropy loss to supervise transformer learning for index prediction, we substitute cross-entropy loss with a feature matching loss to accommodate the continuous learning of our SinCA, as detailed in our appendix (Page 16, line 821). For our feature matching loss, we adhere to the common implementation of code-level losses, using L2 loss.

Inspired by your feedback, we exploring the use of L1 loss as a feature matching loss to supervise the learning of the SinCA on DAEFR. As shown in Tab. 1, L1 loss could potentially enhance the restoration performance on synthetic CelebA-Test. We recognize the potential of further exploring the design of code-level loss and consider it as a promising future direction. Thanks for the constructive comment.

---

W2: Clarify Fig. 3.

We have revised the illustration in Fig. 3 of our main paper to ensure a tighter connection between the images and the text as follows.

T-SNE visualization of VQ process in FeMaSR (a) and CodeFormer (b). Different code items in HQ codebook are marked by "☆" in different colors. The color of LQ feature vector marked by "◯'' or the HQ feature vector marked by "△'' is the same with the codebook item they select in VQ process. Gray dashed lines "- -'' connects the LQ feature vector "◯'' and its selected codebook item "☆" (“VQ Process”). Red dash lines "- -'' connects the LQ feature vector "◯'' and the codebook item "☆" selected by the corresponding HQ feature vector using nearest-neighbor (NN) feature matching in the Prior Learning Stage (“GT Selection”). (a) NN feature matching on LQ feature vectors are inconsistent with “GT Selection”. For a given LQ feature vector, the codebook item selected by NN feature matching (gray dash line, "- -'') is quite different from the corresponding "GT Selection'' (red dashed line, "- -''). (b) Transformer for code index prediction is not robust to image degradation. In a degraded LQ image, an "LQ Eye'' patch looks like skin area and selects the code item represented by many "HQ Skin'' patches. The regions marked by purple dashed box and orange dashed box are enlarged for better view ("Enlarge''). (c) Prediction accuracy of code indices by transformer and SSIM results achieved by CodeFormer using "LQ Indices'' predicted on LQ images, "HQ Indices'' predicted on HQ images, or "GT Indices'' defined in Sec.4.2.

---

W3: More experiments on adverse conditions for BSR.

We conduct more experiments on the real-world dataset to evaluate the proposed SinCA under adverse conditions on both $\times 2$ and $\times 4$ task. We summarize the quantitative results in Tab. 2, verifying that our SinCA can achieve superior performance on adverse senarios. The qualitative results are provided in Figs. 9-12 of the attached PDF file (File name:'Visual_Resulst_3.pdf') in our anonymous link, which also demonstrate the effectiveness of our SinCA. Clicking “view raw” can provide a better view.

---

Tab. 1 Implementation of feature matching loss on DAEFR

Tab. 2 Results of BSR on real-world datasets

Once again, we sincerely thank you for affirming our work and providing constructive feedback and insightful suggestions, which have been invaluable in helping us improve the study.

Thanks for the insightful feedback. We are grateful for your positive evaluation of the detailed analysis of the side effects of discrete VQ and the effectiveness of the proposed SinCA module demonstrated with experiments.

W1: Lacks controlling between quality and fidelity.

As illustrated in the Sec 5.3 of our paper, for different LQ images, SinCA adaptively balances the fidelity and quality by the self-part and cross-part in attention map. This adaptive mechanism ensures an implicit yet effective trade-off between using the HQ codebook and LQ features for BIR. Different CodeFormer, which employs an external parameter w for fidelity-quality balance, SinCA achieves this balance inherently. The capability of SinCA on adaptive quality-fidelity trade-off has been validated in Sec. 6.2 (Tabs.1-2) in our main paper.

We agree that the explicit quality-fidelity balance in CodeFormer is meaningful for BFR. However, our work aims for general replacement of VQ by SinCA for VQ-based BIR methods, not only for methods like CodeFormer. Many VQ-based BIR methods like FeMaSR, AdaCode, DAEFR, and RestorFormer do not have explicit quality-fidelity balance. To accomodate SinCA for general VQ-based BIR methods, we do not provide explicit balance between quality and fidelity in SinCA. In fact, it is not difficult to provide explicit quality-fidelity balance in SinCA. The key is how to strengthen the information from the self-part or the cross-part of the attention map in SinCA. This can be achieved by multiplying a weight hyper-parameter with the key matrix $\mathbf{K}_\mathbf{X}$ in Eqn. (4) of our main paper. We will consider it as a future work. Thanks again for your constructive feedback.

---

W2: Further exploration of benefits from continuous space.

To study the benefits from continuous space through attention, we conduct experiments where cross-attention (CA) replaces the discrete VQ process in the Prior Learning stage of FeMaSR and CodeFormer. Here, we freeze the encoder and decoder of the FeMaSR and CodeFormer in the Prior Learning Stage and only train the cross-attention module. In this setup, the representative space of codebook is expanded from discrete code items to continuous space through attention, which is an essential improvement. The results in Tab. 1 validate that spanning codebook space into a continuous space indeed improve the representation capability of codebook with better reconstruction performance.

Additionally, our ablation study has provided the image restoration results of four VQ-based BIR methods in Tab. 1 and their variants using CA in Tab. 4 of our main paper. For convenient readability, we summarize the above results in Tab. 2. Compared to the retrained baselines, all variants of four VQ-based BIR methods with a transformer using CA achieves clear improvements in most cases. This also validates the benefits of extending discrete codebook items to a continuous space for VQ-based BIR methods, further supporting our approach and motivation.

Tab.1 Reconstruction Results

Tab.2 Restoration Results

Thank you for taking the time to consider our response. We would like to address the remaining concerns.

1. HQ codebook alone is insufficient for BIR and previous work also using LQ feature

Our work does not aim to point out that HQ codebook alone is insufficient for BIR. Instead, our work focuses on tackling the limitations of VQ for general BIR methods using VQ: for W1, our work claims that the representation capability of HQ codebook is confined by VQ (our Observation 1). This point has be validated by replacing VQ by cross-attention (CA) between LQ feature and HQ codebook: the results in Tabs. 1 and 2 of the response to W2 show that continuous CA outperforms discrete VQ on both image reconstruction and restoration.

As presented in Sec. 4.2 of our paper (Page 5, line 267), we agree that previous work used LQ feature for quality-fidelity trade-off, but the feature fusion is implemented after VQ. However, these methods still suffer from the limited representative capability and under-utilization of LQ feature in the VQ process itself, and they do not directly address the root cause within the VQ process. On the contrary, our work reveals that LQ feature is under-utilized in VQ (our Observation 3). Thus, we propose to augment HQ codebook with LQ feature and replace discrete VQ process by continuous feature attention in SinCA (as a better alternative of VQ) for better quality-fidelity balance. This not only directly addresses the under-utilization issue of LQ feature in the VQ process, but also expands the codebook’s representational capacity to effectively tackling the key limitations of VQ we illustrated in our three observations.

On core contribution, we clarify that our work is motivated by three key observations on the limitations of VQ process in general VQ-based BIR methods. To address these limitations for better BIR performance, our core contribution is to replace discrete VQ by continuous feature learning, which avoids these limitations and is more reasonable for BIR task. Our SinCA is a "simple but effective" exploration in this direction. SinCA is able to perform self-attention on LQ feature itself for restoration fidelity and cross-attention between LQ feature and HQ codebook for restoration quality, and meanwhile achieving adaptive balance between restoration quality and fidelity. To achieve these goals, we propose SinCA to perform cross-attention between input-augmented codebook and LQ feature of input LQ image.

While CodeFormer explicitly attempts to achieve trade-off between fidelity and quality, other VQ-based BIR methods like FeMaSR and AdaCode do not achieve this trade-off but employ feature fusion for performance enhancement. The results of FeMaSR and AdaCode are deterministic, whereas SinCA achieves input-adaptive enhancement on both quality and fidelity by integrating LQ information into SinCA for feature transformation. Moreover, DAEFR does not have a fusion module to connect encoder and decoder. **Our SinCA replaces VQ and thus can be applied to general VQ-based BIR methods, whether or not a feature fusion module is used after VQ. **This makes our SinCA highly versatile and compatible with existing methods that already incorporate fusion modules, providing meaningful improvements on both quality and fidelity.

Furthermore, as referred by the reviewer in W1, CodeFormer explicitly achieves the fidelity-quality trade-off through an LQ feature fusion module. When LQ images are severely degraded, removing the fusion module can lead to higher-quality restored images but at the cost of fidelity. To demonstrate the impact of our approach, we compare "CodeFormer+SinCA" with CodeFormer both with and without the fusion module in Figs. 3-5 of the anonymous link (File name: 'Visual_Results_1.pdf'). Clicking "view raw" can provide a better view. Notably, the visual results of "CodeFormer+SinCA w/o fusion" are competitive with, or even surpass, those of "CodeFormer w/o fusion" and "CodeFormer w fusion" (vanilla CodeFormer). It is important to emphasize that our goal is not to replace or remove the fusion modules in previous works, but rather to demonstrate that by replacing the discrete VQ process with our proposed SinCA, we can address the performance bottleneck of the VQ process and unlock new potential for improving blind image restoration.

2. Concurrent use of self-attention and cross-attention.

Thank you for bringing the referred papers to our attention. Our work is not motivated by these references, but by our observations regarding the VQ process in VQ-based BIR methods. After carefully reviewing the literature, we believe that the novelty of these works does not diminish the contributions of our paper. Our work aims to study the limitations of VQ for BIR and give a novel solution to replace the VQ process, extending the codebook representative space and adaptively preseving the fidelity while enhancing the quality. SinCA is one solution we propose to achieve our goal.

The referred papers, while useful in their respective contexts, do not address the limitations of VQ process, which instead is the goal of our work. The use of self-attention and cross-attention in those works is not relevant to the limitations of the discrete VQ process in VQ-based BIR methods.

For example, Animate Anyone [1] uses spatial attention to extract spatial information, cross-attention for useful guidance from reference images, and temporal attention to capture dependencies during denoising steps for character animation. MagicAnimate [2] employs mutual self-attention and cross-attention to ensure temporal consistency while incorporating guidance from the reference image during denoising steps for character animation task. These two approaches focus on different objectives and do not tackle the limitations of the VQ process in BIR methods.

Different from these two work, our work introduces a general replacement of the VQ process in VQ-based BIR methods, achieving promising improvements of these methods on BIR tasks. Since the contributions of our work are very different from those of the above-referred works, we believe that our work is novel and meaningful in advancing the field of BIR, especially the popular branch of BIR methods developed under the VQ framework (e.g., VQVAE, VQGAN, or others).

---

Error in the response to Q3.

Thanks for pointing out this. It's a slip of a pen. We want to say that the reduced diversity in codebook usage contributes to the relatively higher accuracy. We apologize for the oversight and appreciate your careful attention to detail. We'd like to correct the response to Q3 as follows:

As shown in Fig. 2(a) and Fig. 2(b), the codebook usage ratios in BSR methods of FeMaSR and AdaCode are significantly lower than those of BFR methods of CodeFormer and DAEFR. For instance, the codebook usage ratio of FeMaSR is only $3.32\%$ (34/1024) while DAEFR uses $100\%$ (1024/1024) codebook for image restoration. The lower codebook usage ratio in BSR reduces the difficulty of code prediction task due to fewer choices of available code items. For FeMaSR, 48 code items are active in the Prior Learning stage while the Image Restoration stage further narrows this scope to a subset of 34 code items. This reduced diversity in codebook usage contributes to the relatively higher accuracy observed in Fig. 2(c).

---

We hope our responses have effectively addressed your concerns regarding the contributions of SinCA. Considering the strengths of our work, we kindly request you to reconsider your rating and recommend acceptance. Your insightful review and constructive feedback have been very helpful in improving our paper.

---

[1] Xu, Z., “MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model”, CVPR, 2024.

[2]Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., and Bo, L., “Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation”, arXiv e-prints, Art. no. arXiv:2311.17117, 2023.