Dual Associated Encoder for Face Restoration

We extend our gratitude for the positive and constructive feedback.

We would like to address the raised concerns as follows:

[Q1] The full name of the proposed framework, DAEFR, is missing. It should be mentioned on its first occurrence in the paper.

The acronym 'DAEFR' stands for the title of our paper, which is an initialism of "Dual Associated Encoder for Face Restoration." We will revise these parts in our updated manuscript.

[Q2] The proposed method requires training two sets of encoder and decoder for both HQ and LQ images. This will double the training resource requirements.

In our implementation, obtaining the codebooks, which demand the most training resources, involves training the HQ path for 300 epochs and the LQ path for 200 epochs. In comparison, CodeFormer requires 700 epochs to create its HQ codebook. Despite this, our use of training resources remains less intensive than that of CodeFormer.

[Q3] I think it will be better if there is more elaboration on the domain gap issue that the current works exist, i.e., the motivation of the paper. Currently, it is not intuitive from Figure 1 and the current discussion.

• We thoroughly analyze the domain gap in three distinct pathways: HQ reconstruction path, LQ reconstruction path, and Hybrid path. All three ways involve the utilization of LQ images as input. As Figure 1 shows in our supplementary material, both the HQ and Hybrid paths exhibit limitations in effectively reconstructing or restoring the LQ image due to the domain gap. Notably, the Hybrid path fails to generate any facial features.
• In these experiments, we observe that the LQ path better reconstructs the identity information from the LQ images, and the domain gap actually exists. Based on these observations, we design the additional LQ path to better encode the domain-specific information and introduction association stage and feature fusion to solve the domain gap issue.

[Q4] Check the spelling. For example, "recently" at the beginning of the second paragraph in the "Vector Quantized Codebook Prior" of the related work.

We will revise these parts in our updated manuscript.

[Q5] Why not combine the two stages or apply the association loss in stage 2? Besides, in stage 3, you state in the Training Objectives that the MHCA and transformer module are trained in this stage. However, from Figure 2(c), the two encoders seem not to be frozen during stage 3.

• We would like to clarify the role of each stage. In the codebook learning stage, we aim to obtain a pure HQ decoder from the HQ path for use in the final stage. If we are to simply combine the codebook learning stage with the association stage, there is a risk that the HQ decoder might interfere with the LQ path, leading to a decrease in image quality.
• We also conduct experiments where we apply only the association loss in stage 2. This resulted in less constraint on both paths, potentially causing a misalignment between the HQ and LQ paths. The detailed results are presented in the supplementary material.
• To address your question, we do not freeze the two encoders in stage 3. We believe that fine-tuning these encoders can accelerate the convergence and enhance feature fusion and code prediction in our restoration process.

[Q6] The results in Table 1 indicate that the proposed method does not significantly outperform other methods, especially for the synthetic CelebA-Test dataset.

In Table 1 of the manuscript, our approach outperforms the state-of-the-art methods, especially when dealing with real-world datasets. When it comes to the synthetic CelebA-test dataset, we achieve the best performance when we combine the final feature with the LQ features before passing them to the HQ decoder, as shown in the following table. These results highlight the strength and reliability of our approach.

[1] RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs. CVPR 2022.

[2] Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 2022.

[3] VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. ECCV 2022.

[4] DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration. CVPR 2023.

We extend our gratitude for the positive and constructive feedback.

We would like to address the raised concerns as follows:

[Q1] The contribution looks marginal to me since all the methods used in the different stages are well-designed and demonstrated. Adding another stream for low-resolution might not be a major contribution for a top-tier venue like ICLR.

We would like to further clarify the novelty and effectiveness of this work by putting this work in proper context and emphasizing the empirical results over state-of-the-art approaches.

• We would like to clarify that although some proposed model components are similar to prior arts, it requires meticulous algorithmic design and new modules to integrate them and achieve state-of-the-art results. As shown in Table 2 of the manuscript, the straightforward combination of HQ and LQ features does not improve performance. Thus, we introduce the association stage and the feature fusion modules to exploit the information from both domains effectively. The performance of the proposed model significantly outperforms state-of-the-art approaches, as shown in the following table.

• We propose an additional LQ reconstruction path to specifically encode the LQ domain information, which does not exist in prior works. Furthermore, we provide a new perspective to handle the LQ domain information better. We show the effectiveness of our additional LQ path with extensive experiments. How to better deal with the LQ input, the only source of information in this task, to assist the restoration process is crucial. Our contributions in providing new encoding strategies and viewpoints to handle LQ information are essential to advance the field.

• We demonstrate the novel components of our work by comparing with RestoreFormer [1] and CodeFormer [2], whose network building blocks mostly come from VQGAN [3] and Transformer [4]. Although these models are mostly built upon prior works, they show great empirical results can be achieved by novel and essential integration. Such works are important and make significant contributions. We use these examples to emphasize that sufficient novel components are introduced in proposing an additional LQ encoder to specifically deal with the LQ information and demonstrate its effectiveness with extensive experiments. We hope reviewers can appreciate the contribution of this work from this perspective.

• We show the ability of our method in real-world datasets with significant improvements over SOTA methods in both quantitative and qualitative evaluation. For the qualitative results, please refer to our manuscript and supplementary material. For the quantitative results, please refer to the following tables.

[1] RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs. CVPR 2022.

[2] Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 2022.

[3] Taming Transformers for High-Resolution Image Synthesis. CVPR 2021.

[4] Attention Is All You Need. NeurIPS 2017.

[5] VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. ECCV 2022.

[Q2] In Table 2, it seems like all the altered methods outperform the proposed method in terms of LPIPS. Please give discussions or visualizations to explain why this happens.

The LPIPS calculation involves using deep neural networks to analyze image features and then calculating a distance or similarity score based on these features. In our case, we input entire images, including background elements that are not our primary restoration focus. These background elements can affect the LPIPS score.

In this ablation study, our main goal is to demonstrate significant improvements in identity metrics. Additionally, we present qualitative results in Fig. 5 of the manuscript, showcasing the best visual outcomes achieved with our final setting.

We extend our gratitude for the positive and constructive feedback.

We would like to address the raised concerns as follows:

[Q1] Absence of Future Research Guidance: The paper does not offer any recommendations or insights into potential future research directions or enhancements for the proposed method.

We would like to provide detailed suggestions and directions for future research:

• Codebook of Local Facial Parts: As part of future work, we plan to integrate a codebook of local facial parts. This addition could enhance the accuracy of facial restoration, particularly for detailed regions such as the eyes, nose, and mouth, which are crucial for achieving realistic results.
• Extension to Severe Degraded Video Input: To broaden the applicability of our method, extending it to handle video input is an intriguing direction for future exploration. However, this line of work should consider the consistency of restored results across different time frames. Degradation in videos may vary, and the model should generate coherent and consistent outputs throughout the entire video sequence.

[Q2] Omission of Limitation Discourse: The paper lacks a discussion regarding its limitations and possible factors for analysis. Could you offer instances where the method encountered shortcomings or limitations?

We have provided visual results with large poses in Figure 20 of the supplementary material. While our method demonstrates robustness in most severe degradation scenarios, we also observe instances where it may fail, particularly in cases with large face poses. This can be expected as the FFHQ dataset contains few samples with large face poses, leading to a scarcity of relevant codebook features to effectively address such situations, resulting in less satisfactory restoration and reconstruction outcomes.

We include failure cases in the supplementary material to ensure a more comprehensive understanding of our method's limitations.

[Q3] Could you elaborate on the key distinction between DAEFR and CodeFormer?

• Our method, DAEFR, introduces a crucial distinction by incorporating an additional encoder specifically designed for improved feature encoding in the LQ domain. This allows us to leverage essential information from the LQ images effectively. Moreover, DAEFR uses an association stage that effectively bridges the domain gap between the HQ and LQ domains.
• In contrast, CodeFormer adopts a different approach by utilizing a pretrained HQ encoder, which is then fine-tuned on LQ images. However, the domain gap still exists despite this adaptation, which may present certain challenges.
• By integrating the LQ encoder in DAEFR, we aim to encode LQ features better, thereby facilitating more accurate and refined results. This emphasis on enhancing the LQ encoding process demonstrates the critical distinction between DAEFR and CodeFormer, ultimately leading to improved performance and overcoming potential domain gap issues.

[Q4] Can you provide further experimental details into the "Effectiveness of Low-Quality Feature from Auxiliary Branch" as examined in your ablation studies?

To demonstrate the effectiveness of our auxiliary LQ branch, we conduct validated experiments of fusing LQ features with feature $Z^{c}\_{f}$ in the feature fusion and code prediction stage. These experiments involve extracting LQ features $Z^{c}\_{l}$ from the LQ codebook and adding a control module [1] [2]. Given a feature control module and feature scalar $s\_{lq}$, we can control the scale of the LQ feature $Z^{c}\_{l}$ to fuse with the feature $Z^{c}\_{f}$ before feeding to the HQ decoder. The network architecture is shown in Figure 10 in the supplementary material.

We conduct an additional training stage to enable the control module to fuse different features effectively. In our implementation, we fine-tune the control module for ten epochs and maintain a feature scalar value of $s\_{lq} =1$ during training.

[1] Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR 2018.

[2] Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 2022.

We extend our gratitude for the positive and constructive feedback.

We would like to address the raised concerns as follows:

[Q1] Can you provide a detailed explanation of the primary differentiation between DAEFR and CodeFormer?

• Our method, DAEFR, introduces a crucial distinction by incorporating an additional encoder specifically designed for improved feature encoding in the LQ domain. This allows us to leverage essential information from the LQ images effectively. Moreover, DAEFR uses an association stage that effectively bridges the domain gap between the HQ and LQ domains.
• In contrast, CodeFormer adopts a different approach by utilizing a pretrained HQ encoder, which is then fine-tuned on LQ images. However, the domain gap still exists despite this adaptation, which may present certain challenges.
• By integrating the LQ encoder in DAEFR, we aim to encode LQ features better, thereby facilitating more accurate and refined results. This emphasis on enhancing the LQ encoding process demonstrates the critical distinction between DAEFR and CodeFormer, ultimately leading to improved performance and overcoming potential domain gap issues.

[Q2] The paper does not delve into its limitations or potential factors for analysis, which would greatly enrich its discussion.

We have provided visual results with large poses in Figure 20 of the supplementary material. While our method demonstrates robustness in most severe degradation scenarios, we also observe instances where it may fail, particularly in cases with large face poses. This can be expected as the FFHQ dataset contains few samples with large face poses, leading to a scarcity of relevant codebook features to effectively address such situations, resulting in less satisfactory restoration and reconstruction outcomes.

We include failure cases in the supplementary material to ensure a more comprehensive understanding of our method's limitations.

[Q3] The paper outperforms baseline methods in the downstream face recognition task. Could you provide a comprehensive explanation of these results?

Our quantitative experiments demonstrate the strength of our method. The issue with identity evaluation on the synthetic CelebA-test dataset lies in the impracticality of achieving identical images in real-world scenarios. It only measures the likeness between the original image and the restored one. Real face recognition datasets comprise a vast number of images belonging to the same individual, presenting more significant challenges and closely mimicking real-world scenarios. These results confirm the effectiveness of our added LQ encoder, which extracts valuable details from the LQ domain, thereby significantly enhancing the restoration process.

[Q4] The paper does not provide any suggestions or insights into potential avenues for future research or improvements to the proposed method.

We would like to provide detailed suggestions and directions for future research:

• Codebook of Local Facial Parts: As part of future work, we plan to integrate a codebook of local facial parts. This addition could enhance the accuracy of facial restoration, particularly for detailed regions such as the eyes, nose, and mouth, which are crucial for achieving realistic results.
• Extension to Severe Degraded Video Input: To broaden the applicability of our method, extending it to handle video input is an intriguing direction for future exploration. However, this line of work should consider the consistency of restored results across different time frames. Degradation in videos may vary, and the model should generate coherent and consistent outputs throughout the entire video sequence.