Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model

4. How will the performance vary if different versions of diffusion models are used? That is, would diffusion models be the strong foundation of achieving the good performance?

A: We appreciate your insightful feedback. More advanced diffusion models, such as SDXL used in the SUPIR model and the newer SD3 model, can indeed be applied to our approach and achieved promising results. This is primarily due to the heavy reliance of generative model training on extensive image datasets. Without leveraging pre-trained generative models, we would require significantly more data to achieve comparable performance and intelligence. Many state-of-the-art intelligent image processing diffusion models, such as SUPIR, DiffBIR, PASD, and SeeSR, are based on pre-trained diffusion models.

However, utilizing pre-trained models is only part of what makes our approach successful. The exceptional restoration results of our method are primarily attributed to our unique model design, architecture, and the meticulous choices in training strategies and data. It is worth noting that while adopting more advanced diffusion models could further enhance performance, these models typically come with higher training complexity and resource requirements. This consideration motivates us to explore incorporating advanced diffusion models into future iterations of MGFR.

5. Some figures, say fig.4 and 5, are with too small fonts and unclear color denotions, making it less readable.

A: Thank you for your valuable feedback. In the revised manuscript, we have carefully adjusted the font sizes and included clearer color annotations to enhance readability and clarity.

We sincerely appreciate the reviewer’s recognition of our work, particularly the positive feedback on the novel framework of the MGFR approach, the contribution of the Reface-HQ dataset, and the superior performance of our method. We are also grateful for the reviewer’s acknowledgment of our efforts in integrating multi-modal priors, addressing real-world degradation, and mitigating the generation of false facial attributes.

1. Complexity of Implementation

A: We thank the reviewer for the insightful comments. We acknowledge that the inclusion of additional functionalities in the proposed MGFR model may introduce higher implementation complexity and computational demands. However, for diffusion-based methods like ours, the diffusion model itself is inherently computationally intensive. Recent state-of-the-art models such as SUPIR and DiffBIR, which are also diffusion-based, require substantial computational resources and extensive training.

In response to the reviewer’s suggestion, we have included a more detailed discussion in the revised manuscript, comparing the computational requirements (e.g., FLOPs and parameter counts) of our model with those of other contemporary methods. The results indicate that our approach is not significantly more complex than similar methods. However, in the context of face-focused restoration tasks, our method demonstrates significantly better performance compared to competing approaches.

Additionally, we recognize the potential for further optimizations to make our method more practical for real-world applications. Techniques such as low-bit quantization for inference, model pruning, and distillation can substantially enhance efficiency. While these topics are beyond the scope of this paper for an in-depth exploration, we have added relevant discussions in the supplementary material and plan to address these optimizations in future work.

2. Evaluation Metrics

A: Thank you for your question. Assessing the perceptual quality of images is indeed a challenging task, as extensively discussed in prior works. It is difficult for any single metric to accurately represent objective quality. In our paper, we include six different objective image evaluation metrics: SSIM, PSNR, LPIPS, ManIQA, ClipIQA, and MUSIQ, which encompass traditional image similarity metrics, deep learning-based metrics, and unsupervised metrics that are more suitable for generative models. These metrics are widely used in the field of perceptual image processing. Additionally, we include the ArcFace identity distance metric, which is extensively applied in works related to face recognition and face generation. All objective metrics demonstrate the effectiveness of our method. Furthermore, we conducted a user study to provide additional evidence of the superiority of our proposed approach. The results and analysis from this study have been included in the updated manuscript.

3. Ablation Studies

A: Thank you for your valuable suggestions. In this paper, we conducted the following ablation experiments:

1. Model performance without reference images (Table 1): The model achieves state-of-the-art performance and outstanding restoration quality even without reference image input.
2. Model performance with reference images (Table 2): The model achieves better restoration results and lower identity information loss when guided by reference images.
3. Controlled restoration using facial attribute prompts (Figure 8): MGFR effectively uses attribute prompts to edit and control the restoration results.
4. Ablation study on the effect of reference images (Figure 11): Demonstrates the effectiveness of reference images in guiding restoration results.
5. Ablation study on identity information integration (Figure 15): Incorporating ArcFace identity embeddings significantly reduces identity information loss.
6. Ablation study on negative prompts (Figure 13, Table 4): Negative prompts substantially improve restoration quality and detail reconstruction.
7. Ablation study on additional information exchange (Table 3): Incorporating additional information exchange enhances the control effect of reference images.
8. Experiments on attribute prompts completely inconsistent with low-quality inputs (Figure 10): Shows that the model consistently adheres to low-quality inputs during restoration.
9. Experiments on the sensitivity of attribute text control (Appendix E.2): Validates that the sensitivity of attribute text control is positively correlated with the degree of degradation, as expected.

If the reviewer believes additional ablation studies are necessary, we are more than happy to discuss and conduct further tests if needed.

We thank the reviewer for recognizing our work, the MGFR method, and the Reface-HQ dataset, as well as for the thoughtful feedback.

1. The proposed method appears complex

A: We appreciate the reviewer’s insightful comments. While we recognize that the additional functionalities in the proposed MGFR model may increase implementation complexity and computational demands, it is important to note that diffusion-based methods inherently involve high computational costs. Recent state-of-the-art models, such as SUPIR and DiffBIR, which are also diffusion-based, similarly require substantial computational resources and extensive training. To address the reviewer’s concerns, we have expanded the discussion in the revised manuscript to include a detailed comparison of computational requirements, such as FLOPs and parameter counts, between our model and other contemporary methods. The results indicate that our approach is not significantly more complex than similar methods, yet it demonstrates notably superior performance for face restoration tasks. Additionally, we acknowledge the potential for further optimizations, including low-bit quantization, model pruning, and distillation, to enhance efficiency and make our method more practical for real-world applications. While these optimizations are beyond the scope of this paper, we have included a discussion of these possibilities in the supplementary material and plan to explore them in future work.

2. There is a lack of comparison in FLOPs and Params in the main comparison and ablation study.

A: Thank you for your question. In the revised manuscript, we include detailed information on FLOPs and the number of parameters in the Table 12, along with a comparison to SUPIR. It is important to note that the FLOPs of the diffusion model depend on the number of inference steps, which can vary. To provide consistency, we will report results based on a single inference step.

3. Lacking comparison to recent works (SUPIR, BFRffusion)

A: Thank you for your question. We have conducted supplementary experiments using the same low-quality inputs as in the original study, and the quantitative comparison results with BFRFusion are now included in Tables I, 9, 10, and 11 of the revised manuscript. These results demonstrate that our model continues to achieve state-of-the-art performance.

Regarding SUPIR, its batch inference requires substantial hardware resources and computation time, making it challenging to complete all quantitative evaluations within the short discussion phase. We will continue these tests and include the results in future updates of the manuscript once completed. In the meantime, we have provided qualitative comparison results with the SUPIR model in Appendix K of the revised manuscript. These results show that while SUPIR can achieve high-quality outputs, it often loses significant identity information. In contrast, MGFR effectively preserves this information through the incorporation of reference images, which highlights the key contribution and significance of our approach.

4. how do the authors feel the proposed method can be extended for real-world video face restoration?

A: Thank you for the reviewer’s question. We are currently conducting research on video restoration based on diffusion generative models. The primary challenge in extending single-frame methods to video lies in ensuring inter-frame consistency. Our method, with its ability to inherit key features and identity information from reference images, effectively addresses semantic-level consistency, making it better suited for video restoration compared to other single-frame models. However, our approach still needs to overcome pixel-level consistency issues to prevent flickering between frames. This requires pixel-level referencing and smoothing, as well as handling temporal information effectively. Future improvements may involve incorporating temporal inputs and modifying the decoder to output smoother multi-frame results. Nevertheless, we are confident that using relevant frames as reference inputs, which is a core aspect of our method, is one of the key factors for high-quality video restoration.

5. Overall the paper looks promising and makes meaningful contributions, however, it lacks some important experiments and details.

A: We thank Reviewer 2kxB for recognizing the meaningful contribution of our paper and its potential, as highlighted in your comments. We have supplemented and revised the questions that have been raised, improved the relevant experiments, and included additional experimental data and detailed discussions in the revised manuscript. Should there be any further questions or suggestions, we would be more than happy to engage in further discussions. Once again, we deeply value your review and your recognition of our efforts.

We thank the reviewer for recognizing our work, particularly the motivation for using multi-modal inputs to assist face restoration, the rationale of diffusion prior, and the potential contribution of the Reface-HQ dataset to the community.

1. The method used in the paper is not new.

A: We respectfully disagree with the reviewer’s comment. Our method is not a simple application of existing techniques. Specifically:

1. We are the first to propose to simultaneously use attribute prompts, high-quality reference images, and face identity information in a generative model to guide real-world human image restoration.
2. Based on face image datasets, we extracted facial attributes and integrated them as attribute prompts through LLMs, enabling prompt-based editing and controlled restoration.
3. We designed a counterintuitive strategy that leverages negative prompts to further enhance restoration quality and facial details.
4. MGFR improves restoration results and reduces facial identity loss under the guidance of reference images, without requiring alignment between the reference and low-quality input images.
5. To effectively analyze and integrate multi-modal inputs, we introduced a dual-control adapter and a two-stage training strategy.

We believe our work demonstrates significant innovation.

2. Ablation of the effectiveness of using trained diffusion prior is missing.

A: Thanks for the feedback. Recent advanced face restoration methods based on diffusion models (e.g., DiffBIR, SUPIR, and BFRFusion) leverage pre-trained diffusion priors to achieve robust performance. These pre-trained diffusion priors significantly reduce training difficulty and time, allowing models to better learn data distributions consistent with specific tasks, thereby enhancing their effectiveness. Consequently, the use of pre-trained diffusion priors has become a common practice in diffusion-based face restoration models.

Ablation studies on the effectiveness of pre-trained diffusion priors would require training models from scratch, demanding extensive time, computational resources, and data, making it challenging to complete within a limited timeframe. Furthermore, numerous existing works have demonstrated that training diffusion models from scratch with limited data often fails to produce satisfactory results.

That said, utilizing pre-trained models is only one aspect of our method's success. The exceptional restoration results achieved by our approach are primarily attributed to our unique model design, innovative architecture, and the careful selection of training strategies and data. While incorporating more advanced diffusion models could potentially improve performance, such models generally come with increased training complexity and resource demands. This drives us to consider integrating advanced diffusion models in future iterations of MGFR to further refine our approach.

3. The design of the Dual control Adapter is a little but tricky. Can one just replace it by a transformer?

A: Thank you for your question. We also aimed to achieve the desired functionality with a single control module. However, our experiments showed that using a single controller to simultaneously handle guidance information from multiple modalities led to issues such as color discrepancies in the restored output, loss of control, and even suboptimal results. To successfully accomplish the task of controlling face image restoration using facial identity information and reference image guidance, we carefully designed a dual-controller scheme. Designing a single controller would require further investigation into the source of the control effect in diffusion models. We look forward to exploring whether a specially designed single-transformer architecture could further optimize our approach in future work.

4. There can be different result for every sample. How diverse is the model? And how to decide which sample to use?

A: Thank you for raising this insightful question. To evaluate the diversity and stability of our model, we conducted experiments by performing inference under multiple random seeds. The results demonstrated minimal variation across different runs, indicating that the model produces consistent outputs with negligible randomness. This stability ensures the practical usability of our method, as users can reliably achieve high-quality results without significant deviations.

5. Will be dataset be realised?

A: Thank you for your question. The proposed Reface-HQ dataset is set to be released. Compared to existing datasets, Reface-HQ offers a greater number of identity face images along with increased training and testing samples. In addition to the 512-resolution dataset utilized in this work, a 1024-resolution subset will also be made available for training and evaluating higher-resolution models. This dataset enables previous models to be leveraged to their full potential.