$InterLCM$: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration

We appreciate your constructive comments and will incorporate the discussions mentioned below to enhance the quality of our paper.

W1(1): Why prefer the bfr task not for the general sr task?
For the blind face restoration problem, our method InterLCM can efficiently extract facial information through the Visual Encoder, as human faces are with less complex semantic information compared with real images from diverse scenarios. We show several real-image restoration results in Figure.18. The results are promising for simple textures, but less effective for complex textures. To improve the performance of our method on real image, we plan to use a more powerful VQGAN-LC with 100,000 codebooks to act as the visual encoder for our model in future work.

W1(2): The usage of lcm is straightforward, and the theory analysis and support is absent.
This work is inspired by observations from latent consistency models (LCMs), leveraging their priors alongside perceptual losses for blind face restoration. LCMs demonstrate superior semantic consistency, excelling in subject identity preservation, spatial structure retention, and color stability. In contrast, general diffusion models often fall short in these aspects, struggling with identity preservation, structural stability, and color consistency, as illustrated in Figure.1. By integrating image-level losses, such as perceptual loss and adversarial loss, our InterLCM model significantly improves blind face restoration (BFR) performance.

W1(3): It is better to add more relevant works using the defined priors
Thanks for your advice. We already include and discuss them in the introduction and related work of the updated version.

We appreciate your timely response and constructive comments. General super-resolution requires handling more complex semantic information, and we will explore general super-resolution tasks based on our framework in the future.

We appreciate your constructive comments and will incorporate the discussions mentioned below to enhance the quality of our paper.

W1: Using pre-trained diffusion model as the image prior for blind face restoration is widely studied. The extension from conventional DDPM to LCM is quite straightforward.
This work is inspired by observations from latent consistency models (LCMs), leveraging their priors alongside perceptual losses for blind face restoration. LCMs demonstrate superior semantic consistency, excelling in subject identity preservation, spatial structure retention, and color stability. In contrast, general diffusion models often fall short in these aspects, struggling with identity preservation, structural stability, and color consistency, as illustrated in Figure.1. By integrating image-level losses, such as perceptual loss and adversarial loss, our InterLCM model significantly improves blind face restoration (BFR) performance.

W2: The choices of hyper-parameters.
For the LCM inference step numbers, LCM commonly recommends to employ the 4-step inference process to balance image quality and inference time. In this paper, we use the recommended 4-step LCM, while we also offer an ablation study with a 2-step LCM model. As observed from the table below, the 4-step LCM only works slightly worse than the 2-step LCM on two metrics, which is utilized as the backbone for our InterLCM. For the hyperparameter \lambda, we set $\lambda=0.8$ following CodeFormer[A], where they also apply the perceptual and adversarial losses. This setup achieves a better trade-off for our BFR model in practice.

[A] Towards Robust Blind Face Restoration with Codebook Lookup Transformer, NeurIPS 2022

W3: The computation of perceptual loss is particularly challenging. Several existing works [1][2] on blind inverse problems have successfully integrated this term into a Bayesian framework.
The challenge of the perceptual loss computation mainly lies in the accuracy and consistency of $x_0$ predicted in each inference step. Existing diffusion-based papers[1,2] derive $x_0$ from $x_t$ at an intermediate step by applying the inverse process of forward diffusion: ${\hat{x}\_0}=\frac{1}{\sqrt{\bar\alpha\_t}}(x\_t-\sqrt{1-\bar\alpha\_{t}}\epsilon)$.

By comparison, when applying perceptual loss in diffusion-based models, our approach uses $x_0$​ at the end of the inference steps of the diffusion model, consistent with the standard diffusion model. In contrast, existing works [1,2] derive $x_0$ from $x_t$ at an intermediate step by applying the inverse process of forward diffusion. As shown in Figure.19, we can observe that the $x_0$​ obtained from the SD intermediate steps (the first to fifth columns) has an appearance gap compared to the $x_0$​ obtained using the full sampling process (the last column).

[1] Parallel Diffusion Model of Operator and Image for Blind Inverse Problems (CVPR2023)
[2] Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution (WACV2024)

We appreciate your constructive comments and will incorporate the discussions mentioned below to enhance the quality of our paper.

W1: Is there any comparison for x0-prediction-based diffusion model.
To the best of our knowledge, the Consistency Model (CM) is the only diffusion model that features a x0-prediction branch. Unlike DDPM models, which incrementally perform denoising across multiple steps, CM directly learns to map $z_t$ to the original real image space $z_0$. Taking SD-Turbo as a representative method in xt-prediction-based model, we test the SD-Turbo as the backbone to develop our method for the BFR problem. By the quantitative comparison in Table.6 in the updated Appendix, we show that consistency model, which directly predict the $x_0$ in each step, better suits the BFR problem.

W2: Whether the proposed method could generalize to other natural image dataset?
For the blind face restoration problem, our method InterLCM can efficiently extract facial information through the Visual Encoder, as human faces are with less complex semantic information compared with real images from diverse scenarios. We show several real-image restoration results in Figure.18. The results are promising for simple textures, but less effective for complex textures. To improve the performance of our method on real image, we plan to use a more powerful VQGAN-LC with 100,000 codebooks to act as the visual encoder for our model in future work.

W3: Whether the proposed method can be integrated with LCM-LoRA for more fast inference?
The distillation process of LCM updates the entire model's parameters. In contrast, the distillation process of LCM-LoRA is performed only on the parameters of two low-rank matrices, while the pre-trained diffusion model remains frozen. The advantage of LoRA is to reduce the memory and compute overhead during training. After training, the LoRA parameters are added up to the pre-trained diffusion model before inference. During inference, LCM and LCM-LoRA have the similar number of parameters, resulting in close memory usage (6332.54MB vs. 6334.26MB) and inference time on a single 3090 GPU. Therefore, when combined with our InterLCM, either using the frozen LCM or LCM-LoRA, the memory usage (8334.56.MB vs. 8599.13MB) and inference time remain comparable.

Q1: How long could we get the model?
The InterLCM models are trained for 15K iterations using eight A40 GPUs (48GB VRAM), which takes nearly 9.46 hours for the saturation of the loss convergence.

Q2: What's the difference between the spatial encoder and ControlNet, only incorporating the visual embedding?
Our Spatial Encoder shares a similar structure and feature injection mechanism with the original ControlNet but differs in two key aspects:

(a) Integration of Image-Level Losses: LCMs enable the integration of image-level losses, such as perceptual loss and adversarial loss. In contrast, using ControlNet with the original diffusion loss (Eq. 1) proves ineffective for face restoration, as shown in Table.4. By leveraging the ControlNet architecture—replicated from the Stable Diffusion encoder modules and serving as the Spatial Encoder (SE) in our method, InterLCM—alongside the perceptual loss (Eq. 2), we achieve significant improvements in blind face restoration (BFR) performance.

(b) Input Type: The original ControlNet primarily takes canny edges or depth maps as input. Naively applying ControlNet for blind face restoration, as shown in Figure.9 (the fifth to seventh columns), involves using low-quality (LQ) images as input. While this approach generates high-quality outputs that preserve structural integrity, it often compromises fidelity.

Thanks for the authors' response, which has address part of my concerns. And there remain some concerns that unresolved.

Basically, the x0-prediction and consistency models are different model family with different noise schedules in training and different one-step generation ability. It may better to include the discussion with x0-prediction models, and show some comparisons.

LCM-LoRA is integrated for accelerated inference time with reduced sampling steps, and the memory usage the author presented are somewhat less concerned. I wonder whether the integration of the LCM-LoRA works for the proposed method, and the visual comparison or other quantitative comparison would be better.

Thank you for your response and the insightful question.

1. The x0-prediction and consistency models are different model family with different noise schedules in training and different one-step generation ability. It may better to include the discussion with x0-prediction models, and show some comparisons.

We use one-step models ($x_0$-prediction-based diffusion models) as the backbone to develop our method for the BFR problem. We first move the LQ image to the noise space of the one-step models. We make some comparisons in Table below (Table.8 in main paper) and Figure.14. As shown in Table below, our metrics significantly outperform one-step diffusion models in the BFR task, except for the FID metric on the Synthetic dataset. As shown in the qualitative comparison in Figure.14, results of our method using one-step models (as shown in the second and third rows) indicate that these models face challenges with artifacts and blur when reconstructing high-quality images, while our method can reconstruct high-quality images with detailed textures (the fourth row).

2. LCM-LoRA is integrated for accelerated inference time with reduced sampling steps, and the memory usage the author presented are somewhat less concerned. I wonder whether the integration of the LCM-LoRA works for the proposed method, and the visual comparison or other quantitative comparison would be better.

We test the LCM-LoRA as the backbone to develop our method for the BFR problem. Table below (Table.7 in main paper) and Figure.13 show the qualitative and quantitative, respectively, comparison of using LCM-LoRA and our method. LCM-LoRA does not perform as well as our method in terms of LPIPS and FID metrics, while it achieves better results on the MUSIQ metric for image quality evaluation on real datasets, such as LFW-Test and WebPhoto-Test. The qualitative results in Figure.13 demonstrate that both LCM-LoRA and our method can achieve high-quality reconstructed images.

Q4: Figure.1 is unclear. It says that LCM maps directly to the real image space. Does it imply LCM learns a mapping from LR to HR images directly like [2], rather than progressively denoising as depicted in Figures.2 and 3?
LCM mechanism is different from [2]. The Consistency Model (CM) represents a distinct family of diffusion models. Unlike DDPM models, which perform denoising incrementally through multiple steps, CM directly maps $z_t$ to its original real image space $z_0$. As illustrated in Figure.2, the process begins by predicting the original image from random noise in the first step. In subsequent steps, additional noise is added to the previously produced image, and the model predicts the original image again from this noisy input.

[2] I2SB: Image-to-Image Schrödinger Bridge.

Q5: What is the degradation process considered in the synthetic dataset? Is it a simple interpolation?
We detail our degradation process in Appendix A.2, which follows a typical procedure outlined in existing the state-of-the-art BFR work CodeFormer[A]:

$x\_{l}=$ {[(x_{h} * k_{\sigma}){\downarrow_{s}} + n_{\delta}]_{\text{JPEG}_{q}} }{\uparrow\_{s}}

where $x_h$ and $x_l$ represent the HQ and LQ images, respectively, $k_\sigma$ is the Gaussian kernel with $\sigma\in\{1:15\}$, $\downarrow_s$ represents the downsampling operation with a scale factor $s\in\{1:30\}$, and $n_\delta$ denotes Gaussian noise with a standard deviation of $\delta\in\{0:20\}$. The convolution operation is denoted by $*$, followed by JPEG compression with a quality factor of $q\in\{30:90\}$. Finally, an upsampling operation $\uparrow_s$ with scale $s$ is applied to restore the original resolution of $512\times512$.

[A] Towards robust blind face restoration with codebook lookup transformer, NeurIPS 2022

We appreciate your constructive comments and will incorporate the discussions mentioned below to enhance the quality of our paper.

W1&W2&W6: Discusses accelerating and the use of LQ as a prior.
First, using the intermediate state as a better initialization for diffusion tasks is a common practice in existing works, such as SDEdit[A], DR2[B], and DifFace[C]. However, these approaches often suffer from issues like distribution gaps[D] and signal leakage[E]. In contrast, the LCM model directly maps the noise space to the real image $z_0$, effectively bridging the distribution gap.

As to your referred papers, the literature [1,2] aims to accelerate diffusion-based models. In contrast, our method, InterLCM, is inspired by the latent consistency model (LCM), which demonstrates superior semantic consistency, including subject identity preservation, spatial structure retention, and color stability. However, general diffusion models, including distilled models, often struggle with maintaining semantic consistency, such as identity consistency, structural stability, and color preservation (see Figure.1).

Moreover, our few-step sampling approach enables InterLCM to integrate perceptual loss and adversarial loss for face restoration tasks. While speed improvement is an inherent benefit of the LCM model's properties, it is not our primary contribution. Additionally, we use low-quality (LQ) images as inputs to our proposed Visual Encoder (VE) and Spatial Encoder (SE), which extract semantic and visual information as conditions to enhance face restoration, rather than merely using LQ images as training inputs.

Finally, blind face restoration (BFR) is distinct from other restoration tasks like super-resolution, deblurring, and JPEG restoration, as discussed in [1,2]. BFR addresses complex, real-world degradations (see Appendix A.2), whereas other tasks typically deal with more straightforward degradations.

We already put the discussion about [1,2] in the related work of the updated submission.

[A] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. ICLR2022
[B] DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration. CVPR2023
[C] DifFace: Blind Face Restoration with Diffused Error Contraction. TPAMI, 2024
[D] Exploiting the signal-leak bias in diffusion models. WACV2024
[E] Real-world image variation by aligning diffusion inversion chain. NeurIPS2023
[1] Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. CVPR2022
[2] I2SB: Image-to-Image Schrödinger Bridge. ICML2023

W3: The abstract is difficult to follow.
Thanks for your suggestion. We have modified the abstraction in the updated version.

W4: While the quantitative metrics indicate improvements, the visual results do not convincingly demonstrate superior performance.
The blind face restoration is based on the whole dataset statistics, not merely focusing on improving the restoration quality of a single person. As shown in Table.1, for the evaluation of the synthetic and the real-world datasets, our method achieves the best performance on two metrics IDS and MUSIQ for identity and quality metric.

In Figure.6, our method significantly outperforms others in capturing textural details, despite differing from the Ground Truth. The reason for this difference is that the Ground Truth in this case contains natural noise, while restoration methods based on generative priors struggle to produce results consistent with the Ground Truth. We include more examples in Appendix with detailed metrics to show that our method outperforms other approaches.

As shown in Figure.20, our method shows better hair quality than other methods and better aligns with the Ground Truth. Actually, since the low-quality images are losing high-frequency information, the restoration is a random process to complement the high-frequency details (by varying seeds when adding noise).

Q1: The layout of the new Figure 17 appears somewhat unclear. I am unsure if the authors fully understood my question. My intent was to ask whether LR provides meaningful semantic information compared to its corresponding HR. Here, LR and HR are paired. However, the new figure seems to use unpaired HR for extracting semantic information, which does not address my question directly.

We update Figure.19 with paired HQ and HQ images input for our method of the updated submission. In our method (Figure.19, top(a)), InterLCM, we utilize a Visual Module to extract semantic information from LQ images for HQ reconstruction. To demonstrate that the LQ image suffices to provide a prior for HQ reconstruction, we provide our model with LQ images exhibiting varying levels of degradation, decreasing from left to right (Figure.19, middle(the first row)). The reconstruction results (Figure.19, middle(the second row)) show that the semantic information from the LQ image suffices as a prior for HQ reconstruction when the degradation level is below a specific threshold (Figure.19, middle(the third to fifth columns)).

Meanwhile, we observe that when the HQ image is used as the input to both the Visual Module and Spatial Encoder (Figure.19, top(b)), the reconstructed image displays similar semantic information to that obtained using the LQ image (Figure.19, bottom(the first column)). This result further indicates that the LQ image provides semantic information similar to that of the HQ image (Figure.19, middle(the last column) vs., bottom(the first column)).

Then, we verify the provision of paired LQ and HQ images, which are provided to the Visual Module and Spatial Encoder (Figure.19(c)). We also obverse that the reconstructed result shows similar semantic information to the HQ image (Figure.19, bottom(the second column)).

To further assess the importance of facial semantic information from the LQ image for HQ reconstruction, we supplied the Visual Module with non-facial semantic images (Figure.19, top(d)), such as non-facial semantic images (e.g., a image featuring a tree or a solid color) and unrelated facial images (Figure.19, bottom(third and fifth columns)). Using non-facial semantic images resulted in reconstructed outputs with artifacts (Figure.19, bottom(third and fourth columns)), whereas unrelated facial images provided sufficient semantic priors for generating HQ reconstructions with facial features (Figure.19, bottom(fifth columns)).

Q4: The response is clear and aligns with my understanding of LCM. In this case, would it be more precise to phrase it as, “the network used in LCM maps …”?
Yes, we are pleased that we have reached alignment on this point. Based on your suggestion, we have already updated the caption of Figure.1 accordingly.

W5: Regarding the response to the second statement I mentioned, predicting x0 or noise in diffusion models is effectively equivalent, making it possible to apply perceptual loss in any diffusion model. Therefore, it is not a convincing response. Please see “Three Equivalent Interpretations” in https://arxiv.org/pdf/2208.11970 for more details.

Firstly, we agree with your point that predicting $x_0$ or noise is theoretically equivalent. However, in practice, the distribution gap [D] (e.g., Figure.2 in [D]) and signal leakage [E] (e.g., Figure.1 in [E]) accumulate because the training images introduce image priors during the training process. As a result, $x_T$ does not strictly follow a normal Gaussian distribution. This discrepancy underscores why the consistency model focuses on directly mapping $x_t$ to $x_0$, effectively bridging the gap and mitigating these issues.

[D] Real-world image variation by aligning diffusion inversion chain. NeurIPS2023
[E] Exploiting the signal-leak bias in diffusion models. WACV2024

Secondly, we want to claim that LCM was originally proposed to speed-up DM computation. However, in this paper, we show that there is another aspect of LCM which is very crucial for BFR problem: it allows for true end-to-end training for image generation, whereas DMs only optimize single denoising steps. The end-to-end training is also shown in Figure 3 where we can apply losses after the n-step generation of the final image, and backpropagate through all the image generation steps. This end-to-end nature of LCM allows us to do things which are not possible with standard DMs (where only singel step from $x_t$ to $x_{t-1}$ is optimized).

As a practical usage in this paper, it allows us to train with perceptual losses in the image domain, which is crucial for BFR where faithfulness is of utmost importance (compared to general image generation tasks). We can apply the perceptual and adversarial losses directly in the image domain and backpropagate through all image generation steps. It is unclear how to introduce a perceptual loss in a traditional DM training, where the noise-estimation of a single time step is optimized at the time. In this case, we have no results in the image domain on which to apply the perceptual loss. Another alternative ,as you mentioned, could be to estimate $x_0$ also from the intermediate steps, however, this would yield perceptually very low-quality images for most time steps. In this case, the perceptual loss is not expected to work due to the low semantic consistency and worse image quality. In this case, we would like to claim that predicting $x_0$ is not perpectly equivalent to predicting the noise.

As a conclusion, the usage of LCM allows training with losses directly in the image domain; to the best of our knowledge, we are the first to show this.

W4: I acknowledge that the proposed method performs quantitatively better in simulations. However, in real cases—which are central to BFR—I do not observe a notable improvement in visual results. For instance, in Figure 7, particularly in the top two setups, it would be difficult to distinguish whether RestoreFormer, CodeFormer, or the proposed method performs better if the labels were blinded. Even when the ground truth is available, the proposed method appears to over-sharpen the results.
Thanks for your acknowledge that the proposed method performs quantitatively better in simulations.

We also want to emphasize that the blind face restoration is based on the whole dataset statistics, not merely focusing on improving the restoration quality of a single person. As shown in Table.1, for the evaluation on real-world datasets, our method achieves the best performance for three datasets on two metrics: FID and MUSIQ, except for the FID value on the LFW-Test dataset.

Qualitative results: In Figure.7, to show that our method significantly outperforms baselines in reconstructed textual details, we zoomed in on other areas of detail. For example, in the first case (the first and second rows), our method can reconstruct the better detail in hair (red box), and eyes with the surrounding details (green box), while the baselines (including RestoreFormer and CodeFormer) generate blurr hair and meet artifacts for eyes and its surrounding details. In the second case (the third and fourth rows), our method can reconstruct more realistic crow's feet (red box) and hair details (green box). In contrast, the baseline methods, including RestoreFormer and CodeFormer, generate blurry crow's feet and hair. Although DifFace is capable of generating detailed hair, it exhibits noticeable artifacts around the eyes and in the wrinkles.
We present additional qualitative comparisons of the baselines on real-world images from the LFW-Test, WebPhoto-Test, and WIDER-Test datasets in Figure.23. As shown in Figure.23, our method can reconstruct more realistic details in forehead wrinkles (first and second rows), eyes and eyebrows (third and fourth rows), and hair (fifth and sixth rows). These results demonstrate that our method outperforms the baselines in real-world scenarios.

Quantitative results: Furthermore, we provide the top two setups with the MUSIQ metric of Figure.7. Our method achieves the best MUSIQ scores in both setups compared to the baselines, indicating that our results have the highest perceptual quality.

W1&W6(2)： The authors, in their response, argue that existing methods face issues with distribution gaps and highlight that LCM addresses this by mapping noise directly to real images. Firstly, in the multi-step case, LCM still alternates between denoising and noise injection steps. More importantly, I am unsure how this new argument adequately addresses my concern nor how LCM can address the distribution gaps. While I appreciate the point that extracting semantic and visual information from LR is a contribution, I struggle to see other significant contributions in this paper.

Noise injection: During the training phase of LCM, noise is injected into the real image, and then directly maps any point in the ODE trajectory back to its origin, facilitating semantic consistency generation compared to SD, as also shown in Figure.1. In the inference phase of LCM, the same noise injection step used during training is inherited, while the inference of SD does not use a noise injection like in its training phase. Existing work [1,2] uses noise injection that the inference of SD does not include. [A] find that a bias exists between the latent distribution in the vanilla generation chain and the image noise injection chain, resulting in a significant domain gap between the denoised images. In contrast, the LCM inference already includes a noise injection step (see Figure.2.).

We utilize the semantic consistency properties (Figure.1) and the noise injection step during inference (Figure.2). Additionally, thanks to the few-step generation of LCM, we can integrate the perception loss in the image space while avoiding the distribution gap when injecting noise into the LQ image. It is more suitable to combine with perception loss. Additionally, to the best of our knowledge, we are the first to leverage the consistency properties (see Figure.1) of LCM for exploring BFR tasks.

From the philosophical aspect of machine learning, a cornerstone principle is to train algorithms under the same circumstances as they are tested. We therefore think, that the fact that we can actually train our network with the LQ image (which is applied during inference) is an advantage, and could explain the excellent results we report in our experiments.

[A] Real-World Image Variation by Aligning Diffusion Inversion Chain.
[1] Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.
[2] I2SB: Image-to-Image Schrödinger Bridge.

Table.B Quantitative comparison on the synthetic and real-world dataset

Thank you for your feedback and the opportunity to further clarify our work.

W1&W6(1)： My concern is that using better initialization from a different estimator is not a novel concept.

Using a better initialization is not the primary contribution of our method. More importantly, our approach leverages the inherent semantic consistency properties of LCM, as observed in our experiments. A good initialization alone is not enough to ensure good performance. We test SD Turbo as the backbone to develop our method for the BFR problem. In detail, we use a 4-step SD Turbo pretrained model, where the LQ image is treated as the input at an intermediate step of the SD Turbo process. The HQ image is then obtained by performing the remaining denoising steps (i.e., 3 steps) in the 4-step SD Turbo model. Unlike LCM, which incorporates a noise injection step during inference, SD Turbo only includes denoising steps and no noise addition step during inference. Therefore, we adopt the noise injection step from the diffusion training process. We evaluated the initial performance of LCM and SD Turbo on the COCO2014 and COCO2017 datasets. As shown in Table A, SD Turbo outperformed LCM in terms of FID and CLIPScore on both datasets as the initialization points. Compared to LCM, SD Turbo demonstrated superior generative capabilities. Therefore, SD Turbo should provide a better initialization.

Table.A The initial performance of SD Turbo and LCM on COCO2014 and COCO2017

However, the quantitative analysis presented in Table.B indicates that SD Turbo is not optimally suited for BFR problems. For instance, SD Turbo fails to achieve state-of-the-art (SOTA) performance across all metrics; it only attains the second-best score on the MUSIQ metric for the real-world WebPhoto-Test dataset. In contrast, our approach, which incorporates LCM, surpasses SD Turbo in all metrics on both synthetic and real-world datasets, with the exception of the MUSIQ metric on the WebPhoto-Test dataset.

Dear Reviewer 43VF,

Thank you once again for your constructive comments. Your constructive comments have been very helpful.

In response to your remaining questions, we have conducted experiments to clarify that (W1&W6(1)) using a better initialization (i.e., SD Turbo) is not optimally suited for BFR problems and is not the primary contribution of our method. We have also clarified (W1&W6(2)) the distribution gaps, (W5) the prediction of $x_0$, and (Q1) provided additional explanations for Figure 17. (W4) Additionally, we zoomed in on other areas of detail in Figure.7 and provide the additional visual results for real cases. We would like to know if our responses have addressed your concerns.

As the rebuttal window is closing soon, we genuinely appreciate your feedback. We hope our response has addressed your concerns.

Best Regards,
Authors of submission 536

We appreciate your constructive comments and will incorporate the discussions mentioned below to enhance the quality of our paper.

W1: What does "back to the low-quality image initialization" in line 100 mean? Does Figure 3 show all trainable components?
In our method InterLCM, by integrating both perceptual loss and adversarial loss, backpropagation is performed from the real image $z_0$ to the initial low-quality image $x_{\tau_n}$. During this process, the Visual Encoder (VE) and Spatial Encoder (SE) are updated using the computed gradients. In contrast, the diffusion loss commonly used in ControlNet and standard diffusion model training performs backpropagation from $z_{t-1}$​ to $z_t$, with the primary goal of updating the Diffusion-UNet. Notably, in our approach, the latent variables remain unchanged throughout the process.

We already rewrote the corresponding sentence in Line 100 for clarity.

W2: Table.2 should be reorganized. The current version is squeezed and difficult to read.
Thanks to your advice. We reorganize and enlarge the font of the table in the updated version.

W3: The method's performance on LQ faces with textures like tattoos or face paint.
As shown in Figure.16 (the third and fourth rows), our method, InterLCM, demonstrates the ability to reconstruct high-quality details even in challenging cases, such as images featuring tattoos or festival-style face paint. However, when tattoos contain intricate details, such as text (e.g., the last column), accurately recovering these ambiguous elements during high-quality face reconstruction becomes challenging. This limitation may stem from the scarcity of such textures in the training dataset. An illustration of the complex textures in our training dataset FFHQ is also shown in Figure.16 (the first and second rows), where the festival-style face paints and rich-color hair appear multiple times during training.

W4(1): Does the author's method design draw significant inspiration from general image restoration approaches built on diffusion models (e.g., ControlNet) ?
This work is inspired by the observation that latent consistency models (LCMs) demonstrate greater semantic consistency in subject identity, spatial structure, and color preservation. However, general diffusion models often exhibit inferior semantic consistency, including issues with identity preservation, structural stability, and color consistency, as demonstrated in Figure.1.

Additionally, InterLCM enables the integration of image-level losses, such as perceptual loss and adversarial loss. In contrast, using ControlNet with the original diffusion loss (Eq. 1) proves ineffective for face restoration, as evidenced in Table 4. Instead, leveraging the ControlNet architecture—replicated from the Stable Diffusion encoder modules and serving as the Spatial Encoder (SE) in our method, InterLCM—alongside the perceptual loss (Eq. 2) significantly improves blind face restoration (BFR) performance.

W4(2): It seems that the main performance improvement comes from the spatial encoder.
We employ a Visual Encoder to extract semantic information from faces. Initially, faces are embedded by the CLIP image encoder as image embeddings, providing the Latent Consistency Model (LCM) with face-specific semantic priors. To preserve structural integrity, we utilize a Spatial Encoder. As illustrated in Fig. 8③, using only the Spatial Encoder can effectively maintain the structural information of the face but fails to reconstruct finer facial details, such as the eyes. These intricate details primarily rely on the Visual Encoder, as evidenced by the results when solely using the Visual Encoder (Fig. 8①).

W4(3): The second main source of performance appears to be the LPIPS loss.
As mentioned above, our approach does not utilize the LPIPS loss. Instead, we are the first to integrate perceptual loss and adversarial loss into diffusion-based blind face restoration (BFR) solutions. The effectiveness of this integration is demonstrated in the ablation studies provided in Figure.9 and Table.3.

W4(4): Could the performance improvements be easily achieved by using a ControlNet alone?
Naively applying ControlNet for blind face restoration, as shown in Figure.9 (the fifth to seventh columns), uses the low-quality (LQ) image as input. While it can generate high-quality outputs that preserve structural integrity, it often compromises fidelity.

W1&W6(2) (Reviewer 43VF)： The authors, in their response, argue that existing methods face issues with distribution gaps and highlight that LCM addresses this by mapping noise directly to real images. Firstly, in the multi-step case, LCM still alternates between denoising and noise injection steps. More importantly, I am unsure how this new argument adequately addresses my concern nor how LCM can address the distribution gaps. While I appreciate the point that extracting semantic and visual information from LR is a contribution, I struggle to see other significant contributions in this paper.

Noise injection: During the training phase of LCM, noise is injected into the real image, and then directly maps any point in the ODE trajectory back to its origin, facilitating semantic consistency generation compared to SD, as also shown in Figure.1. In the inference phase of LCM, the same noise injection step used during training is inherited, while the inference of SD does not use a noise injection like in its training phase. Existing work [1,2] uses noise injection that the inference of SD does not include. [A] find that a bias exists between the latent distribution in the vanilla generation chain and the image noise injection chain, resulting in a significant domain gap between the denoised images. In contrast, the LCM inference already includes a noise injection step (see Figure.2.).

We utilize the semantic consistency properties (Figure.1) and the noise injection step during inference (Figure.2). Additionally, thanks to the few-step generation of LCM, we can integrate the perception loss in the image space while avoiding the distribution gap when injecting noise into the LQ image. It is more suitable to combine with perception loss. Additionally, to the best of our knowledge, we are the first to leverage the consistency properties (see Figure.1) of LCM for exploring BFR tasks.

From the philosophical aspect of machine learning, a cornerstone principle is to train algorithms under the same circumstances as they are tested. We therefore think, that the fact that we can actually train our network with the LQ image (which is applied during inference) is an advantage, and could explain the excellent results we report in our experiments.

[A] Real-World Image Variation by Aligning Diffusion Inversion Chain.
[1] Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.
[2] I2SB: Image-to-Image Schrödinger Bridge.

Table.B Quantitative comparison on the synthetic and real-world dataset

Thank you for acknowledging our quantitative and qualitative evaluation results. This acknowledgement is very important to us. For your convenience, we have provided below the weaknesses 1 and 6 from Reviewer 43VF, along with our responses.

W1&W6(1) (Reviewer 43VF)： My concern is that using better initialization from a different estimator is not a novel concept.

Using a better initialization is not the primary contribution of our method. More importantly, our approach leverages the inherent semantic consistency properties of LCM, as observed in our experiments. A good initialization alone is not enough to ensure good performance. We test SD Turbo as the backbone to develop our method for the BFR problem. In detail, we use a 4-step SD Turbo pretrained model, where the LQ image is treated as the input at an intermediate step of the SD Turbo process. The HQ image is then obtained by performing the remaining denoising steps (i.e., 3 steps) in the 4-step SD Turbo model. Unlike LCM, which incorporates a noise injection step during inference, SD Turbo only includes denoising steps and no noise addition step during inference. Therefore, we adopt the noise injection step from the diffusion training process. We evaluated the initial performance of LCM and SD Turbo on the COCO2014 and COCO2017 datasets. As shown in Table A, SD Turbo outperformed LCM in terms of FID and CLIPScore on both datasets as the initialization points. Compared to LCM, SD Turbo demonstrated superior generative capabilities. Therefore, SD Turbo should provide a better initialization.

Table.A The initial performance of SD Turbo and LCM on COCO2014 and COCO2017

However, the quantitative analysis presented in Table.B indicates that SD Turbo is not optimally suited for BFR problems. For instance, SD Turbo fails to achieve state-of-the-art (SOTA) performance across all metrics; it only attains the second-best score on the MUSIQ metric for the real-world WebPhoto-Test dataset. In contrast, our approach, which incorporates LCM, surpasses SD Turbo in all metrics on both synthetic and real-world datasets, with the exception of the MUSIQ metric on the WebPhoto-Test dataset.

Dear Reviewer FpYB,

Thank you once again for your constructive comments. Your suggestions regarding the clarification of the sentence (line 100), Table 2, and the method design have been very helpful. We would like to know if our responses have adequately addressed your concerns. We are also eager to hear any additional concerns or suggestions from you that could help us further strengthen our paper. Your feedback would be invaluable in making this work even more compelling and worthy of acceptance.

As the rebuttal window is closing soon, we genuinely appreciate your feedback. We hope our response has addressed your concerns.

Best Regards,
Authors of submission 536