InstaRevive: One-Step Image Enhancement via Dynamic Score Matching

We thank Reviewer ZCNg for the valuable feedback. We address your concerns and questions below.

Q1: About more extend tasks

[Reply]: Thank you for your valuable suggestion. We have conducted further experiments to evaluate the applicability of our method to additional tasks, including various degradation types and real image-to-image transitions. In Section A.3 of the appendix (Page 19), we have added four new tasks: (1) denoising (salt-and-pepper noise), (2) deblurring, (3) image-to-image transition (day-to-night), and (4) deraining. Due to time constraints, we provide qualitative results for these tasks in Figure 15 (Page 20). These results demonstrate the versatility and adaptability of our framework to a broader range of applications.

Q2: About modifications for other tasks

[Reply]: As discussed earlier, our framework can produce plausible results for a variety of tasks without requiring modifications. However, we did encounter certain challenges when extending to these tasks.
First, our method depends on a substantial number of paired images for generator tuning, and the lack of suitable data can limit its performance in tasks like deraining and style transfer. Second, maintaining consistency with the source image, such as preserving spatial layouts and objects, poses additional challenges, particularly in style transfer tasks. To address these issues, we propose the following potential solutions:

• Enhanced Image Conditioning: Utilizing more advanced image conditioning techniques, such as ControlNet and IP-Adapter, can significantly improve consistency by strengthening the image condition injection. We explored integrating IP-Adapter in Section A.5 (Page 21), which helped preserve facial identity in source images.

• Self-Supervised Learning: By relying solely on the dynamic score matching loss in our framework, self-supervised learning without paired data is achievable. However, we observed that this approach can result in unstable loss curves and convergence issues. Further research into stabilizing and refining this technique will be a focus of our future work.

About the identity consistency

Thank you for your observation. Yes, we acknowledge that the original implementation of face cartoonization struggles to preserve facial identity. We attribute this to several factors:

• Dataset Limitations: The main reason lies in the dataset we employed. To produce the stylized face dataset from the FFHQ dataset, we utilized a common diffusion-based image-to-image model. While this model effectively transfers styles, it fails to retain the source identity, leading to a generator trained on this dataset lacking the capability to preserve identity. We believe that using a more specialized model, such as InstantID [a], to construct the dataset could significantly improve identity consistency.
• Simple Image Conditioning: Our current approach involves concatenating the input image with the latent features at the generator’s initial stage. While this straightforward approach is effective for tasks like image restoration and super-resolution, it can lead to the loss of high-level spatial features, including those critical for preserving identity. Recent works, such as IP-Adapter and InstantID, have demonstrated that maintaining facial identity in image-to-image tasks often requires incorporating a specialized network with cross-attention layers. These layers infuse identity features extracted by a face encoder into the latent features at every block of the denoising model. The absence of such advanced architectural designs in our implementation limits identity preservation.
• Loss Function Design: Our generator’s loss design, originally tailored for the blind face restoration (BFR) task, employs regression loss and dynamic score matching loss. With this setup, we achieve highly competitive identity similarity (IDS) scores, as reported in Table 1 (Page 7). However, tasks like face swapping [b, c], which especially emphasize identity consistency, typically use an identity loss to measure discrepancies in identity features between the input and output. As our framework is end-to-end, incorporating identity loss could enhance identity consistency.

Our primary goal in exploring this extended task was to evaluate the generalization capability of our framework across diverse tasks. While face cartoonization prioritizes style transformation over identity preservation, we believe improvements can be achieved by:

• Adopting identity-consistent datasets.
• Introducing more advanced architectures.
• Designing task-specific loss functions.

As demonstrated in Section A.5 and Figure 17 (Page 21), integrating IP-Adapter into our framework has proven effective in preserving identity during image-to-image transitions. We are optimistic that further experiments along these lines will yield better identity consistency in face cartoonization.

[a] "Instantid: Zero-shot identity-preserving generation in seconds." arXiv, 2024.

[b] "Simswap: An efficient framework for high fidelity face swapping." ACM MM, 2020.

[c] "HifiFace: 3D shape and semantic prior guided high fidelity face swapping." IJCAI, 2021

We sincerely thank Reviewer LCnj for the valuable insights and constructive feedback. Below, we address your concerns and questions in detail.

About comparisons with the zero-shot methods

[Reply]: Thank you for your insight. Zero-shot methods such as DDRM, DDNM, DPS, GDP, and PSLD indeed provide valuable approaches by leveraging pre-trained diffusion models for various tasks, including super-resolution, deblurring, and inpainting, using a single model without requiring fine-tuning. However, these methods also face inherent limitations that affect their practical applicability:

• Performance: Due to the absence of model fine-tuning, their performance is generally lower compared to tuned methods.
• Inference Speed: They rely heavily on diffusion sampling, resulting in significantly longer inference times (e.g., GDP requires 1000 steps, DPS over 250 steps, and PSLD around 200 steps).

In contrast, approaches that involve fine-tuning diffusion models—such as ours, DiffBIR, ResShift, and SinSR—achieve substantially better performance and faster inference, although they require task-specific restorers. We believe this trade-off highlights the strengths of our method in delivering efficient and high-quality results for real-world applications. As shown in Table 2 and Figure 14 (Page 19), the zero-shot GDP struggles to deliver satisfactory performance in complex scenarios, whereas our fine-tuned model consistently produces robust and reliable results.

About the initial restorer

[Reply]: Thanks for your suggestions. We implemented the initial restorer to enable the recovery of coarse information for both the text captioner and the generator. However, as shown in Figures 5 and 10, as well as Table 2, this restorer (SwinIR-GAN) provides only limited enhancement, resulting in noticeable blurring and suboptimal quantitative metrics. To further support this observation, we evaluated the performance of the initial restorer (SwinIR-GAN) on the ImageNet-Test dataset, using metrics such as PSNR, SSIM, LPIPS, CLIPIQA, and MUSIQ, as reported in Table 4 of the appendix (Page 18). The results highlight the generator’s ability to significantly improve image quality and consistency with the ground truth, as reflected by higher PSNR, SSIM, and CLIPIQA scores. These findings underscore the critical role of the distilled generator in producing more realistic and consistent results:

We thank Reviewer mTLA for the valuable feedback. We address your concerns and questions below.

Discussion and comparisons with recent methods

[Reply]: Thank you for your suggestion and for highlighting relevant works. We acknowledge that recent works like OSEDiff and img2img-turbo share similarities with our framework in employing diffusion distillation. However, we would like to emphasize that our work specifically addresses challenges encountered during score distillation, such as inaccuracies and over-smoothing, as demonstrated in Figure 2. To provide a more comprehensive comparison, we have included both qualitative and quantitative results in Figure 12 (Page 18) and Table 5 (Page 20), alongside further discussions in Sections A.1 (Page 16) and A.4 (Page 19). Furthermore, we have also provided comparisons with SUPIR in the same figure and table for additional context:

About more details in Figure 7

[Reply]: Thanks for your valuable advice. We agree that presenting two additional curves (without noise control and without loss control) in Figure 7, as you suggested, makes the analysis more detailed and clear. We have reviewed our training logs and updated Figure 7 accordingly. As shown in the revised figure, both dynamic control approaches significantly enhance loss convergence. We appreciate your suggestion, as it helps to make our paper more comprehensive.