Feature out! Let Raw Image as Your Condition for Blind Face Restoration

Thank you for your careful review of the paper structure and formatting, which is greatly appreciated.

Q1. In Table 1, the reference numbers for SoTA methods should be listed.

• Due to the ICML citation format not using numbers, the references were too lengthy to include directly in the table. Instead, we noted the references in the Comparison Methods section of Sec. 5.1. We have now supplemented the references and adjusted the font size in the table to ensure that they can be properly displayed in the main paper in future versions.

Sec.5.1 (Line 361 - Line 370) Comparison Methods We compare Pseudo-Hashing with recent BFR methods, including PSFRGAN (Chen et al., 2021a), GFPGAN (Wang et al., 2021), GPEN (Yang et al., 2021), VQFR (Gu et al., 2022), CodeFormer (Zhou et al., 2022), RestoreFormer (Wang et al., 2022), DMDNet (Li et al., 2022), DAEFR (Tsai et al., 2023), DifFace (Yue & Loy, 2022), DR2 (Wang et al., 2023), PGDiff (Yang et al., 2024), DiffBIR (Lin et al., 2023), PMRF (Ohayon et al., 2024), FlowIE (Zhu et al., 2024) and I2SB (Liu et al., 2023) .

Thank you for your comprehensive review and valuable insights.

Q1. Their approach resembles a data augmentation technique, like in DiffBIR, suggesting that pairing I2SB with it might yield similar results, indicating potential simplicity.

• Our P-I2SB is not a data augmentation method. Data augmentation involves using certain properties of images, such as rotation and brightness, to augment the actual numerical distribution of training data without altering the semantic information of the images. In contrast, the strategy design in pseudo-hashing module (PHM) is intended to satisfy the boundary constraints in SB models, requiring the direct or indirect involvement of degradation representations, and this hashing design is reversible.
• DiffBIR consists of two stages: first, the removal of degradation information, followed by generating a high-quality image that balances quality and identity. This is completely different from PHM. In our first stage (PHM), we do not remove any degradation information; instead, we rely on an SB-based model to learn indirect restoration relationships.

Q2. They provide limited evidence that the paths are optimal; additional analyses, like those in Appendix Figure 6, would help substantiate these claims.

• Existence of Optimal Path in P-I2SB: We first establish the existence of the optimal path in our P-I2SB. Our model is constructed based on SB theory, as outlined in the Preliminaries (Sec.3). The consistency between Equation (7) and the optimal transport problem in Equation (2) reveals why the solution of the SB model is the optimal transport path.
• Monge's Problem, linked to image restoration, seeks the optimal mapping $T$ between two distributions under a certain cost metric. While it can theoretically map low-quality to high-quality images, its practical scope is limited by mapping constraints. Therefore, the optimal transport problem introduces a version that finds the optimal joint distribution, ensuring a solution always exists. This is depicted in Equation (2). In BFR problems, the relationship between low- and high-quality images is more complex. To leverage the existing I2SB model, as discussed in Preliminaries (Sec.3), we propose PHM to address these complexities.
• Additional Experiments and Analyses: To analyze the superiority of the path found by our P-I2SB, we compared P-I2SB with the baseline method on the loss analyses along the path $\{x_t\}, t \in [0,1]$. The results can be found at the following link: https://anonymous.4open.science/r/P-I2SB.

$$forward: X_t\sim q(X_t|X_0, X_1)=\mathcal{N}(X_t;\mu_t(X_0, X_1), \Sigma_t), inverse: \hat{X_t} \sim p(X_t|X_0^\epsilon,X_{t+1})=\mathcal{N}(X_t;\mu_\sigma(X_0^{\epsilon}, X_{t+1}), \Sigma_\sigma).$$

Q3. Lines 73 and 147 contain incorrect double quotation marks.

Thank you for pointing this out. Due to compilation issues, we have used bold and italic formatting instead to avoid errors related to the use of double quotation marks. The revised version is as follows:

• the optimal transport between images is not unique
• a relaxed version introduced by Kantorovich

Q4. Line 164 references “problem (2),” but the text does not provide a corresponding explanation.

• Thank you for your insightful question, which helped us identify an error in Equation (7). The corrected version is as follows, where $\bar{v}(t,x):=f_t(x)-\frac{\sqrt{\beta_t}}{2}\nabla\log\bar{p}(t,x)$ and $p(t,x)\mathrm{d}x = \mu_t(\mathrm{d}x)$.

$$\inf_{(p,v)} \int_0^1\int_{\mathbb{R}^{d}} [\frac{1}{2}||v(t,x)-\bar{v}(t,x)||^2 + \frac{\beta_t}{8}||\nabla\log\frac{p(t,x)}{\bar{p}(t,x)}||^2]p(t,x)\mathrm{d}x\mathrm{d}t. \tag{7}$$

• The text "as shown in problem (2)" means it is actually the same as equation (2). This implies that the SB problem is linked to the optimal transport problem. Furthermore, $p(t,x), (t\in[0,1])$ obtained by solving the SB model represents a form of optimal transport path. The phrase "as shown in problem (2)" is incorrect, and we will correct it to "just like equation (2)".

$$\inf_{(\mu,v)} \int_0^1{\int_{\mathbb{R}^{d}} ||v(t,x)||^2 \mu_t(\mathrm{d}x)}\mathrm{d}t. \tag{2}$$

Q5. What is the significance of designing Cat-I2SB? If only the first three channels are retained, equation (12) can be simplified to $(I_{hq}, I_{lq}^d)$.

• The Cat-I2SB is designed to implement PHM strategies by hashing training image pairs from $(I_{hq}, I_{lq}^d)$ to $(I_{hq} \oplus I_{lq}^d, I_{lq}^d \oplus I_{lq}^d)$, enabling the creation of varied training pairs for different degradations $d$.
• In Equation (12), only the first three channels are used in the final data. However, this hashed data is utilized during P-I2SB training, where the diffusion network's input and output channels are adjusted. The original pairs $(I_{hq}, I_{lq}^d)$ remain unhashed. This modification increases the parameter size of Unet, approximately 558.6MB.

We sincerely appreciate the thorough review and insightful comments you have provided.

Q1. The paper does not thoroughly analyze the computational overhead of the pseudo-hashing strategies (Cat/Res/Noise-I2SB) compared to baseline methods, despite claiming retained inference speed.

• Computational Complexity: In comparison to the baseline methods, our approach incorporates an additional pseudo-hashing module (PHM). The computational complexity of this module is as follows: for Cat-I2SB, it is $\mathcal{O}(n)$; for Res-I2SB, it is $\mathcal{O}(n)$; and for Noise-I2SB, it is $\mathcal{O}(nT)$, where $T=10$ denotes the local number of steps in DDIM, and $n$ denotes the number of input images. This computational complexity is calculated with respect to processing $n$ input images. During inference, our method only adds the Inverse-PHM process relative to the baseline methods, without significantly increasing the complexity of this model or inference time.

Q2. What about the sensitivity to degradation types and the scalability to non-face domains?

• Sensitivity to Degradation Types: The pseudo-hashing module (PHM) is specifically designed for image restoration under various unknown degradation. The three hashing strategies presented in the paper introduce degradation feature representations either directly or indirectly. This enables the module to perceive both the type and degree of degradation, thereby effectively handling low-quality image restoration across different degradation types.
• Scalability to Non-Face Domains: Our method possesses scalability, and we have verified its effectiveness primarily in blind face restoration (BFR), where the demand for naturalness is exceedingly high. As noted in "Limitations of Face Image Generation" [1], face restoration is more challenging than non-face tasks due to the need for naturalness and identity consistency. The restoration of identity, micro-expressions and other details is crucial, significantly validating the effectiveness of our method. Meanwhile, non-face models emphasize robustness, and currently, no general model can be directly applied to face restoration. We plan to extend our method to general restoration tasks in the future. The advancement of BFR is also critical to the development of face restoration field, and we will continue to advance BFR and expand our research into general domains.

reference: [1] Rosenberg H, Ahmed S, Ramesh G, et al. Limitations of face image generation. In AAAI, 2024, 38(13): 14838-14846.

Q3. Res-I2SB assumes that the degradation process can be modeled as a linear residual from HQ to LQ, but in reality, degradation (such as JPEG compression and non-uniform blurring) is mostly a nonlinear transformation, which may lead to insufficient fitting ability of the model for complex degradation.

• It is instructive for us. The hashing module in Res-I2SB is not designed to construct a linear residual from LQ to HQ. Instead, it aims to model the transformation relationship between two data distributions using a SB-based model. This transformation is often a complex nonlinear one, which necessitates the use of the SB model for resolution. In Res-I2SB, $(I_{lq}, I_{lq} - I_{hq})$ is treated as a new image pair. We do not further explore its linear relationship but require that $I_{lq} - I_{hq}$ serve as a new boundary distribution to help find the optimal path.
• In this context, the SB model aims to find the optimal joint distribution between given distributions, not to map low-quality to high-quality images. This joint distribution perspective is a broader generalization. Our goal is to establish the optimal dynamic diffusion path between these images, without assuming any specific mapping relationship, such as a linear constraint. Preliminaries (Sec.3) in main paper shows the evolution from a mapping to a joint distribution perspective in optimal transport problems.

Q4. Cat-I2SB, Res-I2SB, Noise-I2SB still require manual prior selection. I wonder if it would be better to introduce pre-trained degeneration classifiers to guide strategy selection?

• Since we are dealing with a blind task where the degradation parameters and types are random and uncertain, potentially involving multiple combinations, classifiers cannot be used to clearly distinguish each category.
• In terms of strategy design, Noise-I2SB handles differences caused by various degradations using degradation representation. We employ the formula $I_{hq}^{noise} = I_{lq}^d + \lambda_d \epsilon$, where $\lambda_d$ represents the noise level, which is directly related to $I_{lq}^d$. Here, we set $\lambda_d$ as the degeneration representation feature of $I_{lq}^d$ with an MLP layer.
• Looking forward, we will focus on researching how to design PHM strategies without relying on manual prior selection. This will require further exploration of the relationship between the conditions for the optimal solution in SB and the boundary constraints.

Thank you for your detailed feedback and constructive suggestions. Your input is crucial to refining and enhancing the quality of our paper.

Q1. The computational complexity of PHM preprocessing is not well analyzed. What is the computational cost of PHM preprocessing relative to standard diffusion models?

• Computational Complexity: In comparison to the baseline methods, our approach incorporates an additional pseudo-hashing module (PHM). The computational complexity of this module is as follows: for Cat-I2SB, it is $\mathcal{O}(n)$; for Res-I2SB, it is $\mathcal{O}(n)$; and for Noise-I2SB, it is $\mathcal{O}(nT)$, where $T=10$ denotes the local number of steps in DDIM, and $n$ denotes the number of input images. This computational complexity is calculated with respect to processing $n$ input images. During inference, our method only adds the Inverse-PHM process relative to the baseline methods, without significantly increasing the complexity of this model or inference time.

Q2. More ablations on different hashing strategies are required. The paper could include more ablation studies to quantify the impact of PHM on transport efficiencyexplicitly.

• We compared the ablation experiments of different strategies in Table 3 in main paper and Figure 7 in the appendix. The related comparison results are provided here again. Additionally, we considered combining all three strategies. Due to time constraints, this was only validated in a toy experiment, as shown in Sec.4.3 (Toy Exploration and Analysis), and more results are shown at the following link: https://anonymous.4open.science/r/P-I2SB.

• Table.3 Ablation studies on CelebA-Test. (a) donates Vanilla-I2SB as the baseline, (b)-(c) compare different condition guidance, and (d)-(f) compare different Pseudo-Hashing strategies. "lq" donates LQ images as the condition in forward and reverse process and "da" donates degradation-aware representation as condition.

Q3. Limited real-world evaluations—most experiments use synthetically degraded images. Could adaptive pseudo-hashing improve restoration in real-world scenarios?

• We conducted tests on four datasets in total. Among them, CelebA-Test consists of synthetically degraded images, while LFW, CelebChild, WebPhoto-Test, and Wider are real-world image datasets. Table 2 and Figure 5 in main paper quantitatively and qualitatively compare the restoration performance on these four real-world datasets. Additionally, in Appendix Sec. J, we provide a detailed comparison of the restoration effects across the four real-world datasets using Figures 9, 10, 11, and 12.
• The reason for selecting these four datasets for testing is that previous SOTA methods have been validated on these publicly available datasets. They include various low-quality images with different levels of degradation found in real-world scenarios. This allows for a fairer comparison between our method and the SOTAs.