Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models

Thanks for your recognition of this paper and the valuable comments and suggestions. Our responses to the main concerns are given as follows.

(The metrics shown do not include image perceptual quality metrics, like FID or KID. & Evaluation datasets are limited in scope. Can the method operate on general images (such as ImageNet)?)

Thanks for the comments. We perform the experiments for linear motion deblurring on the ImageNet dataset, with reporting the FID metric. Due to the time constraint of the rebuttal period, we follow DiffPIR (https://arxiv.org/abs/2305.08995) to calculate the FID on 100 validation images. The following results show that our DMILO method performs well in terms of both realism and reconstruction metrics, demonstrating its ability to effectively balance perceptual quality and distortion. We will present relevant results for more tasks and more datasets in the revised version.

Table D1: Experimental results for the linear motion deblurring task on 100 validation images from ImageNet.

(How does the method compare to alternative approaches in terms of NFEs? As a method that utilizes multiple derivatives of each diffusion step may be computationally prohibitive.)
Our methods build on DMPlug and reduce its computational overhead. For instance, in the inpainting task, our methods outperform DMPlug while using significantly fewer NFEs. Specifically, our methods need only 3,000 NFEs in total, compared to the 15,000 NFEs required by DMPlug. Even when the number of NFEs is the same, our methods are computationally more efficient. This is because our methods employ a smaller gradient graph, which lessens the burden of gradient computation. In Table D2 below, we present the computational cost of reconstructing a validation image from the CelebA dataset for different methods for inpainting using an NVIDIA RTX 4090 GPU. The results demonstrate that our methods require less computational time than DMPlug.

Table D2: Computation cost for different approaches.

(The authors do not mention the Perception-Distortion Tradeoff [1], which is key is analyzing inverse problem solutions using both distortion and perception metrics.)
Thank you for pointing this out. In the revised version, we will cite the mentioned paper [1] for the Perception-Distortion Tradeoff, and comprehensively explain the performance of our methods in terms of both distortion and perceptual quality.

(Could the authors explain why it is insufficient to only optimize through the last timestep of the diffusion process?)
Thank you for the insightful question. We conduct an ablation study on only optimizing through the last timestep of the diffusion process and find that it also works, although with slightly degraded reconstruction performance (see Table D3 below). We conjecture that the degraded reconstruction performance is primarily attributed to improper initialization. Specifically, when optimizing only through the last timestep of the diffusion process, in principle, the procedure should be initialized with a vector that lies within the range of the composition of functions corresponding to all timesteps except the last one. However, identifying such an initial vector seems to be a challenging task. We plan to delve deeper into this aspect in future research.

We believe this insufficiency arises because a single sampling step may introduce errors in detail, which are difficult to correct with only sparse deviations and accumulate during optimization. We will further explore this in the future study.

Table D3: Experimental results for the inpainting task on 100 validation images from CelebA.

("LTS" denotes methods that optimize only through the last timestep of the diffusion process.)

Thanks for your useful comments and questions. Our responses to the main concerns are given as follows.

(The analysis does not explicitly address whether the bound generalizes to N > 2, leaving open questions about scalability.)
We follow the work for ILO (Daras et al., 2021) to set $N = 2$. We believe that this is sufficient for an intuitive theoretical analysis of the effectiveness of our methods (e.g., Reviewer ve9x found that our claims are “well-supported by extensive empirical experiments and intuitive theoretical analysis”, Reviewer D1CM found that our theoretical analysis “justifies the effectiveness of the proposed methods”, and Reviewer oTx5 found “the theoretical analysis sufficient”).

(The theorem uses ℓ₁-regularization for ν, but the paper’s experiments employ Adam optimization without explicit guarantees of sparse recovery. This creates a gap between theory (exact ℓ₁ minimization) and practice (approximate optimization).)
Firstly, it should be noted that even the globally optimal solutions to the $\ell_1$ minimization problem might not exhibit sparsity. Secondly, given that the objective function of the $\ell_1$ minimization problem is highly non-convex and obtaining its globally optimal solutions is not feasible, in practical applications, we employ the Adam optimizer to approximately solve the $\ell_1$ minimization problem.

(It is recommended that the authors expand their scope to include inverse problems in other modalities, such as linear sparse view CT reconstruction and nonlinear metal artifact reduction in CT imaging. The selected inverse problems (super-resolution, inpainting, nonlinear deblurring, and blind image deblurring) are limited in scope and lack diversity in modalities (e.g., medical imaging, audio, or 3D data).)
We agree that tasks such as sparse-view CT reconstruction and metal artifact reduction in CT imaging are of practical importance. However, prior works closely related to our study, such as DDRM, DPS, $\Pi$GDM, DiffPIR, and DMPlug, have concentrated their experiments on natural images for tasks including super-resolution, inpainting, nonlinear deblurring, and blind image deblurring. As is evident from the fact that these recent papers have been published in the topmost venues in ML or CV, and/or have been highly cited, this line of investigation has been widely accepted in the community and is a highly active area of research. We believe that extending the experiments to tasks like linear sparse-view CT reconstruction and multi-modal data such as medical imaging, audio, or 3D data is beyond the scope of the current work.

(The abstract is overly lengthy and contains redundant information. It should be more concise and focused on the key contributions and findings. Additionally, the inclusion of citations within the abstract is unconventional and should be removed.)
Thank you for the comment. In the revised version, we will shorten the abstract to enhance its conciseness, highlight key contributions, and remove the citation.

(The experimental setup is limited and does not comprehensively address the claimed issues of heavy computational demands and suboptimal convergence in DM-based methods.)
Our methods build on DMPlug and reduce its computational overhead. Due to the character limit, please refer to Table D2 in our responses to Reviewer oTx5 for an illustration. As PGD is known for its capacity to alleviate the problem of suboptimal convergence (Shah & Hegde, 2018), the effectiveness of addressing suboptimal convergence in DM-based methods can be observed from the experimental results presented in Table 2 in the main document. Specifically, for super-resolution and inpainting tasks, DMILO-PGD yields the best reconstructions, thereby demonstrating this advantage.

(The proposed methods, DMILO and DMILO-PGD, exhibit noticeable performance fluctuations across different tasks. The authors should provide a detailed analysis of why these fluctuations occur and whether they are related to specific properties of the tasks.)
We found that for linear deblurring and BID tasks, DMILO achieves the best reconstruction performance, whereas DMILO-PGD gives comparatively inferior results. As mentioned in Section 5.3, performance fluctuations may arise from the naive gradient update, which may not be well-suited for all tasks. Nevertheless, across most tasks, both DMILO and DMILO-PGD show competitive performance. We leave the detailed analysis of why these fluctuations occur and whether they are related to specific properties of the tasks to future work.

(The paper does not adequately discuss the limitations of the proposed methods. For example: Can the methods handle highly ill-posed problems or tasks with significant noise?)
We primarily follow the experimental settings of DMPlug, which does not deal with highly ill-posed problems or tasks with significant noise. We leave the research on these aspects to future work.

Thanks for your positive assessment of this paper and the valuable comments and suggestions. Our responses to the main concerns are given as follows.

(Table 2 should be completed with linear deblurring (Gaussian or Motion), and baselines such as DPIR and/or DiffPIR. Would the authors be able to add comparisons to DPIR/DiffPIR?)
We conduct additional evaluations on linear Gaussian and motion deblurring tasks, comparing our methods with DPIR and DiffPIR on CelebA and FFHQ. The results are shown in Tables B1-B4, which reveal that while DiffPIR demonstrates outstanding performance in terms of the perceptual quality metric LPIPS and DPIR shows superiority in image distortion metrics such as PSNR and SSIM for linear Gaussian deblurring, our DMILO method generally attains the best performance across nearly all metrics, especially for linear motion deblurring. Note that we follow DMPlug to set the kernel size to $64 \times 64$, which differs from the $61 \times 61$ kernel size initially employed in DPIR and DiffPIR. Such a difference might impact the results.

In the revised version, we will further incorporate comparisons with DPIR and DiffPIR for other applicable tasks.

We sincerely hope that these additional comparisons have appropriately addressed your comments, and that you can kindly consider increasing your initial rating accordingly.

Table B1: Comparisons of different methods for linear Gaussian deblurring on 100 validation images from CelebA.

Table B2: Comparisons of different methods for the linear Gaussian deblurring task on 100 validation images from FFHQ.

Table B3: Comparisons of different methods for the linear motion deblurring task on 100 validation images from CelebA.

Table B4: Comparisons of different methods for the linear motion deblurring task on 100 validation images from FFHQ.

(I disagree with the authors in Paragraph 3.2 when they state: "A key difference from conventional PGD is that…")
Thank you for the detailed comment. We will remove the phrase “a key difference” and reword the sentence to enhance its precision.

(Would the authors be able to precisely state the functional Algorithm 2 minimizes (which is not precisely (12))?)
Thank you for the question. In (12), we use $\hat{\mathbf{x}} _ {t _ 0}$ to represent the estimated signal. As Algorithm 1 does not provide an explicit form for $\hat{\mathbf{x}} _ {t _ 0}$, we substitute the inaccessible $\mathcal{A}(\hat{\mathbf{x}} _ {t _ 0})$ with the observed vector $\mathbf{y}$. In Algorithm 2, we first calculate $\mathbf{x} _ {t _ 0}^{(e)}$ via simple gradient descent. This $\mathbf{x} _ {t _ 0}^{(e)}$ serves as an explicit approximation of $\hat{\mathbf{x}} _ {t _ 0}$. We then fix $\mathbf{x} _ {t _ 0}^{(e)}$ and use $\mathcal{A}(\mathbf{x} _ {t _ 0}^{(e)})$ in place of $\mathcal{A}(\hat{\mathbf{x}} _ {t _ 0})$ in (12).

Thanks for your recognition of this paper and the valuable feedback and suggestions. Our responses to the main concerns are given as follows.

(My main concern is the computational cost of the proposed method. If the total number of function evaluations (NFEs) is fixed, does the proposed method still achieve better performance? Could the authors provide additional insights into the computational efficiency of their approach?)
Our methods build on DMPlug and reduce its computational overhead. For instance, in the inpainting task, our methods outperform DMPlug while using significantly fewer NFEs. Specifically, our methods need only 3,000 NFEs in total, compared to the 15,000 NFEs required by DMPlug. Even when the number of NFEs is the same, our methods are computationally more efficient. This is because our methods employ a smaller gradient graph, which lessens the burden of gradient computation. In Table A1 below, we present the computational cost of reconstructing a validation image from the CelebA dataset for different methods for inpainting using an NVIDIA RTX 4090 GPU. The results demonstrate that our methods require less computational time than DMPlug.

Table A1: Computation cost for different approaches.

(Did the author compare in terms of memory and computation cost by applying checkpointing to DMPlug?)
Following your suggestion, we compare in terms of memory and computation cost by applying gradient checkpointing to DMPlug (see Table A2 below). The gradient checkpointing strategy effectively reduces the memory burden significantly. However, it simultaneously increases the computation cost because of the overhead associated with saving and loading gradients. In contrast, our methods manage to decrease both the memory and computation costs compared to DMPlug.

Table A2: Memory cost for different approaches.

("DMPlug-Ckpt" denotes DMPlug with gradient checkpointing)

(There is no ablation study on the effect of sparse deviations., which is a major component in proposed method. Could the authors provide insights into how adding sparse deviations influences performance?)
Thank you for your insightful comment. We perform an ablation study on the impact of sparse deviations in the super-resolution task on the CelebA dataset (see Table A3 below). The results demonstrate the effectiveness of adding sparse deviations. In our understanding, adding sparse deviations not only broadens the range of diffusion models but also alleviates error accumulation resulting from inaccurate intermediate calculations, thus leading to improved reconstruction performance.

Table A3: Ablation study on the effect of sparse deviations for the super-resolution task on 100 validation images from CelebA.

("w/" denotes methods employing sparse deviations, and "w/o" denotes methods without adding sparse deviations.)

(Figure 1 appears inconsistent with its caption.)
Thank you for pointing this out. In the revised version, we will correct this to make Figure 1 consistent with its caption.