Zero-Order Diffusion Guidance for Inverse Problems

Major Comments/Questions:

• The forward model in 131 is typically defined as $c = f(x_0)+noise$. Is noise considered here?

• Where is the discrete case? I feel that the most practical case for the setting of the paper is when exact gradients can not be computed, which has many applications as the authors mention. However, I don't see the proposed method tried in such scenarios.

• The use of the word "adversary" in the abstract is strange. For black box attacks on Neural Networks, the use of "adversary" makes sense because the goal of the attacker is to generate adversarial examples that fool the classifier without accessing the gradients (or only can perform forward passes to the network). But in this paper, the assumption is that we can only make forward passes to the DM network. The purpose is completely different.

• Citations, support, and/or discussion are needed here "Out of the box, zero-order methods are slow". This is lines 149 to 150.

• $l$ was initially defined to take $c$ and $f(x_0)$ which means that it is $\mathbb{R}^m \times \mathbb{R}^m \rightarrow \mathbb{R}_+$, then a different mapping is defined later in line 154. The authors afterwards used $l$ with different arguments.

• In line 103, the score function is typically defined as the mapping $\mathbb{R}^n \times \{0,...,T\} \rightarrow \mathbb{R}^n$. It takes $t$ in the second argument of the function like in Equation (3).

• Estimating the gradients (or finite difference methods) have a long history and rich literature. What do authors mean when they say "novel"? Do the authors mean they adopt an existing method for estimating gradients?

• Same above point applies to adopting bi-linear down- and up-sampling for dimensionality reduction. The authors can say that we adopted these strategies in this particular setting.

• The authors do conduct ablation and sensitivity studies for ZORO. While the selection of the parameters to ablate for $\delta$ is convincing, the selection of $(\alpha_f,\alpha_c)$ is not convincing as the authors only consider $\alpha_f \in \{0,10\}$ and $\alpha_c \in \{0, 1000\}$.

• DMs are trained on downscaled images. Why is that? If it is because ZORO can not handle larger spaces, then this should be clarified and discussed earlier in the paper. This is a major limitation in ZORO. I am not against the fact that ZORO can only apply to lower dimensions (if it is the case), but i believe this should be stated and discussed clearly.

• While the authors report qualitative results against the oracle and other baselines, run-time and NFEs are not reported in the tables.

• Insufficient baselines: Even with the reduction of the FFHQ dataset and the trained DM in the paper, why the oracle isn't Diffusion Posterior Sampling (as an example of a leading DM-based Inverse Problem Solver)? DPS only requires a simple gradient step at each sampling iteration. I am not saying that ZORO must outperform (or even be on par) all DM-based methods (see 'https://arxiv.org/abs/2410.00083'), but it should be compared to existing leading methods.

• In section 4.1, it was never mentioned how the synthetic dataset was generated. Appendix B does not have it either. Furthermore, the write-up for Section 4.1 never refers to Table 1 which include the results for this experiments.

• Does the "query count" account for diffusion sampling?

Minor Comments/Questions:

• Repeated sentences "we may be working with sensitive information from data obfuscated by a black-box program" in lines 36 (Introduction) and 144 (section 3). Another example is "Our key idea is to approximate classifier-based guidance by using zero-order gradient estimators".

• Is $\mathcal{D}$ the training set or a distribution? $p_0$ is later used for distribution.

• Why cosine similarity is used in Figure 1? Typically, SSIM, PNSR, and LPIPS are used. To compare the distributions of reconstructed and ground truth images, FID might be used (if a sufficient amount of test data points is accessible). Furthermore, can the authors include the degraded images?

• In Figure 1, what is the difference between row 1 and columns 2 to 6? For the introduction section, this is not explained. A brief description is required here.

• A dot after Equation (1).

• No $i$ in the numerator of the equation in line 158. It should be $u_i$.

• Can't the authors use limit to describe the nablas in line 161?

• Isn't $f(x)$ the forward operator? what is $\partial f / \partial x$ in line 163?

• What is $h$ in line 170? There is no $h$ is Equation (6). $h$ was only defined in subsequent discussion.

• It is recommended that the JPEG restoration visualizations (for example Figure 4) to include baselines and LPIPS (and PSNR) scores.

• Equation (9) or Equation 9 in line 240.

• In line 299, what do you mean by "we train by 200M images"?

• In lines 310 to 312, the authors say "We observe that ZORO nearly matches the performance of oracle qualitatively and quantitatively as shown in Sec. 1.". Do you mean as stated in Sec.1? Where in Section 1 the authors show this?

• $\alpha_u$ is missing in Algorithm 1.

• How does the performance of ZERO scale to standard image dimensions (256x256 and above)?
• It is somewhat unclear why the autoencoder underperforms the proposed down/upsampler in dimensionality reduction. The autoencoder should provide better compression as it has been trained on the target dataset. How does the autoencoder perform if we add the correction term, similarly to the proposed technique?
• How have the hyperparameters (different guidance weights) been tuned? Is $\alpha_f$ set to $0$ or $500/1000$ (possible typo)?
• What is the compute cost of inversion? How much time does a sample take?
• Why is there no CFG in the JPEG restoration experiments?
• In Table 5, the sensitivity of gradient estimation to the step size is depicted. However, what would be more interesting to see is how it impacts performance.
• Why are there 'steps' in the cosine similarity plots in Figure 3?
• The effect of the error correction term I believe is important and the corresponding experiment could be moved from the appendix to the main paper.

• Since the proposed method estimates the gradient of $f$ using the finite difference method, one would expect performance to improve as the discretization step $\delta$ decreases. Has this relationship been explicitly discussed in the paper? Additionally, Table 5 suggests that the method may not perform well when $\delta$ is small.

• Table 6 indicates that the algorithm is highly sensitive to the guidance rate, suggesting a dependency on numerous hyperparameters that require fine-tuning.

• Is there any theoretical justification provided for the transition from Equation 7 to Equation 9?

• In Equation 5, replacing “=” with $\approx$ might better reflect the intended approximation.

References

[1] InverseBench: Benchmarking Plug-and-Play Diffusion Models for Scientific Inverse Problems, preprint, 2024.

[2] Tang et al.,, Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint, AISTATS, 2024.

[3] Huang et al., Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion, ICML, 2024.

[4] Dou et al., Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective, ICLR, 2024.

[5] Zheng et al., Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problems, preprint, 2024.

See weaknesses.

1. *Clarification of Novelty of the Proposed Method

   Although coordinate-based approximations of gradients are a well-established method, the authors should highlight the novelty of their approach and its contribution to the existing literature. Specifically, it would be beneficial to elaborate on how their method differentiates from or improves upon traditional coordinate-based gradient approximation techniques.

2. Reproducibility Statement

   The authors should provide a reproducibility statement to enhance the credibility and utility of their work. This statement could include details about the availability of code, datasets, and any specific implementation guidelines that would allow other researchers to replicate the results presented in the paper.

3. Clarification on Variable $c$ Reassignment

   There is confusion regarding the reassignment of the variable $c$ throughout Section 3. Please refer to the weakness section above for context.