We sincerely thank the reviewer for your insightful and constructive feedbacks. We have added more experiments, baselines, and ablation studies.

Weakness 1: Assumption of Lipschitz continuity

The assumption of Lipschitz continuity is a standard prerequisite for analyzing diffusion models (e.g., [A]). Without this assumption, constructing a similar argument becomes significantly more challenging (comparing [A] to [B]). Additionally, this assumption is frequently utilized in the analysis of neural networks (e.g., [C]). Therefore, we have incorporated this assumption to facilitate deeper insights while maintaining a streamlined proof structure.

Weakness 2: The experiment presented in Figure 1 is somewhat confusing.

Thank you for your assistance in clarifying Proposition 3.1 and Figure 1. The outcomes (1) and (2) are direct consequences of Proposition 3.1, and Figure 1 serves merely as an illustration of the first claim. We acknowledge the reviewer's point that verifying the first claim necessitates more experiments, and we have included additional experiments in Figure 1 within the PDF of the global rebuttal. Claim (3) is discussed in Section 3.2, and we will ensure to revise this paragraph in the updated manuscript accordingly.

Weakness 3: Quantitative experimental evidence of two drawbacks.

The qualitative evidence supporting the presence of adversarial gradients and slower convergence rates is presented in Figures 2 and 3, respectively. We concur with the reviewer on the importance of quantitative evidence and we include quantitative experiments in Table 3, 4 in PDF of global rebuttal. To evaluate adversarial gradients, we employ a classifier setting with pre-trained adversarially robust networks and test on 1,000 generated images. When utilizing ResNet-50 as the guidance network, it minimizes its own loss, yet the loss on a Robust ResNet-50 remains high, indicating the existence of adversarial gradients. Our proposed random augmentation method is effective in mitigating these adversarial gradients.

Regarding the slower convergence, the simplicity of the classifier setting may obscure this observation. Therefore, we shifted our focus to the more challenging segmentation map guidance. Here, we observed a noticeable gap in the objective values between the training-free FreeDoM and the training-based PPAP [D] when the number of steps is small, demonstrating the slower convergence. The implementation of the Polyak step size helps to narrow this gap.

Weakness 4: Description of UG

We thank the reviewer for correcting this point and we will modify the description of UG in Appendix.

Weakness 5: Classifier-based tasks and ablation studies

Thank you for your valuable suggestion. Similar to FreeDoM and LGD-MC, our work is concentrated on tasks where training-based methods have yet to be fully developed. Given that both FreeDoM and LGD-MC have demonstrated commendable performance on these benchmarks, we believe that surpassing these established baselines is indicative of our method's strong performance.

We concur with the reviewer on the importance of incorporating training-based methods and conducting ablation studies. To address this, we have included training-based methods and an ablation study of our approach in Table 2 of the PDF in the global rebuttal. Polyak step size expedites convergence and significantly enhances the objective value. While random augmentation does improve the Fidelity (FID), it can adversely affect the objective value due to the introduction of randomness. When these two elements are combined, they yield the best performance.

Weakness 6: Hyperparameter choice

Regarding the step size selection, we initially explored a range from $10^{-5}$ to $10^3$. If a particular step size $\lambda$ yields valid images, we then refine our search within the interval [$\lambda$, $10\lambda$] using 20 evenly spaced values. In the case of Monte Carlo samples, the LGD-MC study noted negligible differences between using $n=10$ and $n=100$, and thus we adopt $n=10$. We sourced the implementations of UG, FreeDoM, and MPGD directly from their respective GitHub repositories.

Weakness 7: Limitations of random augmentation

Regarding the first limitation, there exist data augmentation methods within the domains of molecular [A] and offline reinforcement learning [B]. For the second limitation, we concur with the reviewer that the adoption of a foundation model may result in increased computation time. However, it is important to note that random augmentation processes are highly parallelizable, and the guidance models utilized for current training-free guidance are not so large. We have provided detailed timings for the CLIP model in Table 4 of the original paper.

Question 1: Rationale behind Polyak

Training-free guidance is indeed challenged by a slow convergence issue, as outlined in Proposition 3.3. The simplest remedy for this is to adjust the step size. The Polyak step size has been demonstrated to be optimal under a variety of conditions [13], which is why we have chosen to implement it in our approach.

Question 2: Clarification on MDM

Unlike image generation, motion diffusion involves creating $192$ frames of motion where consistency between neighboring frames is crucial to produce a viable trajectory. This requirement significantly complicates the optimization challenge, making the selection of an appropriate step size even more critical. Due to this complexity, MPGD struggles to generate a feasible motion, as detailed in Appendix B.2.

Question 3: the value of $\eta$

The value of $\eta$ is set the same as the released code of these baselines.

[A] AugLiChem: data augmentation library of chemical structures for machine learning. MLST 2022.

[B] Reinforcement learning with augmented data. NeurIPS 2020.

[C] On the spectral bias of neural networks. ICML 2019.

[D] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.

We greatly appreciate your insightful suggestions and comments, which have significantly contributed to the refinement of our paper. We are also thankful for your acknowledgment of our rebuttal efforts.

Comment 1: Regarding Table 3, it is designed to evaluate the adversarial gradient issue in training-free guidance. To assess the presence of adversarial gradients, we utilized an adversarially robust ResNet-50 [33], which is trained to resist adversarial gradients. Since ResNet-50 serves as our guidance network, a large loss gap between the standard ResNet-50 and the robust variant indicates a severe adversarial gradient issue. In Table 3, the values represent the loss value. The columns denote the guidance network type, while the rows indicate the loss tested on either ResNet-50 or Robust ResNet-50. The values reported are the average over 1000 images, providing a quantitative evaluation. When using ResNet-50 for guidance, its loss on ResNet-50 is as low as that of real images from the same class (value 5.91). However, the loss for Robust ResNet-50 is significantly higher (value 6.17), suggesting susceptibility to adversarial gradients. By employing ResNet-50 with random augmentation for guidance, we observe a marked reduction in this gap (from 5.91 to 5.98), underscoring the effectiveness of our proposed method.

Table 4 aims to compare the convergence of training-free guidance with training-based guidance, specifically PPAP [D]. Given that classifier guidance presents a simple scenario and all methods converge relatively fast, it tends to obscure the observations. Consequently, we have directed our attention towards the more complex task of segmentation map guidance in Table 1. The columns in Table 4 list different methods, while the rows indicate the sampling steps. The values represent the "distance" mentioned in Table 1 and are averaged over 1000 images under the same conditions as those in Table 1. Our observations reveal a significant discrepancy in the objective values between the training-free FreeDoM and the training-based PPAP [D], particularly at a lower number of steps (20 or 50 steps), which indicates slower convergence. However, when we incorporate the Polyak step size into FreeDoM, this convergence issue is substantially mitigated.

Comment 2: As for the second concern, we are thankful for your acknowledgment of our efforts in the rebuttal. We have also conducted comparisons with training-based methods, as illustrated by PPAP in Table 1 of the PDF provided in the global rebuttal. PPAP extends classifier guidance to accommodate a broader range of conditions by fine-tuning the pretrained model on clean images to create different experts, each responsible for a specific range of noise level. The gradients from the corresponding experts are then used for guidance at corresponding noise levels. To optimize performance of PPAP, we have set the number of experts to $10$, which is the maximum reported in [D]. As indicated in Table 1, our proposed method shows performance on par with the training-based PPAP. With respect to the reviewer's request for additional experiments under class conditions, we are currently conducting these experiments. However, due to the high number of required samples and the author-reviewer discussion coming to an end, it is possible that they cannot be completed before the reviewer-author discussion period closes. We assure you that the results of these experiments will be included in the revised manuscript.

Comment 3: Regarding the third question, the effectiveness of random augmentation is supported by Proposition 4.1. This proposition does not rely on the assumption of Lipschitz continuity or any specific distribution for the augmentation, suggesting its broad applicability across various modalities. Consequently, the theoretical foundation guarantees the effectiveness of random augmentation in AI4Science or RL without imposing any strong assumptions. As we approach the end of the author-reviewer discussion and considering AI4Science or RL represents a novel application setting for our paper, it is regrettably unlikely that additional experiments can be completed prior to the submission deadline. Nonetheless, we are committed to including these experiments in the revised manuscript and we appreciate the reviewer's understanding in this matter.

We sincerely thank the reviewer for your insightful and constructive feedbacks. Based on the comments, we have added more baselines.

Weakness 1 and 2: Comparison with [A,B,C,D]

We appreciate the reviewers for highlighting these pertinent references. We have now included [B] (training-based PPAP) and [D] (SAG) as additional baselines, with the results presented in Table 1 of the PDF for global rebuttal. Regarding hyperparameters, we have opted for a solve step of $n=4$ and selected the number of experts in PPAP to be $10$, consistent with the original papers.

[A] primarily focuses on employing a classifier as the guidance network. For instance, the adoption of a joint direction, a softplus classifier, and the adjustment of classifiers' temperature are approaches that are difficult to be applied to general loss functions. Since [D] builds upon the general methodology outlined in [C], we have chosen to include only [D] in our analysis. Our findings indicate that SAG outperforms FreeDoM, particularly in terms of the objective value, aligning with the results reported in the SAG paper. However, it is surpassed by our proposed method. Moreover, the performance of our method is comparable to that of training-based approaches.

We would also like to clarify the differences between our work and [A]. Firstly, regarding the smooth classifier, the methodology employed in [A] involves using a softplus classifier and adjusting the classifier's temperature, which may not be readily adaptable for general conditions. In contrast, the random augmentation method we propose is independent of the loss function and guidance network. From a motivational standpoint, our paper introduces an additional significant motivation: to eliminate the adversarial gradient, an aspect that has been overlooked by existing studies. Secondly, concerning the scheduler, our paper proposes a step size choice that is orthogonal to the scheduler design.

Weakness 3: Class condition as benchmark

Similar to FreeDoM and LGD-MC, our work concentrates on tasks for which training-based methods are not yet fully developed. We have utilized the same benchmarks as those employed in FreeDoM and LGD-MC. Given that both FreeDoM and LGD-MC have demonstrated commendable performance on these benchmarks, we believe that surpassing these established baselines is indicative of our method's robust performance. We are open to different viewpoints and welcome the opportunity for further discussion.

Weakness 4: Polyak Step size

We are grateful for the opportunity to clarify the novelty of our approach. Due to space constraints in the paper, we were unable to delve into the specifics of the Polyak step size, which may have led to some misunderstanding. The Polyak step size is a method of choosing step sizes, distinct from normalization, and it unifies various step size choices with a proven optimal convergence rate under a range of conditions [13]. In contrast, normalization methods like those used in other works (e.g., DSG) typically focus on projection to the manifold. It's important to note that the Polyak step size and manifold projections are orthogonal concepts [E], and their combination in our method is more than a mere amalgamation of the two methods' codes. Furthermore, we have compared our method against the state-of-the-art DSG [F], as shown in the global response PDF, and our proposed method demonstrates superior performance.

Typo

Thank you for your careful reading. We will carefully correct the typos in the revised manuscripts.

[E] Iteration-complexity of the subgradient method on Riemannian manifolds with lower bounded curvature. Optimization 2019.

[F] Guidance with spherical gaussian constraint for conditional diffusion. arXiv:2402.03201.

We sincerely thank the reviewer for your insightful comments and recognition of this work, especially acknowledging our theoretical contributions.

Misaligned gradient.

Thank you for suggesting a better name. We agree that the misaligned better captures the behavior and will modify it in the later versions of the manuscript.

I'm not sure ... which is pretty intuitive.

We agree that the statement pointed out by the reviewer is more appropriate and will change the manuscript accordingly.

Proposition 3.2 and 4.1 and mentioned reference.

We thank the reviewer for pointing out this important reference. Proposition 3.2 is a special case of Proposition 4.1 and Proposition 4.1 follows the proofs in Appendix E of the mentioned reference. We will discuss this reference in the revised manuscript.

The role of time-travel section in the appendix.

Time travel is a widely adopted trick for training-free guidance, but there is no theoretical justification. In the baselines, UG, FreeDoM, and MPGD all adopted it. As a result, we include the theoretical analysis in the appendix to support when it works.

$g(x_t,t)$ is L-Lipschitz w.r.t. which parameter?

We assume that $g(x_t,t)$ is L-Lipschitz w.r.t. its first parameter.

How the discretization error relates to the rate of convergence

We apologize if our initial description of the discretization error was not sufficiently clear, potentially leading to confusion. To clarify, the discretization error mentioned in Proposition 3.3 refers to the discrepancy between the solution provided by DDIM and the optimal solution. In the context of diffusion, the maximum step size, denoted as $h_{\max}$, is inversely proportional to the number of steps taken. Consequently, the convergence rate can be expressed as $O((L+1)/T)$.

Is it really the case that the analysis presented "completes the picture" for improving training-free guidance?

We apologize for any confusion caused by our initial explanation. In our paper, we provide theoretical justifications for the techniques used in existing training-free guidance literature, such as step size adjustments and 'time-travel.' Additionally, we have identified and analyzed key limitations. Our work aims to offer a more comprehensive understanding of the current training-free guidance approaches. We acknowledge, however, that the strategies for addressing misaligned gradients and enhancing convergence is not optimal require further development. We will ensure to clarify these points in the revised manuscript.

Presentation

We sincerely thank the recommended changes. We will change the manuscript accordingly.

Nitpickings

We sincerely thank the reviewer for the careful reading of the paper. We will correct these typos accordingly and have a careful check in the revised manuscript.

We sincerely thank the reviewer for your insightful and constructive feedbacks. We have added the baselines and integrated the proposed methods into MPGD and LGD-MC.

Question 1: MPGD and LGD-MC with random augmentation and Polyak step size

We are grateful for your suggestions to enhance our experimental section. As requested, we have included additional experiments in Table 1 in PDF file of the global rebuttal. In these experiments, when integrating LGD-MC, we set the number of random augmentations to 5 and the Monte Carlo iterations to 2. The results show that both LGD-MC and MPGD significantly outperform their original counterparts when combined with random augmentation and Polyak step size adjustments. This indicates that our proposed methods can complement existing techniques. Through further analysis, we determined that the Polyak step size is the primary factor contributing to these improvements. Notably, MPGD-Z, when used in conjunction with our approach, delivers exceptional performance, even surpassing the training-based PPAP [B].

Weakness 1 and Question 2: Comparison with DPS, DSG, and RED-diff

Thank you for highlighting the necessity of including additional baselines. We have conducted further experiments with the DSG algorithm and presented the results in Table 1. It is important to clarify that the DPS algorithm is essentially the FreeDoM method, minus the time-travel techniques. Since DPS is analogous to FreeDoM without the time-travel aspect, we believe that the performance of FreeDoM can be indicative of DPS's performance. As for RED-diff, it employs diffusion as a form of regularization for the least squares problem. It is more suitable for linear inverse problems rather than conditional generation tasks. In our testing, RED-diff failed to generate valid images under conditions such as segmentation, sketch, or CLIP guidance. Consequently, we have decided not to include RED-diff in our baseline comparisons. We have incorporated the DSG results into Table 1 in the PDF file of the global rebuttal. While DSG shows an improvement over FreeDoM, it still falls short when compared to our proposed methods.

Weakness 2: Proposed solution is not novel compared with LGD-MC and DSG.

We appreciate your assistance in elucidating the distinct contributions of our work. The approaches of LGD-MC and DSG differ in their motivations and are orthogonal in their methodologies. In LGD-MC, the MC method is utilized to approximate the posterior distribution, which necessitates that the variance of the Gaussian noise be proportional to $\sigma_t/\sqrt{1 + \sigma_t^2}$. On the other hand, the rationale behind employing random augmentation is to mitigate adversarial gradients, allowing for the consistent application of the same augmentation across different steps. It is worth noting that random augmentation and MC can be employed concurrently, although this may result in an increased sampling time. The essence of DSG lies in the projection onto the manifold, a process that is orthogonal to the choice of step size. Moreover, various manifold projections can be seamlessly integrated with the Polyak step size, as detailed in [A].

[A] Iteration-complexity of the subgradient method on Riemannian manifolds with lower bounded curvature. Optimization 2019.

[B] Towards Practical Plug-and-Play Diffusion Models. CVPR 2023.