Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generation

Q7: What are the specific advantages of guiding $\epsilon_k$ compared to directly guiding $x_k$?

A7: Guiding on the noise space allows us to project the guided $\epsilon_k$ back to its original scale. This serves as a simple regularization technique to prevent the guided $\epsilon_k$ from deviating too far from its initial value. Moreover, it is convenient as it does not require additional hyper-parameters.

Additionally, since the diffusion model is trained with $\epsilon_k \sim \mathcal{N}(0, I)$, where the norm follows a Chi distribution, $||\epsilon_k|| \sim \mathcal{X}_d$, projecting the guided noise back to its original scale ensures that the guided noise also adheres to the same Chi distribution, $|| {\epsilon’}_k || \sim \mathcal{X}_d$. This may help prevent the guided $x_k$ from falling outside the support of the pre-trained model, which could otherwise lead to degeneration.

Q8: The contribution summary in the introduction is not in a clear summary form. It is recommended to reorganize the detailed contributions into a separate paragraph and condense the summary into two or three concise lines.

A8: Thanks for the suggestion, we have revised the contribution summary accordingly.

Q9: Is the value of used in line 13 fixed as a value pre-calculated in line 3? Or is it a value that continuously changes as is updated by the loop in line 4?

A9: The value of $|| \epsilon_k ||$ used in line 13 is fixed to the pre-calculated value from line 3.

Q10: Why was the experiment conducted on instead of (predicted clean image) in Figure 1 (ablation)? Given that represents a clean image, comparing it to the predicted clean image at the same noise level appears to be a more intuitive approach for assessing the difference.

A10: Thanks for the suggestion. We further evaluate the suggested $\hat{x}_k$ case. The results are updated to Figure 1 (page 5). From the experiment, we observe that it can produce target images of comparable quality. This insight inspires an alternative guidance direction within our Fast Direct framework, which has the potential to further enhance performance. We leave this as one of our future work.

We sincerely appreciate the reviewer's constructive advice and valuable comments. Our detailed responses to the reviewer's questions are presented as follows:

Q1: The number of experimental tasks is insufficient, and the number of prompts used in the experiment is limited. A direct comparison with reinforcement learning methodologies for diffusion models, such as those in [1], would benefit from evaluating common tasks like compressibility, incompressibility, and aesthetic scores. Additionally, the manual selection of a small number of prompts by the author raises concerns about potential bias toward the proposed methodology.

A1: Thanks for the suggestion. We further evaluate compressibility, incompressibility, and aesthetic quality tasks. The experimental results are now included in Appendix A (Page 14). We have used the 45 common animal prompts proposed by [1] to avoid potential experimental bias.

From the experiments, we observe that Fast Direct achieves better scores than the baselines. Moreover, we observe that Fast Direct with the GP model has a generalization ability for unseen prompts. In contrast, DNO is an instance-level method, which cannot generalize to generate images using unseen prompts without re-running the query (optimization) process.

[1] Black, Kevin, et al. "Training diffusion models with reinforcement learning." ICLR(2024).

Q2: The methodology's optimization of noise during sampling is incompatible with the ODE sampler, a popular choice due to its deterministic nature and high performance [2]. This limitation could hinder the method's practical applicability.

A2: Thank you for raising this point. Fast Direct indeed relies on a stochastic diffusion process for effective stochastic exploration. This requirement is shared by other approaches such as DDPO, DPOK, and D3PO, which also depend on stochastic diffusion for exploration. However, unlike these methods, which require closed-form calculations of the log probability for each diffusion step and therefore being limited to specific samplers like DDIM; Fast Direct treats the SDE sampler as a black-box. This flexibility allows Fast Direct to directly support any SDE sampler.

Q3: In the text-image alignment experiments, the proposed method utilized the EDM sampler, while the baseline employed the DDIM sampler. This experimental setup may favor the proposed method, potentially compromising the fairness of the comparison.

A3: We further evaluate Fast Direct with the DDIM sampler in image experiments. The experimental results are updated in Fig. 2 and Fig. 4 of the main paper and Appendix A. This would ensure a fair comparison. From the experiment, we observed that Fast Direct achieves similar performance with the DDIM sampler as it does with the EDM sampler.

Q4: How was the point in time considered complete for guidance iterations determined? Explicitly stating this criterion within the text would provide greater clarity.

A4: Thank you for the question. In our experiment, the guidance iterations stop once the pre-set batch query budget (N) is exhausted. In practice, as it is an online learning algorithm, the user may choose to terminate the guidance integration upon achieving satisfactory results.

Q5: Are there any comparison results with the inference time guidance method? Although online black box inference time guidance methods are limited, incorporating a comparison with existing methods, even if the performance of the presented methodology is lower, would offer valuable insights.

A5: Thanks for the suggestion. We further include DNO, an inference-time guidance method, as our new baseline method. The experiment results are updated in Fig. 2, Fig. 4 of the main paper, and Appendix A.

It is important to note that DNO runs at the instance level, meaning each run produces only one image, and its performance is highly dependent on the initial prior. To ensure a more accurate evaluation, we run 16 independent experiments to generate 16 images and reported the average results. This evaluation required 16$\times$ more batch queries compared with our Fast Direct and all other baselines.

From the experiments, we observe that Fast Direct achieves better objective values than DNO. Furthermore, the learned GP in Fast Direct generalizes well to unseen prompts, whereas DNO being an instance-level method, cannot generalize to generate images using unseen prompts without re-running the query (optimization) process.

Q6: What is the number of diffusion sampling steps used in each experiment?

A6: We use K=8 diffusion sampling steps for all image experiments (stated in Section 4.1: Pre-trained Model), and K=200 for molecules experiments (stated in Section 4.2: Pre-trained Model).

Thanks for the reviewer's feedback. We hope the following detailed clarification addresses your question.

Definition. We refer instance-level method as an approach that only optimizes a single data instance in a complete execution cycle. Although the process can be parallelized, each instance is optimized independently, without sharing information across instances. In contrast, a batch-level method optimizes multiple data instances simultaneously in a single execution cycle, often sharing the learned information to reduce redundant efforts across instances.

Here we briefly describe the DNO for black-box objective. Let $\mathcal{E} = [\epsilon_0^\top, \cdots, \epsilon_K^\top]$ denote the noise vector instance, where $\epsilon_0$ represents the diffusion prior, and $\lbrace \epsilon_1, \cdots, \epsilon_K \rbrace$ represent the intermediate diffusion noise. The generated image $x$ is uniquely determined by the diffusion model as $x = M_\theta(\mathcal{E})$. The noise vector can be optimized w.r.t. the objective $f(x)$ by gradient descent: $\mathcal{E} \gets `optimizer-step`(\mathcal{E}, \nabla_{\mathcal{E}} f(x))$. Since $f$ is black-box, DNO uses ZO-SGD to approximate the gradient. Specifically, the gradient w.r.t. $x$ is estimated as:

where each $x_i$ is the image generated by perturbed noise vectors:

where $\xi_i \sim \mathcal{N}(0, I)$ is sampled from a Gaussian distribution, and $\mu > 0$ is a small coefficient to ensure accurate estimation. The gradient w.r.t. the noise vector is estimated by $\nabla_{\mathcal{E}} f(x) \approx \hat{H}(x) \cdot \nabla_{\mathcal{E}} M_\theta(\mathcal{E})$ to optimize the noise vector. As a result, each gradient estimation consumes $q$ objective evaluations.

After $T$ optimization iterations, DNO consumes $T \times q$ objective evaluations and yields an optimized noise vector $\mathcal{E}'$, which generates one optimized image $x' = M_\theta(\mathcal{E}')$. Since DNO produces one image per execution, it is categorized as an instance-level method.

In contrast, Fast Direct optimizes a batch of noise vectors in parallel. In each optimization iteration, a batch of $B$ noise vectors $\lbrace \mathcal{E}_1, \cdots, \mathcal{E}_B \rbrace$ is sampled to generate a corresponding batch of images $\lbrace x_1, \cdots, x_B \rbrace$. These images are then evaluated for their objective values, $\lbrace y_1, \cdots, y_B \rbrace$, where $y_i = f(x_i)$. Then, the pseudo-target model is updated with the batch of labeled data $\lbrace (x_1, y_1), \cdots, (x_B, y_B) \rbrace$ to further guide the diffusion process, which shares the information across all the instances. Notice this step consumes $B$ objective evaluations.

After $T$ iterations, Fast Direct consumes $T \times B$ objective evaluations and generates $B$ images. This makes Fast Direct a batch-level method, as it can generate $B$ images simultaneously. Moreover, at the end of the execution, Fast Direct not only produces images but also produces a pseudo-target model (e.g., GP model) that learns from $T \times B$ data points. This pseudo-target model can be used in Fast Direct to generate new target data (even for new prompts) without requiring additional objective evaluations. Specifically, this can be implemented by running Algorithm 2 without Line 17 (query for black-box objective) and Line 18. We emphasize that the produced pseudo-target model of Fast Direct is now used in Line 11 without further updating.

Summary. DNO produces one image per run, so it is an instance-level method. While the process can be parallelized, each image is optimized independently, with no information sharing across instances. To generate $B$ images, DNO requires $T \times B \times q$ objective evaluations.

In contrast, Fast Direct can generate a batch of images in parallel with information shared across all the instances, so it is a batch-level method. To generate $B$ images, it only consumes $T \times B$ objective evaluations. Moreover, at the end of the execution, Fast Direct produces a pseudo-target model that learns from $T \times B$ data points. This pseudo-target model can be used in Fast Direct to generate new target data (even for new prompts) without requiring additional objective evaluations.

We sincerely appreciate the reviewer's constructive advice and valuable comments. Detailed responses to the reviewer's questions are given below.

Q1: Why were DDPO, DPOK, and D3PO chosen as baselines? Are they the most relevant and competitive in black-box target generation? If there are other methods that could be appropriate, a brief comparison or justification for their exclusion could enhance the robustness of the evaluation.

A1: DDPO and DPOK are two representative RL-based methods for fine-tuning diffusion models. Both approaches support online learning and black-box objectives. D3PO focuses on relative human rewards and also supports online learning and black-box objectives. We selected these methods as baselines because they have been shown to effectively handle black-box objectives in an online manner.

Most recently, DNO[1] introduced a method to guide the diffusion process using ZO-SGD, enabling support for black-box objectives. We further include DNO as a new baseline in our experiments. This would enhance the robustness of the evaluation. The new experiments are included in the Fig. 2 and Fig. 4 of the main paper, and Appendix A.

It is important to note that DNO runs at the instance level, meaning each run produces only one image, and its performance is highly dependent on the initial prior. To ensure a more accurate evaluation, we run 16 independent experiments to generate 16 images and reported the average results. This evaluation required 16$\times$ more batch queries compared with our Fast Direct and all other baselines.

From the experiments, we observe that Fast Direct achieves significantly better objective values than DNO. Furthermore, the learned GP in Fast Direct generalizes well to unseen prompts, whereas DNO being an instance-level method, cannot generalize to generate images using unseen prompts without re-running the query (optimization) process.

[1]: Tang, Zhiwei, et al. "Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization." arXiv preprint arXiv:2405.18881 (2024).

Q2: Have you considered alternative pseudo-target selection strategies? While the paper presents GP and historical-optimal updates, discussing the potential effects of alternative methods (e.g., reinforcement learning-based or clustering-based approaches) might highlight the generalization of Fast Direct.

A2: Thank you for the suggestion. It is interesting to consider alternative pseudo-target selection strategies under our Fast Direct framework, which may further improve the query efficiency and we believe this would be a promising direction. We leave this as one of our future work.

Q3: Could you provide more insights into the parameter tuning process for Fast Direct? As query efficiency is a primary focus, understanding the sensitivity of Fast Direct to parameters like batch size and step size would clarify the required fine-tuning efforts.

A3: Thank you for your question. We have conducted an ablation study about the hyper-parameters in Appendix B (page 18), which shows that Fast Direct is not sensitive to the batch size and step size settings. The batch size can be adjusted based on available GPU memory and the query budget.

Q4: Could you expand on the potential for scalability? Especially in high-dimensional tasks like high-resolution image generation, it would be helpful to know if there are constraints or limitations for Fast Direct in terms of computational efficiency.

A4: Thank you for your question. We have conducted an ablation study on computational time in Appendix B (page 18). The results show that Fast Direct requires approximately 6.4 hours per experiment with a budget of 50 batch queries to generate 32 images simultaneously. Experiments show that Fast Direct scales linearly with batch size for small historical query dataset (e.g., 3.2k data points). However, because Fast Direct relies on a GP model, its computational time scales cubically. This limitation can potentially be addressed using kernel approximation techniques [2,3]. We leave this as one of our future work.

[2] Rahimi et al. Random Features for Large-Scale Kernel Machines. NeurIPS (NIPS) 2007.

[3] Yu et al. Orthogonal random features. NeurIPS 2016.

Dear Reviewer hGsH

We truly appreciate the effort and time you have devoted to providing constructive reviews, as well as your positive evaluation of our submission.

We have now provided more clarifications, explanations, and experiments to address your concerns and followed your advice to improve our paper.

Would you mind checking our responses and confirming if they have addressed your concerns? We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.

We sincerely appreciate the reviewer's constructive advice and valuable comments. Your support is greatly appreciated. Detailed responses to the reviewer's questions are given below.

Q1: For comparison with three baselines, were there reward models finetuned with Gemini scores? Or were the Gimini scores directly used for finetuning the diffusion model? It was not really clear to me from my reading.

A1: We have clarified the baselines details in Appendix D (page 18). All baselines use the Gemini rating to directly fine-tune the diffusion model. No additional reward models are trained/fine-tuned for all methods.

Q2: In the claim of selecting K instead of k for the update direction in Section 3.1 and the observation from Figure 1, I'd really love to also see the effect of choosing K/2 and K/4 to get a sense how the degree of "far away from the data manifold" correlates quality degradation.

A2: Thank you for the suggestion. We have included this experiment in Appendix C (page 18). We observe that the generated image quality decreases as $K’$ decreases. This is because as $K’$ decreases, the images $\boldsymbol{x}_{K'}$ become noisier, and move further away from the data manifold.

Q3: While aiming to deal with the situation of missing a guidance model, could you also add a comparison of the results using this method versus using a trained guidance model? That would make the story more convincing.

A3: Thanks for the suggestion. Due to a lack of data before the query process, we were unable to obtain a pre-trained guidance model. Instead, we have added DNO [1], an online guidance-based approach, as a new baseline method for image tasks. The new experiment results are shown in Fig. 2 and Fig. 4 of the main paper, and Appendix A.

It is important to note that DNO runs at the instance level, meaning each run produces only one image, and its performance is highly dependent on the initial prior. To ensure a more accurate evaluation, we performed 16 independent experiments to generate 16 images and reported the average results. This evaluation required 16$\times$ more batch queries compared with our Fast Direct and all other baselines.

From the experiments, we observe that Fast Direct achieves significantly better objective values than DNO. Furthermore, the learned GP in Fast Direct generalizes well to unseen prompts, whereas DNO, being an instance-level method, cannot generalize to generate images using unseen prompts without re-running the query (optimization) process.

[1]: Tang, Zhiwei, et al. "Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization." arXiv preprint arXiv:2405.18881(2024).

We thank the reviewer for dedicating time and effort to review our paper. Detailed responses to the reviewer's questions are given below.

Q1: Lack of ablation studies of the hyper-parameters used in the proposed pipeline (step-size alpha, lambda). Additional experiments on the chosen hyper-parameters (especially for the step-size alpha since this is the guidance scale) are encouraged.

A1: Thank you for the suggestion. We further include a detailed ablation study about the step size and batch size in Appendix B (Page 18). From the experiment, we observe that Fast Direct is not sensitive to the step size and batch size settings.

Q2: For both image and molecule generation tasks, the evaluation is based on the same metric used in the optimization, which may enrol incomplete or unfair comparisons to the other baselines. The paper could use the same LLM score during the inferences of the baselines (if it is available for them) or post additional evaluation metrics (such as the score from different LLMs and human evaluation scores) to strengthen the argument.

A2: We clarify that all baselines are optimized and evaluated by the same metric. For image tasks, we use the same LLM score, Gemini rating, to evaluate the generated images; for molecule tasks, we use the Vina docking score to evaluate the generated molecules. The evaluation is consistent across all baselines and Fast Direct to ensure a fair comparison.

Q3: In Figure 1 (also in the figures in the Appendix), the initial image of the proposed method is different to the other baselines, which seems to be a factor that influences the final generation performance. Can the authors clarify here?

A3: The initial image is slightly different because we use EDM sampler for Fast Direct but use DDIM sampler for baselines. We have added DDIM sampler for Fast Direct in image experiments to ensure the initial images are consistent with the baselines. The results are now included in Fig. 2 and Fig. 4 of the main paper, as well as in Appendix A. This would ensure a fair comparison. From the experiment, we observed that Fast Direct achieves similar performance with the DDIM sampler as it does with the EDM sampler.

Dear Reviewer TjGV

Thank you again for your time and efforts in reviewing our submission!

We have now provided more clarifications, explanations, experiments, and discussions to address your concerns and followed the advice of all reviewers to improve our paper.

Would you mind checking our responses and confirming if they have addressed your concerns?