Stochastic Adaptive Sequential Black-box Optimization for Diffusion Targeted Generation

We sincerely appreciate the reviewer's constructive advice and valuable comments. Our detailed responses to the reviewer's questions are presented as follows:

$**Q1**:$ "The authors should aim to articulate their primary contributions and the uniqueness of their proposed methodology more clearly, focusing on how it addresses the identified challenges."

$**A1:**$ In this paper, we focus on the problem of how to employ diffusion models to generate user-preferred content with black-box target scores while avoiding re-training. This problem is essential but unexplored in the literature. The related DDOM needs a reweighted sampling of training samples to re-train a conditional diffusion model from scratch.

Moreover, the black-box optimization methods in the literature can not be directly applied for diffusion model fine-tuning due to ignoring the sequential nature of the diffusion sampling process. To address this problem, we $**are the first to**$ formulate the diffusion model black-box fine-tuning problem as a sequential black-box optimization problem with an adaptive diffusion noise process. We propose a novel covariance adaptive sequential black-box optimization algorithm to solve this problem. Furthermore, we prove a $O(\frac{d^2}{\sqrt{T}})$ convergence rate of our algorithm for convex functions without smooth and strongly convex assumptions. To the best of our knowledge, our algorithm is the $**first**$ covariance adaptive black-box optimization method that achieves a provable $O(\frac{d^2}{\sqrt{T}})$ convergence rate for general non-smooth convex functions.

We firmly believe that our work explores a crucial area underexplored and provides solid technique contributions to this area.

$**Q2:**$ "The rationale for using text-guided generation as a demonstration of the proposed method is not entirely clear. The authors should elaborate on why they believe this setting poses a challenging black-box optimization score. Does this scorer belong to those touch non-smooth cases?"

$**A2:**$ The black-box targeted image generation is challenging due to the very high dimension of images. Specifically, the dimension of the images employed in our work is $256\times256\times3 = 196608$. In contrast, most of the tasks employed in the DDOM paper are less than $100$-dimension, and the highest dimension is $5152$. The image generation task is more challenging than the one studied in the DDOM paper.

"Does this scorer belong to those touch non-smooth cases?"

Most scorers in real applications are non-smooth cases. For example, the scorers based on neural networks with ReLU function are non-smooth. To be more precise, note that the ReLU function is non-smooth. Thus, the neural network using the ReLU function is also non-smooth. As a result, the score functions that rely on the neural network using the ReLU function are also non-smooth.

$**Q3:**$ "The quality of generated images is damaged. As evidenced in Figure 2, the generated images of baseline methods are more natural and clear than those of the proposed method. Is there any solution to resolve this problem?"

$**A3:**$ We have solved the loss of visual quality issue and include additional experiments. Please check our overall response and Section A in the Appendix of our revised paper.

$**Q4:**$ "The single-domain experiment on text-guided generation does not sufficiently prove the effectiveness of the method proposed. The authors should consider introducing more diverse black-box scorers to exhibit the method's versatility. For example, DDOM conducted experiments across multiple domains; perhaps a similar approach could be beneficial here. "

$**A4:**$ Thanks for your suggestion. We include additional experiments on the tasks studied in the DDOM. Please check our overall response and Section A in the Appendix of our revised paper.

$**Q5:**$ "The derivation process in Equation (10) and how this objective aligns with Equation (8) is not clear. Can the authors provide further clarification on this?"

$**A5:**$ Note that $J(\bar{\theta}_K^t)$ is a constant when we minimize $J(\bar{\theta}_K)$ w.r.t. $\bar{\theta}_K$. Thus, minimizing Eq.(10) (Eq.(9) in the revised paper) is equivalent to minimizing objective Eq.(8) (Eq.(7) in the revised paper)

We sincerely appreciate the reviewer's constructive advice and valuable comments. Our detailed responses to the reviewer's questions are presented as follows:

$**Q1:**$ "The organisation and writing of the paper need to be improved. The notations and equations need to be better defined. For example, the authors should define all terms appeared in equation 1, 2, and 3, such as $g(t)$ and $\alpha_t$ and so on. The writing made it very difficult for readers to follow the proposed method."

$**A1:**$ Thanks for your suggestion. We have revised the paper accordingly. The notation in Eq.(1)~(2) is clearly defined in the revised paper.

$**Q2:**$ "The loss of perception quality of the targeted generation is a hurdle for this approach to have any practical use. It'd be better if the authors explored this trade-off a bit further."

$**A2:**$ We solve the loss of visual quality issue and include additional experiments. Please check our overall response and Section A in the Appendix of our revised paper.

We sincerely appreciate the reviewer's constructive advice and valuable comments. Our detailed responses to the reviewer's questions are presented as follows:

$**Q1**$: What do you mean by a "full matrix update" and an "adaptive update"? When making claims such as "algorithm is the first full matrix adaptive black-box optimization with x convergence rate," it is important to explain the qualifiers and the relevant literature.

$**A1**$: Our algorithm updates the full covariance matrix at each step during the optimization process. The “adaptive update” means that the covariance matrix is updated according to the function values of the quires at each step instead of a fixed one.

In the literature, NES, CMAES and INGO are methods that update the full covariance matrix for black-box optimization, which has been discussed in our related work section. In the NES and CMAES papers, the authors did not provide convergence analysis. In INGO, the convergence rate is analyzed for standard black-box problems under strongly convex assumptions. Our convergence rate holds for sequential black-box optimization problems without strongly convex assumptions. In particular, when the total step of the sequential problems is one, i.e., K=1. We achieve a convergence rate for standard black-box problems.

$**Q2**$: “Can the proposed method be adapted to the tasks studied in DDOM and if so, how do they compare?”

$**A2**$: Thanks for your suggestion. We include additional experiments on tasks studied in DDOM. Please check our overall response and Section A in the Appendix of our revised paper.

We sincerely appreciate the reviewer's constructive advice and detailed comments. Our detailed responses to the reviewer's questions are presented as follows:

$**W1**$: “The major weakness lies in the clarity of writing. … In particular, (1) the indexing of k and t are confusing. and (2) which of the algorithm 1 and 2 is the practical algorithm? How are they different? ”

"(1) the indexing of k and t are confusing."

$**A(1)**$: Thanks for your suggestion. We have revised the paper accordingly. The index $k$ denotes the sequential step of the diffusion sampling. The index $t$ denotes the number of iterations for parameter updates in the black-box optimization process (see Alg.1 and Alg.2). To clarify the confusion, we revised section 2.1 to avoid the double definition of $t$ in Eq.(1) $\sim$ (2).

"(2) which of the algorithm 1 and 2 is the practical algorithm? How are they different? ”

$**A(2)**$: Algorithm 2 is a general framework for sequential black-box optimization with a flexible set of parameters. Algorithm 1 is a practical algorithm of our framework (Alg.2) with a particular parameter setting for diffusion model targeted sampling. In Algorithm 1, the constraint of $\boldsymbol{H}_k^t$ in our framework(Alg.2) is relaxed, and the step size depends on the SDE solver coefficient. In addition, the black-box function at each sequential step in Alg.1 is a particular function that depends on the diffusion model prediction.

$**W2**$: Loss of visual quality issue. "The generated images have higher CLIP scores but lower visual quality. "

$**A2**$: We have solved this issue and included additional experiments. Please check our overall response and Section A in Appendix of our revised paper.

$**Q3**$: “Equation 4: describe the indexing of t ”.

$**A3**$: The index $t$ denotes the number of iterations for parameter updates in the black-box optimization process.

$**Q4**$: “How is the proposed method different from DDOM?”

$**A4**$: Our method is a novel sequential black-box optimization method that adaptively updates the mean and covariance matrices of a series of Gaussian distributions for candidate sampling. We employ our sequential black-box optimization method to fine-tune the inference of a pre-trained diffusion model for targeted sampling.

In contrast, the DDOM employs a weighted sampling of the training samples to train a conditional diffusion model from scratch and utilizes the conditional diffusion model for a targeted generation.

$**Q5-Q6**$: “Section 3 Notation and symbols: … what "Gaussian distribution" is this referring to? define d (the dimension of data?)”

$**A5-A6**$: Thanks for your suggestion. We have revised the paper accordingly. Please check our revision.

$**Q7**$: “Section 3: what value does k take in ? What does k and t mean in $\bar{\theta}_k^t$?”

$**A7**$: $k$ take values from 1 to $K$, where $K$ denotes the number of diffusion sampling step. Note that the diffusion sampling is in a sequential manner. In $\bar{\theta}_k^t$, index $k$ denotes the $k^{th}$ sequential sampling step, where $t$ denotes the $t^{th}$ parameter update iteration in black-box optimization process.

$**Q8**$: "But equation (1) (and equation (3)) uses $t$ to denote step and suggests a "reverse-time" manner, which seems not consistent."

$**A8**$: We have revised Eq.(1) to Eq.(3) to avoid the double definition of $t$ (See Sec.2.1 in the revised paper). The symbol $t$ denotes the number of iterations in the black-box optimization process (See our Alg.1 and Alg.2).

$**Q9-Q10**$: "When first introduced the black-box target score function (in the line above equation (6)), describe its argument"

$**A9-A10**$: Thanks for your suggestion. We have revised the paper accordingly.

The remaining concerns of the reviewer will be addressed in the next comment.

$**Q11**$: “Equation 9, I wonder if it is correct to factorize the distribution w.r.t. $\bar{\epsilon}_k$ across $k$”

$**A11**$: Note that $\bar{\boldsymbol{\epsilon}}_k=[\boldsymbol{\epsilon}_1^\top,\cdots,\boldsymbol{\epsilon}_k^\top]^\top$ for $k \in \{1,\cdots,K\}$ , from the linear property of the expectation operation, we know that

$\mathbb{E}_{\bar{\boldsymbol{\epsilon}}_K \sim \mathcal{N} (\bar{\boldsymbol{\mu}}_K,\bar{\boldsymbol{\Sigma}}_K) } [ \sum _{k=1}^K f_k( \bar{\boldsymbol{\epsilon}}_k ) ] = \sum _{k=1}^K \mathbb{E} _{\bar{\boldsymbol{\epsilon}}_K \sim \mathcal{N} (\bar{\boldsymbol{\mu}}_K,\bar{\boldsymbol{\Sigma}}_K) } [ f_k( \bar{\boldsymbol{\epsilon}}_k ) ]$. Note that $f_k( \bar{\boldsymbol{\epsilon}}_k)$ depends on $\bar{\boldsymbol{\epsilon}}_k =[\boldsymbol{\epsilon}_1^\top,\cdots,\boldsymbol{\epsilon}_k^\top]^\top$, but it does not depends on $[\boldsymbol{\epsilon} _{k+1}^\top,\cdots,\boldsymbol{\epsilon}_K^\top]^\top$. Then we know $\sum _{k=1}^K \mathbb{E} _{\bar{\boldsymbol{\epsilon}}_K \sim \mathcal{N} (\bar{\boldsymbol{\mu}}_K,\bar{\boldsymbol{\Sigma}}_K) } [ f_k( \bar{\boldsymbol{\epsilon}}_k ) ] = \sum _{k=1}^K \mathbb{E} _{\bar{\boldsymbol{\epsilon}}_k \sim \mathcal{N} (\bar{\boldsymbol{\mu}}_k,\bar{\boldsymbol{\Sigma}}_k) } [ f_k( \bar{\boldsymbol{\epsilon}}_k ) ]$. Thus, the Eq.(9) (Eq.(8) in the revised paper) holds.

$**Q12-Q13**$: “How is Algorithm 1 and 2 different from each other?”

$**A12-A13**$: Algorithm 2 is a general framework for sequential black-box optimization with a flexible set of parameters. Algorithm 1 is a practical algorithm of our framework (Alg.2) with a particular parameter setting for diffusion model targeted sampling. In Algorithm 1, the constraint of $\boldsymbol{H}_k^t$ in our framework(Alg.2) is relaxed, and the step size depends on the SDE solver coefficient. In addition, the black-box function at each sequential step in Alg.1 is a particular function that depends on the diffusion model prediction. Our convergence analysis is conducted on the general framework Alg.2 without relying on diffusion model prediction.

$**Q14**$: "Algorithm 1: K should be an input argument."

$**A14**$: Thanks for the suggestion. We have revised the paper accordingly.

$**Q15**$: “Page 7 Section 5.1, what does the subscript m mean? Is it suppose to index the dimension of data. It seems to encode a different meaning of subscript k”

$**A15**$: $m$ denotes the index of the element in the input vector $\boldsymbol{x}$. $k$ is the index of the step in the diffusion sampling process.

$**Q16:**$ "In both experiments, how do you choose the total number of steps, and how do you choose the fine-tune steps to present, e.g. tabel 1 (steps =1596, 2646). Are they chosen randomly?"

$**A16:**$ Yes. The parameter is chosen randomly. We have reported additional experimental results in the Appendix of our revised paper.

$**Q17:**$ “Does Figure 2 show independent draws?”

$**A17:**$ Yes. The images are drawn independently. We report more comparisons and demonstrations in our revised paper.

$**Q18:**$ "Loss of visual quality issue."

$**A18:**$ We have solved the loss of visual quality issue and included additional experiments. Please check our overall response and Section A in the Appendix of our revised paper.