RectifID: Personalizing Rectified Flow with Anchored Classifier Guidance

Thank you for the very constructive comments. Below are our responses to the raised concern.

[L1] Generalization to broader diffusion models

• While our classifier guidance is derived based on rectified flow, the same idea can be generalized to some few-step diffusion models by assuming straightness of their trajectories within each time step. We empirically demonstrate this in Figure 2 in the global response with two popular diffusion models, SD-Turbo [1] and phased consistency model [2]. As the results indicate, our method effectively personalizes these diffusion models to generate identity-preserving images. We will continue to explore this approach for other generative models in future research.

[1] Sauer, Axel, et al. Adversarial diffusion distillation. arXiv 2023.

[2] Wang, Fu-Yun, et al. Phased consistency model. arXiv 2024.

sorry for the initial short review and thanks for the author's rebuttal. After reading other reviewer's comments and the author's rebuttal, I don't think I have more insights to add here. And I would like to keep my rating.

We deeply appreciate your valuable suggestions, and we would like to address your main concerns as follows:

[W1/Q1] Theoretical justification for Equation (6)

• The intuition for Equation (6) is to shift the velocity toward data regions with higher class likelihood. Formally, we verify this using the ODE formulation in EDM [1], which states that: $$v(z_t,t)=-\\dot\\sigma(t)\\sigma(t)\\nabla_{z_t}\\log p(z_t),$$ where $\\sigma(t)$ is a noise schedule. By Bayes' theorem, the desired class-conditional distribution satisfies: $$\\nabla_{z_t}\\log p(z_t|c)=\\nabla_{z_t}\\log p(z_t)+\\nabla_{z_t}\\log p(c|z_t).$$ It turns out that the new velocity $\\hat v(z_t,t)$ in Equation (6) generates the desired distribution when the guidance scale is set to $s=-\\dot\\sigma(t)\\sigma(t)$: \\begin{aligned} \\hat v(z_t,t)&=-\\dot\\sigma(t)\\sigma(t)\\nabla_{z_t}\\log p(z_t)+s\\nabla_{z_t}\\log p(c|z_t)\\\\ &=-\\dot\\sigma(t)\\sigma(t)\\nabla_{z_t}\\log p(z_t|c). \\end{aligned} In practice, a different scale can be used to adjust the guidance, resulting in the form of Equation (6).

[W2/Q2] Issues with the "Time" column in Table 1

• The "Time" column in Table 1 refers to the inference time. Notably, our method is applied only at inference, without requiring additional training of the diffusion model. We will revise the table caption to make this clearer.
• Regarding the inference time, there is indeed an overhead of about 0.2 seconds per iteration in calling the gradient w.r.t. the classifier. Nevertheless, the iterative process can quickly converge within 10 seconds over 20 iterations, which is comparable to other baselines in terms of efficiency . This efficiency is attributed to our proposed anchored classifier guidance, as confirmed by the ablation study, and we expect further speedup with algorithm improvements.

[1] Karras, Tero, et al. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.

Thank you for providing valuable feedback. Here are our responses to the concerns raised.

[W1/Q1] Domains lacking pre-trained discriminators

• In the short term, we suggest first training a specialized discriminator and then applying our classifier guidance. There are two reasons for doing this instead of finetuning the generator directly: (1) training/finetuning a discriminator is usually more efficient and stable than training/finetuning a generator; (2) it can take full advantage of domain images that have no captions or even labels by using standard contrastive learning loss.
• In the future, scaling up vision-language models may be a general solution for these domains. The current models such as GPT-4V have demonstrated certain generalizability across visual understanding tasks. As they continue to improve in generalizability and robustness, they will become a viable source for guiding diffusion models in new domains.

[W2/Q2] More controllable generation tasks

• Following your suggestion, we've extended our method to more controllable generation tasks by directly using the guidance functions from Universal Guidance [1]. The experimental results under the guidance of segmentation map or style image are illustrated in Figure 1 in the global response. As shown, our classifier guidance can perform both tasks without additional tuning. This confirms the adaptability of our approach for various controllable generation tasks.

[1] Bansal, Arpit, et al. Universal guidance for diffusion models. ICLR 2024.