Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models

Dear reviewer XWYq,

Thank you for valuable comments and suggestions in reviewing our work. We address each of your questions and concerns individually as follows.

---

[W1] Regarding computational cost

We will move the limitation into the main text in our final revision.

---

[W2] References

Thank you for the relevant work. While several papers have proposed different adapter designs (e.g., SVDiff, DiffuseKrona, B-LoRA) for improving personalized T2I synthesis, we introduce a new fine-tuning objective that is independent of adapter design. We appreciate the mention of the MaPO paper, which was unavailable at our submission time but will be included in our final manuscript as it provides a different fine-tuning objective for T2I diffusion models.

---

[W3] Experimental dataset

We elaborated the experimental dataset for each subject and style personalization experiment in line 200-211 and line 241-245, respectively. We will include details on the dataset for each figure and table in our final manuscript.

---

[W4] The value of w_text

$w_{text}$ is kept with 7.5 throughout the experiments (see Appendix D.4). We will revise our draft to include this information in figures.

---

[S1] Regarding Figure 1

We will add details to the captions of Figure 1 to better deliver the limitations of previous works and demonstrate effectiveness of our method.

---

[S2] Details on classifier-free guidance

We will add technical details of classifier-free guidance.

---

[S3] Technical implementation

Following your suggestion, we compare the training memory consumptions of suggested technical implementation and our implementation. Note that the suggested method use

unet.disable_adapters()
unet.enable_adapters()

to compute loss from pretrained models. Originally, we used ‘cross_attention_kwargs’ to control the LoRA scale. For each method, we measure the allocated and reserved memory. All experiments were conducted on PyTorch v2.3.0, diffusers v0.27.2 on 1x A100 40GB GPU. The results are shown in the table below.

The table shows that the suggested method saves more memory than our implementation. We will update technical implementation to our code, and release it with our final manuscript.

---

[S4] Cite libraries

We will add citations to the libraries that we used in our experiments.

---

[S5] Loss landscape of DCO with different $\beta$

We provide the loss landscapes by using different values of $\beta$ in Figure B in our attached file. For a small value of $\beta$, the regularization strength is small, thus the loss results in higher variance. For a large value of $\beta$, the regularization is stronger, which results in smaller variance.

---

[S6] Highlight scores

We will highlight the scores in Table 1.

---

[S7] Training prompts

Training prompts are manually engineered using vision language models like LLaVa or ChatGPT because their generated captions are often too long for SDXL text encoders. A more effective approach could be to use captioning models (e.g., in DALL-E 3) to generate prompts for personalized images similar to the pretraining datasets.

---

[Q1] Consistency guidance

InstructPix2Pix takes additional input (i.e., source image) to control the similarity of the edited image with respect to the source image. On the other hand, consistency guidance controls the fidelity to the reference image without any additional input. We will add discussion in our final revision.

---

[Q2] Context of ‘omitting’

To clarify, we omit the usage of textual inversion technique and comprehensive captioning, which we used for all baseline experiments for fair comparison.

---

[Q3] Usage of LoRA

As we mentioned in our paper, we used LoRA for all experiments.

---

[Q4] Consistency sampling for DreamBooth

For qualitative results, we show the results by using classifier-free guidance (CFG) sampling of fine-tuned models. In our quantitative results, we provide results of using both CFG sampling (diamond) and consistency guidance sampling (dots with solid lines) for all baselines including DB and DB+p.p.

---

[Q5] Reference for pivotal tuning

We will add citation of pivotal tuning for the usage of textual inversion and LoRA fine-tuning.

---

[Q6] Regarding $w_{\text{con}}$

Yes, as $w_{con}$ increases, the image similarities become higher. In Figure 4, the image similarities increase (i.e., towards upper left) as $w_{con}$ increases from 2.0 to 5.0.

---

[Q7] Different merging method experiments

Following your suggestion, we conduct experiments on comparing different merging methods. Specifically, we apply SVD based LoRA merging (i.e., TIES_SVD) [1]. As shown in Figure C, SVD merging improves quality for both DB and DCO compared to direct merging. We observe that LoRA trained with DCO shows better performance, particularly in following complicated prompts while maintaining identity. We will add this to our final revision.

[1] Yadav, Prateek, et al. “Ties-merging: Resolving interference when merging models.” NeurIPS 2023.

---

[N1-3] Thanks for pointing this out. We will revise in our final manuscript.
[N4] Note that $\lambda_1$ is the maximum log-SNR for the noise scheduling function, which should be large enough so that the distribution of $z_1$ follows pure random Gaussian noise. Conversely, $\lambda_0$ is the minimum log-SNR for the noise scheduling function, which should be small enough so that the distribution of $z_0$ matches the data distribution.

Dear reviewer ayXu,

Thank you for valuable comments and suggestions in reviewing our work. We address each of your questions and concerns individually as follows.

---

[W1] Improvement by using DCO.

We claim that DCO uses a novel fine-tuning that pushes the frontier of the pareto curve of image similarity and prompt similarity, outperforming previous baseline methods DreamBooth and DreamBooth with prior preservation loss, as well as their variants, e.g., early stopping or different prior preservation loss weights (e.g., see Figure. 7). Not only that, DCO enables direct merging of two independent trained LoRAs, where DreamBooth trained models often fail to preserve the subject or style fidelity during merging.

---

[W2] Other experiments

To show the effectiveness of our method, we provided 1-shot subject personalization experiments as shown in Figure.14 and Figure.15 of Appendix. By using a single subject image, our method can generate diverse personalized images that change the style (e.g., photo to 2d animation style, or 3D animation style to photo) or visual attributes (e.g., actions or outfits) by using textual prompts. Specifically, as shown in Figure. 14, we show the capability of our method in learning from a single human image to stylize or change actions. We believe our method can be further explored for human specific datasets, which we leave for future work.

---

[W3] Reproducibility

For reproducibility of our method, we provide implementation details in Appendix D, where we describe the compute resources and training schedules. For our codebase, we use PyTorch and Huggingface diffusers library for the implementation. We will release our code in the final manuscript.

---

[Q1] Personalization of image-conditioned models

We believe our method can be utilized for personalization of image-conditioned diffusion models (e.g., Kandinsky 2.2 or Dalle-2). Similar to fine-tuning with DCO objective for text-to-image diffusion models, one can consider fine-tuning image conditioned diffusion models that use image embeddings. Specifically, given a pair of reference images (img1 and img2) that share the concept (e.g., subject or style), one can fine-tune the diffusion model on img1 conditioned with image embeddings from img2. Here, DCO loss could be used to restore the pretrained knowledge, which we believe would be effective to prevent overfitting and improve compositional generation of personalization image synthesis. We believe it is an interesting direction, and leave it for future work.

Dear reviewer HfJW,

Thank you for valuable comments and suggestions in reviewing our work. We address your questions and concerns as follows.

---

[W1] Human evaluation

Following your suggestion, we conduct a user study to compare DCO (ours) against DreamBooth (DB) and DB with prior preservation loss (DB+p.p.) on subject personalization tasks. As in Section 5.1, we train three models (DCO, DB, DB+p.p.) per subject with the identical experimental setup (i.e., all models are trained with comprehensive caption and textual inversion, and compare by generating images with the same random seed. We asked the participants to choose the best one out of three images from each model. Raters are asked to compare images on the following three criteria: Subject fidelity: Which image most accurately reproduces the identity (e.g., item type and details) of the reference item? Prompt fidelity: Which image most closely aligns with the given prompt? Image quality: Which image exhibits the highest quality (e.g., overall visual appeal and clarity)? We asked 22 users to evaluate 60 sets of images, resulting in a total 1320 responses per query. The table below shows the result.

Table 1. Subject fidelity, prompt fidelity and image quality user preference.

We see that DCO (ours) largely outperforms DB and DB+p.p. in prompt fidelity and image quality, while showing comparable performance with DB on subject fidelity. This result is consistent with Figure. 1 of our manuscript, where we have shown that DCO outperforms others in image-text similarity (i.e., prompt fidelity), while exhibiting comparable performance in image similarity (i.e., subject fidelity). Interestingly, we find that the users find DCO-generated images are of better quality than others, which demonstrates that DCO mitigates image quality degradation, which often happens when fine-tuning from a few, low quality reference images. We will include our results as well as the detailed information on our user study format in the final revision.

---

[W2] Ablation study

As we mentioned in our manuscript (e.g., line 210-211), using comprehensive caption (instead of compact caption used in DreamBooth paper) improves the performance of both DreamBooth and DCO methods. To support our claim, we provide an additional ablative study on the effect of comprehensive caption. We select 10 subjects from DreamBooth dataset and compare with compact captions on both DreamBooth (DB) and DCO, using the same experimental setup as in Section 5.1. For compact captions, we do not use rare token identifiers and learn textual embeddings as in Section 5.1 Figure A in our attached file shows the Pareto curves with consistency guidance of varying scales ($\omega_{\textrm{con}}$=2.0, 3.0, 4.0, 5.0). We observe that a comprehensive caption (solid line) forms an upper-right frontier compared to compact caption (dashed line) for both DB and DCO.

---

[Q1] Prior preservation loss hurts subject fidelity

We hypothesize that learning from both reference and synthetic images as in prior preservation loss hurts subject fidelity due to the concept leakage from the synthetic images. Note that such a behavior (e.g., losing subject fidelity at the cost of generation diversity) has been already observed from the DreamBooth paper (e.g., see Section 4.3 and Table 3 of the DreamBooth paper). On the other hand, DCO preserves the prior without synthetic data by minimizing the shift from the pretrained model while fitting on reference images.

Dear reviewer aXKh,

Thank you for valuable comments and suggestions in reviewing our work. We address each of your questions and concerns individually as follows.

---

[W1] Experiment with LoRA

We clarify that all our experiments were conducted using LoRA. This has been explicitly mentioned in our manuscript, experimental setup section (e.g., see line 205 and line 242).

---

[W2] Diversity of generated images.

Our method is able to generate personalized subject images of various poses, e.g., see Figure. 19 and Figure. 20 in Appendix. Since our method is better than previous methods in terms of prompt fidelity and subject fidelity, one can provide prompts to change the poses or actions to generate diverse images while preserving identities.