Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.

Q: More visual results

We have added 32 images generated by RPO to the attached PDF on global response, specifically focusing on prompts that are both unseen in the training data and highly imaginative.

Q: Improve DINO scores

To preserve the details of the reference images, we may need to use ControlNet [1] or incorporate an additional cross-attention layer for the reference images within the U-Net component [2]. Mathematically, these methods will modify the distribution from $p_{\theta}(x_{t - 1} | c, x_t)$ to $p_{\theta}(x_{t - 1} | c, x_t, x_{\text{ref}}),$ allowing the model to capture extra information from the reference images during the inference phase. However, RPO does not have any assumptions about the model architecture. Therefore, in future work, we will integrate the ControlNet or cross-attention layer approach with RPO to improve DINO scores.

[1] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Chen, Wenhu, et al. "Subject-driven text-to-image generation via apprenticeship learning." Advances in Neural Information Processing Systems 36 (2024).

Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.

Q: How does the method allow for adjusting $\lambda$ during inference, and how does it compare to classifier-guidance inference?

Our contribution is on the fine-tuning step on a pretrained diffusion model. Currently, our method (RPO) only provides reward or preference signals during the fine-tuning phase. $\lambda_{\text{val}}$ allows flexibility in the model-selection process for the trade-off between image-image alignment and text-image alignment for subject-driven tasks (details can be seen in Table 3). The reward function selects the optimal checkpoint during fine-tuning, and we do not modify the diffusion reverse process for the inference phase. Therefore, our inference is the same as the SD, which is classifier-free.

Q: Wall-clock comparison between DreamBooth. We define the fine-tuning time as the sum of the preparation time and training time. The preparation time refers to the time spent generating negative samples. We use RPO and DreamBooth to fine-tune SD on 4 TPU v4 nodes and report the wall-clock fine-tuning time for these two methods in the following table.

Q: Experimental results of arithmetic mean is missing.

We set the harmonic mean as the arithmetic mean and test the $\lambda_{\text{val}} = 0.3, 0.5, 0.7$. We report the results of the arithmetic mean reward function in the following table. We refer readers to Table 3 in our paper for comparison. We discussed the difference between the harmonic mean and the arithmetic mean from lines 139 to 143. The arithmetic mean is not very sensitive to smaller values; it tends to maximize higher values to achieve a better final score. In practice, ALIGN-I will receive a higher value (this effect can be seen from CLIP-I and CLIP-T in Tables 1 to 3). Thus, the model will tend to optimize image-to-image alignment and achieve good results on DINO and CLIP-I but have a lower score for text-to-image alignment.

Q: Extensive comparisons with other baseline methods.

We encourage readers to refer to the Extensive comparisons with other baseline methods section in the global response.

Q: Discussion of PALP

We encourage readers to refer to the Discussion of PALP section in the global response.

Q: Table values seem to be inconsistent

Table 1 results from $\lambda_{\text{val}} = 0.5$ (Details show in Table 3). Table 2 results from $\lambda_{\text{val}} = 0.3$, which is our default setting.

Q: Unclear about the rationale behind adding "+1" to the similarity calculations.

Firstly, harmonic mean only supports elements that are non-negative [1]. Secondly, our measure of alignment is $\text{CosSim}(u, v) \in [-1, +1]$. Therefore, we add 1 then divide the sum by 2 to ensure each element is between 0 and 1.

[1] Scipy Library. https://github.com/scipy/scipy/blob/v1.14.0/scipy/stats/_stats_py.py#L215-L307

Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.

Q: Limited novelty of reward function and loss.

Yes, the $\lambda$-harmonic reward function is indeed a classical Pythagorean mean with weights. In fact, when $\lambda = 0.5$, it is exactly the classical Pythagorean mean. The novelty of our contribution comes from the combination of using this reward function along with reinforcement learning-based finetuning approach, not from the reward function alone. There are two aspects to consider in the design of the $\lambda$-harmonic reward function: $\lambda$ provides flexibility to the user in skewing the relative importance of image-to-image alignment and text-to-image alignment. Also, as mentioned in lines 139 to 143, the harmonic variant is preferred over geometric and arithmetic counterparts because we do not want the fine-tuning process to ignore text-to-image alignment since they tend to yield lower reward values.

While $\mathcal{L}_{\text{pre}}$ is a variation of binary cross-entropy (BCE) and DPO, it has an important difference albeit the seemingly unimpressive formulation. Validation in DPO is done using an external model or human evaluation [1], while trained using a different approach using raw preference data. Our approach uses binary labels (hence requiring the BCE loss) sampled from the reward model to fine-tune, but the validation also comes from the same reward model. This reduces distributional shifts between training and validation.

Using the $\lambda$-Harmonic function and our simple loss function (Equation 9), RPO allows for early stopping and requires only 3% of the negative samples used by DreamBooth, as well as fewer gradient steps, making it highly efficient.

Q: Quantitative comparisons do not show the improvement.

For Figure 3, as we discussed in our paper (Lines 234 to 236), the first prompt is not grammatically correct: "A dog eating a cherry bowl." The correct prompt should be "A dog eating a cherry from a bowl." The trained model was confused by the original incorrect prompt and generated this image. As for the concern over the misalignment with "table" in the second row, the image generated by our model clearly shows the object standing on a table (at the bottom of the image). Furthermore, we also provide an additional comparison in the Appendix A.3. RPO can address some of the failure cases in DreamBooth and SuTI.

Q: Comparison to DPO.

The original DPO is not suitable for subject-driven tasks because the datasets do not contain preference labels. We introduce the $\lambda$-harmonic function and design a variant of DPO for this task. To address your concern, we implemented a pure DPO for diffusion [2] (without image similarity loss) using preference labels for image-to-image similarity and text-to-image alignment. We chose $\lambda_{\text{train}} = 0.5$ because this value assigns equal weights to image-to-image similarity and text-to-image alignment. For a fair comparison, we also report the results from RPO with the same $\lambda_{\text{train}} = 0.5$. The results on DreamBench for these two methods are shown in the following table.

These results show that DPO can capture the text-to-image alignment from the preference labels. However, without $\mathcal{L}_{\text{sim}}$, DPO faces a significant overfitting problem; i.e., it achieves high text-to-image alignment but cannot preserve the subject's unique features.

Q: Undefined function.

Thank you for pointing out the typo. $\mathcal{L}_{\text{ref}}$ should have in fact been $\mathcal{L}_{\text{pre}}$.

Q: Details on CLIP and DINO.

CLIP consists of two encoders: one for the image and one for the text [3]. Suppose a dataset contains pairs of image and caption. The objective is to ensure that the encoding of the image and the encoding of the text (caption) are aligned in terms of cosine similarity. This is achieved by contrastive learning using positive and negative pairs, which naturally proves useful in our case since this cosine similarity can be used as a reward signal for both image-image and image-text alignment. DINO, on the other hand, encodes only images and in a significantly different manner [4]. Image augmentations are made with random perturbation and cropping, passed through the student and teacher models to produce representations that summarize the image. This makes a better fine-grained understanding of images than CLIP in terms of producing accurate rewards, but cannot provide signals on text-image alignment.

[1] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36. 2024.

[2] Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning, PMLR. 2021.

[4] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.

Q: Limited evaluation metrics.

Limited evaluation metrics are a common issue in subject-driven tasks. We appreciate you raising this issue and providing reference metrics, such as image quality. In the table below, we report the average aesthetic scores [1] of the real reference images in DreamBench and the average aesthetic scores obtained with the SOTA CLIP I/T lambda configuration ($\lambda_{\text{val}} = 0.5$) in DreamBench. We observe that RPO does not decrease the quality of images; instead, the generated images achieve slightly better quality than the reference images.

Q: Overfitting risk.

We observe this problem and propose a hypothesis in lines 228 to 231. To solve this overfitting issue, we may need to use ControlNet [2] or add an additional cross-attention layer to the reference images within the U-Net component [3]. Mathematically, using these methods, the distribution will be changed from $p_{\theta}(x_{t - 1} | c, x_t)$ to $p_{\theta}(x_{t - 1} | c, x_t, x_{\text{ref}}),$ which can capture the information from the reference images during the inference phase. However, RPO has no assumptions about the model architecture. Therefore, in future work, we will combine the ControlNet or cross-attention layer approach with RPO to alleviate this overfitting risk.

Q: Unseen prompts visualization results.

We have added 32 images generated by highly imaginative prompts to the attached PDF on global response.

[1] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. "Laion-5b: An open large-scale dataset for training next generation image-text models", 2022.

[2] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Chen, Wenhu, et al. "Subject-driven text-to-image generation via apprenticeship learning." Advances in Neural Information Processing Systems 36 (2024).

Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.

Q: Comparison to DCO.

Lee et al. [1] use SDXL [2] as the backbone model and apply LoRA to fine-tune the pretrained model. For a fair comparison, we only use LoRA to fine-tune the U-Net component for both methods (DCO and RPO). Furthermore, we implement RPO with SDXL as the backbone model, training the LoRA parameters with 1000 gradient steps and saving the checkpoint from the last gradient step, i.e., without early stopping. RPO achieves better image-to-image similarity results than DCO (higher DINO and CLIP-I). Even without early stopping, the CLIP-T score is only slightly lower (by 0.016) than DCO.

Q: What happens when $\lambda_{\text{train}} \neq 0$ during training?

We test four different values of $\lambda_{\text{train}} = 0.3, 0.5, 0.7$ with the default $\lambda_{\text{val}} = 0.3$ and report their results in DreamBench. We observe that the image-to-image alignment increases with larger $\lambda_{\text{train}}$, but the text-to-image alignment decreases because the preference model tends to favor alignment with reference images and ignores the prompt alignment.

[1]: Lee, Kyungmin, et al. "Direct consistency optimization for compositional text-to-image personalization." arXiv preprint arXiv:2402.12004 (2024).

[2]: Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).