PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction

To Reviewer PSkP

Weakness 1: Different hyper-parameters may affect the alignment of the previous methods, such as the Rank of LoRA and fine-tuning steps. Authors may need to provide more clarification on how these hyper-parameters are chosen for the compared methods to make sure they achieve good enough alignment in the experiments. Thank you for your positive recognition of our work. We have provided an ablation study of these hyper-parameters in Appendix Sections C and D. We have further clarified the selection of hyper-parameters in these sections in our revised version to ensure a comprehensive understanding of their impacts on our comparisons.

To Reviewer TfPo

Weakness1: ……it's not clear what are the foundational differences, and how the authors come up with the current solution. One important motivation is that taming a T2I model toward a novel concept implies a small generation space. In linear algebra terms, the dimension of the output space of a matrix is equal to its rank. In the LoRA approach with $W+BA$, while the matrix $BA$ has a rank of $r$, the overall rank of the sum $W+BA$ is not necessarily limited by $r$. Typically, the rank of $W+BA$ remains the same as the dimension of $W$, independent of the rank of $BA$. In contrast, PaRa's matrix operations, specifically through transformations such as $W-QQ^TW$, result in a reduction of the rank of the matrix. This is what we have proven in Appendix Section 1. The originality and significance of our approach are highly appraised by other reviewers: "The idea of achieving T2I model personalization by controlling the rank of diffusion model parameters is highly innovative” (Reviewer gJ5u), "The proposed Parameter Rank Reduction (PaRa) method is a creative solution to the challenge of T2I model personalization, offering a new perspective on controlling the generation space” (Reviewer h7W1).

Weakness2: SSIM is used in this paper for image diversity evaluation. I'm not quite sure whether SSIM is the best one…… We agree that SSIM may not capture semantic-level differences, as it primarily focuses on pixel-level changes. To address this limitation, we also evaluated image diversity using CLIP image similarity, which is more sensitive to semantic content. This comparison is presented in Table 2. We analyzed both CLIP image similarity (e.g., as used by papers like SVDiff[1] and DiffuseKronA[2]) and SSIM to provide a more comprehensive assessment of image diversity.

Weakness3: Writing should be improved to highlight the contributions, especially how the solution is formed, and how it is different from the existing techniques. In Lines 088-099 of the Introduction in our initial submission, we explicitly outline our key contributions point by point. Among these, the second point, "explicit rank control," and the fourth point, "restricted small generation space," represent the fundamental differences between our method and existing techniques.

Question1: ……I'm curious about the relationship between edibility and the optimal rank of B. Is it always possible to learn a good low-rank B to perform reasonable editing in one-shot manner…… We cannot guarantee that setting a specific rank $r$ will always result in a reduction of the output rank by exactly $r$; instead, the rank of $Q$ is constrained to be less than or equal to the rank of $B$. As a result, the choice of $r$ is indeed influenced by empirical considerations. In our experiments, when performing few-shot generation, a rank up to $r=64$ still yielded usable results (as shown in Figure 8). However, for image editing, it is generally preferable to keep $r$ below 16 (see Figure 9). For tasks involving combination, a rank of less than 10 is recommended (refer to Figure 10).

Question2: ......how the model perform with different one-shot pairs? Please explain robustness of the model. Thank you for your question regarding the robustness of our model. To demonstrate this, we evaluated its performance on different one-shot pairs derived from subjects in the DreamBooth dataset. This analysis showcases the model's ability to consistently handle diverse input pairs across various subjects while maintaining fidelity to the target images. To provide further evidence and clarity, we have included additional image examples in Appendix J.

Question3: …… what are the limitations of Para? Please provide a rough failure case analysis.

In both multi-subject generation and single-subject generation, the richness of the images may be affected. As mentioned in lines 12 and 47 of the paper, generation diversity and alignment with the target concept are always a trade-off. PaRa prioritizes alignment with the target image while placing less emphasis on generation diversity. In addition, image editing might not precisely modify results. As shown in Figure 9, PaRa may lead to some loss or alteration of texture information. While it is already effective as a method that does not require an inversion process, it might not be precise.

[1] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7323–7334, 2023.

[2] Marjit, Shyam, et al. "DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Model." arXiv preprint arXiv:2402.17412 (2024).

Dear reviewer TfPo,

Since it is close to the discussion deadline, we are writing to check whether our rebuttal has addressed your concern. Thanks!

To Reviewer h7W1

Weakness 1: The authors have not compared their method with SOTA such as LyCoris, DiffuseKronA, etc. Including these results would provide a better comparison of the method. Thank you for your positive recognition of our work. We have supplemented our comparisons with LYCORIS, in Table 2, Figure 3, and Appendix Section E, Figure 11, of our revised version. DiffuseKronA has not released its official code yet, so we are unable to make a comparison.

To Reviewer gJ5u

Thanks for the constructive comments. Here are our responses to your concerns.

Weakness1: The methodology section includes extensive explanations of the mathematical…… section would make the concepts more intuitive. Thank you for your suggestion. We understand that your comment refers to how the inclusion of [V] in the prompts interacts with text embeddings within the PaRa framework. This treatment of text embeddings, particularly through the use of [V] tokens, aligns with established practices found in related works such as LoRA, SVDiff, and Textual Inversion. Aside from the newly introduced [V] tokens, other text embeddings remain unchanged during this process. Regarding the overview diagram for the model, our approach, much like LoRA and SVDiff, focuses on a purely computational innovation at the weight level. As such, the overall model architecture remains the same as Stable Diffusion. In the revised version's Appendix Section I, we add an overview of PaRa to provide readers with a visualized summary, helping them better understand the model.

Weakness2: ……However, the experiments do not provide sufficient comparisons to support this claim. Firstly, there is no comparison with encoder-based personalization methods mentioned in the introduction. Secondly, the fine-tuning-based personalization methods used in the experiments are not the latest approaches in this field……

We have supplemented our comparisons with the latest fine-tuning-based personalization approaches LYCORIS[1] , in Table 2, Figure 3, and Appendix Section E, Figure 11, of our revised version. to strengthen our experimental results. Regarding the encoder-based personalization methods mentioned in the introduction, we acknowledge the importance of including these comparisons. However, the implementations for these methods have not been released, which prevents us from conducting a meaningful and fair comparison at this stage. Additionally, we would like to re-emphasize the advantages of PaRa, such as its low computational complexity and efficiency in handling personalized fine-tuning tasks. Furthermore, PaRa achieves state-of-the-art performance in terms of image fidelity, as clearly demonstrated in Table 2 and Figure 3 of the main text.

Weakness3: …… it is mentioned that “B is initialized to zero and fine-tuned with a few text-image pairs...... fine-tuning time, and other relevant information…… parameter efficiency, a computational efficiency analysis should be provided.

Thank you for pointing this out. In the experimental section in the revised paper, under "Implementation Detail," we stated that we evaluated the effects of PaRa on customized single-subject generation using the Dreambooth dataset, where each label consists of five to six images.” This is a few-shot training based on all data from the Dreambooth dataset. For the computational efficiency analysis, we have added training time data in Appendix Section H Figure 5 to demonstrate the computational efficiency.

Weakness4: …… PaRa may result in some loss of texture information……

We acknowledge that our approach may not achieve state-of-the-art performance in terms of texture retention for image editing. However, our method offers a distinct advantage: unlike many state-of-the-art image editing approaches, PaRa does not require a noise inversion process, making it a simpler and more efficient alternative. To provide a fair comparison, we focused on methods that similarly avoid noise inversion, such as SVDiff. In these comparisons, PaRa demonstrated superior performance.

This has been mentioned multiple times in our paper, such as in lines 178 to 180:“omitting the inversion process ... significantly impacts its faithfulness to the target image. In contrast, our model PaRa can maintain editability after only one-shot learning, and eliminate the need for the inversion process to achieve image editing.” As well as in line 428: “on the same task without DDIM inversion.”

[1] Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023.