Efficient Personalized Text-to-image Generation by Leveraging Textual Subspace

Q1: Regarding the use of subspace learning...

R1: In the reverse process of diffusion models, to obtain the text condition $y$, all word vectors need to be combined and input through a pre-trained text encoder. In the literature of NLP, all word vectors are assumed to lie in a shared space. One of the motivations of our method is to construct a textual subspace according to the provided initial word and optimize the target vector on it. For multi-concept generation, several trained vectors are combined with other word vectors and input through the text encoder, with each trained vector is searched and optimized in a independent textual subspace. However, although multiple vectors are represented in their textual subspace, they do not need to be trained at the same time. For Figure 6, we have used $M$ basis vectors to represent the object to showcase more qualitative results. However in Figure 8, we only use $1$ basis vector to perform an ablation study to demonstrate the necessity to utilize multiple basis vectors to represent the target concept.

Q2: Addressing missing objects in generated images...

R2: It is widely stated in the literature of text-to-image personalization that TI falls short of text-to-image alignment ability since it only optimizes the word vector in a reconstructive manner, while the proposed method, successfully improves the text-to-image alignment score with low computational cost by constraining it in a textual subspace. The ability of generating missing objects can be contributed to the high text-to-image alignment ability of our method, which successfully reflects all text information.

Q3: Clarification on optimization problem and loss...

R3: From Eq. (7), the target word vector $v$ is represented by the weights $w$ and its corresponding basis vectors. By switching the optimization target from $v$ to $w$, the proposed method represents and optimizes the target vector in a textual subspace, which offers high text-to-image alignment and low computational cost.

Q4: Discussion of limitations...

R4: Due to page limit, the limitations have been discussed in Appendix A. Please head there for details.

Q1: The significance of improving training time may be less important considering the already inexpensive nature of personalization...

R1: In the literature of text-to-image personalization, TI is often considered to be slow since it requires nearly an hour to train a single object, thus hindering it to be applied to large-scale applications. We have experimentally demonstrated that by applying the proposed BaTex method, the training budget can be reduced to nearly 10 minutes, which is only 1/6 of TI. Also, we have stated in Section 3.2 that the textual subspace is designed using the input word and the vocabulary provided by the pre-trained models. Thus, when optimizing the target vector in this subspace, the text-to-image alignment can be well preserved, which results in high text-to-image alignment score. Conversely, the objective of TI solely depends on the reconstruction loss, which only concentrates on the input image. In a word, our method demonsrates that, by constraining the training process in a textual subspace induced by the vocabulary, the learned vector retains high text-to-image alignment score with less computational cost.

Q2: It is hard to tell if Batex outperforms TI in Figure 2...

R2: Take the upper figures as an example. We have added "with a glass of wine talking to a lady'' and "chatting with a student at his desk'' as additional text conditions. It can be seen that only the results of BaTex successfully reflect the corresponding text conditions while other three methods all fail to achieve it. Although TI does follow the provided style, the results are missing various degree of text information due to its low text-to-image alignment ability.

Q3: How is the results of other baselines in fig 3?

R3: In this paper, we focus on learning a single concept by a word vector in a induced textual subspace to improve its text-to-image alignment with less computational overhead. Thus, we only showcase multi-concept results as a complement to demonstrate the performance of our method. We leave investigation of more detailed multi-concept generation to future study.

Q1: Could the authors provide visualizations of both learned weights and corresponding basis vectors (i.e., words)?

R1: We have provided the learned weights and their corresponding basis vectors for example ``cat'' in Appendix F.10. As can be seen, starting from the initial word “cat”, BaTex learns a reasonable combination to obtain the target vector while TI tends to search around the initial embedding using only the reconstruction loss, which results in low text-to-image alignment score. Please head there for more details.

Q2: Too few methods are compared in this paper...

R2: Thanks for advice. We have added both qualitative and quantitative comparisons with three concurrent works: SVDiff [1], XTI [2] and NeTI [3]. For encoder-based methods mentioned above, since they require to train a domain encoder for each object category, it is not suitable for most cases in our experiments. We will try to compare with these methods in the next revision. Results can be seen in Appendix F.9. It has shown that our method achieves superior results over all three concurrent works. Please head there for details.

Q3: The expressiveness of words combination may be limited, especically in reconstructing image detalis, e.g., human faces.

R3: We have demonstrated the expressiveness of our method from qualitative and quantitative comparisons. As can be seen in Table 2, our method achieves superior text-to-image alignment score and comparable image-to-image alignment score compared to baseline methods. We have also included additional qualitative and quantitative results in Appendix F, including object composition and more qualitative comparisons.

Q4: Is there too much redundancy among these basis embeddings?

R4: We have conducted an ablation study about the number $M$ using the text-to-image alignment score and image-to-image alignment score in Table 4 in Section 5. It can be seen that setting $M=576$ achieves a balance between high generation quality and fast convergence speed. We leave further analysis of $M$ to future study.

[1] Ligong Han, et al. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning. ICCV 2023.

[2] Voynov, et al. P+: Extended Textual Conditioning in Text-to-Image Generation. arXiv:2303.09522.

[3] Alaluf, et al. A Neural Space-Time Representation for Text-to-Image Personalization. SIGGRAPH 2023.

Q1: I remain unconvinced regarding the claimed time efficiency of the proposed method...

R1: We have revised Table 1 to show the total training time compared with other three methods. It can be seen that our method is 6 times faster than the baseline method TI and achieves competitive results when compared with Dreambooth and Custom-Diffusion. Also, although it requires users to define the number $M$, we have experimentally observe that the embedding matrix $V_M$ in Algorithm 1 is highly linearly-independent, thus further releasing the constraint of checking the rank $r(V_M)$. Furthermore, it can be seen from Table 2 that BaTex outperforms TI in text-to-image alignment score with no performance degradation in reconstruction. Compared with DB and CD, it clearly shows that we reach the state of the art performance in text-to-image alignment score with competitive image-to-image alignment score. Thus, it can be concluded that BaTex achieves high alignment ability with significantly less computational overhead.

Q2: I also have reservations about the qualitative performance of the proposed method...

R2: Due to page limit, we have only provided limited results in the main text. However, diverse results have been placed in Appendix F, including ablation studies, qualitative results about more objects and styles and quantitative comparisons. Please head there for details.

Q3: I am unclear as to why the authors assert that previous methods solely focus on image reconstruction...

R3: As observe in the previous work [DreamBooth Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation] and [Multi-Concept Customization of Text-to-Image Diffusion], TI often get lower text-to-image alignment score due to its nature of only training the vector by reconstruction objective. As can be seen in Figure 2, when input the same complex text prompt to all four methods, BaTex generates reasonable results according to the input text conditions while TI occurs missing some objects mentioned, including ''lady'', "student'' and "table''. Thus, one of our motivations is to represent the word vector in a textual subspace to enhance its ability to combine with other words with less computational overhead.

Q1: The novelty is limited...

R1: First of all, we claim that BaTex is the first method to introduce a specific textual subspace for personalization. Due to the slow training time of the baseline methods, which requires nearly an hour to train a single object, text-to-image personalization is ineffective to be applied in large-scale applications. In this paper, we propose a simple yet effective method to search the embedding in a subspace, which results in significant reduced training time and fast convergence speed. For the two paper [Medical diffusion on a budget] and [LaDI-VTON] mentioned above, they apply the personalization mechanism to other fields rather than improving the baseline methods. Thus, the novelty of the proposed method can be guaranteed by its novel way to represent and optimize the word vector in a textual subspace. Since our method offers fast convergence speed compared to baseline method, it naturally requires less training steps.

Q2: Please highlight the priority and novelty of the proposed method from the whole diffusion procedures

R2: As stated in R1, we are the first to propose a method to represent and optimize the word vector in a meaningful textual subspace. From the perspective of the whole diffusion procedure, we introduce a mechanism to optimize the word vector $v$ in textual subspace, which represents a pseudo-word. The vector $v$ is then concatenated to the embedding matrix $V_M$ and input into the text encoder to get the final condition $y$, and input into the noise predictor $\epsilon_{\phi}(z_t;\epsilon, t, y)$ to predict the corresponding noise in the reverse process at timestep $t$. Since the original personalization method needs up to an hour to optimize $v$ for each object, the proposed method significantly improves the training speed with no performance degradation.

Q3: Please add more analysis of the training details of the proposed methods comparing with the textual inversion.

R3: In Textual Inversion, it represents and optimizes the target vector in the original textual space, while in BaTex, a specific subspace is established and utilized using the proposed selection strategy. In each iteration, the trainable weights $w$ are updated using stochastic gradient descent with other modules frozen. After training, the weights are combined by the basis vectors to obtain the target word vector, which lies in the textual subspace.

Q4: Please add more experimental results of more conditions of the text-to-image tasks.

R4: Experimental results of more conditions have been added in Figure 4 and 5 in the supplementary material. As can be seen, the proposed method successfully reconstruct the input concept while generating diverse conditions following the provided text prompt.