OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance

We greatly appreciate your insightful review of our work. For the weakness and questions, we will response to each of them individually.

Weakness 1: Comparison with multi-concept customization pipeline

We provide a qualitative comparison with FreeCustom[1], the current state-of-the-art multi-concept customization method, in the Figure 22 of the global rebuttal pdf. Note that FreeCustom is officially implemented on StableDiffusion V1.5 and thus exhibits averagely lower image quality. In terms of subject consistency and prompt conformity, which are backbone-irrelevant, our method still outperforms FreeCustom. This reaffirms the effectiveness of the proposed cluster guidance method. Due to the time limit, we are not able to perform more comprehensive experiments of multi-concept customization methods. We will add these methods to the Related Work section and add more experiments in the new version of our paper.

[1] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024

Weakness 2: Typo

We will carefully review our paper sentence-by-sentence and try to correct every type error. Thanks for your meticulous review!

Weakness 3: Imperfect consistency

For the "hobbit" case comparison between DB and our method, solely from the perspective of subject consistency, DB indeed beats our method. Yet we believe our method achieves a more balanced performance of subject consistency, background and layout diversity. We believe this balanced performance is more meaningful in the consistent subject generation task. Due to the non-ideal distribution of the clusters, there will be some inconsistent minor details such as the beard or clothing decorations. Addressing this challenge will be the focus of our future work.

Question 1: Connection with score-SDE

In our opinion, the basic logic of our method is the same as that of the score-SDE. The methodology of score-SDE [2] is to transform the log-likelihood into the predicted score of the denoising network in a score-matching manner. We are inspired and employ the transformation in the derivation from Equation (5) to Equation (6) in our submission. The training of $\epsilon_{\boldsymbol \theta,\boldsymbol \phi^*}(\boldsymbol z_t,t,\boldsymbol c^{sub},c_\Delta^{tar})$ and $\epsilon_{\boldsymbol \theta,\boldsymbol \phi^*}(\boldsymbol z_t,t,\boldsymbol c^{sub},c_\Delta^{aver})$ are essentially two score-matching processes of the target sub-cluster score $\nabla_{\boldsymbol{x}}\log p(\mathcal{S}^{tar}\mid\boldsymbol{x})$ and the auxiliary sub-cluster score $\sum_{i=1}^{N-1}\nabla_{\boldsymbol{x}}\log p(\mathcal{S}_i^{aux}\mid\boldsymbol{x})$.

[2] Score-based generative modeling through stochastic differential equations

Question 2: Formula expression

There are many different ways to express the expectation of the task including $p(\boldsymbol{x})\cdot\frac{p(\mathcal{S}^{tar}\mid\boldsymbol{x})}{\prod_{i=1}^{N-1}p(\mathcal{S}_i^{aux}\mid\boldsymbol{x})+p(\mathcal{S}^{tar}\mid\boldsymbol{x})}$. We eventually adopted the simplified Equation (5) for the following reasons:

• Retaining only the multiplication and division terms simplifies the subsequent formula transformation.

• $p(\boldsymbol{x})\cdot\frac{p(\mathcal{S}^{tar}\mid\boldsymbol{x})}{\prod_{i=1}^{N-1}p(\mathcal{S}_i^{aux}\mid\boldsymbol{x})+p(\mathcal{S}^{tar}\mid\boldsymbol{x})}$ can be transformed into

  $p(\boldsymbol x)\cdot\frac{1}{\prod_{i=1}^{N-1}p(\mathcal{S}_i^{aux}\mid\boldsymbol x)/p(\mathcal{S}^{tar}\mid\boldsymbol x)+1}$.

  Thus, the core expectation is also to increase $\frac{p(\mathcal{S}^{tar}\mid\boldsymbol x)}{\prod_{i=1}^{N-1}p(\mathcal{S}_i^{aux}\mid\boldsymbol x)}$ as the Equation (5) does.

• During implementation, the target and auxiliary samples all contribute to the average condition, which completes the center of the whole cluster. So the simplification is more like a theoretical trick.

Question 3: Radius concept

We believe there exists the concept of radius in the latent space, yet it’s hard to precisely determine the radius. Because if we consider the $(\boldsymbol z_t,t)$ as a whole, the latent space is complicated and might not be a Euclidean space. The radius might appear to be a geometric manifold that extends continuously over $t$. It will be a interesting yet challenging future topic to develop a strategy to determine the radius.

Question 4: Advantages over attention manipulation methods

Compared with the training-free pipelines based on attention manipulation, our tuning-based pipeline has the following advantages:

• Our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which implements this research task into more practical applications.
• Based on the stable tuning process of an auxiliary network rather than the U-Net backbone, our method demonstrates more robust performance across different settings. By contrast, the attention manipulation to the U-Net backbone utilized by training-free baselines exhibits occasional degradations of the image quality (see the first and fourth images of the ‘man’ case generated by StoryDiffusion in the Figure 21 of the global rebuttal pdf; also see the first image of the ‘girl-cat’ case generated by FreeCustom in the Figure 22).
• Though our method takes averagely 5 minutes to tune, it doesn’t increase the inference time as training-free baselines do. Thus, as the generation number increases, our method outperforms the training-free methods in terms of total generation time (see the Table 3 in the global rebuttal pdf).

We believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!

We sincerely thank you for your valuable reviews. For the weakness and questions, we will response to each of them individually.

Weakness: Novelty of semantic interpolation

We believe our discovery is fundamentally different from previous works. Ordinary diffusion process can be denoted as $p(\boldsymbol{z}\mid \boldsymbol{c},s)$, where $\boldsymbol{c}$ is the semantic condition and $s$ is the weight of classifier-free guidance score. Previous works [1,2,3,4] prove that the semantic interpolation $\boldsymbol c'=\boldsymbol c_1+\alpha\cdot(\boldsymbol c_2-\boldsymbol c_1)$ between two conditional embeddings $\boldsymbol c_1$ and $\boldsymbol c_2$ shows a mixed visual effect and a gradual visual change, which can be written as: $p(\boldsymbol z\mid\boldsymbol c’,s)\approx p(\boldsymbol z\mid\boldsymbol c_1,s)\cdot p(\boldsymbol z\mid\boldsymbol c_2,s)$. This discovery is utilized to mix different input conditions for a fancy hybrid output image. By contrast, our contribution lies in proving that the semantic interpolation $\boldsymbol c'=\boldsymbol c_\emptyset+\alpha\cdot(\boldsymbol c_1-\boldsymbol c_\emptyset)$ between the unconditional embedding $\boldsymbol c_\emptyset$ and conditional embedding $\boldsymbol c_1$ has the same effect with the guidance interpolation $s'=0+\beta\cdot(s_1-0)$ between no guidance $s=0$ and standard guidance scale $s=s_1$. The above conclusion can be written as: $p(\boldsymbol z\mid \boldsymbol c',s_1)\approx p(\boldsymbol z\mid \boldsymbol c_1,s')$. It indicates that semantic manipulation has the same effect with the sampling guidance strength, which we believe is essentially different from the conclusion of previous works. Yet we apologize for the misinformation caused by our inappropriate wording and we will polish our phrasing to be more precise and objective. We will also add the above-mentioned works to the Related Work section in the new version of our paper. Thanks so much for your nuanced review!

[1] FaceStudio: Put Your Face Everywhere in Seconds

[2] PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding

[3] InstantID: Zero-shot Identity-Preserving Generation in Seconds

[4] FlashFace: Human Image Personalization with High-fidelity Identity Preservation

Question 1: Multi-entity scenes

We provided illustrations of double subjects generation and triple subjects generation in Figures 17 and 18 in the Appendix C of the submission. The results show that our method is able to deal with the complex 2-entity and 3-entity scenarios. The storybook generation in the Appendix B.2 also proves that our method is capable of complex sequential generation. Yet when dealing with more numerous subjects (usually >4), as discussed in the Appendix D, some subjects might be neglected. We also observe this defect in the original SDXL model and addressing this foundational limitation will be the focus of our future work.

Question 2: Comparison with StoryDiffusion

We carry out qualitative comparison with the contemporaneous work StoryDiffusion [5] in Figures 20 and 21 in the global rebuttal pdf. Generally, we think our method performs better in terms of subject consistency and background diversity. In a broader context, compared with the training-free pipelines like StoryDiffusion, our tuning-based pipeline has the following advantages:

• Our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which implements this research task into more practical applications.
• Based on the stable tuning process of an auxiliary network rather than the U-Net backbone, our method demonstrates more robust performance across different settings. By contrast, the attention manipulation to the U-Net backbone utilized by training-free baselines exhibits occasional degradations of the image quality (see the first and fourth images of the ‘man’ case generated by StoryDiffusion in the Figure 21 of the global rebuttal pdf).
• Though our method takes averagely 5 minutes to tune, it doesn’t increase the inference time as training-free baselines do. Thus, as the generation number increases, our method outperforms the training-free methods in terms of total generation time (see the Table 3 in the global rebuttal pdf).

[5] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

We believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!

Dear Reviewer 7cPE,

We sincerely thank you for your valuable reviews. To each of the question, we have provided detailed respones in the rebuttal stage, which we hope could address your concerns. If you have any further questions or there is any omission in our responses, please feel free to let us know. And we would be extremely grateful if you could consider increasing the rating after reviewing our responses.

Thanks again for your great efforts!

We sincerely thank you for your valuable reviews. For the weakness and questions, we will response to each of them individually.

Weakness 1: Limited styles and objects in the main part of the paper

The 9-page main part of the submission is limited in space so we placed the most crucial comparisons with the baselines in it and placed more visual results in the Appendix of the submission. Because the baselines (TheChosenOne and ConsiStory) have not yet open-sourced their codes, we can only compare to the results of their written materials, which accidentally limits the variety of the style and central subject in the main part. We showed a little more results of photorealistic styles and objects in the Appendix. B and C. Additionally, we provide more visual results concentrating on objects and photorealistic style in Figure 21 of the global rebuttal pdf. The results demonstrate that our method performs well in this setting.

Weakness 2: CLIP score is not insufficient

CLIP-I-score is the metric that the baselines utilize so we followed and reported it in the main part of our submission. Yet we completely agree that CLIP-I score is insufficient to measure the subject ID. Thus in addition, we provided a more comprehensive quantitative evaluation in the Appendix A.5, where we reported DINO score and LPIPS score as well. Beside these common metrics, we also notice that recently some human-aligned benchmarks are proposed to specially evaluate the image consistency, such as DreamBench++[1]. Due to the time limit, we are not able to perform comprehensive experiments on such benchmarks in the rebuttal stage, but we will try to add such experiments in the new version of our paper.

[1] DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation, arXiv2406.16855

Weakness 3: Cluttered figure

We will revise the Figure 2 to highlight the projector and the cluster guidance methodology. Thanks for your valuable advice!

Weakness 4: Qualitative ablation study

Since there is no space in the main text, we provided the illustration of component ablation study in the Appendix A.3 and parameter analysis in the Appendix A.4. Please refer to it for further review.

Weakness 5: Quantitative efficiency results

We provided an efficiency analysis in the Appendix A.2 and add an efficiency statistics in Table 3 of the rebuttal pdf. Though our method takes averagely 5 minutes to tune, it doesn’t increase the inference time as training-free baselines do. From the user’s perspective, when generating <20 images for one subject from scratch, ConsiStory is the fastest, followed by our method and then TheChosenOne. However, considering the application scenarios, the user is very likely to have a demand for large-quantity generation (>50 images). In this case, our method demonstrates definite advantages. For example, when generating 100 images, ConsiStory takes about 35 minutes while our method only needs about 23 minutes in total. If we set $\eta_2=0$, with an affordable compromise of generation details, our method will reduce the total time to about 18 minutes. With the generation number increasing, our advantages will continue to grow.

We believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!

Dear Reviewer SGyo,

We sincerely thank you for your valuable reviews. To each of the question, we have provided detailed respones in the rebuttal stage, which we hope could address your concerns. If you have any further questions or there is any omission in our responses, please feel free to let us know. And we would be extremely grateful if you could consider increasing the rating after reviewing our responses.

Thanks again for your great efforts!

We greatly appreciate your commendation of our work. It is without doubt a tremendous encouragement for us. For the weakness and question, we will response to each of them individually.

Weakness: Typo

We will carefully review our paper sentence-by-sentence and try to correct every type error. Thanks for your meticulous review!

Question: The design of the projector network

This is a very meaningful question which we've discussed a lot ourselves. In our pipeline, the role of the projector network is to transform a token embedding to another adjusted token embedding with respect to the visual feature. Its essence is a transformation inside one modality (semantic) conditioned on another modality (visual). It's similar to the style transfer task, which transfers one image style to another style with respect to the image elements. Inspired by this, we utilize a projector based on AdaIN, which is a popular transformation layer in the style transfer task. Then the ResNet and linear layer are naturally chosen to extract the visual feature and output the $\beta$ and $\gamma$ of the AdaIN layer. According to the experiments, our design functions well in the one-shot tuning setting. Furthermore, our method is potential to be utilized to pre-train a consistent subject generation model from scratch. In thus setting, we do believe there might be a more perfect network design for the projector.

We believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!