One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt

Thank you for your appreciation of our work and for your insightful feedback. Below, we outline our responses to the weaknesses and questions you brought up.

W1 & Q1: The Prompt Consolidation method concatenates all text prompts into a single input before feeding it to the text encoder. However, in my view, this approach faces a significant limitation for generating longer stories due to the token length restrictions of the diffusion model’s text encoder.
This is also a key challenge addressed in our method. To counter the token length limit of the CLIP model, we introduced a sliding window technique to overcome this limitation, enabling infinite-length story generation. The details of this technique are elaborated in Appendix D.2 and illustrated in Fig. 17.

W2 & Q2: Could you further clarify why the EOT needs to be initially expressed and then suppressed? Will you provide a brief explanation of the role of EOT tokens in the model?
The ablation study presented in Fig.11(left) of the Appendix confirms that the order of expression and suppression has minimal impact on the output, demonstrating the robustness of our approach. Additionally, the EOT token's capacity to encapsulate all preceding text aligns with observations in prior research [1,2]. By aggregating the cumulative frame prompt information, the EOT token serves as a dependable anchor for achieving consistent T2I generation. Our SVR method utilizes this property effectively, isolating and amplifying the expressed components while diminishing the suppressed ones, all while preserving coherence across frames.

[1]Senmao Li, Joost van de Weijer, Fahad Khan, Qibin Hou, Yaxing Wang, et al. Get what you want, not what you don’t: Image content suppression for text-to-image diffusion models. (ICLR2024)
[2]Yinwei Wu, Xingyi Yang, and Xinchao Wang. Relation rectification in diffusion model. (CVPR 2024)

W3: The user study sample size of 20 participants seems relatively small, which may limit the generalizability of the findings.
For the initial user study, we randomly selected 20 prompt sets across various categories, with each example comprising story generation images from four methods. The detailed information about the user study is introduced in Appendix.F and illustrated in Fig. 19.

To enhance the reliability and breadth of the study, we expanded it to include 37 questionnaires, each with 30 prompt sets of questions. The updated results, summarized below, provide more robust evidence of the effectiveness of our approach across diverse scenarios.

Thank you for recognizing our work and for your valuable feedback. Below, we offer a point-by-point response to the weaknesses and questions you raised.

Minor: L 353 (K, V) notations are different from those used later in the paragraph.
Thanks for your remind, we have already fixed the typo in Line 353

W1&Q1&Q2&Q3&Q4: Can you please confirm that you simply zero-out the token features of all prompts P_i, i \in [1,N] ,except for P_0 and EOT ? What is the rationale behind zeroing out features of frame-prompts p_i rather than removing them entirely? How do the P_0 and EOT features alone enhance identity consistency across frames? Can you please provide further motivation to the KV extension and why this generates more consistent characters.

Sorry for the misunderstanding and our vague description. Here we reply to the weakness point and your questions 1 to 4 in detail. What we meant by “zero_out” is to set the weight of ( K ) in the attention map for ( p_i ) to zero. In practice, we do not set the corresponding value of ( K ) to zero. Instead, we design an attention mask where all values in the column corresponding to ( p_i ) are set to zero, and other positions are set to one (for unchanging them). Then, we add its natural logarithm value to the original attention map. That effectively makes the position corresponding to ( p_i ) negative infinity, resulting in a probability of zero after softmax. In the implementation, we program it as:

“attn_weights_pos = torch. softmax(scores_pos + torch.log(mask), dim=-1)”

which corresponds to line 246 in utils.py in our code of the supplementary material. We already make the description clearer in the implementation details in the Appendix B.1.

Regarding the P0 and EOT features, it is another point that may lead to your misunderstanding. To clarify, after SVR, the EOT feature is modified, but the P0 feature remains unchanged as we do not apply any adjustments to it. During IPCA, we concatenate P0 and EOT before applying SVR. This approach has an additional advantage: the ID information in the unmodified P0 and EOT features may more closely align with the original shared identity in a single long prompt. Such observation can be seen in the ablation study in Table 3. In practice, we employ an additional branch to retrieve the original P0 and EOT features, the corresponding implementation is detailed in the supplementary material (util.py, line 665).

W2&Q6: The authors claim their method to be memory-efficient compared to existing methods. I think it is nice to know the memory complexity of the proposed method compared to existing ones.
We provide a GPU memory usage comparison across all methods. Among these, "The Chosen One" is a training-based approach. In contrast, our method only requires a single concatenation operation in the cross-attention of IPCA, resulting in a negligible increase in memory consumption for the cross-attention module. On the other hand, training-free methods like Consistory and StoryDiffusion execute operations within the self-attention mechanism, which significantly increases memory usage, especially when generating more than three frames. Unlike these methods, our approach maintains a constant memory footprint regardless of the number of generated images, allowing it to operate efficiently on a standard 24GB graphics card for consistent T2I generation with numerous frames.

Thank you for acknowledging our work and for your thoughtful comments. Below, we provide a detailed response to the weaknesses and questions you raised.

W1: Long prompts might pose a challenge to the applicability of the proposed method due to the limit of number of tokens
This is also a key challenge addressed in our method. To counter the token length limit of the CLIP model, we introduced a sliding window technique to overcome this limitation, enabling infinite-length story generation. The details of this technique are elaborated in Appendix D.2 and illustrated in Fig. 17.

Q1: For the experiment in Section 3.1.1, it seems that the t-SNE visualization is conducted for only a single set of prompts. It would benefit the clarity if more details can be provided.
Indeed, our primary focus is on internal identity consistency within the storytelling scenario. Since diverse prompt sets are with different text embeddings and are hard to draw in a single figure. Therefore, it is logical to visualize the textual embeddings within a single prompt set under two different configurations: our single-prompt setup and the conventional multi-prompt setup. t-SNE is a non-linear dimensionality reduction method and relies on the overall context during decomposition, which makes it suitable to measure the internal prompts similarities as shown in Fig.2. Furthermore, we present the average distance metrics in Fig. 2-right, showcasing the statistical results over the Consistory+ benchmark.

Q2: Instead of the proposed Identity-Preserving Cross-Attention technique, is it possible to improve the consistency by performing SVR on C_P0, C_EOT
Your intuition is absolutely correct, and this approach does show potential. In our experiments, we explored concatenating C_{P0} and C_{EOT}, followed by applying SVR+ to enhance identity consistency (ID). While this method helped preserve ID, we observed that it could inadvertently impact the visual quality of the images.

As an example in Fig.12 of the updated submission, performing SVR over P0 and EOT leads to high similarity of the backgrounds and poses across all frame images. In comparison, our method 1Prompt1Story shows better diversity in consistent T2I generation. We also aim to develop this idea in the future work.

Q3: Does the proposed method affect the visual quality/image realism? How is it compared with other methods?
We evaluate the Fréchet Inception Distance (FID) metric to assess the quality of image generation. We use images generated by the base model as the real dataset and images generated by each method itself as the fake dataset. The results indicate that our 1Prompt1Story approach does not compromise the original model's image generation quality. Additionally, our method preserves the diversity of text-to-image (T2I) generation, as demonstrated in Appendix C.4 (Fig. 14), where we present results using various random seeds.

Thank you for your detailed review and for offering such thoughtful and constructive feedback. Below, we present a point-by-point response to the weaknesses and questions you mentioned.

W1: Reweight and Identity-Preserving or their similar ideas have appeared in previous articles
In this paper, we include Naive Prompt Reweighting (NPR) as a baseline method. However, NPR has proven ineffective in story generation scenarios. Our proposed SVR method represents a stronger reweighting approach. Additionally, while common prompt weighting methods operate at the single-word level, our NPR baseline uniquely functions at the sentence level. We are the first to propose an enhanced reweighting technique like SVR, which we plan to extend to future applications.

Regarding identity preservation, our 1Prompt1Story method leverages the inherent properties of text embedding spaces, distinguishing it from JeDi in several ways:
(1) Training-free approach: Our method is inherently training-free as it relies on the identity information embedded within a single joint prompt. In contrast, JeDi depends on the creation of subject-grouped datasets for training coupled self-attention layers.
(2) Objective focus: While 1Prompt1Story ensures consistent T2I generation, JeDi targets T2I personalization. These methods are orthogonal and can complement each other. For instance, our method combined with PhotoMaker (as shown in Fig.8-right) can improve identity consistency while enabling real-image personalization.
(3) Generalizability: 1Prompt1Story is based on the foundational properties of language models, making it applicable to a broader range of scenarios. Conversely, JeDi relies on an LLM-augmented pipeline to generate large-scale, grouped subject datasets.

We already include JeDi in our related work and will incorporate this discussion to clarify these distinctions in the future version.

W2&Q2: The improvements for multi-IDs are NOT mentioned, which is always talked in recent works like StoryDiffusion(The example of a man and a dog).
By specifying multiple characters in the identity prompt, our method naturally extends to achieve multi-character ID consistency within a single generation. For instance, as demonstrated in Appendix D.1 (Fig. 15), we showcase results generated by our approach featuring two distinct characters, further validating its capability to handle complex identity setups.

Q1: Prove your novelty (I know it's a little hard for training-free methods in this field)
(1) We innovatively utilize a single-prompt approach instead of the conventional multi-prompt setups to achieve identity-consistent story generation. Unlike prior methods, which overlooked the inherent context consistency property of language models, our work is the first to explore and leverage this characteristic (Section 3.1).
(2) Both our Singular Value Reweighting (SVR) and Identity-Preserving Cross-Attention (IPCA) techniques are rooted in the context consistency observation. SVR refines the expression of the current frame’s prompt while attenuating contributions from other frames, ensuring frame-specific focus. Simultaneously, IPCA strengthens subject consistency within the cross-attention layers, preserving identity across frames. By applying these techniques, 1Prompt1Story achieves significantly more consistent T2I generation results compared to existing approaches.

Thank you for your thorough review and for providing such valuable and insightful feedback. Below, we provide a point-by-point response to the weaknesses and questions you raised.

W1: Stable diffusion models use a CLIP text encoder, which is limited to 77 tokens. This severely constrains the number of frame prompts that can be used.
This is also a key challenge addressed in our method. To counter the token length limit of the CLIP model, we introduced a sliding window technique to overcome this limitation, enabling infinite-length story generation. The details of this technique are elaborated in Appendix D.2 and illustrated in Fig. 17.

W2: Evaluating on a single benchmark of 200 examples seems insufficient. Similarly, a human study on just 20 examples seems quite limited.
We agree with your observation and have taken steps to address this limitation. To overcome this drawback, we developed the Consistory+ benchmark, which is three times larger than the original benchmark [1] in terms of the number of examples, categories, and overall diversity. Additionally, we expanded our experiments by incorporating more examples using bootstrapping, as per your suggestion.

For comparison, we selected the two best-performing methods and evaluated them under varying conditions. Specifically, we generated additional image sets using different random seeds, referred to as sample2 and sample3, while retaining sample1 as the original results from Table 1. Combined, these sets account for a total of 4,500 images (1,500 per sample set), providing a robust foundation for our analysis.

For the initial user study, we randomly selected 20 prompt sets across various categories, with each example comprising story generation images from four methods. The detailed information about the user study is introduced in Appendix.F and illustrated in Fig. 19.

To enhance the reliability and breadth of the study, we expanded it to include 37 questionnaires, each with 30 prompt sets of questions. The updated results, summarized below, provide more robust evidence of the effectiveness of our approach across diverse scenarios.

[1]Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. 2024. Training-Free Consistent Text-to-Image Generation. ACM Trans. Graph. 43, 4, Article 52 (July 2024), 18 pages. https://doi.org/10.1145/3658157

Q4: In figure 3, I don't see the NPR results as incorrect since the background isn't specified in frame prompts 2, 3, 4 and 5.
Thank you for bringing up this point. We agree that the lack of explicit background specification can lead to inconsistent or suboptimal generation, particularly with the NPR method. As noted, the 2nd and 4th frames in Fig.3 indeed demonstrate worse identity coherence when using NPR, reinforcing the need for a more robust handling of such scenarios. Our preset aligns with the original SDXL's behavior: in cases where no background is specified, it is preferable not to generate one. This ensures better focus on the primary content of the image. However, as highlighted in Fig. 13 and Fig. 20 (the second-to-last row), when backgrounds are explicitly specified in the frame prompts, NPR still encounters background confusion problems, compromising both spatial and thematic consistency across frames. This underscores the limitations of NPR when dealing with multi-frame prompts, particularly in complex storytelling scenarios. Our method 1Prompt1Story addresses this issue by prioritizing both identity consistency and background coherence, as demonstrated through our proposed approach in contrast with NPR in these figures.

Q5: Why does the embedding of the [EOT] token belong to both the expressed and suppressed sets? Won't the effects cancel each other out?
The EOT token's capacity to encapsulate all preceding text aligns with observations in prior research [1,2]. By aggregating the cumulative frame prompt information, the EOT token serves as a dependable anchor for achieving consistent T2I generation. Our SVR method utilizes this property effectively, isolating and amplifying the expressed components while diminishing the suppressed ones, all while preserving coherence across frames.

These two mechanisms do not cancel each other out during the enhancement. The enhanced prompt is combined with the EOT token, followed by singular value decomposition (SVD), where only the main singular values are selectively amplified. This approach assumes that the main singular values correspond to the essential components of the frame prompt requiring enhancement. Similarly, suppression applies the same principle but focuses on weakening less relevant components. Ablation experiments conducted with SVR+ and SVR- independently (Fig. 7, Table 3) demonstrate that combining both operations achieves superior results, highlighting the synergy between the two mechanisms.

[1]Senmao Li, Joost van de Weijer, Fahad Khan, Qibin Hou, Yaxing Wang, et al. Get what you want, not what you don’t: Image content suppression for text-to-image diffusion models. (ICLR2024)
[2]Yinwei Wu, Xingyi Yang, and Xinchao Wang. Relation rectification in diffusion model. (CVPR 2024)

Q6: For prompt alignment evaluation, I would suggest using more recent metrics that are more correlated with human judgment, such as DSG [1] or VQAScore [2].
Thank you for your suggestion. In our method, we have already incorporated the commonly used DreamSim metric to capture human preferences and identity consistency, and the CLIP-T metric is to measure the prompt alignment. Following your advice, we conducted additional comparisons of our approach with other methods using the DSG[1] and VQAScore[2] metrics. The results are presented below, where the SDXL model is listed as the baseline reference not for comparison. These evaluation results, along with the experiments in Table 1, prove the effectiveness of our method 1Prompt1Story in consistent T2I generation.

[1] Cho, Jaemin, et al. "Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation." ICLR 2024.
[2] Lin, Zhiqiu, et al. "Evaluating text-to-visual generation with image-to-text generation." ECCV 2024

Thank you for your thorough response. While some of my concerns have been addressed, I still have reservations regarding the significance of the results. In addition, the justification for expressing and suppressing the EOT token remains unclear. As such, I've increased my soundness score to 3, but I'm maintaining my overall rating at 5, as I'm still not convinced about the impact of this work.

I would like to clarify that bootstrapping is a technique to estimate the variance of a sample, not to increase the sample size. In addition, providing the means of 3 samples is not enough to compute the variance of each method and assess whether the improvements are statistically significant. For instance, the prompt alignment values between storydiffusion and 1prompt1story are very similar for the 3 samples.

In figure 8, I believe the identity preservation metrics are worse for the original PhotoMaker because in the second image the subject is not facing the camera. In addition, it seems that applying the proposed method reduces diversity (the subject has a very similar pose in images 1 and 3), which is not captured in the quantitative results.

In figure 2, I believe you could solve the issue by transforming all embeddings from both methods at the same time. Then the two sets of projected embeddings would be directly comparable. In addition, the same issue with shaded regions is present in figure 3, I encourage authors to fix that one as well.

For the prompt order ablation, testing on a single example is not convincing. To provide evidence that the method is robust to the frame prompts' order, it needs to be validated across a dataset of prompts by computing quantitative results.

Finally, the fact that VQAScore assigns a higher prompt alignment score to storydiffusion is in line with my previous concern that 1prompt1story might not be significantly better than storydiffusion in terms of prompt alignment.

Dear Reviewer Pz4Z,

Thank you for dedicating your time to review our paper, and for your continuous engagement during the discussion period. Your constructive comments have been invaluable, and we genuinely hope to get feedback from you. Regarding the question you mentioned, we include the corresponding experiments and discussion as follows:

1. Ablation Study on EOT: Our experiments (Fig. 13 in the revised version) demonstrate that without modifying the EOT token, the generated images tend to mix semantics across different frame prompts.

2. Bootstrapping: We applied the bootstrapping method to calculate the Clip-T, Clip-I, and DreamSim metrics. The results are consistent with those in Table 1, showing stable improvements of our method over training-free approaches.

3. PhotoMaker with and without 1Prompt1Story: We tested PhotoMaker with and without 1Prompt1Story using the Consistory+ benchmark, including ID similarity and diversity metrics. The results indicate that while diversity metrics are similar, 1Prompt1Story significantly improves ID similarity compared to PhotoMaker.

4. T-SNE Visualization: We applied T-SNE to visualize all embeddings from both settings (Fig. 18 in the Appendix). This confirms our earlier analysis, showing that frame prompts exhibit more consistent semantic information and identity preservation in the single-prompt setup.

5. Prompt Order Ablation: We tested random frame prompt orders on the Consistory+ benchmark, and the results show that prompt order has minimal impact on prompt alignment and identity consistency metrics.

6. Aesthetic Score and ImageReward Metric: We computed two additional metrics, aesthetic score [1] and ImageReward [2], to evaluate image quality, human preference, and prompt alignment. The results demonstrate that 1Prompt1Story outperforms other methods, consistently delivering higher image generation quality across various aspects.

Thank you again for your time and effort in reviewing our work and looking forward to your response.

Best Regards,
Authors of submission 903