StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

We wish to express our sincere gratitude to the reviewer for the thorough review and constructive feedback. Following the suggestion of the reviewer, we provide detailed responses in the hope of addressing the reviewer's concerns.

Q1: Additional Comparison Methods.

We are thankful to the reviewer for the feedback. We have run extra experiments and the visualization results are included in Figure 2 of the uploaded PDF in the rebuttal. We will cite these three papers and include comparisons with them in the revised paper. Compared with them, our method performs better and is more flexible with less inference time as it requires only a single generation pass. In contrast, "The Chosen One" requires time-consuming LoRA self-training for each example; "Zero-shot Generation of Coherent Storybook" requires first generating the images and then performing Iterative Coherent Identity Injection embedding; "Consistory" iteratively calculates the segmentation mask during the diffusion process to improve consistency and can also be considered concurrent work with ours due to the close timing.

Q2: Experiments about the "plug-and-play" characteristics.

We are very appreciative of the reviewer's feedback. We have additionally implemented our StoryDiffusion on SD1.5 and SD2.1, and put the result compared SDXL into Figure 1 of the rebuttal PDF. Our method maintains good performance when integrated into different models, which demonstrates the "plug-and-play" characteristics.

Q3: Efficient analysis.

Based on the reviewer's suggestions, we performed an efficient analysis of consistent image generation and transition video generation:

(1) Our consistent self-attention is training-free, which means we are more efficient than the models needed to train at large datasets (IP-Adaper or PhotoMaker) or tune models at inference time (The chosen one). We have listed the average time required to generate a 1024x1024 image, training parameters, as well as the amount of training data used below. Our inference time is similar to that of the comparison methods, but it does not require extensive additional pre-training, which demonstrates that our method is efficient.

(2) For video, our inference time and parameter count are comparable to other methods; Meanwhile, we achieve significantly better performance.

Q4: “For objective metrics, consider including FVD. For the user study, increasing the number of participants”

(1) We sincerely thank the reviewer for the feedback. We have expanded the quantitative experiments accordingly following the reviewer's suggestions. We first have supplemented the FVD comparison results:

(2) Additionally, to further address the reviewer's concerns, we have also added the FID results calculated on the frames:

(3) Furthermore, for the user study, following the reviewer's suggestions, we have increased the number of participants and organized a study with 79 participants. Our survey also included participants from a diverse range of professions and knowledge backgrounds to avoid cluster bias.

Firstly, we would like to thank the reviewer for acknowledging our efforts and contributions. Additionally, we deeply appreciate the reviewer's constructive feedback and would like to reply to the question raised by the reviewer in detail.

Q1: "The Method section lacks clarity"

We are very thankful to the reviewer for pointing out these detailed issues. We will revise our paper to address the clarity issues pointed out by the reviewer to enhance clarity:

a. We will add the submodule to Figure 2 to illustrate how we use GPT-4 to split the story text into several prompts.

b. We will add the format of prompts to Section 4.1 for clarity. The format of our character prompts is "[adjective] [group or profession] [wearing clothing]" and the format of activity prompts is “[action] [location or object]”.

c. We will add this statement to clarify that T in Line 182 and Equation 6 refer to the prompt corresponding to the previous image.

Q2: "could be time-intensive and potentially result in less smooth transitions."

We would like to address the reviewer's concerns through the following aspects:

(1) For the time consumed, we compared our method with comparable methods of the average generation time for a 16-frame 512x512 video. Our StoryDiffusion's average inference time is comparable to SEINE and significantly better than SparseCtrl, demonstrating that our method is not time-consuming compared to previous approaches.

(2) For the smooth transition, although our method generates transitions for every two consecutive keyframes, as demonstrated in the videos uploaded in the supplementary materials, our method can produce smooth long videos. The reason may be that our method learns from a large number of video clips, making each video clip coherent on its own. Additionally, we acknowledge that long video generation is a very challenging problem in the "Limitations" section of the paper, which may not be completely resolved within a single work. Since the video files in our supplementary materials have already demonstrated effectiveness, we will further explore how to establish connections between multiple short videos to make the transition smoother in our future work.

Q3: "the dog no longer wears the blue vest."

We sincerely thank the reviewer for pointing out the ambiguity. The "blue vest" appears only in the first image just because the corresponding prompt contains "blue vest," while the other prompts do not. We will add the prompts to Figure 6 (a) of the main paper to prevent misunderstanding.

Q4: "Why is the current implementation with a specific tile size preferred?"

We sincerely thank the reviewer for the feedback on the tile size. Overall, we use a fixed tile size for stability and robustness. To better illustrate this, we conducted an additional experiment, which we present below. Increasing the tile size and reducing the sampling rate within certain limits indeed enhances consistency. However, beyond a certain point, further increases result in decreased character consistency. Additionally, we may not need such a large number of images at once in most cases. Therefore, we chose to use a slightly smaller tile size and a larger sampling rate after considering multiple factors. We appreciate the feedback and will make revisions in the method section to better help readers understand this point.

Q5: "Are any segmentation or bounding boxes applied?"

Yes, in the process of calculating character similarity, we first use the SOTA background removal method RMBG-1.4 to eliminate the background. Then, we use only the images of the foreground characters for calculating character similarity. We sincerely thank the reviewer for pointing this out and will add the relevant details to our paper for clarity.

Q6: "How does the paper define an image token?"

We greatly appreciate the reviewer for pointing out this issue. In the attention modules of the UNet, the 3-dimensional HxWxC latent feature in VAE space is flattened into a 2-dimensional sequence of HWxC. We refer to the 1xC vector in this HxW sequence as an image token. We will add the definition of "image token" at the beginning of the Methods section to avoid any confusion

Firstly, we sincerely thank the reviewer for the thorough review and constructive feedback. We also appreciate the positive rating provided by the reviewer. After carefully reading the comments, we hope to address the reviewer's concerns in the following:

Q1: "(whether) consistent self-attention is robust across random seeds, and comparisons with other token schemes."

We are very thankful for the feedback and hope to address the reviewer's concern from three aspects: intuition and insights, the robustness of the method, and the quantitative evaluation of the method as below:

(1) The insight behind random sampling is to avoid reinforcing fixed characters' poses across different images, which is probably reducing the controllability of the text prompts over the generated images. This trend is also reflected in the newly added experiment shown in the table in part (2). The introduction of the sampling ratio is to achieve the best trade-off between character consistency and text controllability. Besides, a lower sampling ratio indeed effectively reduces the computational cost of self-attention, reducing GPU memory consumption.

(2) To further address the reviewer's concerns, we add a quantitative evaluation for the impact of the random sampling ratio and the comparison with the Image Grid method. As shown in the table, while the non-random GridSampling method indeed results in better character consistency, the controllability of the text prompt is reduced. More visualization results are also shown in Figure 3 in the uploaded file in the rebuttal. In summary, the introduction of the sampling ratio is to achieve the best trade-off between character consistency and text controllability.

(3) All reported results in the paper are average values over three different random seeds. We will add more details in the revision to make this clearer. Under this setting, our proposed method still achieves consistently higher performance than the selected baseline methods.

Q2: "Novelty, ... (Consistent Self-Attention) appears to be a post-processing technique"

We respectively remind the reviewer that our contributions are two-fold: (1) a new block to generate consistent sequence of images and (2) a novel semantic motion predictor to convert the sequence of images into dynamic videos with infinite length. The novelty of each point is explained below for your reference:

(1) We hope to clarify that Consistent Self-Attention is not a post-processing trick. It is used to modify the UNet architecture and operate together with the generation process. Even though there are some self-attention mechanisms in video editing tasks, the purpose and focus of our consistent self-attention are different as mentioned in Q3.

(2) We propose a new framework for long video generation with high dynamics. We modify an image model with CSA in a training-free manner and the semantic motion predictor transforms the sequence of images into videos with meaningful and dynamic motions. Compared to traditional text-to-video approaches that treat the entire process as a black box, our two-stage method, offers better controllability. Because it allows dynamic adjustment of keyframes to modify video results, aligning with the demands of film production.

Based on the above justification, we hope the reviewer could re-assess the contributions of the proposed framework.

Q3: "Discussion with other attention mechanisms"

We thank the reviewer for raising those extra works. Some of the papers have been discussed in the main paper (E.g. Tune-A-Video). The major differences between StoryDiffusion and the rest of the works are summarized below and will be added in the revision: The attention mechanisms used in the mentioned papers are designed for highly similar consecutive video frames, and adopt specific methods for selecting keys and values in the attention mechanism to enhance coherence of video: "Tune a video" uses the first frame and the previous frame as references for selection and requires additional fine-tuning of the "q linear" to align queries; "Text2video-zero" uses the first frame as keys and values; "DirecT2V" uses a temporal rotational selection of keys and values; Tokenflow uses a frame set with fixed frame intervals as keys and values.

Differently, our method targets to generate consistent characters with large dynamics that allow strong text controllability. The generated images could vary from each other to a large extent without direct motion continuity between adjacent images. In this case, these methods for selecting keys and values are not applicable. Differently, consistent self-attention emphasizes the mutual interaction between images within a batch to enhance consistency, rather than simply selecting some frames as key and value for unidirectional interaction.

Q4: What is the main motivation for not fully fine-tuning the model with the datasets but only the module?

The main motivation for freezing the spatially related model parameters is to save the computational cost. As we have the first frame and the last frame as prior knowledge, this practice significantly saves the training cost with acceptable video generation performance.

Q5: haven't discussed the potential social impact.

We kindly remind the reviewer that the potential social impact has been included in the Broader Impact paragraph in Sec. 5. (Line 275-279) We will try to make it clearer in the revision.

First, we are extremely grateful that the reviewer recognizes our efforts and contributions. At the same time, we also appreciate the reviewer's constructive feedback and would like to respond specifically to the following aspects.

Q1: Improve ablation experiments of Random Sampling.

Special thanks for your insightful feedback. We also agree that a more systematic quantitative evaluation of the sampling strategy would make the paper better. Thus, we have run some quantitative evaluations as shown below:

Insights & Analysis: The initial reason for using random sampling is to avoid reinforcing fixed characters' poses across different images, which is probably reducing the controllability of the text prompts over the generated images, visualized in Figure 3 in the rebuttal PDF. This trend is also reflected in the newly added experiment shown in the above table. The introduction of the sampling ratio is to achieve the best trade-off between character consistency and text controllability. Besides, a lower sampling ratio indeed effectively reduces the computational cost of self-attention, reducing GPU memory consumption, but this is not the initial purpose of doing so. There are also many engineering implementations to address memory consumption, such as the tiling method demonstrated in supplementary materials, or saving sampled tokens for reuse when inferring other images.

Q2: Transferring semantic motion predictor details from the appendix to the main paper.

We are very thankful for the reviewer's suggestions. We will adjust the structure of the article by moving the implementation details section to the first subsection of the experiments chapter to improve the readability of the paper in the revision.

Q3: Meaning of the word "plug-and-play".

Thanks to the reviewer for pointing out this issue. The difference between plug-and-play and training-free is that plug-and-play implies compatibility with multiple different models, whereas training-free merely emphasizes that no training is required, without indicating whether the method can be extended. We state in the paper that our consistent self-attention (SA) is plug-and-play because our method is implemented based on the properties of self-attention.

In our experiments with consistent self-attention, we found that consistent SA works effectively across multiple models, such as Stable Diffusion 1.5, Stable Diffusion 2.1, and Stable Diffusion XL. Therefore, we believe that SD is plug-and-play. We have presented the experimental results in Figure 1 of the upload Rebuttal PDF.