Multi-Shot Character Consistency for Text-to-Video Generation

Thank you for your detailed feedback and thoughtful suggestions, as well as your willingness to increase your score based on improvements. Your comments on motion dynamics, novelty, and clarity have been instrumental in refining our work. In particular, your suggestion to evaluate our method using a stronger video model inspired us to conduct new experiments with T2V-Turbo-V2, yielding significantly enhanced motion dynamics. Below, we address your comments in detail.

• Lack of Motion Dynamics .. do these issues stem from limitations in the base model’s dynamic capabilities. If so, would using models with stronger dynamic performance … If the video dynamic problem is addressed, I am willing to increase my score.

  Following the reviewer's request, we applied our method to T2V-Turbo-V2 [Li et al., NeurIPS 2024], a recent state-of-the-art video model released in October 2024 that offers significantly larger motion dynamics and improved video quality. Results from this experiment—both qualitative (Figure 8, Appendix) and quantitative (Table 1, Appendix)—show that combining our approach with Turbo-V2 produces consistent subjects while achieving much more dynamic visuals. Specifically, the dynamic degree improves threefold, from 20 to 62, while maintaining the same level of text alignment. This improvement is also evident in the qualitative examples the reviewer ask about, featuring the rabbit and Muppet characters (see Figure 8, Appendix and the online website). There, our method combined with Turbo-V2 demonstrates noticeably enhanced motion dynamics compared to its application with VideoCrafter2.

  The static motion in Ours+VideoCrafter2 results reflects the limitations of the underlying pretrained model. However, our method still achieves a better consistency-dynamics balance than naive consistency approaches, as shown in the ablation studies (Figures 4 and 5) and the user study.

  [Li et al.], T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback, NeurIPS 2024

• Novelty.

  Our key novelties are twofold: (1) we demonstrate that query features (Q) encode both motion and identity—this is our main insight. This insight directly motivates (2) our two-phase query injection mechanism, which is specifically designed to address the challenges arising from this observation, particularly in preserving identity without compromising motion. For that, we adopt components from prior work in non-trivial ways to form a cohesive solution.

  Furthermore, our work is the first to address this problem setup for video generation and the unique challenges it presents. Indeed, we also introduce additional technical improvements, such as the frame-wise SDSA and the use of estimated clean image (pred-x0) for mask generation, rather than relying on internal network activations used in prior methods.

• Clarity.

  We thank the reviewer for this valuable feedback. We have improved the paper's clarity through:

  • Added Notations Section: A dedicated Notations section defining key terms (e.g., $Q_v$, $Q_c$, $Q_f$) to reduce ambiguity.

  • Aligned Structure: Revised the methods section to mirror ConsiStory's three steps using clear bold headers, making connections to prior work explicit: (1) SDSA, (2) Layout Diversity, (3) Refinement Injection.

  • Focused Content: Technical derivations and complex details were moved to the appendix, while the main text emphasizes core challenges before presenting the solutions.

  The project page has also been updated for better navigation.

• Improvement over Baseline Models.

  While we acknowledge that the method is not perfect and there is room for improvement in future work, we emphasize that it already achieves significant advancements over existing baselines. For instance, its consistency is rated higher by 66% of users compared to the best baseline and by 79% of users compared to the base model, all while maintaining competitive text-motion alignment.

• Overclaiming the Benchmark.

  We revised and downplayed the benchmark claim to reflect the scope of our evaluation. Our evaluation, based on 150 generated videos (see L431, L471), provides distinguishable error bars and ensures robust evaluation results. In response to this comment, we also report additional semantic alignment scores, including “text-similarity” (referred to as “overall-consistency” by VBench) and VBench’s “subject-consistency,” which measures the similarity between frames within the same video shot using DINO (see Table 1 in the Appendix).

Thank you for your thoughtful feedback, insightful questions, and support of our work. We greatly value your perspectives on scaling to longer videos, balancing identity preservation and motion quality, and combining methods like ConsiS Im2Vid with our approach. Below, we provide detailed responses to your questions.

• How do you envision this approach scaling to longer video sequences? What additional challenges might arise, and how could the method be adapted to handle them? In long video sequences, subjects undergo viewpoint variations, causing their appearance to evolve due to perspective changes. Extended attention mechanisms, while useful for maintaining consistency, often promote a single dominant viewpoint. This suppresses viewpoint diversity and hinders the ability to handle natural perspective shifts over time.

  One way to address this is to apply extended attention selectively, instead of connecting frames in a non-specific manner. For this, the video can be divided into shorter temporal segments, and attention is extended only across segments detected to share similar viewpoints. This approach may maintain subject consistency within segments while respecting viewpoint diversity across the sequence.

• Have you explored any strategies to further improve the balance between identity preservation and motion quality? Are there other techniques beyond query injection that could be investigated? Yes, there is definitely room for improvement in balancing identity preservation and motion quality. One approach we explored, though it did not yield significant improvements, was a training-free rank-one update to the network weights by copying average features from anchor videos, inspired by [Bau et al. ECCV 2020].

  Several potential strategies for improvement include leveraging temporal attention layers to link features across video shots, such as copying temporal features from one shot to another. Alternatively, generating a single long video, as done in VSTAR [Li et al. 2024], could help maintain stronger identity preservation. However, the challenge lies in creating clear-cut transitions between scenes, avoiding the scene-transition style of VSTAR videos, while ensuring alignment with text prompts.

  Another avenue is a training-based approach, such as those explored in works like [Zeng et. al CVPR 2024], which could allow for deeper optimization of the balance between identity and motion dynamics.

  References: [Bau et. al] Rewriting a Deep Generative Model, ECCV 2020

  [Li et. al] Generative Temporal Nursing for Longer Dynamic Video Synthesis, preprint 2024

  [Zeng et. al] JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation, CVPR 2024

• Comment on the strengths of ConsiS Im2Vid baseline approach and how it might be combined or compared with your Video Storyboarding method? The strength of the ConsiS Im2Vid baseline lies in its ability to generate video shots from a set of consistent reference images. However, it has key limitations: (1) it lacks a mechanism to enforce cross-shot consistency, leading to identity variations, even when conditioned on images with consistent subjects, and (2) it lacks text-prompt control, resulting in static or misaligned motion.

  Combining Im2Vid with our Video Storyboarding method is an excellent idea that could lead to a robust hybrid approach. Framewise-SDSA and feature refinement could enhance subject consistency across shots. That said, the challenge of injecting motion remains, as Im2Vid lacks text-prompt control. This could be addressed by generating a non-consistent video using a text-to-video model and applying an inversion technique to extract and inject motion activations into the Im2Vid framework. Alternatively, combining our approach with a pretrained model conditioned on both image and text (image+text-to-video) could enable simultaneous control over subject identity and motion dynamics.

Thank you for your constructive feedback and thoughtful suggestions, which have helped us improve the scope of our work. We appreciate your detailed comments on expanding baseline comparisons, exploring concise prompts, and evaluating multiple characters, as they helped us strengthen both our experiments and presentation. In particular, your points regarding VSTAR and broader evaluations led us to incorporate new results, which we believe address your concerns. Below, we respond to your comments in detail.

• Experiments: The paper misses some important baseline methods, e.g., VSTAR[1]. The paper only provides several comparison samples.

  We first note that VSTAR's code was released on October 10, after the ICLR submission deadline. However, in response to the reviewer’s request, we have now included both qualitative and quantitative comparisons to the VSTAR baseline. Please see the results in the Appendix (Table 1, Figure 11 and the online website). Overall, while VSTAR produces large motion dynamics, it struggles with prompt-specific control, often resulting in entire videos misaligning with text descriptions. Since it achieves consistency through continuous video generation, VSTAR is better suited for scene transitions rather than independent video shots.

  Additionally, regarding the scope of comparisons, our user study and automated metrics were conducted with 150 generated videos (see L431, L471). This sample size provides distinguishable error bars, ensuring robust evaluation results.

• Would a more concise description impact character consistency? For example … "A woman".

  Following the reviewer’s request, we have added qualitative results demonstrating our method’s ability to handle general subject categories (see Appendix Figure 9 and the online website). These examples show that our approach can successfully generate consistent videos for broad subject types like "woman" and "rabbit," indicating strong performance even with concise, superclass-level prompts.

  Overall, a more concise description generally increases the gap between our method and the baselines, as it amplifies variation between shots in the baseline methods. Note that we intentionally adopted prompts with a complexity level comparable to consistent-character image generation methods, such as Consistory. For example, prompts like "Cinematic, middle-aged female athlete" are not overly complex, especially when compared to the highly detailed descriptions used in works like Sora (e.g., “A stylish woman walks down a Tokyo street... She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick.” or “the 30-year-old space man wearing a red wool knitted motorcycle helmet”).

• How the proposed method performs with multiple characters.

  Following the reviewer’s request, we have added qualitative results demonstrating the capability to handle multiple subjects (see Appendix Figure 10 and the online website). By incorporating two subjects in the prompt of the zero-shot mask, our approach can consistently render multiple characters within the same scene, as illustrated by examples featuring girl-owl and boy-teddy bear pairs.

• Quality of consistency results .. in Fig.3, the color and style of clothes change across different video shots.

  We acknowledge that the method is not perfect and that there is room for improvement in future work. However, it is worth noting that in some cases, changes in color and style, such as the girl wearing different attire while surfing, align with the context of the action and user expectations. A rigid preservation of clothing or style across all shots might not always be desirable.

  That said, our method already provides significant improvements over existing baselines. For instance, its consistency is rated higher by 66% of users compared to the best baseline and by 79% of users compared to the base model, all while maintaining competitive text-motion alignment.

Thank you for your valuable feedback and thoughtful insights. They have greatly helped us refine the clarity, novelty, and evaluation of our work. We appreciate your recognition of our key insight that “query features encode both motion and identity.” We hope that our revisions, including clearer explanations of our method, additional experiments, and expanded evaluations, address your concerns effectively. Below, we address your concerns in detail. Below, we respond to your points in detail.

• Novelty

  Our key novelties are twofold: (1) as the reviewer highlighted in his point #3, the insight that "query features (Q) encode both motion and identity." This key observation directly inspired (2) our two-phase query injection mechanism specifically designed to address the challenges arising from this observation, particularly in preserving identity without compromising motion. We respectfully disagree with the notion that our method is merely an "A+B combination with minor modifications.". While the method builds on components from prior work, they are adapted in non-trivial ways to form a cohesive solution.

  Furthermore, our work is the first to address this problem setup for video generation and the novel challenges it presents. Indeed, we also introduce additional technical improvements, such as the frame-wise SDSA and the use of estimated clean image (pred-x0) for mask generation, rather than relying on internal network activations used in prior methods.

• Clarity of the method part.

  We thank the reviewer for this valuable feedback. We have improved the paper's clarity through:

  • Added Notations Section: A dedicated Notations section defining key terms (e.g., $Q_v$, $Q_c$, $Q_f$) to reduce ambiguity.

  • Aligned Structure: Revised the methods section to mirror ConsiStory's three steps using clear bold headers, making connections to prior work explicit: (1) SDSA, (2) Layout Diversity, (3) Refinement Injection.

  • Focused Content: Technical derivations and complex details were moved to the appendix, while the main text emphasizes core challenges before presenting the solutions.

• The key insight "self-attention query features (Q) encode both motion and identity" lack experimental results to demonstrate.

  Please see the experiment in Figure 4 and the discussion around L375. We demonstrate that injecting the query features from the vanilla videos results in strong motion preservation. At the same time, it causes a loss of subject identity (e.g., the Muppet’s color change), clearly indicating that the Q tokens encode both motion and identity information.

• The evaluation does not contain the overall video generation quality and the qualitative semantic alignment scores.

  In response to this comment, we now provide additional semantic alignment scores, including “text-similarity” (referred to as “overall-consistency” by VBench) and VBench's “subject-consistency,” which measures the similarity between frames within the same video shot using DINO (see Table 1 in the Appendix). We also note that our video generation quality scores are similar to those of the underlying pretrained model (see discussion in L474), which is why we focused on presenting two key metrics — multi-shot set consistency and dynamic degree — that are more distinguishable relative to the pretrained model.

• Results are not perfect, e.g., inconsistent hairstyles.

  We acknowledge that the method is not perfect and there is room for improvement in future work. However, we highlight that our method already provides a significant improvement when compared to existing baselines. For instance, its consistency is rated higher by 66% of users compared to the best baseline and by 79% of users compared to the base model, all while maintaining competitive text-motion alignment.

• Does the overall generation quality decrease after the proposed method?

  No, the overall generation quality does not decrease. Only the generated motion is somewhat reduced (e.g. the dynamic degree) when using the VideoCrafter2 pretrained model (see Figure 6). Following Reviewer ntRv's comment, we included additional results using another SoTA pretrained video model (T2V-Turbo-V2). In this case, the dynamic degree is only slightly affected (see Table 1 in the appendix).

• How does the motion quality changed after the proposed method?

  According to the user study (Figure 7, right), 55% of our generated motions were judged to be of similar or better quality compared to the vanilla model.