VideoAlchemy: Open-set Personalization in Video Generation

"Ablation Study."

"Comparison Fairness."

As suggested, we conduct an ablation study with three control factors. The results are shown as follows.

[Different Image Encoders] We train the models with two encoders, CLIP and DINOv2, and find that the former yields better text similarity while the latter achieves better subject similarity. We hypothesize that DINOv2 is learned to minimize the self-supervised training objective and can capture unique features in an image. In contrast, CLIP is learned to bridge visual and textual modalities and focuses on details typically described in the prompt which can help the model generate text-aligned videos.

[Necessity of Binding Image and Word Concepts] In Section 3.2, we bind word and image concepts by concatenating their embeddings. Empirically, we find that, without such binding, the model could apply image conditioning to an incorrect subject (e.g., put the input human face on a dog), which can lead to the missing of other subjects and lower subject similarity scores.

[Effect of Training Augmentations] Section 3.3 introduces image augmentations during training to mitigate undesirable training data bias. Empirically, we find that training the model without augmentation can make it achieve better subject fidelity by copying and pasting the reference image into videos, which is not preferable and could lead to the degradation of the video dynamic.

We will include both qualitative and quantitative comparisons in the paper.

"​​Lack of comparisons with multi-subject video customization methods."

To the best of our knowledge, the existing multi-subject video customization methods, such as VideoDreamer, CustomVideo, and DisenStudio, do not release either the code or checkpoint. In addition, they are all arxiv papers at this moment.

We also want to highlight that compared to multi-subject video customization, single-subject personalization is a relatively easier task and we have shown that VideoAlchemy surpasses the existing single-subject personalization model with a huge margin.

We thank the reviewer for the insightful and valuable comments. Here are our responses:

"Limited quality and diversity of the training and evaluation datasets."

"The video examples provided are not diverse enough for multi-subject video customization."

To illustrate the diversity of our training dataset, we will demonstrate our model’s generalizability by including more synthetic videos with diverse conditional subjects (e.g., dog, cat, horse, car, dinosaur). We will also include more videos with multi-subject conditioning. In the meanwhile, we would like to emphasize that the word cloud is generated from a subset of the training dataset (10k samples out of a total of 37.8M videos) and thus does not fully represent the entire dataset's distribution.

For the evaluation dataset, Figure 2 in the MSR-VTT paper confirms a diverse range of video content, covering topics such as sports, vehicles, and animals. Empirically, we observe that our testing dataset includes varied animals, including wolves, deer, and horses. We will visualize the qualitative comparison on these samples in the paper.

"Although the differences from the IP-Adapter are mentioned in the method part, the similar idea of using separate cross-attention layers for text and image embeddings has been explored in previous work"

To equip VideoAlchemy with multi-subject, open-set personalization capabilities for both foreground objects and backgrounds, we design the model with two key innovations. One is separate cross-attention for image conditioning. The other is the binding of image and word concepts. To facilitate multi-image conditioning, it is necessary to provide the pairing between the conditional images and entity word tags. As such, we develop a novel algorithm to bind conditional images with entity word tags. We will include an ablation study to demonstrate the necessity of binding images with entity word tags.

"It would be better to introduce new metrics, like temporal consistency or subject motion intensity as in VBench, to enhance the benchmark." "Baseline Selection."

As suggested, we include Dynamic Degree and Temporal Consistency to evaluate the temporal quality. Additionally, we remove the application of the image animation algorithm and directly treat the outputs of the image models as single-frame videos for a fair evaluation. We also include the comparison with two optimization-based baselines: DreamVideo and Magic-Me. The results are shown as follows.

[Subject mode]

[Face mode]

Compared to the existing open-set personalization models, VideoAlchemy achieves notably higher fidelity while simultaneously having better text-aligned and temporally consistent videos with larger dynamics. More interestingly, our open-set model can also achieve significantly higher face fidelity than the models specific to the face domain.

For the fairness of the comparison, the ablation study is conducted with the identical model architecture and the training data and can demonstrate the effectiveness of our proposed component.

We thank the reviewer for the insightful and valuable comments. Here are our responses:

"Lack of Ablation Studies"

As suggested, we conduct an ablation study with three control factors. The results are shown as follows.

[Different Image Encoders] We train the models with two encoders, CLIP and DINOv2, and find that the former yields better text similarity while the latter achieves better subject similarity. We hypothesize that DINOv2 is learned to minimize the self-supervised training objective and can capture unique features in an image. In contrast, CLIP is learned to bridge visual and textual modalities and focuses on details typically described in the prompt which can help the model generate text-aligned videos.

[Necessity of Binding Image and Word Concepts] In Section 3.2, we bind word and image concepts by concatenating their embeddings. Empirically, we find that, without such binding, the model could apply image conditioning to an incorrect subject (e.g., put the reference human face on a dog), which can lead to the missing of other subjects and lower subject similarity scores.

[Effect of Training Augmentations] Section 3.3 introduces image augmentation during training to mitigate undesirable training data bias. Empirically, we find that training the model without augmentation can make it achieve better subject fidelity by copying and pasting the reference image into videos, which is not preferable and could lead to the degradation of the video dynamic.

We will include both qualitative and quantitative comparisons in the paper.

"Unclear Methodological Contribution of the Model"

To equip VideoAlchemy with multi-subject, open-set personalization capabilities for both foreground objects and backgrounds, we design the model with several changes from the existing methods. We highlight two key innovations on the model design:

[Binding of Image and Word Concepts] To facilitate multi-image conditioning, it is necessary to provide the pairing between the conditional images and entity words. As such, we develop a novel mechanism to bind the images with the corresponding entity words. We will include an ablation study to demonstrate the necessity of providing such binding information.

[Separate Cross-attention for Image Conditioning] While IP-Adapter proposes a single decoupled cross-attention layer for both text and image conditioning, we empirically find that separate cross-attention layers work better for our case. We attribute this fact to two reasons:

• First, multi-image conditioning introduces a longer sequence of image conditioning tokens. Therefore, mixing text and image tokens in a shared layer would lead to image tokens dominating the synthesis process and reducing text-video alignment.
• Second, we apply T5 embeddings for text and DINOv2 embeddings for images. The mismatch of the embeddings distribution disables us to use a single cross-attention for both conditioning. Note that IP-Adapter uses CLIP embeddings for both conditional text and image. However, our experiments (see Tables 1 ~ 3 and Figures 5 ~ 6) reveal that DINOv2 image embeddings yield higher subject fidelity than CLIP.

"Limited Baseline Comparisons"

As suggested, we additionally include the comparison with two optimization-based baselines: DreamVideo and Magic-Me. The results are shown as follows.

[Subject mode]

[Face mode]

Compared to the existing open-set personalization models, VideoAlchemy achieves notably higher fidelity while simultaneously having better text-aligned and temporally consistent videos with larger dynamics. More interestingly, our open-set model can also achieve significantly higher face fidelity than the models specific to the face domain.

For the fairness of the comparison, we provide the ablation study above where the model architecture and the training data are identical, demonstrating the effectiveness of our proposed component.

We thank the reviewer for the insightful and valuable comments. Here are our responses:

"The pipeline for dataset construction doesn't impress me much. Using Grounding DINO, SAM, or face cropping are seen in VideoBooth, Boximator, and ID-Animator."

Our goal is to build a video model equipped with multi-subject, open-set personalization capabilities for both foreground objects and backgrounds. This goal distinctly shapes our dataset collection pipeline and sets it apart from existing approaches. We highlight three key innovations:

[Multiple Entity Word Tags Extraction and Classification] Different from previous methods, we utilize a LLM to extract multiple entity words from a text prompt and classify them into three categories: subject, object, and background. Additionally, we apply rigorous filtering criteria (as specified in lines 200 ~ 203 and Appendix A.2) to manage the increased complexity of retrieving multiple entity word tags within a single prompt.

[Learning Background Conditioning through Inpainting] Our model uniquely supports background conditioning. To achieve this, we remove foreground elements by their dilated masks, and apply inpainting to create a clean background (as described in lines 211 ~ 215).

[Subject Image Extraction from Multiple Frames] For each subject, we capture three images from distinct video frames, offering two advantages. First, it introduces variations in pose and lighting, addressing the copy-and-paste problem, a common issue of reconstruction-based personalization models. Second, this dataset enables the training with multi-image conditioning for individual subjects. Tables 1 and 2 demonstrate that multi-image conditioning can enhance subject fidelity in video synthesis.

"The differences between the proposed method and IP-Adapter are minimal, and further discussion would be beneficial."

Compared to IP-Adapter, our approach incorporates three essential modifications to enhance high-quality multi-subject personalization:

[Binding of Image and Word Concepts] To facilitate multi-image conditioning, it is necessary to provide the pairing between the conditional images and entity words. As such, we develop a novel mechanism to bind the images with the corresponding entity words. We will include an ablation study to demonstrate the necessity of providing such binding information.

[Application of Image Augmentation] While both IP-Adapter and VideoAlchemy adopt a reconstruction-based training, we observe that this approach can lead to undesirable training data bias and a copy-and-paste issue, as described in Section 3.3. To mitigate this issue, we introduce image augmentations during training. We will provide an example illustrating the copy-and-paste limitation in IP-Adapter and include the ablation study to demonstrate the effectiveness of our proposed image augmentation strategy.

[Separate Cross-attention for Image Conditioning] While IP-Adapter proposes a single decoupled cross-attention layer for both text and image conditioning, we empirically find that separate cross-attention layers work better for our case. We attribute this fact to two reasons:

• First, multi-image conditioning introduces a longer sequence of image conditioning tokens. Therefore, mixing text and image tokens in a shared layer would lead to image tokens dominating the synthesis process and reducing text-video alignment.
• Second, we apply T5 embeddings for text and DINOv2 embeddings for images. The mismatch of the embeddings distribution disables us to use a single cross-attention for both conditioning. Note that IP-Adapter uses CLIP embeddings for both conditional text and image. However, our experiments (see Tables 1 ~ 3 and Figures 5 ~ 6) reveal that DINOv2 image embeddings yield higher subject fidelity than CLIP.

"Temporal metrics were not used in the experiments, even though temporal consistency is another crucial factor in video alongside quality."

As suggested, we include Dynamic Degree and Temporal Consistency to evaluate the temporal quality. The results are shown as follows.

[Subject mode]

[Face mode]

We show that VideoAlchemy can produce videos with text-aligned content, high-fidelity subjects, and appropriate video dynamics while maintaining temporal consistency.

"​​The experiments lack comparison with multi-subject video customization methods."

To the best of our knowledge, the existing multi-subject video customization methods, such as VideoDreamer, CustomVideo, and DisenStudio, do not release either the code or checkpoint. In addition, they are all arxiv papers at this moment.

We also want to highlight that compared to multi-subject video customization, single-subject personalization is a relatively easier task and we have shown that VideoAlchemy surpasses the existing single-subject personalization model with a huge margin.