ActAnywhere: Subject-Aware Video Background Generation

We thank the reviewer for their thoughtful feedback and for acknowledging that the introduced problem possesses originality, the high quality of the results demonstrates the effectiveness of the proposed model, and that the paper is clearly written. We next address the reviewer’s questions and comments.

Video inpainting / outpainting is a fairly well-known challenge…

As discussed in L40-L44 and L100-L112 of the submission, the key challenge of our task compared to general video inpainting / outpainting lies in generating dynamic interactions and large background regions that follow the image condition. This is acknowledged by reviewers Pq5G and gAcX. It is nontrivial how to extend the video inpainting / outpainting frameworks, which generally tackle a pixel harmonization problem, to solve these challenges. Also, in Appendix 6.1, we showed that our model, once trained, exhibits general video inpainting / outpainting capabilities, while general inpainting / outpainting methods are not able to solve our proposed task.

The modeling approach is very similar to Gen1 [13]

Our key contributions include 1) introducing the novel problem of automated subject-aware video background generation with a background image as condition, and 2) proposing a specific video diffusion-based pipeline to solve this problem. Prior works such as Gen1 [13] are not able to solve the introduced problem, as demonstrated qualitatively and quantitatively in Sections 4.2 and 4.3 of the main manuscript. We found empirically that our proposed framework can effectively tackle the problem.

Unconditional generation with the model

Thank you for the question. We have performed the requested experiment and included the results in Fig. 2 of the attached PDF. Specifically, we set the condition to zeros, same as the random condition dropping described in L192-L194 of the submission. We sampled two subjects from the HiC+ dataset, and for each we ran our model with three different seeds. From each of the three generated videos, we selected one frame to show its input segmentation along with the generation. We observed that the model can generate reasonable random backgrounds that fit the foreground.

Watermark in the generated backgrounds

Webvid* videos are a major source of the HiC+ dataset, which all contain the “shutterstock” watermark. Training on them provides a dataset bias such that when conditioned on segmentations or a condition frame with the watermark, the generated results will also contain and complete (if the watermark appears partially in the input) the watermark. As the reviewer pointed out, this is an example of the model using foreground context to inform the background generation.

Despite that this issue is orthogonal to the focus of this work, we are aware of methods that can alleviate watermarks appearance in the generation, e.g. the “Domain Adapter” in AnimateDiff [17]. We leave the integration of such methods in our framework to future work.

*Frozen in Time: A Joint Video and Image Encoder for End to End Paper. Bain et al. ICCV 2021.

We appreciate the positive feedback from the reviewer and thank them for acknowledging that our introduced problem is novel, the proposed method is effective and makes a significant contribution to the movie and VFX industries, and that our paper is well-written. We address the reviewer's individual questions and comments below.

Lacks sufficient comparison with recent works on video generation and editing

To the best of our knowledge, the work with the closest setting to ours is AVID*, which can perform a task termed “Environment Swap” by editing the background region with a text condition. However, AVID does not take image conditioning, and hence cannot make the generated video follow the strict guidance specified by an image as we do. Moreover, from the results the authors showed on the website, they do not manifest realistic foreground-background interactions as ours (e.g. the sand does not deform according to the woman / tiger’s movement), and fail to generate correct shadows (the shadows are simply carried over from the input). We also note that this work was published recently in CVPR which was after the NeurIPS submission deadline, and is not open sourced yet which makes it hard to compare with.

Apart from this work, we believe others study different settings from ours and are generally non-trivial to extend to work under our setting. We are happy to compare any specific methods that the reviewer may suggest.

*AVID: Any-Length Video Inpainting with Diffusion Model. Zhang et al. CVPR 2024.

It seems that the model relies heavily on the quality of the foreground segmentation

As referenced in L204 of the main manuscript and discussed in Appendix 6.1, our model is in fact quite robust to inaccurate masks (Fig. 8), thanks to our designed data augmentation and processing procedures (i.e. random cut-outs and image erosion to the segmentation and mask) as noted in Sec. 3.4 of the main manuscript and Appendix 6.2.

How scalable is the model for generating longer / variable-length video sequences?

Due to the modular nature of our framework, we can easily scale up to generating longer videos if we swap to a different base model. The choice of generating 16-frame videos aligns with many previous works (e.g. [9, 17] for 16-frame generations, and [24] for 8-frame generations) and is primarily limited by the compute resource we had. Generating longer videos is out of the scope of this work, and we will explore this in future work with the latest DiT backbone.

We thank the reviewer for their positive feedback and for acknowledging that our introduced task is novel, and that the experimental comparisons are under a fair setup. We address the reviewer’s particular questions and comments below.

Training resolution is low… The method of using reference image does not seem fully considered, e.g. the method similar to IP-Adapter?

There is no fundamental limitation for our method to work under higher-resolution settings, granted a base model under a higher resolution and more compute resources.

Compared to the method of condition encoding in IP-Adapter, we only support image conditioning and thus do not need to map image and text to a joint space, hence we do not need the Decoupled Cross-Attention module as in their framework. Similar to the image encoding method in IP-Adapter though, we also have a linear layer to project the CLIP image features to a lower dimensional feature space, as noted in L179-L182 of the main manuscript.

Are we finetuning only the spatial layers, only the motion module, or both?

Thank you for the question. We finetune both the spatial layers and the motion module layers at the same time. We will clarify this in the final version of the paper.

Technical framework is very similar to SD-inpainting

Our key contributions include 1) the introduction of the novel problem of automated subject-aware video background generation with a background image as condition, and 2) the proposal of a specific video diffusion-based pipeline that is carefully designed and tailored to solve this problem. SD-inpainting works only under the image setting, and it is unclear how to extend it to videos. Our proposed framework, along with the designed self-supervised training pipeline and data augmentation strategies, contribute to our solution altogether.

We appreciate the positive feedback from the reviewer and thank them for acknowledging that our introduced task is interesting and that our proposed model achieved competitive results. We next address the reviewer's individual questions and comments.

CLIP encoder alone cannot contain detailed information… Compare to the approach of AnimateAnyone using ReferenceNet.

We thank the reviewer for the suggestion. We would like to note that detail preservation of the condition is still an open question. And because our framework is fairly general and agnostic to the image encoder, we can swap the CLIP encoder with any encoder with ease, e.g., DINO* if object-centric features are more desirable. ReferenceNet is an interesting alternative, but AnimateAnyone is not open sourced, which makes it hard to directly compare. Nonetheless, we are happy to compare if the reviewer insists.

*Emerging Properties in Self-Supervised Vision Transformers. Caron et al. ICCV 2021.

*DINOv2: Learning Robust Visual Features without Supervision. Oquab et al. TMLR, 2024.

Provide video demos. Blurry boundaries may exist in the videos.

We already included extensive video results in the easily accessible supplementary webpage, as mentioned in L206-L208 of the main manuscript. We did not observe any particular cases with blurry boundaries.

Task relevant to human animation. Show results on the TikTok dataset.

We note a key difference from the human animation task that the goal is in fact the opposite - we are given the foreground motion and try to generate the moving background. The TikTok dataset has a different focus to generate the foreground motion instead. Hence, the background in the dataset is often very simple and does not move much, and very little, if any, interaction happens between the foreground and the background. Thus the dataset is not a good fit for our task. And since AnimateAnyone is not open sourced, it is hard to compare experimentally. Nonetheless, as requested by the reviewer, we tested our model on two samples from the TikTok dataset and included the results in Fig. 1 of the attached PDF. Our model demonstrates good generalization over the tested data.