PuppetMaster: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics

Thank you for your time devoted to reviewing our work and your feedback. We answered some concerns shared by fellow reviewers above. Here, we answer your remaining questions:

Weakness 1

We further provide 6 examples in video-results/additional-result.gif in the revised Sup. Mat. to demonstrate our model’s performance on a wide range of objects. We include 3 samples generated with the same drag prompt to show its ability to generate reasonable yet diverse part-level motion videos (second row). Regarding the slight appearance flickering, we hypothesize the culprit is the pre-trained SVD model. The videos generated by SVD usually feature this appearance flickering (e.g., the rocket here). Fine-tuning from a better video generator could potentially resolve this problem.

Weakness 2

Thank you for the suggestion! We provide 8 comparison examples in video-results/comparison-DragAPart.gif in the revised Sup. Mat. We will make room to provide a few such examples in the main paper.

Q1

Thank you for reading our paper so carefully! To clarify, the number before the slash (12.21 for PuppetMaster) is averaged among conditioning drag points (L410), while the number after the slash (3.53 for PuppetMaster) is averaged among all foreground points (L411). So the flow error is smaller when averaged with all foreground points. This is because PuppetMaster models part-level motion – the parts that do not move with the drag (e.g. the 2 other covers of the box in the 3rd row of Figure 4) remain static throughout the video. The points on those static parts have almost 0 flow error, so when included (when flow error is averaged among all foreground points), they make the flow error lower. We explained this in L413-416; we will make the explanation clearer in the revision.

Q2

Here, the end location refers to the termination point of the entire drag. Recall in line 237-238 we defined each drag as $v^{1:N}$. For each video frame $n$, the encoding always encodes the starting point $v^1$ and the ending point at the current frame $v^n$. “w/ end loc” refers to the fact that the encoding further encodes the final termination point $v^N$. We empirically found that adding $v^N$ facilitates learning and yields better generation quality. This was specified in L241-246; we will make it clearer in the Experiment section.

Q3

The modulation is implemented as in Eq (3). We regress the shift and scale term from the drag encoding with a few convolutional layers (L254-256) to modulate the diffusion hidden features. This design follows from DragAPart [1], where they only use a shift term, i.e., $f_s\leftarrow f_s + \beta_s$. We empirically found that adding the scale term yields significantly better generation quality (Table 2 design A vs. B).

[1] DragAPart: Learning a Part-Level Motion Prior for Articulated Objects, Li et al.

Thank you for your time devoted to reviewing our work and your feedback. We answered some concerns shared by fellow reviewers above. Here, we answer your remaining questions:

Weakness 1

“Progressively fewer animations” means Objaverse-Animation-HQ goes through an extra filtering step (L347-354), and hence contains fewer but higher-quality animations.

We would like to think of the second challenge as an optimization challenge, i.e., how to facilitate learning so that the model doesn’t get stuck in local optimum. We then propose architectural changes (i.e., All-to-First Attention) to resolve this. That being said, it’s not wrong at all if you would like to think of them as a single challenge under the umbrella of “effectively injecting drag control”. We will provide this alternative interpretation in the text in the revision.

Weakness 2

In this work, our goal is to synthesize internal, part-level motion, as opposed to shifting or scaling an entire object (L86-90). We stated this again in L191-192: “generate a part-level animation of the object”. You are absolutely correct that high-quality training data is the key to achieve this. To this end, we dedicated enormous efforts into data curation to learn motion where object parts are manipulated (Sec. 4).

We believe what you pointed out is a really interesting direction! We have had many internal discussions about learning a model that can interpret whether the drag is intended for part manipulation, object zooming, or object moving. An illustration from our discussion is provided in future-work.png in the revised Sup. Mat. However, this is beyond the scope of our submission.

Weakness 3

Please refer to the Novelty of the All-to-First Attention section of the shared thread.

Q1

There are 3 key differences:

(1) For each video frame $n$, in addition to encoding the starting point $v^1$ and the ending point at the current frame $v^n$, we also encode the final drag termination point $v^N$ (L241-246).

(2) We use modulation instead of a shift-only approach proposed by DragAPart to modify the internal diffusion features (L248-258).

(3) We introduce the use of drag tokens to make the cross attention modules in SVD spatial-aware (L259-269).

Q2

We provide 8 comparison examples in video-results/comparison-DragAPart.gif in the revised Sup. Mat. We agree that including them could better visualize our model’s improved temporal consistency.

Q3

The model might generate videos of a higher level of artifacts if the input image features an object rarely represented in the training data, such as non-articulated objects like balls or phones. A few examples are provided in video-results/non-articulated.gif in the revised Sup. Mat.

Thank you for your time devoted to reviewing our work and your feedback. We answered some concerns shared by fellow reviewers above. Here, we answer your remaining questions:

Weakness

Please refer to the Novelty of the All-to-First Attention section of the shared thread.

Q1

The model generates 14 frames per video.

Yes, All-to-First Attention can be replaced by full 3D attention. In fact, a recent work CAT3D [1] claims full 3D attention is essential for the generation quality. However, full 3D attention is compute intensive, as the compute (time and memory) required in attention goes up quadratically with the sequence length. CAT3D does inference on 16 A100 GPUs, whereas our model was trained entirely on a single A6000 GPU.

Q2

We provide 3 examples of non-articulated objects in video-results/non-articulated.gif in the revised Sup. Mat. The videos do show some motion towards the intended directions. However, since these cases were not seen during training, the generated videos can have artifacts (e.g., the iPhone and the tree).

Q3

The architecture of our model is designed so that having a zero-length drag is different from having no drags at all. The former specifies that no motion is wanted for that point, so the generated video features minimal motion. The latter specifies no motion control, thus an arbitrary motion is generated. An example distinguishing the two is provided in video-results/corner-cases.gif in the revised Sup. Mat. Such corner cases were rarely sampled during training, so the generated videos can exhibit a slightly higher level of artifacts.

[1] CAT3D: Create Anything in 3D with Multi-View Diffusion Models, Gao et al.