MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance

1. Is confidence-aware pose guidance the optimal approach?

We evaluated depth information as an additional input and found it does not significantly improve the resolution of front-back ambiguity or incorrect pose issues beyond what our confidence-aware approach already achieves. Depth estimation introduces its own prediction errors and computational overhead without commensurate benefits. While alternative guiding modalities have been explored in prior work (e.g., Champ), each presents inherent limitations—depth varies with camera distance, and part segmentations may compromise action template transferability across different contexts. The reviewer's suggestion, though theoretically plausible, lacks empirical substantiation. We maintain that our approach represents an effective balance of performance and efficiency within the scope of our research objectives.

2. Why hand region enhancement improves all evaluation metrics?

We respectfully note that our approach does not solely assign higher loss weights to hand regions. Rather, it strategically enhances weights for clear hand regions, encouraging the model to generate more defined hands. Given the iterative nature of the diffusion process, we observe that improved hand clarity appears to serve as a quality signal that positively influences overall video generation quality across successive denoising steps.

3. Progressive latent fusion results in lower quantitative performance.

To address this, we have supplemented the website with comparison videos corresponding to Fig. 9's progressive latent fusion to better illustrate the qualitative differences.

The discrepancy in quantitative metrics stems from inherent limitations in evaluation metrics: SSIM and PSNR are inadequate for evaluating our progressive latent fusion approach, as they assess image-level quality rather than transition smoothness between denoising windows. Similarly, Fréchet distance-based metrics rely on video classification models pre-trained on datasets like Kinetics, which primarily leverage spatial rather than temporal features for classification. These datasets can often be classified accurately from single frames, indicating their limited sensitivity to temporal consistency issues.

4. Post-processing

Thank you for raising this concern. We have addressed this by replacing all videos on our website with versions without post-processing.

We emphasize that the referred Figure 4 and all other comparison and ablation results in the paper and website are presented without post-processing. The identity preservation challenges visible in our results and baselines represent the actual performance of these methods, not artifacts of face swapping or other post-processing techniques. The clothing color shift observed in Figure 4 occurs to varying degrees across multiple methods, including ours, confirming the fairness and transparency of our evaluation.

1. Out-of-Domain Generalization

The model is trained exclusively on human dancing videos. There are two reasons for this generalizability:

First, the SVD backbone can generate cartoon and animal videos. Our fine-tuning enhances the model with pose controllability, and since the pose feature space of humans, cartoons, and animals may share a common subspace that is preserved in the fine-tuned model, human control signals can also guide cartoons and animals. The results in AnimateAnyone and MagicPose also verify this point.

Second, we map the limb lengths of the pose template to the reference character following MuseV, making the pose guidance signal closer to the reference character. This helps the model preserve the character's appearance and also helps to adapt one pose to another with different body proportions, enhancing generalization.

We thank the reviewer for raising these concerns and will add these explanations in the revision.

2. Missing Baseline Comparisons

We have enhanced our evaluation by adding comprehensive comparisons with Champ both quantitatively (see table below) and qualitatively (refer to the Comparisons section on our website). Regarding DisPose, we respectfully note that it was released within 4 months of our submission date, which falls outside the comparison requirements according to the conference guidelines. For VividPose, despite our efforts, its unavailability as open-source presents reproduction challenges.

Our method specifically focuses on 2D DWPose-guided video generation, which offers practical advantages by eliminating SMPL and other inference overhead, and better handling out-of-domain characters where SMPL-based approaches face significant limitations.

3. Supplementary Website Demos

Thank you for the suggestion. We have replaced all demos on the website with their raw versions without post-processing. It needs to be clarified that, for the cases of comparison and ablation, no post-processing was applied in the original web page. We used post-processing operations in the original web page only to showcase better visual effects in demos, without affecting the contributions of this paper.

4. Ablation Analysis of SVD and SD Backbone

Thank you for your suggestion. We have added videos corresponding to Fig. 9's progressive latent fusion on the website.

Regarding the SD Backbone, we believe that progressive latent fusion would not provide significant improvement. Videos generated with the SD backbone exhibit high-frequency flickering within a single window (see baseline comparison on the website). Progressive latent fusion only enhances transition smoothness between denoising windows, without improving smoothness within windows. Therefore, the SD backbone would likely still suffer from flickering. We have added a Moore AnimateAnyone (SD backbone) + proposed strategy result on the website, which verifies this point.

Like other research in this field, our paper deliberately focuses on a specific backbone (SVD), which is both well-pretrained and open-source. The issues with the SD Backbone fall beyond the scope of our current research objectives, as it is standard practice to focus on optimizing a single backbone network.

1. Essential References Not Discussed: DiT

Thank you for your effort in reviewing our paper. We will revise our manuscript and add a discussion about DiT. Compared with U-Net, DiT is considered a more novel and promising foundation model, especially in video generation tasks, where it demonstrates scalable capabilities.

2. Is the image encoder in Figure 2 CLIP?

Yes, it is a typical CLIP image encoder.

3. The authors mention in line 206 that "Pose guidance is not added to every U-Net block". What kind of degradation effects would occur if pose control were introduced in multiple layers of the UNet structure? Can comparative examples be provided?

We apologize for the confusion. Here, we mean pose guidance is only added to the output of the first convolution layer of U-Net, as we found this is sufficient for pose control in experiments.

This design choice is based on two hypotheses: First, introducing sequence pose information extracted without temporal interaction could potentially disrupt the spatio-temporal representations within subsequent U-Net layers. Second, since our image-to-video base model was pre-trained without posture information, excessive integration of pose sequence data may degrade pertained performance.

To ensure clarity, we will update the relevant section accordingly.

1. Essential References Not Discussed: StableAnimator, ControlNeXt

Thank you for your effort and valuable feedback during the review process. We will add a discussion of these two related works and revise our manuscript accordingly.

2. The influence of hand region enhancement.

Does the hand region enhancement suppress the generation of other areas to some extent? I hope to see whether videos without hand enhancement could achieve better results, and whether the non-hand regions in videos with enhanced hands still maintain good quality.

Thank you for your valuable suggestion. We have updated the supplementary website with full-frame and video comparisons (see Sec. Ablation). As demonstrated by multiple examples, we observe no significant quality degradation in non-hand regions while maintaining enhanced hand generation.

3. The claim of long-video generation.

How long can other methods generate high-quality videos? It would be useful to show how the performance of the proposed method changes as the video length increases, both quantitatively and qualitatively.

The video lengths of video generation base models are as follows. AnimateDiff: 2s, SVD: 3.6s, CogvideoX-5B: 6s, LTX-Video: 10.6s. We discussed long-video generation in a sub-section of Related Work. Existing methods adopt a windowed generation approach to produce longer videos, which can be extended to infinitely long videos; however, there are still issues with abrupt changes at the window boundaries. In contrast, our proposed progressive latent fusion mitigates this problem.

To further show how performance changes as the video length increases, we have added comparative videos to the website. This method has an independent good case rate for each window. While bad cases may occur over time, the quality of video clips will not deteriorate as the total video duration increases.