SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios

• Q1: Discussion of VideoJAM [1] (ICML 2025). The concurrent work VideoJAM generates video alongside 2D optical flow to enhance visual fidelity and motion plausibility. In contrast, our approach generates video with explicit 3D motion. This enables direct perception of object geometry, scale, and spatial relationships, resulting in superior physical plausibility for complex scenarios involving occlusions or intricate interactions. The core challenge in 3D joint video-motion generation lies in the significant domain gap between visual and dynamic information, as well as the difficulty in ensuring consistency between both outputs. To address this, we bridge the multimodal domain gap through adaptive modulation and establish a closed-loop co-evolutionary pipeline. This framework effectively captures richer complementary information between video and motion. To validate our improvement over VideoJAM, we replace our default synchronized video-3D-motion diffusion model with a video-2D-motion variant to mimic VideoJAM. Results demonstrate that incorporating 3D motion diffusion achieves more reasonable motions, faster convergence speed, and improved overall video score, and significantly higher human perceptual quality. We will cite and discuss VideoJAM in the revised manuscript.

• Q2: Human perceptual evaluation of 3D motions. For 3D motion evaluation, we employed MPJPE, motion smoothness, chamfer distance, and FID metrics, detailed in Sec. 4.1 and Tables 2 & 3. To provide evaluation more consistent with human perception, we have conducted a user study (Fig. 5(b) in the main paper) involving 410 participants, which demonstrates our method's superiority over the baselines in 97.56% of cases.

• Q3: Zero-shot generalization to more distant HOI categories. Our train-test split mitigates HOI category overlap by reserving all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) exclusively for the test set, ensuring these categories remain unseen during training. Thereby our evaluation on the TACO test data could demonstrate zero-shot generalization capabilities to some extent. To further validate this capability in real-world scenarios, we conducted experiments using daily HOI examples (Fig. 6). For more rigorous generalization evaluation, we constructed an additional test set of 100 instances comprising entirely unseen HOI categories (novel objects, actions, and diverse camera viewpoints), sourced from TACO, OakInk2[2], and GigaHands[3] datasets. As shown below, while video and motion metrics for this unseen set show a slight performance decline compared to the main TACO test set, they remain reasonably close. This robust zero-shot generalization capability stems from two critical design principles: (1) Our plug-and-play synchronized diffusion architecture leverages the visual-semantic priors of large video foundation models. (2) Our 3D point cloud object representation offers enhanced spatial geometry and physical awareness, and supports viewpoint generalization.

• Q4: Demonstration of more diverse videos. As suggested by the reviewer, we will include more representative video samples, including both successful demonstrations (cases mentioned in Q3) and systematically analyzed failure cases, in the upcoming supplementary material to further substantiate the practical versatility of our approach.

• Q5: Validity of the VBench metric (background consistency). Firstly, as defined in VBench [4], the "background consistency" metric calculates global CLIP image feature differences between consecutive video frames, reflecting overall video stability across both foreground and background. Secondly, in the field of HOI generation, cropping the HOI region is a common practice[5], as the global background context has limited impact on fine-grained local interactions. Our experiments reveal that despite cropping the HOI region, baseline models still exhibit hallucinated artifacts, such as spontaneous object emergence, violating temporal coherence (See Appendix Fig. 9, rows 1-2). Therefore, this metric remains valuable for video consistency evaluation.

• Q6: Ablation study of the multimodal fusion machinism. As suggested by the reviewer, we replaced the adaptive modulation with a simpler linear projection (Linear Proj. (Suggested)) mapping features to a shared embedding space. The results below demonstrate that the linear projection leads to significant degradation in motion FID scores, slower convergence speed, and moderate declines in overall video quality. That is because our triple-modality adaptive modulation has the advantage of inherently decoupling appearance and motion, thereby enabling the extraction of discriminative and complementary video-motion representations.

• [1] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. In Forty-second International Conference on Machine Learning, 2025.
• [2] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024.
• [3] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga-hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025.
• [4] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024.
• [5] Youxin Pang, Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12209–12219, 2025.

• Q1: Ablation study of the multimodal fusion mechanism. We replaced our adaptive modulation with a simpler linear projection (Linear Proj. (Simpler)) mapping features to a shared embedding space. The results below demonstrate that the linear projection leads to significant degradation in motion FID scores, slower convergence speed, and moderate declines in overall video quality. That is because our triple-modality adaptive modulation has the advantage of inherently decoupling appearance and motion, thereby enabling the extraction of discriminative and complementary video-motion representations.

• Q2: Chamfer distance loss is employed in Eq. 6 (not Eq. 2) in the main text with its detailed formulation presented in Eq. 13 in the Appendix. This loss measures the Chamfer distance between the generated object point clouds and the ground-truth point clouds, which also serves as a metric for 3D motion generation quality.

• Q3: Discussion of concurrent works Vlogger[1] (CVPR 2025) and VideoJAM[2] (ICML 2025). VideoJAM generates video alongside 2D optical flow to enhance visual fidelity and motion plausibility. In contrast, our approach generates video with explicit 3D motion. This enables direct perception of object geometry, scale, and spatial relationships, resulting in superior physical plausibility for complex scenarios involving occlusions or intricate interactions. Vlogger utilizes a two-stage pipeline: first, it generates 3D human poses with one network, then feeds the motions into another separate network for video synthesis. In contrast, we unify video and motion generation within a single synchronized diffusion model. Since HOI video and motion inherently share the same physical laws of the real world, this joint modeling enables the learning of intrinsic co-evolutionary relationships, enhancing the quality of both outputs. We will cite and discuss Vlogger and VideoJAM in the revised manuscript.

• Q4: Extension potential to more sophisticated HOI scenarios. Our method inherently supports extension to HOI scenarios involving objects of complex geometries, or even articulated objects, and full-body interactions. Extending the approach requires only minor adjustments to point cloud density or quantity, or pose representation in our vision-aware 3D interaction diffusion model. Specifically, (1) Complex geometries or deformable objects require increased point density per object for precision. (2) Articulated objects can be decomposed into sub-components and increase the total number of point clouds. (3) Full-body interaction requires extending the current hand pose to a full-body skeleton pose. Notably, human-scene interaction is distinct from our task. Those works typically take static 3D scenes as input and generate plausible human motions[3][4].

• Q5: Convergence behavior on the 2D and 3D data. Our model comprises two core components: synchronized diffusion for generating 2D video and motion video, and 3D interaction diffusion for synthesizing 3D HOI motions. To analyse the convergence behavior, we conduct comparative experiments using only 2D data to train the synchronized diffusion versus using joint 2D/3D data to train the whole model. Results show that at identical training steps (1K, 3K), integration of 3D data achieves lower training loss, faster convergence, and superior final performance.

• Q6: Comparison setting for 3D motions generation between ours and baselines. Our method generates motion from both images and text (image+text-to-motion), whereas the original MDM and EMDM are text-to-motion systems. For fair comparison, we adapted both baselines into text-and-image-to-motion generators by additionally adding the reference image into their CLIP encoders (details in lines 253-260).

• Q7: Memory and time consumption for training and inference. We compared VRAM consumption and training/inference time between Animate Anyone (as suggested by the reviewer) and CogVideoX-5B (our backbone). Results show that Animate Anyone requires significantly lower VRAM consumption and shorter time for both training and inference. This efficiency stems from its incremental architecture design: it extends a 2D image generation model with temporal attention blocks to enable video synthesis. In contrast, both CogVideoX and our method employ unified spatial-temporal full attention modules. Compared with CogVideoX, our approach can simultaneously generate both 2D videos and 3D motion sequences while introducing only minimal additional parameters and keeping computational cost within a reasonable range.

• [1] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. In Forty-second International Conference on Machine Learning, 2025.
• [2] Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15896–15908, 2025.
• [3] Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1737–1747, 2024.
• [4] Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say interact as you can: Language-guided human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 433–444, 2024.

• Q1: Diversity and physical metrics of 3D motions. As suggested by the reviewer, we added three new evaluation metrics: diversity, penetration rate, and contact rate. The following table shows that our method achieves lower diversity scores than baselines. This occurs because baseline methods produce 3D motions with visible artifacts, including chaotic object point clouds and significant shape/scaling distortions relative to reference images (Fig. 4). Consequently, higher diversity scores don't necessarily indicate better motion quality. Additionally, our method achieves superior performance in both penetration rate and contact rate metrics, indicating superior physical plausibility.

• Q2: Memory and time consumption for training and inference. We compared the VRAM usage and training/inference time with our backbone CogVideoX-5B, and found that our method simultaneously generates both 2D video and 3D motion sequences while introducing only minimal additional parameters, and maintaining computational cost within a reasonable range.

• Q3: Generalization to actions with novel verbs. Our train-test split mitigates HOI category overlap by reserving all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) exclusively for the test set, ensuring these categories remain unseen during training. Thereby the evaluation on test data (Tables 1 & 2 in the main paper) could demonstrate generalization capabilities to novel verbs to some extent. For more rigorous generalization evaluation, we constructed an additional test set of 100 instances comprising entirely unseen HOI categories with novel verbs, sourced from TACO, OakInk2 [1], and GigaHands [2] datasets. As shown below, while video and motion metrics for this unseen set show a slight performance decline compared to the main TACO test set, they remain reasonably close. This robust zero-shot generalization capability stems from two critical design principles: (1) Our plug-and-play synchronized diffusion architecture leverages the visual-semantic priors of large video foundation models. (2) Our 3D point cloud object representation offers enhanced spatial geometry and physical dynamic awareness, and supports viewpoint generalization.

• [1] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024.
• [2] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga-hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025.

• Q1: The influence of motion information on baseline video models. Compared to baseline methods performing video generation alone, our approach jointly generates video and motion, and learns a visual-motion joint representation through adaptive multimodal modulation. To enable fair comparisons with baselines and assess the contribution of the modulation mechanism to our performance gains, we augmented baseline video models with the motion modality. Specifically, Animate Anyone inherently uses GT Motion as input. For EasyAnimate and CogVideoX, we trained them using both the original video and the rendered motion video. Results show that Animate Anyone produces videos of notably lower overall quality due to severe inter-frame flickering (see Fig. 3 in the main text and supplementary videos). For EasyAnimate and CogVideoX, while motion-aware training yielded marginal gains, improvements remained modest. This suggests that multimodal modulation fusion plays a more critical role in our approach than the motion modality alone.

• Q2: Evaluation on HOI datas with more camera viewpoints. The TACO dataset we used contains 12 camera views. To assess our method on HOI datasets with greater viewpoint diversity, as suggested by the reviewer, we curated approximately 70 instances from OakInk2[1] (4 views) and GigaHands[2] (51 views). As shown below, both the video and motion metrics on this unseen set are comparable to those on the main TACO test set. This robust generalization capability to camera views stems from two critical design principles: (1) Our plug-and-play synchronized diffusion architecture leverages the visual-semantic priors of large video foundation models. (2) Our 3D point cloud object representation offers enhanced spatial geometry and physical dynamic awareness, and supports viewpoint generalization.

• Q3: Requirement of the rendered motion video during inference. Our method does not require the rendered motion video as input during inference. We only require a reference image and a prompt as inputs, and we initialize the target outputs (2D video, 2D motion video, and 3D motion) with pure noise, and iteratively denoise them to obtain the final results. As detailed in Appendix Alg. 2, Line 4, denote the triple noisy latents at timestep $t$ as $[z_t^V, z_t^M, (h_t, o_t)]$ , we first apply single-step denoising using VID $\mathcal{M}_\phi$ to produce the refined (but still noisy) 3D motion ($\tilde{\boldsymbol{h}}_0$,$\tilde{\boldsymbol{o}}_0$). The refined 3D motions are then rendered back into a noisy 2D motion video latent $\tilde{\boldsymbol{z}}_0^M$. Since $\tilde{z}_0^M$ represents a more refined estimate than the original noisy latent $z_t^M$, we concatenate it with the latter as auxiliary information and input them into SViMo $\mathcal{G}\theta$.

• Q4: Train-test dataset splitting. We implemented a two-stage data partitioning strategy. First, to mitigate overlap between test and training sets that might compromise evaluation of generalization, we reserved all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) as the initial test set, ensuring these elements are absent from the training set. Second, from the remaining instances grouped by <action, tool, object> triplets, we applied weighted sampling according to group sizes to obtain additional test data. Finally, the ratio of the training set to the test set is 9:1 (Line 203 in the main text).

• Q5: Extension potential to HOI scenarios with objects of different numbers and shapes. Our method inherently supports extension to HOI scenarios involving diverse object numbers, shapes, complex geometries, or even articulated objects. Extending the approach requires only minor adjustments to point cloud density or quantity in our vision-aware 3D interaction diffusion model. Specifically, (1) Object number expansion merely needs additional point clouds. (2) Articulated objects (e.g., drawers, flip phones) can be decomposed into sub-components and handled via approach (1). (3) Complex geometries or deformable objects require increased point density per object for precision.

• Q6: Limitations about training efficiency and the reliance on 3D data. We discussed the limitations in training efficiency of the video foundation model, and the requirement for paired video-3D motion data in Appendix Sec. D. These issues will be alleviated with the development of video foundation models, as well as advances in 3D hand pose recovery and object pose estimation methods.

• [1] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024.
• [2] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga-hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025.

The rebuttal clearly resolve all of my concerns regarding the (1) fair comparison with video models, (2) view point generalization; (3) inference pipeline. I support the acceptance of the paper.