Single Motion Diffusion

“It seems like there is a trade-off to reducing the temporal receptive field. … Can authors discuss this? This may indicate that we need more metrics to measure the visual quality of motions.”

Following this question, we added Appendix F, presenting a study about motions smoothness. In this study, we describe a metric that measures the smoothness fidelity between the generated motion to the ground truth on which the network has been trained. We show that using a wider receptive field improves smoothness fidelity but degrades the other metrics, hence reducing motion quality. Finally, we introduce a method to enhance smoothness fidelity while maintaining a narrow receptive field, thereby preserving the quality of the results.

“I initially thought "arbitrary topology" meant cases like generating the dragon motions from the model trained on human skeletons.…”

We added a clarification in the introduction (in blue). Thank you for bringing the need for this clarification to our attention.

“Are there motion representations other than the adopted one from HumanML3D tested for the proposed method? How do you expect the motion representation could affect the generation performance?”

SinMDM is experimented with three different representations: (1) The HumanML3D representation, including joint rotations, global locations and velocities; (2) The Mixamo representation, which includes rotations and global location for the root joint only; and (3) The Zoo representation, which requires extra degrees of freedom and includes rotations + xyz relative location for all joints.

For all datasets, we use rotations of the 6D format, which has been proven by prior work to yield optimal results (See Table 2 in the GANimator paper).

Overall, we showed that our approach is robust with different motion representations.

“In what circumstances does the method fail?”

In addition to the limitations mentioned in the conclusions section, we observe a failure case that is common to prior works as well - sub-motions are played in an arbitrary order, and this can be a disadvantage when a sequence of sub-motions needs to be executed in a precise order. For example, while doing the bird dance, one should clap hands immediately after shaking the hips. However, single motion algorithms will place these two sub motions in a temporally arbitrary order.

This can be addressed by enlarging the receptive field (at the cost of lower diversity). We added this limitation to the conclusions section (see in blue).

“Results of some editing applications... Similar results can be achieved by simple manipulations”.

Many of the applications enabled by SinMDM may also be implemented using designated algorithms. However, these designated algorithms are neither simple nor general, while ours provides a one-stop-shop for all applications, and yields better quality results when compared to simple manipulations.

For spatial composition, using cut-and-paste may yield a motion where the hands and the legs are un-synched, for example, in a walk motion, we expect that when the left hand is at the front, the right hand will also be at the front. In addition, when using cut-and-paste there is no balance compensation when the body leans to a specific direction, for example, when the upper body leans to the right, the pelvis should balance by leaning to the left. The motion generated by cut-and-paste is merely a copy of the upper/lower part of the inputs, hence has no flexibility to adjust them to match each other. On the other hand, the motion generated by SinMDM can be different from the one on which it was trained, thus there can be seamless synchronization between the upper and lower body parts. In the context of the style transfer application, while manually synchronizing the pace of steps may yield comparable results, SinMDM achieves this straightforwardly, offering a solution that would otherwise demand a designated, complicated approach, as shown in prior works (Menardais et al., 2004; Lee et al., 2020; Panagiotakis et al., 2013). Temporal synchronization requires the identification of both motions' rhythms, a non-straightforward task by itself. Moreover, stretching one motion according to the rhythm of another is naively done by a linear interpolation, which can cause motion blur, and may not be linear by nature (say, accelerating a leg motion at the beginning of the step and slowing it as it continues). Our model has no such challenges. It generates the required motion such that it is natural, maintaining the motion motifs of the learned input.

[Stephane Menardais, Richard Kulpa, Franck Multon, and Bruno Arnaldi. Synchronization for dynamic blending of motions. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 325–335, 2004.]

[Juyoung Lee, Myungho Lee, Gerard Jounghyun Kim, and Jae-In Hwang. Effects of synchronized leg motion in walk-in-place utilizing deep neural networks for enhanced body ownership and sense of presence in vr. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology, pp. 1–10, 2020.]

[Costas Panagiotakis, Andre Holzapfel, Damien Michel, and Antonis A Argyros. Beat synchronous dance animation based on visual analysis of human motion and audio analysis of music tempo. In International symposium on visual computing, pp. 118–127. Springer, 2013.]

“Why is the vanilla UNet considered in the first place?”

“Why is the UNet the default option here? Wouldn't LSTMs / WaveNet / TCNN would be a better option?”

We have considered several backbones, focusing on architectures that have demonstrated the highest scores in previous research and have already won ablation studies against alternative models. Our initial choice for a backbone has been a UNet, a selection that aligns with the consensus among imaging diffusion works (Ho et al., 2020; Song et al., 2020a; Dhariwal & Nichol, 2021; Saharia et al., 2022) and has demonstrated effective performance.In addition to the empirical results, Jolicoeur-Martineau et al. (2020) demonstrate that UNets significantly enhance sample quality compared to previous architectures used for denoising score matching. Our second choice of backbone is the Transformer Encoder, used by most motion diffusion works (Kim et al., 2022;Tevet et al., 2023, …). Tevet et al. (2023) conduct ablation studies and show that a Transformer Encoder performs better than a Transformer Decoder and better than a GRU (a GRU is similar to the LSTMs mentioned in R3’s question). In our ablation studies we show that in the case of a single motion input, a UNet performs better than a Transformer.

[Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioan- nis Mitliagkas. Adversarial score matching and improved sampling for image generation. arXiv:2009.05475, 2020.]

“afaik vanilla UNet [Ronneberger'2015] does not have attention layers, do you imply the unet similar to that used in stable diffusion implementation?“

Our "vanilla UNet" is the one employed in early imaging diffusion works, e.g., DDPM (Ho et al., 2020), which includes attention layers. We acknowledge the concern raised by R3 that the term "vanilla UNet" might lead the reader to question whether we are referring to Ronneberger2015's UNet or Ho2020's UNet . We have added a clarification in Section 4 (in blue).

“What exactly is meant by "single input UNet" in 4?”

It means a UNet trained on a single input. We revised Section 4 for better clarity. Thank you for bringing it to our attention.

“As the receptive field is narrowed, how to guarantee temporal smoothness? As shown in Table 3, using a wide receptive field gives better FID.”

Following this question, we added Appendix F (in blue), presenting a study about motion smoothness. In this study, we describe a metric that measures the smoothness fidelity between the generated motion to the ground truth on which the network has been trained. We show that using a wider receptive field improves smoothness fidelity but degrades the other metrics, thus reducing motion quality. Finally, we introduce a method to enhance smoothness fidelity while maintaining a narrow receptive field, thereby preserving the quality of the results.

As noticed by R2, using a wide receptive field results in better FID. However, a 'near-perfect' FID indicates overfitting to the input motion, as evidenced by poor scores in diversity metrics.

“Comparison with Ganimator in HumanML3D”

We added a row to Tab. 2, exhibiting the metric scores of Ganimator on the HumanML3D dataset, and demonstrating once again that SinMDM performs better. Note that Ganimator’s scores demonstrate a very high FID, indicating divergence from the input motion. The updated radar plot reveals a relatively small area for Ganimator, suggesting that it faces challenges in generalizing to other datasets. Running Ganimator on HumanmML3D is not straight forward, as Ganimator expects an entirely different data format. To ensure a fair comparison, we transformed the HumanML3D benchmark into the data structure utilized by Ganimator, enabling Ganimator to leverage its skeleton-aware framework. To evaluate the results using the HumanML3D metrics, we reverted them to the HumanML3D format.

“could you elaborate on this, and emphasize the contribution of this work?”

Indeed, the performance of SinMDM is built upon the fundamental concepts of Motion Diffusion Models. However, as Motion Diffusion Models typically rely on large datasets, adapting them to a single motion required a thoughtful redesign. In Section 2, we note that Wang et al. (2022) and Nikankin et al. (2023) use a UNet with limited depth, a strategy not directly applicable to the motion domain. Unlike images with a regularized 2D spatial structure, motion data consists of non-regularized skeletal joints with a temporal axis and fewer degrees of freedom. To bridge this gap, we propose a novel architecture that combines the strengths of the traditional UNET and recent QNA architectures. This effective adaptation to the motion domain enables the application of advanced techniques such as ILVR, as credited in Section 5.2.

We have revised Section 2 to provide an explanation of the distinction between the imaging and motion domains. Thank you for bringing the need for this clarification to our attention.