MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence

We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:

1. More specific implementation details

a) The motion vectors are flattened to $n \in \mathbb{R}^{T_m\times(J\times2)}$ first.

b) No. We use the same downsampling rate for both motion and music inputs (Table 8 in the Appendix) and align the frame rates by pre-processing techniques.

c) Yes. The latent representations from different modalities are normalized before forwarding the diffusion model.

Thanks for your comments. We will add these explanations in the revised manuscript to make it clearer.

2. Why incorporate the kinematic amplitude indicator and how the latent clip is sampled?

Why: In lines 111-117, we mentioned that existing audio-visual temporal contrastive pre-training methods are not feasible for motion-music alignment. For example, the temporal contrast, which seeks to maximize the similarity of audio-visual pairs from the same time segment while minimizing the similarity of pairs from different segments, may push away two clips that sampled from different time segments but shared similar rhythmic beats. That is why we incorporate the kinematic amplitude indicator to enable motion-music contrastive pre-training. As for the illustration of kinematic amplitude, you can refer to the line chart example in Figure 2(a), whose values are normalized within the range of (0,1).

How: In Figure 2(a), we have vividly illustrated the sampling process. For a motion-music latent representation pair, we compute the 2D motion directogram according to Eq. (1) first, and then derive the kinematic amplitude indicator for each temporal step by aggregating the bin-wise directogram difference according to Eq. (2). Finally, we randomly sample the motion-music clips from different categories to construct the contrastive learning objective according to Eq. (4). Note that the classification of the different categories is based on the kinematic amplitude indicator values, which undergo in-clip max-pooling, as described by Eq. (3). We will include the example in Figure 2 along with its accompanying explanations to make the Section 3.2 easier to understand.

We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:

1. Comparison with other methods on the demo page.

Thanks for your suggestions. Since the illustrated video samples are pretty long, the existing demo page has exceeded 567 MB. We will work to add the generated samples of SoTA methods: LORIS in motion-to-music and D2M in music-to-motion and re-organize the demo page. Currently, you can refer to their released samples from https://justinyuu.github.io/LORIS and https://github.com/NVlabs/Dancing2Music for a preliminary qualitative comparison.

2. Add a specific example of the different frame rates of motion and music inputs.

Assuming that we have a ten-second video and process it at 60 fps for pose sequence extraction, which results in 600 motion frames. For the corresponding audio, we resample it to 16khz and extract the mel-spectrogram with a hop length of $256$. Then, the audio frames are $10\times16000/256=625$. Finally, to align the frames from different modalities, we evenly drop $25$ audio frames to ensure the temporal dimensions of motion and music inputs are equal. In the revised manuscript, we will add an example to make it clearer.

3. Frame rate alignment between motion and music inputs in the pre-processing stage.

Due to the different sampling rates of video and audio, it is hard to ensure the audio and visual inputs are strictly temporally aligned. We ensure that the timing error is within a small range when we implement cross-modal temporal alignment in the pre-processing stage. For example, following the example mentioned above, we denote the audio frame rate as $f_1$ and the motion sequence frame rate as $f_2<f_1$. The minimum timing error is $|1/f_1-1/f_2|$ and the maximum timing error is $|k/f_1-k/f_2|$, where $k$ is the dropping interval and is computed by $k/f_1=(k-1)/f_2$. Recalling that $f_1=16000/256$ and $f_2=60$, the range of timing error is ($0.6$ms, $17$ms). For human beings, a timing error of less than $30$ms is typically hard to capture. Also, we will improve the pre-processing techniques to reduce the timing error in future work.

Thanks again for your valuable comments. We are happy to engage in discussion if you have any further questions.

We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:

1. Ablation study of the hidden size and layers of Transformer

We appreciate your suggestion on the ablation study of model size. Due to the time limit, we may be not able to present comprehensive results (e.g., model params $\textgreater$ $1$B) in rebuttal within a short response period. We implement on the AIST++ dataset and study four model configurations: the combination of hidden size in $[576,768]$ and Transformer layer in $[4,8]$:

We can observe that there is a slow performance boost when scaling the model size. The scaling law usually suggests that the performance can be improved by scaling up the training data and model size, simultaneously. It is reasonable to hypothesize that the bottleneck for the performance of motion-music generation lies in the data size. For existing public datasets like AIST++, LORIS, and BHS Dance, we think the base model configuration (DiT: 158M, BiCoR-VAE: 213M) is matched.

2. How long can we generate the video in the joint generation setting?

In joint generation, the length of the generated video is suggested to be less than 25 seconds, which is mainly caused by the video length of training data (i.e., AIST++). Currently, for motion-music generation, the major difficulty in achieving longer video generation is the lack of corresponding training data. In terms of algorithm design, It is reasonable to speculate that DiffCollage is effective in dealing with longer video generation (more than 1min) thanks to its parallel sampling and smaller sequence length. For the experiments in our paper, the end-to-end diffusion formulation can effectively address the problems and it is not necessary to split the sequence into several clips. In future work, we are going to build a larger dataset with longer video length and are willing to incorporate DiffCollage for a faster and longer generation.

Thanks again for your valuable comments. We are happy to engage in discussion if you have any further questions.

We thank the reviewer for his/her insightful comments and for appreciating the technical contributions of our paper. Here, we respond to your questions as follows:

1. How to handle the cases where the beat in motion or music is not clear and achieve/evaluate natural generation beyond beat alignment?

Training Phase: The diffusion model's goal is to learn the conditional distribution between motion and music, encompassing not only beat alignment but also the synchronization of temporal and semantic elements. While additional alignments beyond the beat are not explicitly modeled during the pre-training phase, they are implicitly achieved through the diffusion model. The incorporation of supplementary learning objectives, such as semantic contrast as utilized in Diff-Foley, into the BiCoR-VAE could lead to challenges in hyper-parameter tuning and optimization. Therefore, we only emphasize the beat alignment in the pre-training stage and implicitly model the cases where the beat is not clear in the diffusion training. In future work, we will conduct an in-depth study on stably combining both beat and non-beat alignments during the pre-training stage.

Evaluation Phase: In evaluating the quality of the generated samples, we have implemented a suite of beat-matching metrics (e.g., beat F1 score) and metrics that assess the overall generation quality (e.g., FAD). Compared to introducing additional objective metrics, we posit that the design of subjective evaluations holds greater significance in assessing motion-music generation. Drawing a parallel with the well-researched domain of text-to-speech synthesis, despite numerous objective metrics such as word error rate and speaker similarity, subjective assessments are still significant for measuring the naturalness of synthesized samples. Consequently, in the context of motion-music generation, we have conducted subjective A/B testing to compare our approach with existing state-of-the-art methodologies, as depicted in Figure 6.

2. Including an example to make the Section 3.2 easier to understand.

Thanks for your suggestions. For a motion-music latent representation pair, we compute the 2D motion directogram according to Eq. (1) first, and then derive the kinematic amplitude indicator for each temporal step by aggregating the bin-wise directogram difference according to Eq. (2). Finally, we randomly sample the motion-music clips from different categories to construct the contrastive learning objective according to Eq. (4). Note that the classification of the different categories is based on the kinematic amplitude indicator values, which undergo in-clip max-pooling, as described by Eq. (3). We will include the example in Figure 2 along with its accompanying explanations to make the Section 3.2 easier to understand.

Thanks again for your valuable comments. We are happy to engage in discussion if you have any further questions.