HyDance: A Novel Hybrid Dance Generation Network with temporal and frequency features

Reviewer vNB2,

Thank you for reviewing our paper and your valuable suggestions. We have carefully considered your feedback and provide our responses below:

Weakness:

1. Overall contribution is limited.

We thank the reviewer for the insightful suggestions, which have effectively helped us to improve our work.

a. Overall contribution

Our motivation is to address the issue of models generating overly monotonous dance movements, rather than focusing on the application of PAE. Our contribution is not an exploration of the application of PAE. Instead, we propose a framework that integrates frequency-domain and time-domain features for dance generation, and we have conducted experimental tests to validate this framework.

In our experiments, we observed that the more compact frequency-domain feature space indeed helps the model better capture motion transitions within dance sequences. However, using only frequency-domain representations does not yield satisfactory dance movement results, especially concerning root node movement and limb positioning. Therefore, we designed a generation framework that leverages the complementary strengths of both representations.

b. The scope of taking the advantage of frequency domain motion features should not be limited to dance generation.

This is an excellent idea. However, our current work focuses on generating high-quality, expressive dance motions. We plan to open-source our code to support further research in complex motion generation.

2. Generalization ability.

We agree with the reviewer that generalization ability is a critical aspect of dance motion generation models. To demonstrate the generalization capability of our method, we compared the dance generation results of our approach with those of the reference method under randomly selected in-the-wild music conditions. We have uploaded the result videos to a fully anonymous repository, which can be accessed via the following link: https://github.com/null770/submission5560/blob/main/inthewildtest.mp4

From the generated result videos, our method demonstrates decent performance under in-the-wild music conditions of varying styles. It produces expressive and rhythmically aligned dance sequences, indicating the good generalization capability of our approach.

3. More analyses for frequency-related performance.

We appreciate the reviewer’s novel and constructive suggestions. We have added comparisons of dance generation results under different BPM music conditions in the revised version of the paper (Section 4.2, Table 2, Line 393). The revisions can also be found in the table below:

Table 2. Quantitative evaluation on subsets of AIST++ test set with different BPMs

Low BPMs

---

High BPMs

4. Implementation details

In the revised paper, we have included a description of trainable parameters. Regarding inference speed, it is highly dependent on the GPU and memory size, making it challenging to provide a specific description. For reference, generating a 15-second dance sequence on an NVIDIA 3090 GPU takes approximately 2 to 3 seconds.

5. Definitions and analyses of DIV metric.

Thank you for pointing out the shortcomings. We have refined the descriptions of the evaluation metrics in the revised paper(Section 4.2, Lines 374-377).

6. Details of user study.

We are grateful for the reviewer’s constructive feedback. We re-conducted subjective evaluation experiments and enhanced the descriptions of the experiments.(Section 4.2, Lines 417-425)

7. The Figure 4 is not really informative.

In Figure 4(Section 4.2, Line 469), we have added ground truth data to facilitate better understanding.

We sincerely appreciate the reviewer’s insightful suggestions, which have been instrumental in helping us improve this work.

Reviewer wujW,

Thank you very much for your thorough review of our paper and the valuable suggestions for improvement.

Weaknesses:

1. Qualitative results demonstration:

We agree with the reviewer's perspective. We have included comparison figures of the dance rendering results alongside those from other methods.(Section 4.2, Figure 4, Line 342)

2. w/o Dual-Domain Hybrid Encoder, the model seems to achieve comparable performance against the full model except the DIV_k metrics.

Thank you for your constructive suggestion. We have re-conducted the subjective evaluation experiment and added subjective result comparisons for the ablation studies. (Section 4.2, Table 4, Line 487) The revisions can also be found in the table below:

Table 4. Ablation study results

We greatly appreciate the reviewers’ insightful suggestions, which have helped us enhance the quality of the manuscript.

Reviewer LHkM,

Thank you very much for your thorough review of our paper and the valuable suggestions for improvement. We have carefully considered your feedback, and here is our response:

Weaknesses:

1. Discussions with other approaches exploring a similar idea

We agree with the reviewer and have added a discussion of related work in the revised manuscript (Section 2.2, Lines 160-161).

2. The key contribution of utilizing frequency-domain information can be explained in more depth.

Thank you for your constructive suggestions. We have included comparisons of results under different tempos of music in the updated version of the paper (Section 4.2, Table 2, Line 392). The revisions can also be found in the table below:

Table 2. Quantitative evaluation on subsets of AIST++ test set with different BPMs

Low BPMs

---

High BPMs

We aimed to combine time-domain and frequency-domain representations to address the issue of overly smooth dance movements in generation networks. The more compact frequency-domain feature space allows the model to better capture motion transitions, resulting in dance movements with increased motion transitions.

However, during experiments, we also observed that frequency-domain representations alone were insufficient for achieving satisfactory dance movement results, particularly in root node movement and limb positioning. Thus, we designed the model to combine the strengths of both representations.

3. Experiments can also be explained in more detail.

Thank you for your insightful suggestions for improving this paper. We have provided an explanation for the DIV metric in the revised paper(Section 4.2, Lines 374-377). Regarding instances where certain metrics exceed the ground truth, this is due to the calculation principles of these metrics. For example, the PFC score tends to be lower if dance sequences have less movement of the dancer’s root node, which is also reflected in our evaluation of dance results generated under different tempos of music (Table 2 Above). As for the BAS metric, since it measures the distance between music beats and dance beats, the integration of frequency-domain representation helps our method generate dance sequences with more reasonable movement transitions, leading to potentially higher BAS scores.

4. Some details on the user study are also missing.

We have re-conducted the subjective evaluation experiment, increased the number of dance sequences for comparison, and introduced a neutral option. These changes are reflected in the revised version of the paper(Line 379, Table 1 and Line 392, Table 2).

Questions:

1.About the contact loss term

The contact loss term originates from EDGE [A], aiming to encourage the model to predict contact labels and ensure that the generated movements are consistent with its own predicted labels. When the model predicts foot contact with the ground but the foot velocity remains high, this loss increases.

2.About freq. analysis figure

We have added Ground Truth data to Figure 4 in the revised manuscript (Line 469) to show the difference and improvement.

3.Details of user study

In the ablation study of the hybrid encoder, we do not use other encoders. Instead, we directly input the results from the frequency-domain and time-domain representation encoders into the Dance Decoder as the query and key, respectively.

We are grateful for the time and effort you have invested in reviewing our work. Your suggestions have been instrumental in improving our paper. We hope that our responses are satisfactory, and if possible, please consider increasing the score for this submission.

[A] Tseng, J., Castellon, R. and Liu, K., 2023. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 448-458).

Reviewer YEBL,

Thank you very much for your thorough review of our paper and for the valuable suggestions. We have carefully considered your comments, and here is our response:

Weakness:

1. & 2. Description of representations

We agree with the reviewer that the descriptions of the contact label and the duration of the music could be more detailed. The foot contact label is the heels and toes of the left and right feet the same with previous work EDGE[A]. “a music sequence of length L” wil be modified to ““a music sequence of L frames”. Accordingly, we have made the necessary revisions in the revised version of the paper (Section 3.1, Line 192 and Lines 198-199).

3. Misalignment between the figure and description

We apologize for the incorrect descriptions. The confusion in the description has been corrected in the revised version of the paper (Section 3.4, Lines 287-288). The relevant description has been revised to: “Finally, the noisy motion sequences in two different representations, are fed into the dance decoder together with the unified motion feature sequences obtained from the hybrid encoder”

4. Figure 4 could incorporate the spectrum of the Ground Truth.

We agree with the reviewer and have added Ground Truth data to the spectrogram in the revised version of the paper (Figure 4, Line 469).

5. Some parts are not very clear

The figures in the paper have been revised, and we have provided explanations for the meanings of the arrows and the DIV metrics (Section 4.2, Lines 374-377, Lines 379-380).

Questions:

1. Clarification between notations

The notation $d_f$ in Section 3.1 is a typo and should be corrected to $d^f$, consistent with the notation in Section 3.3, where it represents the sequence of dance in the frequency domain representation. $\hat{d^f}$ denotes the dance sequences in the temporal domain representation that is obtained by using $d^f$ as the query input to the Dance Decoder.

2. In Figure 2, why does d^f input into the “Temporal Encoder”?

In Figure 2, we mistakenly depicted the two encoders. We apologize for the confusion caused by it, This has been corrected in the revised version of the manuscript.

3. What is the meaning of the arrows from the “Temporal Encoder” and the “Freq Domain Encoder” modules to the “Dance Decoder”?

The arrows from the “Temporal Encoder” and the “Freq Domain Encoder” modules to the “Dance Decoder” indicate that the outputs at the tails of the arrows serve as the query inputs to the Dance Decoder.

Once again, we express our sincere gratitude for the time you have invested in reviewing our work. Your feedback have significantly contributed to the enhancement of our paper. If our revisions and responses meet with your expectations, please consider increasing the score.

[A] Tseng, J., Castellon, R. and Liu, K., 2023. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 448-458).

Reviewer YEBL,

We hope our previous response addressed your concerns. If there are any additional points or suggestions you'd like us to address, please feel free to let us know. Thank you again for your valuable feedback!