InterDance: Reactive 3D Dance Generation with Realistic Duet Interactions

Thank you for taking the time to review our work and for your valuable feedback. We greatly appreciate your words of “dance dataset is valuable, a nice touch,  interesting, etc”. We hope the following clarifications help to address your concerns.

W&Q1 Significance and contribution

Duet dance generation is a challenging task, not only due to the high cost of acquiring high-quality duet datasets but also because current methods do not adequately consider the motion representation and the strategies for generate strong interaction. Our work, on one hand, provides a high-quality duet dataset, and the ablation study in Table 3 of the appendix demonstrates that the inclusion of InterDance effectively alleviates the issue of limited duet dance data. On the other hand, we are the first to incorporate body surface information in duet motion generation tasks and propose a reasonable representation along with a duet motion generation network that supports both reactive generation and simultaneous duet motion generation.

W&Q2 Importance of the task

Duet dance is widely used in dance education. However, directly asking two beginners to practice duet dance can be challenging, as it typically requires an experienced dance partner to assist with the teaching, significantly increasing the teaching costs. Therefore, an important application is to generate the follower's movements in a VR environment through human motion generation, helping dancers learn different duet dances.

Meanwhile, our method can also directly generate duet dances. To the best of our knowledge, we are the first to use a network-based approach for generating music-driven duet dances, and we have achieved promising results. This approach can be widely applied in dance animation and film production.

W&Q3 Long Dance Generation

Our work focuses primarily on addressing the fine-grained interaction quality issue. Our method also supports autoregressive generation of long dances using a sliding window approach. Please refer to our project page for videos showcasing long dance generation.

W&Q4 Qualitative results

Thank you for your suggestion. We have added more visual results to our project page(https://inter-dance.github.io/).

Thanks for the response from the authors. I acknowledge the new dataset can be a nice asset for the community, and the generation of duet dance is a nice trial. However, my concern regarding this work remains as follows:

1. This work claimed two contributions, one is on a large-scale dataset, while the other is on better reactive duet dance generation. However, these two contribution are both be claimed in Duolando, which proposed duet dance dataset (117 mins), as well as the first effort for reactive dance generation. I would not say a 235 min dataset makes much difference on the quantity comparing to duolando. Meanwhile, though the authors claimed a better motion representation on contact, I did not say particular visual improvement in close-proximity contacts. The generated results remain surreal. Some visual ablations may be helpful. However, the insignifiance of this work is doubtful.
2. Reactive duet dance, the main focus of this paper, still looks a relatively minor problem to me. The motions of close-proximity interactions, especially in duet dance, are highly correlated. They are even temporal codependent. It's quite weird to generate the whole other sequence given one's movement. Though this work seems to address the additional task of both duet dance generation, they are not claimed as main contribution.

Overall, I acknowledge that the data are the more the better for the community, and the nice trial on duet dance generation as a side-product. I would like to rise my rating to borderline reject.

Dear reviewer jdiH,

We sincerely appreciate you for raising the score. Your support and encouragement will greatly motivate and advance our work.

Sorry for the delayed response, as we wanted to address your concerns with experimental data. First, we would like to thank you for recognizing our dataset. We believe that both our dataset and code will significantly contribute to research on interactive human motion.

Second, regarding the improvement of our method, as shown in Table 3 of our manuscript and the demo video on our website, we observe that our method outperforms previous SOTA methods on most metrics. We agree with your point that there is still much room for improvement, especially in terms of the close interactions. However, our method has already made an important step forward in interactive motion generation, particularly by considering body surface information for the first time, which will provide a solid foundation for future research. To further demonstrate the effectiveness of our method, we further trained and tested our method on the DD100 dataset to provide an additional comparison with Duolando. The results are as follows, on the DD100 benchmark, our method also outperforms Duolando on most metrics.

Third, we agree with your point that generating duet motions simultaneously is an important issue. In this work, we have also focused on this problem, as shown in Table 7 of the manuscript and in the supplementary materials under MORE DETAILS OF DUET DANCE GENERATION and MORE DETAILS OF THE INTERACTIVE DECODER BLOCK.

To our knowledge, we are the first to use a single framework to address both reactive motion generation and duet motion generation. We have provided the community with a new approach that unifies these two tasks. As you mentioned, our work places more emphasis on reactive dance generation because duet interactive motion generation is still in the early stages of research, and there is still significant room for optimization in some strong interactive motions. We first focus on improving the quality of interactive motions generated by our method through reactive dance generation. Thanks to our unified framework, both tasks can benefit. Additionally, in VR scenarios, there are also some applications where the leader's dance of human is used to generate the follower's dance of virtual character, such as for duet dance practice or entertainment purposes.

We hope that these additional experiments have further addressed your concerns. If there are any other questions or suggestions, please do not hesitate to let us know.

Thank you once again for your invaluable contribution to our research.

Warm Regards,

The Authors

We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We appreciate your recognition of our dataset, paper, quantitative evaluations and the positive feedback on the motion representation being "makes sense". We hope the following clarifications help to address your concerns.

W1 Overall performance.

See the first on the left of Figure 7, the image here demonstrates that after applying the interaction guidance we proposed, we effectively resolved the self-penetration issue, and this movement is physically plausible. Duet dance generation is a challenging task. Although there are still some shortcomings in performance, our method represents a significant improvement over previous work, especially with the inclusion of body surface modeling.

W2 Comparasion with Duolando

In our demo video, our method demonstrates reasonable interaction quality, while Duolando sometimes fails to track the leader's motion effectively, as seen between 2:20 and 2:50.

In our comparison experiments, we achieve state-of-the-art results in most metrics for both reactive dance generation (Table 3) and duet dance generation (Table 7), with our method significantly outperforming all previous methods, including Duolando, especially in terms of interaction quality.

W3 Technical contributions.

Regarding motion representation, our proposed approach is not a simple combination of existing methods. Unlike HumanML3D[1], which uses velocity to express root translation and rotation but lacks surface vertices, and COINS[2], which does not account for canonical space processing, our representation introduces a novel approach specifically designed for interactive motion generation.

Regarding the method, our framework uniquely supports both reactive generation and duet dance generation within a single structure. It effectively leverages contact guidance and penetration guidance to optimize fine-grained interactions, marking the first time these techniques have been introduced in duet motion generation research.

W4 Qualitative results

Thank you for your suggestion. We have added more visual results to our project page(https://inter-dance.github.io/).

Q1 FID calculation

We calculate the Fréchet Inception Distance (FID) between the generated motion and the real data across the entire InterDance dataset following previous works [3,4]; therefore, the FID of the GT test set is not zero. We use motion kinematic features [5] and graphical features [6] to compute the FID distance.

Due to the complexity of dance movements and the scarcity of dance data, directly comparing the FID difference between the generated motions and the dance data test set is unlikely to provide an effective evaluation. Therefore, comparing the FID between generated motions and the entire dataset is a widely used approach. A large number of dance generation works also follow this FID calculation standard, as it provides a more accurate and objective evaluation of dance generation algorithms.

Q2 Contact label

Our motion representation includes contact labels, and we leverage contact guidance in our method to further utilize this information for promoting more accurate and fine-grained interactions. In duet dance generation, the motion features of both dancers, including the contact labels, are predicted. In reactive dance generation, the the leader’s motion is provided, which also include contact label. This setup is motivated by the following considerations:

1. In some practical applications, even given a leader motion without contact label, we can retrieval the nearest neighbour contact labels from the training set, making our method suitable for more practical applications.
2. By replacing the contact labels in the leader’s motion, we can do data enhancement to get more diverse and high-quality interactive motions. Obtaining paired interactive motions is inherently challenging, and this approach allows us to overcome this difficulty while advancing the research on duet motion generation. We will explore this issue further in our future work.
3. A unified representation and algorithmic framework can be applied to both reactive dance generation and duet dance generation.

[1] Generating Diverse and Natural 3D Human Motions From Text.

[2] COINS: Compositional Human-Scene Interaction Synthesis with Semantic Control.

[3] Bailando: 3d dance generation by actor-critic gpt with choreographic memory.

[4] Duolando: Follower gpt with off-policy reinforcement learning for dance accompaniment.

[5] Fmdistance: A fast and effective distance function for motion capture data.

[6] Efficient content-based retrieval of motion capture data.

Dear Reviewer 5vWh,

Thank you for reviewing our response and engaging in a positive discussion. We are currently conducting additional experiments based on your suggestions and striving to complete them as quickly as possible. Thanks for your patience.

Once again, we sincerely appreciate your invaluable contribution to our research.

Warm regards,

The Authors

Dear reviewer 5vWh,

Tanks for your further valuable suggestions.

We followed your suggestions and conducted experiments by removing the contact label from the given leader. In the table below:

1. The first row Ours (Repaint w/o leader label), indicates that we applied the diffusion repainting technique during the inference phase to generate the follower dance under the control of leader motion (without the leader contact label).

2. The second row, Ours (Trained w/o leader label), shows the results when we retrained our network without providing the leader contact label to generate the follower dance, and then use the new trained model to generate follower dance conditioned by the given music and leader dance (without the leader contact label). The videos results are also updated to our project page (https://inter-dance.github.io/).

3. The third row represents the metrics of our method from the manuscript, where the given leader motion includes the leader contact label.

4. The last row shows the metrics of Duolando, trained and tested on the InterDance dataset, also without given contact label. Ours (Train w/o leader label) outperforms Duolando in metrics such as FID, PR, CF, and others. It even achieves the highest metrics in BED and BAS.

Additionally, we trained and tested our method on the DD100 dataset to provide a more fair apple-to-apple comparison with Duolando, both methods generate follower dance conditioned on the given music and leader dance without contact label. The results are as follows, on the DD100 benchmark, our method also outperforms Duolando on most metrics.

We hope that these additional experiments have further addressed your concerns. If there are any other questions or suggestions, please do not hesitate to let us know.

Thank you once again for your invaluable contribution to our research.

Warm Regards,

The Authors

We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback.*  We greatly appreciate your recognition of a new large-scale dataset, a new motion representation and an interaction refine guidance sampling strategy. Please see below for our responses to your comments.

W1: Comparison on DD100 dataset

Thanks for pointing this. While we conducted comprehensive comparisons with various SOTA methods on the InterDance dataset, including reactive and duet dance generation, we did not include this comparisons due to we focus on our dataset and method.

W2: The effectivess of $\mathcal{L}_{RO}$ and vertices

The $\mathcal{L}_{RO}$ and vertices are highly effective, particularly in optimizing strong interactive motions. As shown in Table 3, Motion Quality primarily evaluates the quality of individual motions, while Interaction Quality assesses the quality of interactions. Although adding them results in a deterioration in FID}_k, the metric of Interaction Quality, such as FID_cd , shows a significant improvement.

W3: The relationship between interaction and penetration

We agree with your comments and have made the corresponding modifications. Please refer to our manuscript, where the revised sections are highlighted in blue.

W4: The novelty

Regarding the dataset, we conducted ablation studies as shown in Table 3 of the appendix. Compared to training solely on the DD100 dataset, joint training on both DD100 and InterDance significantly improves various metrics. Additionally, our dataset features a broader range of dance styles, making it a substantial contribution to the fields of duet dance generation and interactive motion synthesis.

For motion representation, our proposed approach is not a simple combination of existing methods. Unlike HumanML3D, which uses velocity to express root translation and rotation but lacks surface vertices, and COINS, which does not account for canonical space processing, our representation introduces a novel approach specifically designed for interactive motion generation.

Regarding the method, our framework uniquely supports both reactive generation and duet dance generation within a single structure. It effectively leverages contact guidance and penetration guidance to optimize fine-grained interactions, marking the first time these techniques have been introduced in duet motion generation research.

Finally, we have also followed your kind suggestions and added references to strengthen our work.

Dear reviewer UHsx,

We appreciate your responses and further valuable suggestions.

Apologies for the delayed response, as we have struggle to conducting experiments over the past few days. We have finally tested our method on the DD100 benchmark. As shown in the table below, we transformed the DD100 data into our motion representation and coordinate system for training, and then converted the generated results back to the original motion space of DD100 for testing. Compared to the metrics listed in the original Duolando paper, our method outperforms Duolando on most metrics. This is because our method’s motion representation incorporates human surface information, and the diffusion structure makes it easier to optimize fine-grained interaction quality.

Regarding to the technical contribution, we are the first to use a single framework to address both reactive motion generation and duet motion generation. We have provided the community with a new approach and motion representation that unifies these two tasks. We believe that our dataset and code will significantly contribute to research on interactive human motion.

We hope that these additional experiments have further addressed your concerns. If you have any other questions or suggestions, please do not hesitate to let us know.

Thank you once again for your invaluable contribution to our research.

Warm Regards,

The Authors

Thanks for your positive recommendation and valuable suggestions. We are happy to see your recognition of our proposed dataset, motion representation and method. Please see below for our responses to your comments.

W1: More details of dataset

We summarize the dance data for different genres in terms of total duration (in minutes), number of frames, number of sequences, and their percentage of the entire InterDance dataset. We have added this table to our manuscript.

W2: How to select the 655 vertices?

We follow the settings of COINS[1] and PyMAF-X[2] to downsample the vertices of SMPL-X. These 655 vertices are not randomly sampled and preserve the topology, allowing for direct rendering of the human mesh on these 655 vertices. We have added these details in our manuscript.

W3: Foot contact loss is not clearly stated

Thanks for your suggestion, we have add the formulation to our manuscript. Please see Line 318-319, the revised texts in our manuscript are marked in blue.

W4: The symbol $f$ used for foot contact and follower motion

Thanks for your thorough review. We have marked the symbols related to foot contact as $foot$ and made the revisions in our manuscript.

W5: One important realted work

Thanks for your reminder. This is indeed an important work. We have added a citation for it and included a comparison with Motorica Dance in Table 1 of our manuscript.

W6: Line 72 priors, without -> prior without Thanks for your detailed review. We have fixed this in our manuscript.

Questions of dancers

Each category has multiple pairs of dancers. All the dancers have signed consent forms, allowing us to publicly release the data.

References

[1] Compositional humanscene interaction synthesis with semantic control

[2] PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images

Dear reviewer yXpV,

We wish to express our sincere appreciation to you for recognizing the substantial significance of our InterDance's contribution.

Your acknowledgment holds great importance to us and serves as a meaningful validation of our dedicated efforts to advance this research.

If you have any other questions or suggestions, please do not hesitate to let us know.

Thank you once again for your invaluable contribution to our research.

Warm Regards,

The authors

Response of Questions:

Dataset

Thanks for your suggestion. As shown in Table 3 of the appendix, we have already jointly trained the neural network on DD100 and InterDance. We will provide the code to support the combined use of both datasets.

Motion representation

Decoupling root translation and angle information is more conducive to learning motion priors, and this is exactly what we have done. Our representation records the body’s joint and vertex information relative to the root node, excluding global orientation. For root translation and angle, we used absolute position and angle rather than velocity and angular velocity because, in duet interaction motions, the relative position and orientation between the two individuals are critically important. Using velocity and angular velocity would make it challenging to optimize their relative position and orientation effectively.

Evaluation 1: Metric

• We calculate the Fréchet Inception Distance (FID) distance between the generated motion and the real data in whole InterDance dataset following Duolando. We use motion kinematic feature [3] and graphical feature [4] to calculate the FID distance.
• Please see weaknesses response above.

Evaluation 2: Baselines

• For EDGE, we encoded the leader motion features using a Transformer and concatenated them with music features as the new condition to generate follower motion. For InterDance, we replaced the text encoder with a Transformer-based music encoder. These changes have been updated in the manuscript. We have added these details to the manuscript.
• Duolando was trained on the InterDance dataset and evaluated on its test set to ensure a fair comparison.
• We argue that Duolando’s VQ-VAE+GPT framework encodes motion into a coarse-grained codebook. After training, the weights of the VQ-VAE are frozen, and therefore the following GPT network is challenging to learn fine-grained interactions. Although Duolando incorporates some optimization modules to enhance duet interaction performance, our method demonstrates superior fine-grained interaction quality compared to Duolando.

Result videos

• The generated videos we showcase are all produced conditioned on the test set's music or leader dance
• To ensure a fair visual comparison, we trained other methods on InterDance and tested both our method and the others on the InterDance test set under the same conditions.
• EDGE is a single-person motion generation framework, and while it demonstrates excellent performance in solo dance generation, it lacks optimization for duet motion generation. As a result, the generated duet dance is noticeably inferior to other methods. We believe this is not comparable, so we did not showcase EDGE's generated results.
• In our result videos, you can see some generated outputs. We have added more generation results. Please refer to our project page https://inter-dance.github.io/ for further details.

References

[1] Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion.

[2] Jukebox: A Generative Model for Music.

[3] Fmdistance: A fast and effective distance function for motion capture data.

[4] Efficient content-based retrieval of motion capture data.

Thank you for taking the time to review our work and for your valuable feedback. We greatly appreciate your recognition of our contributions, including the mocap dataset with fine-grained contact information, the proposed motion representation correctly predicting contact, and the diffusion-based method and achieve competitive results. We hope the following clarifications help to address your concerns.

Response of Weakness:

Evaluation 1: Simple strong baselines

We have followed your suggestions and added these baselines in our manuscript. The table below shows the results, tested on the InterDance dataset.

NN-motion refers to “Return a NN motion clip of a follower from the training set where the distance is calculated based on motion-based distance”. Given an input leader motion, we find its nearest neighbor from the training set and use its corresponding follower segment as the prediction.The distance is calculated based on the MSE distance of the motion clips.

NN-music refers to “Return a NN motion clip of a follower from the training set where the distance is calculated based on a SOTA music embedding distance”. Given an input music clip on InterDance test set, we find its nearest neighbor music clip from the training set and use its corresponding follower motion as the prediction. The distance is calculated based on the MSE distance of the music features extracted by the advanced music feature extractor Jukebox[2].

As table below shows, Ours (mirror) refers to “Mirror the motion of the leader”. We use the mirrored leader motion and corresponding music from the InterDance test set as conditions to generate the follower motion and test its metrics.

---

Evaluation 2: Metrics

We used a comprehensive set of metrics to evaluate Interaction Quality, including FID_cd to measure interaction quality, DIV_cd to assess the diversity of interaction motions, PR and CF to evaluate the physical realism of the interaction, and BED to measure the rhythmic alignment of the duet dance.

The PCC in [1] can be used to measure the covaries of two 1D time series data. However, in duet dance, a more appropriate metric is BED.

The TLCC correlation in [1] can analyze the correlation between two sequences at different time lags, making it more suitable for measuring the correlation of two time series with causal relationships, therefore is commonly used in two-person dialogue scenarios. However, many movements in duet dances occur synchronously, involving more complex interaction dynamics. Therefore, measuring the FID distance of the cross-distance can better assess interaction quality by comparing the similarity between the generated motions and the Ground Truth dataset.

Evaluation 3: Baselines

To ensure a fair comparison, we trained and tested several SOTA methods, including Duolando, on the InterDance dataset. The experimental results are presented in Table 3 of the manuscript. Since Duolando was also trained on the InterDance dataset, this constitutes an apples-to-apples comparison.

Exposition: Details of the user study

We recruited 40 participants for the user test, including 14 individuals with a dance background, consisting of students with dance experience and professional dancers from dance training schools, as well as 26 randomly recruited participants with no dance background. Each participant received a payment of 50 RMB. None of the participants had prior knowledge of the project before the test. To ensure fairness, we randomly shuffled the order of the results generated by our method and the different comparison results. The videos were presented in 30fps, 1280x1024 mp4 format, accompanied by background music. We have updated the details of the user test in the manuscript.

Smaller notes: The caption of Table 1

Thank you for your thorough review. We have corrected this description in the manuscript.

Limitations

Thank you for your suggestions. We have added more discussion in the Limitations section.

Dear Reviewer oJee,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insightful questions and valuable feedback have been extremely beneficial.

During this period, we have made consistent efforts to add supplementary experiments and videos to enhance the quality of our work. To better response your suggestion regarding the Baseline of Weaknesses, we trained and tested our method on the DD100 dataset to provide a more fair apple-to-apple comparison with Duolando. The results are as follows, on the DD100 benchmark, our method also outperforms Duolando on most metrics. Especiallyin the $\text{DIV}_g$, $\text{CF}$ and $\text{BED}$, our method significantly outperforms than Duolando.

We hope that these additional experiments have further addressed your concerns. If you have any other questions or suggestions, please do not hesitate to let us know.

Thank you once again for your invaluable contribution to our research.

Warm Regards,

The Authors