Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment

Q1: "it would be beneficial to provide more data statistics, such as movement speed and action distribution"

A1: We compute the statistics of speed distributions for different types of dances. This part is revised in Appendix B.1 (pp.14).

Q2: "In Equation 1, it's stated that both p and M are inputs, but in Figure 3, only p is shown as an input"

A2: The VQ-VAE is designed to encode $p$ and subsequently decode it into both $p$ and $M$. $p$ and $M$ in Equation 1 are ground truth targets in the learning, not inputs.

Q3: "It would be interesting to explore which signal, between music and leader motion, has a more dominant influence."

A3: As required by the reviewer, we randomize music during the testing phase. The experimental results are shown in Table 4 （pp.17 Appendix D.1). For “Duolando w/o. RL”, incorrect music input causes a reduction in all motion metrics and music metrics. Specifically, the BAS decreases by 12%, while the Beat Echo Degree (BED) decreases relatively less (8%), indicating that leader motion is more crucial in our proposed framework.

Q4: "There is some ambiguity in the terminology between 'duet dance generation' and 'dance accompaniment'."

A4: We appreciate the reviewer indicating that. We have revised our paper.

Q1: "How is music represented? "

A1: We apologize for the oversight.

The music features $\bf{\it m}$ are extracted from the audio signal using the public audio processing toolbox Librosa (McFee et al. 2015), including mel frequency cepstral coefficients (MFCC), MFCC delta, constant-Q chromagram, onset strength and onset beat, which are 54-dimension in total, and are mapped to the same dimension （438）of GPT via a learned linear layer.

This part was originally in the main text but was inadvertently omitted when moving it to the appendix due to page limitation. We have added this back in the revised version (pp.15 Appendix B.2).

Q2: "What if the music is not fed as input?"

A2: (1) To explore the effect of the music signal input, we train a new variant GPT without the slots for music: Duolando w/o. RL $m$. The experimental results have been updated in the revised draft (Table 4, Appendix D.1, pp. 18). Without the music input, the follower achieves a comparable Beat-Align Score (BAS) to that with music input since the follower is explicitly anchored to the leader's position and will follow the rhythm indirectly.

(2) The above alignment is under the condition that the follower's motion aligns with the music well. In the other experiment, we randomized the music input in the test phase. As shown in Table 4, the BAS of Duolando w/o. RL $m$ is 0.1657, lower than that with music input (0.1840), which shows the music encoding will significantly impact the motion when music is not aligned with the movement.

Q3: "... ablations against on-policy RL nor ablations on the reward design"

A3: (1) We present the reward curves of the proposed off-policy RL and the on-policy AC (Siyao et al., 2022) on test set in Figure 8, pp.18 and Appendix D.2, pp. 19. The average sequantial reward for the off-policy RL shows a steady increase; in contrast, the on-policy RL trend is less stable and even collapses at around the 30-th epoch. This aligns with the phenomena reported in (Siyao et al., 2022) that excessive fine-tuning in on-policy AC leads to instability.

(2) Given that our reward system comprises only a single reward, conducting a compositional ablation study is impractical. On the reward itself, we introduced a threshold of for the penalty. If without this threshold, the follower tend to perform excessively redundant small steps to align the translation strictly.

Q4: "... whether ... performed by professional dancers"

A4: All dancers involved in our recordings are professionally trained and have won awards in professional competitions. After a thorough comparison, we confirm that the segments provided in our demo align with the live video footage. The perception of misalignment between motion and music that the reviewer notes might be attributed to the concept of 'rubato'. In ballroom dancing, particularly in smooth styles, dancers may intentionally delay a step or movement to coincide with a specific musical element or to create a desired effect or expression. This artistic choice enhances the performance's expressiveness but might appear as a misalignment to those unfamiliar with the technique.

Q5: "... outputs have penetration."

A5: Indeed. As the first solution to dance accompaniment, while our method is not yet perfect, it effectively tackled some key issues through the incorporation of Interaction Coordination (IC), relative translation (tr), and Reinforcement Learning (RL). These improvements resolve issues like action irrelevance, inappropriate relative displacement, and follower's lower body incompatibility, making our approach significantly superior to existing solo dance methods in the interactive scenarios. It can serve as a solid baseline for this newly proposed task.

Dear Reviewer CR8y,

We sincerely thank your great effort in reviewing our work. We have improved and revised our submission based on your comments. As the deadline is approaching, please let use know if you have any follow-up concerns. We sincerely hope you can consider our reply in your assessment. Thank you!

Sincerely, ICLR 2024 Conference Submission2432 Authors

Q1: "The paper only addresses the problem of misalignment of lower-body motion and full-body motion; why the author just stopped here? ... Has the author tried to address such artifacts? Does the author believe such off-policy tuning combined with some heuristic reward function can reduce most artifacts?"

A1: We appreciate the reviewer's suggestions for a more in-depth exploration of the RL rewards to enhance dance quality.

(1) Our primary focus in this work is to enable the follower to dance responsively to the leader. Beginning with a solo dance GPT structure similar to Bailando (Siyao et al., 2022), we employ Interaction Coordination (IC) and explicit relative translation (tr) to make the follower correspond with the leader on movement and positioning, respectively. The proposed RL concentrates on solving/suppressing the side effect of tr -- the skating effect, caused by misalignment between lower body and global shift.

Regarding penetration issues, potential post-processing exists in the industry and the academy [1,2,3]. A feasible RL solution can be rewarding non-intersection while penalizing the corresponding intersecting parts based on penetration depth. For instance, intersections on hands and arms should lead to a penalty on the upper body, while that on the trunk should impact translation (tr).

(2) We acknowledge that RL alone may not resolve all issues. In practical applications, graphical post-processing is expected for further quality enhancement.

Reference

[1] Smith et al. Open dynamics engine, 2005

[2] Pavlakos et al.. Expressive body capture: 3d hands, face, and body from a single image. CVPR 2019.

[3] Müller et al. On self-contact and human pose. CVPR 2021.

Q2: "It would be beneficial to add some new metrics to evaluate this specific improvement (of RL)."

A2: To show effect of RL on reducing skating effect, we propose a metric named slipping ratio (SR). Specifically, we compute the percentation of the length where the feet/legs do not move obviously but the global shift is significant (> 1 cm/frame). The results are shown in Table 4, Appendix D.3, pp.19. Experimental results show our proposed RL can reduce over 60% of skating in that without RL.

Q3: "Why not use 3D-rendered motions (for user study)?"

A3: As required by the reviewer, we conduct a new user study using 3D-rendered SMPL-X models. We also updated the results on the revised draft. Generally, the average winning rates do not change greatly and do not change the conclusion.

Q4: "What is the training time for each of the stages?"

A4: The training time for VQ-VAEs is about 2 days. The training time of GPT in the supervised learning stage is around 1.5 days and that of the RL stage is about 18 hours. Four NVIDIA V100 GPUs are used for training.

Q5: "How to evaluate the quality of the motion-capture? It seems there are some flickering frames and some exaggerated hand motions"

A5: Thanks for bringing up the issues with the mocap evaluation. We have observed similar phenomena in our data and have undertaken methods to refine mocap data quality along two dimensions:

1. Flickering frames: Inaccuracies arise when marker points are potentially obscured during the collection process, resulting in inaccuracies in specific frames. To address this, we introduce two kinds of information: (1) Single-frame confidence. We leverage marker point confidence during the fitting process, excluding points with low confidence. (2) Temporal smoothness. If most of the markers within a frame have low confidence, the fitting outcome may be suboptimal. We introduce temporal information in post-processing to first detect abrupt changes from the temporal axis. Subsequently, we eliminate such frames and employ interpolation to complete the missing frames.
2. Exaggerated hand motion: This issue typically arises from either accumulated errors in the original glove data or a lack of constraint information for joints. To avoid this, explicit constraints are imposed on each finger and wrist joint according to human anatomy. For instance, finger joints are restricted from undergoing self-rotation, and hand wrists are also constrained against self-rotation.).

Dear Reviewer otBs,

We sincerely thank your great effort in reviewing our work. We have improved and revised our submission based on your comments. As the deadline is approaching, please let use know if you have any follow-up concerns. We sincerely hope you can consider our reply in your assessment. Thank you!

Sincerely, ICLR 2024 Conference Submission2432 Authors

Q1: “Output quality issues”

A1: As the first solution to dance accompaniment, while our method is not yet on par with professional human dancers, it effectively tackled some key issues through the incorporation of Interaction Coordination (IC), relative translation (tr), and Reinforcement Learning (RL). These improvements resolve issues like action irrelevance, inappropriate relative displacement, and follower's lower body incompatibility, making our approach significantly superior to existing solo dance methods in the interactive scenarios. It can serve as a solid baseline for this newly proposed task.

Q2: “Potential copyright issue of the music mp3… why use copyrighted music”

A2: Choice of Music. These tracks were chosen by the performers from their familiar collections to ensure the quality of performance.

‘Fair Use’ [1, 2] Justification. The music segments are modified (clipped, and partially slowed down), deviating from their original form. These modifacations, coupled with their use solely for model training/testing rather than entertainment, comply with fair use criteria. Additionally, the academic nature of our work and the absence of public searchability (we won’t provide music info beyond the audio) avoid potential commercial impact. Therefore, the music included in our dataset falls under ‘fair use’ provisions, while it also align with previous practices [3, 4] in dance geration domain, presenting a relatively low risk. We have also updated these facts in the ethical statement in our paper.

Contingency Plans. Should the need arise to remove certain tracks, we can provide them in a non-playable, digitized format suitable for model training/testing. Or at least, we can offer only the titles and relevant details (like YouTube links and specific segment/speed information) with automatically downloading/processing scripts, a common method used in prior academic datasets [5, 6].

Reference

[1]https://guides.nyu.edu/fairuse#:~:text=Fair%20use%20allows%20limited%20use,automatically%20qualifies%20as%20fair%20use

[2] https://en.wikipedia.org/wiki/Fair_use

[3] Lee et al., Dancing to Music, NeurIPS2019

[4] Huang and Hu et al., Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning, ICLR 2021

[5] Siarohin et al., First Order Motion Model for Image Animation, NeurIPS 2019

[6] Miao et al., VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild, CVPR 2021

Q3: “Limited reusability of insights (to broader community other than dance generation).”

A3: Our method can extend beyond dance generation and offer insights for human-human interaction tasks, a key area of 3D human motion and virtual reality. Additionally, it holds promise for integrating 3D human interaction within unified language models (GPT), which could interest a broader range of researchers.

Furthermore, our proposed DD100 dataset contributes strong interaction data with long average durations to the human-human interaction research community.

Q4: “Why not pre-train models on noisy dance data”

A4: This is indeed a great idea. However, at present, there is a lack of such large-scale noisy data on interactive duet dance (e.g., that from monocular video reconstructions).

Q5: “There doesn’t appear to be any details about the music tokenization strategy used in this work - can the authors clarify how music features are represented?”

A5: We apologize for the oversight.

The music features $\bf{\it m}$ are extracted from the audio signal using the public audio processing toolbox Librosa (McFee et al. 2015), including mel frequency cepstral coefficients (MFCC), MFCC delta, constant-Q chromagram, onset strength and onset beat, which are 54-dimension in total, and are mapped to the same dimension of GPT via a learned linear layer.

This part was originally in the main text but was inadvertently omitted when moving it to the appendix due to the page limitation. We have added this back to the revised version (pp.15 Appendix B.2)

Q6: “I was confused by the use of lookahead - would the lookahead model actually work in a real-time dance setting”

A6: The 'look-ahead' feature views future music and leader actions (4 seconds ahead in our work) to generate smoother movements, as demonstrated in Table 2 (Duolando w/o. RL LA) and in the supplementary video at 7'39". However, it indeed does not yet meet real-time requirements.

A potential solution could be to predict future music and leader motions in advance rather than viewing them from the data.