PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance

Q7: Issue of Ground Truth

R: We acknowledge that hand motions can exhibit notable variability among pianists, even when playing the same passage. Despite individual nuances, these movements typically align with established principles of piano playing technique. Instead of attempting to strictly mimic an individual’s motions, our approach leverages a generative model to learn generalized hand movements that encapsulate the essence of piano playing techniques. This enables the method to produce results within a distribution of natural human variation, aligning with the diverse range of human movement patterns.

Q8: Difference of Minor Quantitative Variation

R: A minor quantitative variation, such as a shift from FGD=0.360 to FGD=0.351, suggests a high similarity between the distributions of generated and real motions. It could reflect subtle differences in finer motions, such as minor finger tremors or slight adjustments, which may still impact the perceived naturalness in detailed movements. Minor metric variations observed may stem from noise, random fluctuations, or subtle hand-specific movement discrepancies, rather than intrinsic differences in model performance.

Q9: User Study

R: In the R5 of Global Author Rebuttal, a small range of Mean Opinion Score (MOS) is presented. The human assessment results in the table align closely with our proposed benchmark metrics. For example, in rhythmic synchronization, EmoTalk shows the lowest performance, while our Transformer-based method performs the best—consistent with the FID and WGD scores in the quantitative evaluations, validating these metrics’ ability to assess short-term action coherence. For smoothness, the Transformer-based model is slightly less effective than the Mamba-based model, aligning with the Smooth metric. In motion diversity, the Transformer-based method scores significantly higher than others, aligning with FGD and FID scores and supporting their effectiveness in evaluating overall hand motion distribution.

Q10: The layout of Table 3

R: Thanks for your advice. We have reorgnized Table 3 to place the Right Hand to the right of the Left Hand accordingly.

Q11: Interpretation of Qualitative Results

R: In Figure 4, the results of EmoTalk and LiveSpeaker exhibit some positional differences. In particular, the yellow hands (LiveSpeaker) show a clearer distinction, which emphasizes a positional bias in the simulation.

While variations in hand positions are expected, especially for tasks like piano playing, even small deviations in finger and hand positions are important for maintaining realistic motion. By refining the movement to better match real-life performances, we can achieve more accurate and convincing simulation of human actions.

Q12: Data Availability and Compliance Concern

R: Thanks for your comments. Please refer to our previous response (R3) provided in Global Author Rebuttal. All data in our dataset has been obtained with explicit consent from the creators, who provided their original videos directly. If creators choose to retract their videos from public view, we are committed to honoring their wishes and will remove the data from our dataset as needed.

Q1: Emphasis on the Dataset and Evaluation Metrics

R: Thanks for your suggestion. We have enhanced our description of the dataset and measures. As we recognize the importance of focusing on the dataset and metrics, we believe that including the proposed model serves as a practical demonstration of the dataset’s reliability and potential applications. The model establishes an initial benchmark for future works, offering a foundation for comparative studies. It demonstrates the potential of the dataset to enable learning of complex, high-quality hand motions that are synchronized with music. As mentioned in comments, the comparison results with EmoTalk and LivelySpeaker show that these baseline methods have close performance. We highlight that these methods are served as valuable references, the design of our model along with our collected dataset, showcases its scalability for performance improvements and reveals insights into hand motion generation’s unique demands in music contexts. In terms of the evaluation metrics, the additional details are provided in Q6.

Q2: Dataset Bias and Style Representation

R: Thank you for bringing this to our attention. Our dataset currently encompasses a variety of musical styles, including classical and modern pieces, while specific styles and genres haven’t yet been labeled or quantified. This may introduce certain biases. In future work, we plan to annotate and categorize each music piece by title and style, like the MAESTRO dataset. While this process will require significant time and resources, we consider PianoMotion10M as an evolving project. Moving forward, we plan to reach out to creators on other video sharing platforms, such as YouTube and TikTok, to further enhance the diversity of our dataset.

Q3: Mistype of Learning Rates

R: That was indeed a typo. We are using standard notation, with learning rates of $2 \times 10^{-5}$ and $5 \times 10^{-5}$. The text has already been corrected accordingly.

Q4: More Details of Dataset Process

R: Thanks for your question. As mentioned in the R2 of Global Author Rebuttal, we have incorporated more details of dataset process and revised the manuscipt accordingly.

Q5: More Details of Implementation

R: Our large HuBERT-Transformer model has approximately 557 million parameters, as described in Table 3 of the manuscript. The training process took around 1 day on a GPU with 24GB RAM. Regarding the release of the trained weights, the checkpoint for our base model is available at PianoMotion10M. We intend to release the larger pretrained checkpoints on Hugging Face upon the paper's acceptance.

Q6: More Details of Evaluation Metrics

R: Thanks for your insightful comments. We understand the importance of clarifying the relevance of the metrics to the real problem and addressing the concerns regarding the comparison with the ground truth. To make it clear, we compare the generated hand poses with the ground truth hand poses with joint Euler angles, not the rendered hands. The metrics we used are designed to evaluate different aspects of the generated motion in relation to the true hand poses. Since directly comparing the generated data with the ground truth is unreasonable, we primarily use distributional difference metrics to evaluate our generated results. In piano performance, the left and right hands perform distinct actions, requiring separate evaluations.

• FID (Fréchet Inception Distance) is used to evaluate the overall similarity between the distributions of generated motions and the ground truth, helping to assess the quality of the generated motion sequences at a high level.
• FGD (Fréchet Ground Distance) and WGD (Wasserstein Ground Distance) focus on the similarity of individual hand movements, specifically assessing how well the generated hand poses match the ground truth in terms of the overall motion characteristics. FGD evaluates long-term distribution similarity across entire sequences, providing a measure of global alignment between generated and real data. In contrast, WGD is based on parametric GMM (Gaussian Mixture Model) distributions, which focuses on short-term similarity by analyzing smaller time segments, allowing it to capture finer and localized motion patterns within the sequence.
• PD (Position Distance) is used to evaluate the accuracy of the hand poses at specific locations (e.g., joint positions), ensuring the generated hand poses are spatially accurate. A higher position distance denotes larger errors of the hand position.
• Smoothness assesses the continuity and smoothness of the generated hand motion, highlighting if the motion sequence has unnatural jumps or jitter. A higher smoothness value indicates greater jittering in the generated motion.

Q1: Human Evaluation Results for Generated Motions

R: Thank you for the suggestion. We agree that human evaluation can provide valuable insights into the realism and quality of the generated motions. In Global Author Rebuttal (R5), we provided a small-scale user study in various aspects of generating action quality. Since we currently focus on quantitative metrics, we are planning to conduct more extensive human evaluations to further validate the effectiveness of our model. These results will be incorporated into future versions to enhance our evaluation and provide a more comprehensive assessment of motion realism.

Q2: Ethical Concerns and Copyright Considerations

R: Please refer to our previous response (R3) in Global Author Rebuttal. Our dataset mitigates privacy and copyright concerns by exclusively comprising hand pose data, corresponding audio, and MIDI files, thereby excluding images that could disclose identifiable personal features. This avoids privacy issues related to facial or other identifiable characteristics and adheres to copyright restrictions by not directly distributing visual content.

Q3: Missing Reference

R: Thanks for your recommendation. We have cited the referenced paper in the revision accordingly.

Q1: Physical Verification of Hand Motions

R: Thanks for your contructive feedback. Verification of hand motion sequences with their target audio using a physical model would indeed enhance data reliability. In future work, we plan to expand the dataset and incorporate a simulation environment, such as the one proposed by Zakka et al. (2023), to simulate piano performances and assess the precision of generated motions. This method facilitates dataset refinement by identifying and rectifying inaccuracies. Additionally, simulated environments could facilitate MIDI generation directly, further enhancing dataset precision.

Q2: Multiple Possible Hand Motions for a Given Performance

R: As noted in our abstract, “PianoMotion10M aims to provide guidance on piano fingering for instructional purposes.” Our method enables diverse piano performances for a given audio piece. The diffusion model we developed supports this flexibility by simulating different playing styles under the same audio input, with variations controlled via seed adjustments.

Q3: Baseline with Audio vs. MIDI

R: MIDI does indeed provide a more direct description of notes and positions, which could simplify the estimation of hand motions. We also conducted experiments on hand motions generated from MIDI, which can be found in R4 of Global Author Rebuttal. However, an end-to-end approach from audio to hand motions is generally more practical, which led us to design our baseline model based on this motivation. As for the sustain pedal, we agree that it plays a pivotal role in achieving realistic performances. Currently, our audio-based approach requires the model to implicitly account for pedal use, and we will explore more explicit treatments of pedal events or rhythms in future work.

Q4: Exploration of Model Limitations and Failure Modes

R: Thank you for these thought-provoking questions. Our baseline model is learned on data within typical human performance ranges, while giving challenging cases—such as virtuosic or unnaturally sped-up piano audio—it may struggle to generate realistic hand motions. These cases fall outside the distribution of the training data, making it difficult for the model to handle extreme tempos or hand spans that exceed human limitations. This is an area we recognize as a limitation when dealing with out-of-distribution (OOD) inputs, and future work could explore adjustments for handling such scenarios more robustly. We discuss the failure cases and provide visualization samples in the appendix of the revised submission.

Q1: Outputs with Paired Audio and Visualization

R: Thank you for the feedback. Our project webpage includes paired audio along with the corresponding generated hand motions. Though the volume may be somewhat low; we recommend increasing the volume for optimal synchronization and quality assessment. Additionally, some videos in the supplementary material may have audio playback issues due to encoding format incompatibility. We have found that QuickTime may not support the audio format, but using players like ffplay or IINA can resolve this issue.

Q2: Clarity of Dataset Collection and Filter

R: Thanks for your seggestion. Please refer to our previous response (R2) in Global Author Rebuttal. To enhance the clarity of our dataset collection process: we selected videos that best highlight hand movements by prioritizing those shot from a bird’s-eye view to minimize hand occlusion. We reached out to specific content creators for permission and obtained raw videos directly from them. During manual filtering, we excluded videos that did not meet our angle requirements, involved multiple performers or instruments, or contained distracting sounds. Automated audio event detection alone was insufficient for our needs due to potential inaccuracies in filtering out non-piano sounds but also the additional filtering criteria such as viewpoint, hand visibility, and the number of hands were essential for our dataset quality.

Q3: Annotation Quality and Recent Transcription Methods

R: Thank you for highlighting advancements in music transcription. We initially adopted the method by Kong et al. (2021) due to its demonstrated robustness in various scenarios. Also, we recognize that recent methods like MT3 (Gardner et al.) and DiffRoll (Gardner et al.) may offer the improved accuracy. To investigate this, we randomly selected 100 audio samples in our dataset and transcribed them into MIDI using various methods. The approaches by Maman et al. and Toyama et al. lack publicly available inference code, and Diffroll, which relies on diffusion with a fixed sequence length, poses challenges in smoothly stitching MIDI sequences. We evaluated the method proposed by Kong et al. against MT3, focusing on Mel Spectrogram Loss（L2), rhythm stability, and temporal consistency, as shown in the table below. Rhythm stability was assessed by analyzing note intervals and tempo patterns in the MIDI files, while temporal consistency was measured using Dynamic Time Warping distance to compare the rhythm of the MIDI with the original audio. Although both methods show similar performance in audio estimation, the approach by Kong et al. demonstrates superior stability in rhythm and prosody.

To enhance annotation quality, we manually refined MIDI files with significant discrepancies, ensuring alignment with the video content. Additionally, some creators provided ground-truth MIDI for their performances, and we will acknowledge their contributions in the camera-ready version.

Q4: Hand Motion Generation with MIDI

R: Thanks for your thoughtful comments. MIDI data does offer a direct link to hand movements and positions, while we focused on audio-to-pose generation in this version as audio data is generally more accessible than MIDI. More experimental results can be found in the R4 of Global Author Rebuttal. Consequently, our baseline model was designed to generate hand movements directly from piano audio, which provides broader applicability and ease of use for this dataset.