Realistic Human Motion Generation with Cross-Diffusion Models

We appreciate your enlightening comments, which have facilitated our deeper thinking. Below are our explanations and thoughts on your concerns:

Q1: How does the projection help provide "intricate" details?

As we have mentioned in Response to All Reviewers, When using only 3D representation, the generation model may struggle to relate text semantics to some body part movements with very small movement variations compared to others, which can lead to overlooking important motion details. This is because the model might focus on more dominant or larger movements within the 3D space, leaving subtle nuances underrepresented. Due to different view projections in 2D data, small body part movements can be magnified in certain projections, making them more noticeable and easier to capture. This helps the text-to-motion generation models to better associate text descriptions with a wider range of human body motion details, including subtle movements that might have been overlooked when relying solely on 3D representation.

To provide a more specific explanation, we have included a complete analysis of 2D and 3D motion in Section 4.4.1, which should help in understanding how the projection helps provide "intricate" details. It is the difference in the distribution of 2D and 3D motion data that inspired us to learn the relation between 2D motion and text.

Q2: More analysis of the 2D information.

To evaluate the impact of different views, we conducted tests on four settings: (a) only the front view, (b) only the left view, (c) four views including front, left, back, and right, and (d) five views with an additional top view. When multiple views were used, the 3D motion was paired with a random view projection to formulate the loss. The results is shown below:

It is reasonable to assume that four views contain more 2D information and outperform one view. Furthermore, the front view and left view do not have much distinction in terms of performance. However, the performance actually decreased with the additional top view. This could be due to the fact that the 2D information becomes more difficult to classify without the condition of the camera view. We did not pass along the camera information because it is difficult to estimate the camera view of in-the-wild videos, and injecting camera information could disrupt the structure of the text-to-2D model, making it difficult for the text-to-3D model to follow. In conclusion, the number of views serves as a practical hyperparameter that can be adjusted through enumeration experiments.

We have provided a complete analysis in Appendix A, which we encourage you to refer to. Additionally, we were expecting that adding a top view would improve the performance, but the results showed the opposite. This has prompted us to think that more views do not necessarily lead to better results. A large gap between 2D information may complicate the easy-to-learn text-to-2D relationship. Therefore, the number of views serves as a hyperparameter in our CrossDiff model.

Q3: Is the framework aware of the input/target view?

No, we did not pass along the camera information. During training, each epoch, the 3D motion is paired with a random 2D projection. Our design is to extend our model to learn in-the-wild 2D pose while camera information is difficult to obtain. For further illustration, please refer to Q2 of the response to reviewer W5FS.

Q4: If the $x_{2D}\rightarrow\hat{x}_{3D}$ motion generation is linked to 3D pose estimation via lifting?

Yes, we also find that there is a similarity between $x_{2D}\rightarrow\hat{x}_{3D}$ motion generation and 2D motion lifting. [A] proves that 3D pose estimation via lifting is more accurate than the end-to-end method. The insight behind it might be that 3D motion is more difficult to extract features than 2D motion. So directly estimating 2D motion and lifting it works like specifying a feature that is easy to learn. Our method also considers learning the relation between 2D motion and text is easier, and we think this is helpful for data learning.

Q5: A video in the supp would be helpful.

As you requested, we have presented a video in the supplementary material to compare our results with other methods.

[A] Martinez et al., A simple yet effective baseline for 3D human pose estimation, ICCV'17

Q6: What is the concrete conclusion we could draw from Figure 6(a)(the number of old PDF)?

The figure clearly indicates that enhancing R precision is achieved by increasing 3D sampling steps, whereas optimal FID performance necessitates a balanced combination of 2D and 3D sampling. This finding emphasizes the critical need for custom-tailored sampling strategies to meet specific performance criteria effectively. For a more intuitive understanding, we suggest refering the supplementary material, which includes a video providing visual comparisons.

Q7: Reformatting of Figure 6(the number in old PDF) and root-decoupled diffusion. Thank you for pointing out the problem. We have moved both parts to the Appendix.

Once again, we appreciate your enlightening thoughts and are open to any further questions or suggestions you may have.

We would like to thank you for your valuable comments, and we have provided detailed answers to your concerns below:

Q1: The claim of “using 2D motion data without 3D GT during training”.

We acknowledge that the original description may have been misleading, and we want to emphasize that we have no intention of exaggerating the quality or significance of our work. We have revised the statement to "with a pretrained model, our approach accommodates using in-the-wild 2D motion data without 3D motion ground truth during training to generate 3D motion." We hope this clarification makes our attribution clear and removes any doubts.

Q2: The results showcased in Table 1 are not impressive.

We agree that our results are not impressive. However, we claim that our metrics are on the same order of magnitude as other state-of-the-art works. Additionally, we introduce a new metric measuring the FID of upper and lower body movements. With fewer joints measured, the model needs to perform more details to achieve higher performance. Our method outperforms other methods by a large margin with respect to FID-L, which proves our statement. For generation tasks, visualization sometimes matters more than metrics, and we suggest referring to Figure 4 and the video in the supplementary material for the effect of our method.

Q3: Ablation studies in Sec 4.4 lack a focus.

We appreciate your suggestion, and we have revised Section 4.4 accordingly. Our primary claim is that when training with paired 3D motion and text, projecting the 3D motion to 2D motion and building a connection between 2D motion and text can help boost performance. We have added an analysis of 3D and 2D motion in Appendix A and demonstrated that learning the connection between 2D motion and text is valuable. The control groups of "100% 3D" and "100% 3D + 100% 2D" in Table 2 prove our claim on a quantitative level. As our model is based on MDM, the visualization of "100% 3D" is nearly the same as MDM. From the comparison of our method and MDM in Figure 4 and the video, we can see the effect of learning 2D motion. Additionally, we have conducted more ablation studies in Section 4.4 and Appendix A to explore our model's capacity. Although our method may not be extremely surprising, we provide a simple and novel perspective on motion learning that is also enlightening in other research areas.

Our secondary claim is that additional 2D data can help boost performance, as you mentioned. The additional 2D data indicates other 2D motion without ground truth 3D motion. The control groups of "50% 3D" and "50% 3D + 100% 2D" in Table 2 prove this claim on a quantitative level. Moreover, in Section 4.3, we demonstrate that learning 2D motion from other datasets helps with out-of-domain 3D motion learning.

Once again, we appreciate your valuable comments and are open to any further questions or suggestions you may have.

We thank for your enlighting comments. Below are our response to your concerns.

Q1: The analysis of 2D and 3D motion.

As we have mentioned in Response to All Reviewers, when using only 3D representation, the generation model may struggle to relate text semantics to some body part movements with very small movement variations compared to others, which can lead to overlooking important motion details. This is because the model might focus on more dominant or larger movements within the 3D space, leaving subtle nuances underrepresented. Due to different view projections in 2D data, small body part movements can be magnified in certain projections, making them more noticeable and easier to capture. This helps the text-to-motion generation models to better associate text descriptions with a wider range of human body motion details, including subtle movements that might have been overlooked when relying solely on 3D representation.

We have provided a complete analysis of 2D and 3D motion in Section 4.4.1, which should help in understanding why 2D motion contains more details. We propose the concept that 2D motion can help in 3D motion learning, regardless of the model used.

Q2: The effect of camera information in Mixture Sampling.

Firstly, all 3D motion is preprocessed to face the z+ direction at the starting frame. Therefore, the view of 2D motion would not rotate the 3D pose. This can be seen in Figure 6 (the number in the new PDF), where after learning with in-the-wild 2D pose (where we cannot guarantee the camera view), our 3D generation still faces z+ at the starting frame. Additionally, we plan to extend our work to learn abundant in-the-wild 2D human pose, where the camera information will be a weakness. Therefore, we do not pass along the camera information. However, we agree that learning with accurate camera information would improve the results.

Q3: What is the purpose of the learnable token embeddings?

We do not have any special purpose for learnable token embeddings. Our model is simple and nearly the same as the standard transformer model [1] when viewed in only one domain pipeline. As the original transformer decoder is inputted with the output embedding, we regard learnable token embeddings as a simple method to learn the output feature.

Q4: Why is the text embedding added separately to the 3D and 2D encoder and not “jointly” in the “Shared Weights Encoder”?

As mentioned in our response to Q1, we find that 2D motion contains different attention to the different joints. Therefore, we want to establish a direct connection between the 2D motion and text prompt and enforce the model to learn the connection between the different details and the text. If the text embedding is added to the "Shared Weights Encoder," the whole model is forced to learn the connection between 3D and 2D motion first and is only able to extract the feature of motion, which may result in the same output as extracting the feature from 3D motion alone. This would cause the remarkable details in the 2D domain to be fused and lose their value.

In response to your feedback, we have moved the related experiments from Figure 6 (the number in the old PDF) to the Appendix to make room for other important experiments in the main body. Once again, we appreciate your valuable comments and are open to any further questions or suggestions you may have.

[1]Vaswani et al. Attention Is All You Need

We would like to express our gratitude for your insightful comments and constructive feedback. We have carefully considered each concern and provide our responses below:

Q1: Add significant references.

We acknowledge our oversight and have taken steps to address it. Specifically, we have added the relevant work to both Section 2.1 and Section 4.

Q2: Effect of the shared-weights encoder.

In response to the reviewer's concern regarding the efficacy of the shared-weight encoder, we have conducted a evaluation, the results of which are presented as below.

These empirical findings demonstrate that incorporating a shared-weight encoder significantly boost the the performance across several key metrics, including R Precision, Precision, FID, and DIV. This enhancement is not merely incremental but substantial, as evidenced by the quantitative results presented. Inspired by [1], we found that when learning with data from two modalities, extracting separate and fused feature layer-by-layer is more efficient. The shared-weights encoder serves as a fusing module, while the 3D/2D motion decoder acts as a separate decoder module. The goal is to ensure that the decoder layers follow the same extraction patterns as the shared-weight layers, rather than simply gaining deeper embeddings.

For an in-depth analysis and discussion of these outcomes, we kindly direct the reviewer's attention to Section 4.4.3 of our revised manuscript, where these aspects are elaborated upon in detail. We hope this additional data addresses the concerns raised and further substantiates the efficacy of our proposed method.

Q3: Add the User study.

Thank you for the suggestion. We conducted a user study on motion performance, in which participants were asked two questions to assess the vitality and diversity of the motions. The first is "Which motion is more realistic and contains more details?" and the second is "Which generations are more diverse?". The results are presented as below.

Our method produces more realistic and vivid generations while maintaining the diversity of the diffusion model. For visualization results, please refer to the supplementary material.

Thank you once again for your valuable advice on the experimental settings. We greatly appreciate your feedback and are open to any further questions or suggestions you may have.

[1] Xu et al. Building bridges between encoders in vision-language representation learning.