ImDy: Human Inverse Dynamics from Imitated Observations

Dear Reviewer yxxH:

First and foremost, we extend our deepest gratitude for your thorough review and helpful advice in improving the paper. We are thankful for recognizing our work as a novel attempt with comprehensive analyses. Herein, we respond to each of your concerns.

Data analysis. Thanks for the advice. We update Figure 12 with ImDy data distribution analysis according to BABEL annotations. As shown, ImDy is less dominated by gaits with higher diversity.

Data flow charts. Thanks for the advice. We updated a data flow figure in Appendix D, Figure 15.

Discriminator loss. Though the accuracy contribution is not much, we show its potential to produce reasonable frame-level motion assessments in Appendix Section A, which is meaningful for motion capture and generation. We believe a more curated design would be even more beneficial.

Term of "imitation learning". Thanks for your kind suggestion. By this term, we would like to raise discussions on the extent of "imitation". In detail, IL like PHC primarily focused on imitating "results" (kinematics). We would like to investigate whether they could successfully imitate the "production mechanisms" (dynamics) of human motion. We agree that "imitation learning" is overloaded, and would consider replacing it with "IL-based motion tracking" when applicable.

Further analysis on Figure 11. The sample in Figure 11 has approximately 50% frames with quality lower than clinical grade, which makes data quality a possible cause of failure, especially for the lumbar torques. However, we would identify the data distribution as a major cause of the failure for ankle and knee torques in Figure 11, given the sample is rarely seen in gait-dominated AddBiomechanics. And more obvious cases of quality-related failures are shown in the lower half of Figure 13.

Dear Reviewer xT3W:

First and foremost, we extend our deepest gratitude for your thorough review and helpful advice in improving the paper. We are thankful for identifying our work as well-visualized and meaningful. Herein, we respond to each of your concerns.

Data acquisition. We would like to clarify that one of our key contributions is providing initial benchmarks and baselines instead of an imitation algorithm for data acquisition since as pointed out, it is mostly borrowed from PHC. The major difference is covered in Section 5.1 L321-L323, where we retrain PHC with a different control mode. We respectfully emphasize a different view with it. By "imitation", IL like PHC primarily focused on imitating "results" (kinematics) with limited attention on "production mechanisms" (dynamics). The implementations of most IL algorithms validate it, where the IssaacGym function set_dof_position_target_tensor() without access to torques was adopted for better precision. Instead, we adopt the function set_dof_actuation_force_tensor with accessible torques. In view of this, ImDy is proposed as a first step toward large-scale and comprehensive human inverse dynamics analysis, with the major originality of interpreting imitation learning algorithms as an ID knowledge base beyond simple control algorithms.

Imitation success rate. Related to the first question, the source of it is the different control modes. Also as shown in Figure 4 and our attached video, the data in ImDy suffer from more severe jittering compared to the original PHC. However, even with its jittering quality, ImDy is surprisingly effective for real-world scenarios.

ImDy compared to IL.

• Experimentally, some straightforward comparisons are available in Tab. 2-3 and Fig. 4-5. PHC, as a representative instance of IL, could suffer from phase shift when directly adopted for dynamics estimation in Fig. 4. And for real-world scenarios, PHC could produce jittering predictions in Fig. 5, which might be due to the adopted naive PD controller. The quantitative prediction errors are also considerably outperformed by ImDyS.
• Theoretically, given the analysis in [1], only a 2-cm marker error could result in a peak ankle plantarflexion moment error of 26.6 N·m, which is inherently inevitable for imitation learning algorithms with the current tracking precision. Also, the sim2real gap hinders IL from being directly adopted for real-world scenarios like AddBiomechanics.
• Technically, as mentioned, ImDyS could bypass the involvement of simulation pipelines in inference, resulting in higher efficiency and reduced technical complexity.

[1] Thomas K Uchida and Ajay Seth. Conclusion or illusion: Quantifying uncertainty in inverse analyses from marker-based motion capture due to errors in marker registration and model scaling. Frontiers in Bioengineering and Biotechnology, 10:874725, 2022.

Dear Reviewer xT3W,

Thanks for your responses. Here we would like to clarify the following points and answer your further questions.

Data acquisition with original PHC. We clarify that the original PHC could not be adopted for data acquisition because the torques could not be acquired under position-control mode. The underlying reasons are two-fold. First, as mentioned before, there are no APIs for torque access in position-control mode. Second, the torques applied in position-control mode are computed implicitly in PhysX by a mixed linear complementarity problem (MLCP) solver instead of naive PD controllers [1], between which noticeable gaps exist.

ImDy as a data acquisition framework. Here we would like to clarify that the framework of ImDy is composed of an IL-based motion tracking algorithm with torque access (re-trained PHC as an easy-to-use and effective instantiation, with other choices like re-trained PULSE also available) and a high-performance physics simulator (IsaacGym is adopted, but Mujoco could also be applicable). With the submission, we showcase the feasibility of adopting this data acquisition framework for realistic human biomechanics analysis.

PHC as a baseline. For all experiments, we use re-trained PHC as the baseline since the original PHC can't be adopted for torques. However, it is noticeable that the original PHC could be used for ground reaction force estimation. Here we include the results as follows. Some visualizations are included in Appendix F and Figure 17, where jittering predictions are shown for the original PHC even with the MLCP solver, which produces smooth kinematics results. In contrast, ImDyS manages to produce smoother predictions with higher precision.

[1] https://forums.developer.nvidia.com/t/get-torques-in-position-control-mode/209415/21

Dear Reviewer qgUn:

First and foremost, we extend our deepest gratitude for your thorough review and helpful advice in improving the paper. We sincerely appreciate for recognizing our work as novel and straightforward. Herein, we respond to each of your concerns.

FD model analysis. We report the marker RMSE of the FD model on the AddBiomechanics test set in the following table. ImDyS noticeably outperforms the baseline trained on AddBiomechanics only, indicating the importance of ImDy pre-training. Moreover, ImDyS is competitive even compared to the differentiable simulator Nimble.

Differentiable FD simulation. Thanks for the insightful comment. Differentiable FD simulation like Nimble is a feasible design choice. However, the adopted simple FD module shows competitive performance compared to Nimble as in the above table. Also, differentiable simulators could introduce extra complexity and computation load to the compact ImDyS. Given these, we did not include Nimble in our pipeline. While we believe the analytical gradients from differentiable simulators are beneficial, we will consider using them in future works.

More visual samples are included in the attached video rebuttal_video.mp4. ImDyS predictions on samples from AMASS are visualized, including both walking samples that are similar to AddBiomechanics and some upper-body movements that are not covered in AddBiomechanics. As shown, the jittering is noticeably less significant compared to predictions on ImDy. Also, please refer to the updated Appendix C for further sim2real analysis.

I thank the authors for sharing the comparison and thoughts on the FD model. Looks all good.

The rebuttal video is just showing the final ImDyS results. I asked for visual comparisons of ImDyS with and without the AddBiomechanics to support the claim that the two-stage training is indeed necessary. Sorry to bring this up a day before the discussion period deadline.

Thanks for the response!

We would like to clarify that Figure 15 exactly visualized ImDyS before and after finetuning. Without sim2real fine-tuning, ImDyS can't be applied to real-world scenarios because it is not informed with realistic joints (Rajagopal's model in AddBiomechanics). We can only conceptually relate joints like the knee as in Figure 15. Also, the sim2real finetuning helps to reduce the over-powerful and over-active phenomenon in ImDyS when applied to real-world scenarios. The human model differences explain the phenomenon, the necessity of sim2real fine-tuning, and how Figure 15 is designed as a visualization of sim2real fine-tuning ablation.

We adopted markers as the motion representation of ImDyS in L267-275, so the retargeting procedure is not involved.

Dear Reviewer qrg1:

First and foremost, we extend our deepest gratitude for your thorough review and helpful advice in improving the paper. We sincerely appreciate for recognizing our work as interesting and effective. Herein, we respond to each of your concerns.

On the technical contributions. Thanks for your comments. We respectfully clarify that one of our key contributions is providing initial infrastructures, like data acquisition pipelines and baselines, for data-driven human inverse dynamics. As mentioned in the comments and Section 2 L135-146, many previous efforts took dynamics estimation as an intermediate for kinematics-related downstream tasks, with limited analyses and discussions. With its wide application in biomechanics, healthcare, robotics, and sports training, we emphasize that dynamics is more than an intermediate step in understanding kinematics. Given this, ImDy is proposed as a first step toward large-scale and comprehensive human inverse dynamics analysis. Its major originality is interpreting imitation learning algorithms as an ID knowledge base beyond simple control algorithms. Therefore, we claimed our major technical contribution as a novel pipeline for human inverse dynamics data collection with original insights and comprehensive analyses of its feasibility for both direct and downstream applications in Section 5 and Appendix A.

AMASS and KIT. Thanks! By the mentioned KIT, we mean [1], which is not contained in the original AMASS publication but is included by the AMASS team in later updates. The citation in the paper has been fixed.

ImDyS w/o AddBiomechanics finetuning. Due to the human model divergence (SMPL in ImDy and Rajagopal's model in AddBiomechanics), the joint definitions are inconsistent for ImDy and AddBiomechanics. Therefore, the zero-shot quantitative evaluation of ImDyS on AddBiomechanics is invalid. Even though, we visualize a sample for zero-shot ImDyS prediction on AddBiomechanics knee torques, since the knee is the only joint that could be conceptually related. As shown in Fig. 15, zero-shot ImDyS predictions show similar trends with GT. However, the magnitude differs a lot. This is consistent with our recent findings that ImDy tends to be over-powerful compared to real humans. More detailed analyses are updated in Appendix C.

ImDy(S) for downstream tasks. As mentioned in Section 2 L135-146, many previous efforts have shown that dynamics estimation is effective for markerless MoCap. We are also working optimistically on the potential of ImDy for downstream tasks as new projects. Moreover, as demonstrated in Appendix Section A, our learned prior managed to detect frame-level motion artifacts, which could be a meaningful tool for motion capture and generation.

[1] Franziska Krebs, Andre Meixner, Isabel Patzer, and Tamim Asfour. The kit bimanual manipulation dataset. In IEEE/RAS International Conference on Humanoid Robots (Humanoids), pp. 499–506, 2021.