Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos

We thank the reviewer for the thoughtful and constructive feedback. We address the concerns and questions from the reviewer in the section below.

2D overlay of estimated pose on the input image.

Thank you for the suggestion. We plotted the overlay of 3D estimated poses from OSDCap onto the input 2D images in Figure*1. As can be seen from the figure, the re-projected SMPL meshes overlay closely to the human inside the input images. Despite the image-based observations being refined by the Kalman filter, the idea of considering original image observations could potentially benefit the prediction, and will be investigated in future work.

OSDCap cannot prevent foot-ground penetration and scene dynamics.

Yes, foot-ground contact is not explicitly modeled in OSDCap. However, our contact estimation is able to mitigate it and creates physically plausible motion. Please refer to Table*2 in the attached PDF above for our additional physics-based measurements of OSDCap.

Is the final pose obeying physics laws?

OSDCap combines the two data sources and collectively creates the best of both worlds. Naturally, if the GRU model decides that the noise from the physics simulation data is larger than the kinematic input in a specific time step, the resulting human pose $q_t$ and $\dot{q}_t$ after Kalman filtering might not strictly obey physics laws anymore. There always exists a knowledge gap between simulation and real-world that can cause the simulated result to be sub-optimal under the provided constraints. While the image-based kinematic estimation, despite being jittery and noise, reflects real-world observation and can act as an online compensation for the mistakes made by the simulation.

Unnatural leaning due to the usage of external force.

A false external force prediction might result in implausible artifacts such as unnatural leaning. This is one of the motivation for proposing a Kalman filtering approach, where a noisy inputs to the in physics simulation, i.e. false external force prediction, can be monitored and compensated directly by the kinematics input. However, unnatural leaning artifacts often occur more in the kinematic input due to depth uncertainty in the optical axis, especially in scenarios when the person is facing the camera. We show an example of using a physics-based simulation with external forces can actually refine the unnatural pose and prevent unnatural leaning in Figure 3b in the main paper.

Kalman filter on kinematics or simulation only.

The performance of applying Kalman filtering on kinematic motion can only be shown in Table*1 in the rebuttal PDF above. As can be seen, without physics-based knowledge, the Kalman filtering scheme only does smoothing and achieves worse performance than our approach. For a deeper analysis of the classical Kalman filter compare to a learnable one proposed in the paper, we kindly refer to the discussion with reviewer wqMZ.

Adaptability of trained models on different skeleton shapes.

The trained models can be generalized to different skeleton shapes since we are predicting joint angles of the skeletons while training the NN models. Therefore, a motion retargeting to a character with different bone lengths is straight-forward. However, achieving the same performance as in the three used dataset Human3.6M, Fit3D and SportPose for data with a completely different character configuration would require additional training.

We thank the reviewer for the highly positive evaluation and the constructive feedback to our work. For a better clarification, we provide the answers to the reviewer's questions below.

Full-body contacts.

We agree that full body contact is the next logical step towards a fully environment-aware physical model. Note that our approach (though not integrated in the current version) can easily add an additional force to any part of the body. However, this would require contact detection and estimation of the physical properties (e.g. velocity, weight, softness) of the object which is non-trivial. Recently, this field received more attention [C, D] and we are looking forward to integrating these approaches into our model in the future.

For a similar discussion on body models that enable to add external forces we kindly refer to the “Selection of a simple proxy character” section with reviewer BVYQ.

[C] Tripathi et al., DECO: Dense Estimation of 3D Human-Scene Contact In The Wild, ICCV, 2023.

[D] Xie et al., Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation, CVPR, 2024.

Inertia-matrix M.

For a robotic arm, the mass distribution often does not change significantly. However, while a constant limb-wise mass distribution is an acceptable approximation for a human, it is not entirely correct. Due to muscle movement and soft tissue deformation the mass distribution changes which has a direct effect on the estimated motion [E, F]. While our inertia matrix intentionally leaves this additional degree of freedom, we agree that its interpretability is limited. Increasing the interpretability of the inertia matrix will be investigated as the next step of this work.

[E] Pai, D.K., 2010. Muscle mass in musculoskeletal models. Journal of biomechanics 43, 2093–2098.

[F] Featherstone, 2008. Dynamics of rigid body systems. Rigid Body Dynamics Algorithms, 39–64.

Evaluation of related works on Fit3D and SportPose.

Due to the availability of the implementation code of related physics-based methods, we could not replicate all related physics-based methods on Fit3D and SportPose during the rebuttal phase. We hope that these experiments will be finished during the discussion phase and at the latest we will add the evaluation results of works that provided implementation, such as NeurPhys, to the camera-ready version of the paper.

Typos.

Thanks for catching the typos. We will correct them.

We thank the reviewer for the very positive assessment and the recognition of our method's novelty. We would like to provide answers and clarifications to the remaining questions of the reviewer.

Presentation Improvement.

We appreciate the suggestions of the reviewer. We will change the visualization to SMPL model for the GT, kinematics and OSDCap prediction in the camera-ready version of the paper, similar to Figure*2 in the rebuttal PDF document. We are also open to any other recommendations to further improving the presentation quality.

Additional physics-based metrics.

We provide additional metrics for physic-based measurements in Table*2 of the rebuttal PDF above. OSDCap helps refining the input kinematics on most of the physics-based metrics. Note that TRACE outperforms our approach in the ground penetration metric. The reason is that in most cases the TRACE predictions float above the ground, which gives a low penetration error but can be seen as equally bad. Thus, we additionally provide a ground distance metric (GD) to reflect the correct foot-ground quality during contact. The value is computed as the mean absolute vertical differences between foot contact points and ground plane during contact duration, expressed as:

$GD=\frac{1}{6} \sum_{c=1}^{6} \rho_{c} |p_{c}^{OSD} - p_{c}^{GT}|$,

where $\rho_{c}$ is the binary label of contact $c$, $p_{c}^{OSD}$ and $p_{c}^{GT}$ are the 3D vertical positions of contact $c$. There are a total of 6 contact points considered, 3 in each foot accounting for heel, foot and toe.

Differentiable simulation in OSDCap.

The differential simulation in line 38 refers to the usage of rigid body dynamics equations and Euler's integration on a proxy character, not an actual physics engines like TDS or PyBullet.

Selection of a simple proxy character.

We based the calculation of the inertial properties of the proxy character on the Rigid Body Dynamics Library (RBDL) which was shown to be effective by Shimada et al. [PhysCap]. Since the simpler character definition (as for example compared to KinPoly) yielded good results, we decided to stick with it and mainly focusing on implementing our core contribution of the learnable Kalman filter. During experimentation we found that prediction errors mostly occur from insufficient kinematic or dynamic predictions than from using a more sophisticated proxy character. However, in the future we plan to extend this approach to other external forces, e.g. from full body contact, which will strongly benefit from a detailed human body model such as KinPoly.

How did foot-ground contacts are modelled?

The foot-ground contacts are modelled directly by the NN model and automatic data annotation, using the ground truth 3D poses from the training data set.

Additional baselines.

Thank you for the additional baseline suggestions, MultiPhys and PhysPT. We will include their results and contributions into the related section. We would like to briefly discuss the suggested paper due to their high relevance:

• MultiPhys introduces a method for multi-person physics-based 3D human motion capture that mainly address plausibility of the captured multi-human poses, inside an physics engine and a reinforcement learning framework. This work could bring new insights about inter-person body interaction, and a more detailed proxy character such as KinPoly (as suggested by the reviewer) is needed.

• PhysPT provides a pre-trained transformer model specifically for human dynamics capture. Instead of explicitly defining physical constraints, as in our or related [NeurPhys, DnD], PhysPT realizes the equations of motion as the main training objective, forcing the transformer model to be physics-aware, thereby allowing a violation of physics laws. The pretrained model can be used to refine kinematics input, acting as an NN-based approximation of a physics simulation.

Typos.

Thanks for catching the two typos. We will correct them.

We thank the reviewer for the feedback and constructive review. We would like to address questions and clarify the concerns below.

Proxy character creation.

For character construction, we used the Human 3.6M metadata skeleton as the initialization of our character. The bone lengths are treated as extra learnable parameters and optimized along with the model during the training process. During testing, the bone lengths are fixed to the ones learned during training, i.e. no ground truth bone lengths are used when testing.

Why TRACE?

The reason for choosing TRACE as our kinematic estimation input is their additional global translation estimation (w.r.t the first frame), while for example VIBE only predicts root relative poses. We use this global information to enable the integration of physics laws into the system. Moreover, TRACE is a well-established and new (CVPR'2023) baseline. The additional benefit of TRACE being robust to camera motion is an advantage we plan to explore in future work.

OSDCap estimation is agnostic to initial calibration.

The statement in line 473 explains how we preprocess the data. It indeed only removes the translational component of the camera calibration. As correctly stated in the review, the rotational component is important for the physics simulation. OSDCap will still work in the case of highly tilted camera, as long as the camera pose is provided. If a fully automatic calibration process for tilted cameras is desired, an automatic calibration with humans as calibration targets can be employed [B].

[B] Tang et al., CasCalib: Cascaded Calibration for Motion Capture from Sparse Unsynchronized Cameras, 3DV, 2024.

The performance of the PD-only method.

We thank the reviewer for pointing out this potentially misleading table.

The PD implementations in Table 3 are different from other PD controller-based works (PhysCap, NeurPhys, DnD). While NeurPhys (cf. their implementation on Github) assumes that gravitational forces and external forces are cancelling out, which is not true in the real world, we decide to keep them to maintain physical plausibility. Moreover, to make the PD controllers work, PhysCap, NeurPhys, and DnD add an additional predicted offset term to the controller output which introduces another source of implausibility. By contrast, we maintain the physical plausibility of the simulation and the PD controller which leads to state-of-the-art performance among physics-based approaches. We will clarify this in the final version.

Additional baselines.

We will add the suggested kinematics estimation methods in the comparison. However, similar to what have been done in related physics-based methods such as NeurPhys, DiffPhy or DnD, it is not trivial to directly compare physics-based motion reconstruction to image-based pose estimators, since the latter do not predict the global 3D trajectory in world coordinate, making it impossible to evaluate the global motion quality (MPJPE-G, GRP). We kindly refer to the comments to all reviewers above for a detailed discussion.

Is GRU uni-directional?

Yes, the GRU unit is uni-directional.

We thank the reviewer for the positive assessment to our proposed idea and writing quality. For a better interpretation of our work, we clarify and answer the questions of the reviewer below.

Comparison to a classical Kalman Filter.

Classical Kalman filters have been used traditionally for all types of tracking problems, including human pose tracking. More recently, the only work we could find that utilizes a Kalman filter is from Buizza et al. [A], where a classical parametric Kalman filter is employed to track 2D keypoints estimated from an off-the-shelf 2D human pose estimators. We are happy to take further suggestions for related work that we might have missed.

We agree that our learnable Kalman filter should be compared to a classical Kalman filter. Therefore, we conducted an additional experiment where we replace our learnable filter by a traditional one. The biggest challenge of using classical Kalman filter is the tuning of unknown noise covariances of both the kinematic input TRACE and the simulated result from PD controller. Assuming noise covariances that are constant over time and equal in all directions, the ratio between the noise covariance of the simulated PD controller (process noise) and the noise covariance of the kinematic input TRACE (measurement noise) governs the quality of the Kalman filter estimates. The evaluation results can be seen in Table*1 in the PDF file, where we use constant noise covariances with ratios 100/1, 10/1, 1/1, 1/10, 1/100 between process noise and measurement noise. While a classical KF approach helps increase the result marginally, optimal results are difficult to find. Our choice of a learnable Kalman filter relieves us from trial-and-error process of finding the correct noise covariance matrices and achieves the best results.

[A] Buizza et al., Real-Time Multi-Person Pose Tracking using Data Assimilation, WACV, 2020.

Additional baselines.

We will add the suggested baselines and others suggested by reviewer DQhK to the paper. Please refer to our global rebuttal response for references about their performances. We would like to briefly discuss the two suggested approaches in relation to our work.

• PointHMR (CVPR 2023) is a frame-wise mesh and keypoint regression method that can work in an online setting. However, unlike TRACE, PointHMR does not produce global root translation of the estimated poses, therefore, can only be used for MPJPE reference, not for physics-based reconstruction methods.

• KTPFormer (CVPR 2024) predicts 3D keypoints directly instead of 3D joint angles. This might lead to implausibilities in the skeletal configuration, for example in changing bone lengths, and is therefore not transferable to a dynamics model. Since it appears to be the current state of the art in terms of the common metrics like MPJPE we will include it.