ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation

We thank the reviewer for their comments. We answer their remarks below, following the same order.

1. Citation of SMPL-based methods:

• We will properly cite these important works. We agree with the reviewer that SMPL-based methods share the same constant-bone-length assumption that we present, and that it represents an extensive line of work which is worth mentioning. We thank the reviewer for pointing this out to us.

• We would like to highlight however a few differences between these works and ours:

  • HMR is a field solving a more complex task: “Human body pose and shape estimation” (a.k.a. “human body reconstruction”, “human mesh reconstruction”, “HMR”, …) is a separate field, different from our “3D human pose lifting”: the objective of the former is to predict whole 3D body meshes from images, as opposed to 3D joint positions from 2D keypoints for the latter. This means that these works tackle a different task, which is more challenging.
  • HMR is often frame-based: Because their task is more challenging, SMPL methods are more computation-heavy and often restricted to single-frame predictions, which are then smoothed across frames using optimization-based post-processing. A good example of this methodology can be seen in [A].
  • SMLP methods deliver worse MPJPE: Of course, predicting the human mesh across time includes the skeleton pose, which means that some of these methods do evaluate MPJPE. Note however that, while [A] is SOTA in Human3.6M data last time we checked, their performance (MPJPE=44.8 mm) lags behind 3D human pose lifting methods (c.f. Table 2 and 8 in our paper).
  • Our novelty lies on multiple hypotheses with constraints: Furthermore, please note that our work proposes not just to restrict predicted poses to have constant bone length, but proves that this is not sufficient to optimize both pose fitting (i.e., joint position error) and consistency (i.e., avoiding bone stretching). We prove that multiple hypotheses are needed to accommodate both objectives and propose a practical implementation, show-cased in our experimental results.

2. Concerns regarding rotations parametrization:

• Learning 3D directions instead of full rotations yielded poorer results: We understand the reviewer’s concern. It is true that, when compared to full 3D rotation matrices, predicting simple direction vectors (as done in Anatomy3D [4] for example) is a simpler way of parametrizing poses where joints are connected by segments. As advised by the reviewer, we hence performed a new ablation study, where we compare our parametrization to the one used in Anatomy3D, i.e., where the rotations module predicts normalized vectors representing the direction of each segment. As shown in Table 1 of the pdf attached to our global answer, the latter leads to poorer oracle MPJPE of 39.6 mm when compared to our rotations parametrization (39.1 mm) in a multi-hypothesis setting.

• Our choice was guided by relevant previous works: To clarify our choice, we opted for our rotations representation building on results from [43], where the benefits of 6D representations of SO(3) matrices is studied. Please note that [43] includes good results in an inverse kinematics problem with "stick" human poses similar to ours.

3. Concern regarding results with aggregated poses:

• Values for MPI-INF-3DHP are reported in Table 4: The penultimate row corresponds to a weighted averaged pose, while the last row corresponds to the oracle pose.

• Values for Human3.6M are reported in Table 8 in the appendix: Metrics of the aggregated pose can be found at the penultimate row of Table 8. We understand that this can be missed since it is in the appendix.

• We will clarify this in our revision. We apologize if the results with aggregated poses are not stated with enough clarity in our manuscript.

• MPSCE performance is reported above: As explained to reviewer 5BLY in Q3, both MPJPE and MPSCE performance are hurt when using the averaged pose. While the latter is expected according to Proposition 4.2 in the paper, the MPJPE degradation is explained by a new theoretical result proved in the answer to reviewer 5BLY.

4. Concern regarding temporal consistency of predictions:

• The training objective includes a temporal term in the Human 3.6M and MPI-INF-3DHP experiments. We apologize if this was not very clear in our manuscript. It is mentioned in Appendix D.3 lines 602-609, where we explain that TCloss and velocity loss terms were added to our original objective in order to have a fair comparison with existing art. We chose not to add this in the main manuscript because of space limitations and because they are relatively standard practice. We agree that it may lead to confusion and will hence add a longer note on this in our revision to the main paper.

• We provide a gif demonstrating temporal consistency: Although we understand the reviewers concern, we found no important issues regarding this point. We believe this to be explained by the aforementioned loss terms and the fact that the model predicts sequences instead of single frames. The reviewer can find a gif in the pdf attached to the main response (NeurIPS rules forbid us from providing a link to a video). Note that Adobe Reader is needed to visualize it.

[A] Goel, Shubham, et al. "Humans in 4D: Reconstructing and tracking humans with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Chen, Tianlang, et al. "Anatomy-aware 3d human pose estimation with bone-based pose decomposition." IEEE Transactions on Circuits and Systems for Video Technology 32.1 (2021): 198-209.

[43] Zhou, Yi, et al. "On the continuity of rotation representations in neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

We thank the reviewer for their comments. We answer their remarks below, following the same order.

1. New theoretical analysis on the advantage of multi-hypothesis methods over single-hypothesis:

We agree with the reviewer and provide the proposed theoretical result hereafter.

• Let $\mathcal{X}=\mathbb{R}^{2 \times J}$ denote the space of input 2D poses and $\mathcal{P}=\mathbb{R}^{3 \times J}$ the space of 3D poses. Following Rupprecht et al. 2017 [31] and Letzelter et al. 2024 [A], we define the “oracle risk” for a K-hypothesis model $f_{\text{WTA}} = (f_{\text{WTA}}^1, \dots, f_{\text{WTA}}^K)$ as:

We thank the reviewer for their comments and provide hereafter our response to their concern, in the same order.

1. How to constrain the rotation space during training?

• Our method can be adapted to incorporate angle constraints. This is possible for example if one chooses to use rotation representations where angles appear explicitly (Axis-angle, Euler angles, …). This is not straightforward when using the 6D representations that we chose in this work though, where angles are implicit. We chose these representations because there is a bijection between them and rotations matrices, which presents optimization advantages and allows us to avoid training instabilities (cf. [43] On the continuity of rotation representations in neural networks. CVPR 2019.)
• A possible solution: If we chose to predict directions instead of rotations (cf. answer to reviewer fpqy Q2), then angles could be easily made explicit in our representation (using spherical coordinates for instance) and we could constrain them to stay within a certain interval by using simple sigmoid activations at the end of our rotations network. This idea could be a nice future extension of our work.

2. Is ManiPose a multi-hypothesis extension of Anatomy3D?

• Yes, in a sense: It is true that, similar to Anatomy3D, ManiPose disentangles limbs length and orientation in order to constrain predicted poses to lie in an estimated manifold. In this sense, we can indeed see it as an extension to the multi-hypothesis setting.

• But with new theoretical results and a very different message: Please note however that, unlike Anatomy3D, we provide theoretical proofs and empirical evidence that both constraints and multiple hypotheses are needed if one wants to optimize joint position error and pose consistency together. This is quite a different message than the one in Anatomy3D paper.

• And a different way of constraining poses: Also note that we use a different representation of limbs orientations than Anatomy3D to constrain our predicted poses (cf. answer to reviewer fpqy Q2). New ablation results in Table 1 of the pdf attached to our main response show that our representation yields better results in the multi-hypothesis setting.

3. Concern with SMPL-methods and Pose-NDF citation:

• We will eagerly cite Pose-NDF. We knew and hold this work in high regard. We agree with the reviewer that it is related to our work, as it implicitly learns a plausible sub-manifold of $SO(3)^J$ for poses.
• Please note however that Pose-NDF is part of the “body pose and shape estimation” field, which is related but different from 3D human pose lifting. The objective of most SMPL-based methods is to predict whole 3D body meshes from images instead of 3D joint positions based on 2D keypoints. The task is more challenging, which means that algorithms are heavier and more reliant on optimization-based post-processing. For instance, Pose-NDF estimates 3D meshes in a single image by initializing pose angles and using gradient descent to project them into the learned pose manifold, which is different from doing a simple forward pass with ManiPose.

4. Question regarding synthetic data and MSE loss:

• We can guarantee that if multiple hypotheses and constraints are not used together, then the predictions of the lifting function learned using just MSE loss on the synthetic data proposed by the reviewer will not lie on the manifold (unless the distribution of 3D poses conditioned on 2D poses are Dirac measures, i.e., if there is always a single possible 3D pose projecting into 2D space).
• This is proved in Proposition 4.2 and shown (with simplified 2-joint and 3-joint articulated objects) in experiments of sections 4.2 and C.2.

5. Optional question regarding MoCap:

• The Human3.6M dataset, as many other datasets, does not contain raw MoCap measurements, but rather post-processed estimations of 3D joint positions obtained through heavy optimization. One of the constraints that is enforced during such post-processing steps is precisely that limb lengths do not vary over time for a given subject. We verify this for ground-truth poses of Human3.6M dataset in figure 5 of our paper (cf. legend).

6. Citation style

• We apologize for the citation style and will correct it in our revision to comply with the NeurIPS format.

We thank the reviewer for their comments and answer here their concerns in the same order.

1. Diversity and fixed number of hypotheses concern:

• SOTA oracle MPJPE is evidence of good diversity: It is true that ManiPose produces a fixed number of poses per forward pass. While methods based on generative models (GFPose, DiffPose, etc.) typically control the number of samples drawn at test-time, we argue that in predictive tasks, selecting the right number of samples is important to capture the diversity of the plausible targets, i.e., the modes of the conditional distribution. It was indeed proven in prior works that the winner-takes-all (WTA) training scheme, on which ManiPose is based, can achieve an optimal quantization of the target distribution [A] given sufficient data. This diversity in the predicted poses can be measured through Oracle MPJPE performance, for which ManiPose achieves state-of-the-art results.

• Few hypotheses with scores are more informative: Moreover, note that generative methods assign equal weight/likelihood to all sampled poses for a given input. Since they require to sample a large number of poses (~200) to achieve competitive results, it becomes difficult to practically use such an uniform and high-cardinality output in a real scenario. ManiPose, on the other hand, provides more information to the user by predicting just a few relevant consistent poses with their corresponding scores/likelihoods, which is easier to process.

• Similar coverage of 3D poses distribution as DiffPose: In an attempt to quantify the diversity of poses predicted by ManiPose by other means than oracle MPJPE, we have computed the coverage (c.f. [B]) of generated poses over the ground-truth test distribution. For computation cost reasons (it grows quadratically with sample size), we limited our analysis to 5 actions from subject S11 of Human 3.6M. We compare ManiPose to [2] DiffPose: Toward More Reliable 3D Pose Estimation, using 5 hypotheses for both, and observe similar diversity on average (c.f. Figure 1 in the pdf attached to the main response).

2. Concern with Table 4 and trade-off between consistency and accuracy:

• With a single hypothesis, one has to choose between minimizing MPJPE or pose consistency: We agree with the reviewer reading of Table 4 and would like to highlight that these are precisely the main messages of our work: 1) in a single-hypothesis setting, constraints will necessarily trade MPJPE for consistency (2nd row) and 2) multiple hypotheses (1st row) are needed to conciliate both MPJPE and consistency metrics.

• Table 4 is hence empirical evidence supporting our proposition 4.2 and corollary B.1.: This result is precisely what makes our approach principled and novel, since there are previous works proposing to constrain predicted poses to estimated manifolds (e.g. Anatomy3D) or proposing to use multiple unconstrained hypotheses (e.g. Diffpose, GFPose, …), but never both together. Please let us know if we misunderstood the reviewers’ point.

3. Concern with missing comparison with two recent multi-hypothesis methods:

• We will add GFPose and DiffPose to our experimental results. We thank the reviewer for pointing us to these excellent recent works. We have used their official code and checkpoints to generate 3D poses over the test split of Human3.6M and computed their pose consistency metrics.

• GFPose delivers worse pose consistency and joint position error in a comparable setting: Indeed, we measured an MPSCE of 16.5 mm and an MPSSE of 13.1 mm, considerably worse than ManiPose (MPSCE=0.5 and MPSSE=0.3). In terms of MPJPE, the paper reports an impressive 35.6 mm, but this score makes use of 200 sampled hypotheses, which is 2 orders of magnitude higher than us. In a fairer setting, when using a similar number of hypotheses for both methods, GFPose then stands at 45.1 mm of MPJPE with 10 hypotheses, which is considerably worse than ManiPose (39.1 mm with 5 hypotheses).

• DiffPose delivers worse pose consistency and competitive joint position error when using their checkpoint and code: We computed an MPSCE = 6.1 mm and MPSSE = 5.2 mm for them, which is again inferior to ManiPose. In terms of MPJPE, the authors also report an impressive number of 36.9 mm in their paper for their video model. However, using their official code and checkpoint, and setting the number of hypotheses to 5 as mentioned in the paper, we could only obtain 39.3 mm, which is competitive with ManiPose. Note that there are two other issues on their github page (which we can’t link here) mentioning precisely that the code and checkpoint do not allow to reproduce their reported results.

[A] Letzelter, V., Perera, D., Rommel, C., Fontaine, M., Essid, S., Richard, G., & Perez, P. Winner-takes-all learners are geometry-aware conditional density estimators. In Forty-first International Conference on Machine Learning.

[B] Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., & Yoo, J. (2020, November). Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning (pp. 7176-7185). PMLR.