ExtPose: Robust and Coherent Pose Estimation by Extending ViTs

We express our sincere appreciation for the helpful reviews and tackle the concerns below:

[Design 1] Accuracy-efficiency tradeoff curve?

A: Thanks for this suggestion. Table 9 in our submission already follows Xu et al. (2022) and Wang et al. (2024) to demonstrate computational efficiency. For clarity, we plot them along with accuracy in Fig. a (anonymous link). As the amount of attention computation increases with the addition of 2D pose modality and the number of frames, the error does decrease steadily, but the accuracy gain brought by the increase in computation gradually diminishes until the $T=16$ plateau, so it is necessary to weigh the actual accuracy against the efficiency required. It is worth mentioning that the introduction of the 2D pose branch will roughly double the amount of data that needs to be processed; as keeping the total number of frames (batch size$\times$sequence length) unchanged, the latency load brought by $T$ is not as obvious as it seems. Despite of increasing computation, the overall efficiency is still acceptable to real-time. In this work, our primary goal is to achieve high accuracy, and we also mentioned that efficiency could be improved in future work.

References:

• Wang et al. YOLOv10: Real-time end-to-end object detection. NeurIPS 2024.
• (VIMO) Wang et al. TRAM: Global trajectory and motion of 3D humans from in-the-wild videos. ECCV 2024.

[Design 2] ExtPose underperforms lifting-based methods?

A: Thanks for your careful review. A short answer is, model- and SOTA lifting-based paradigms are usually not fairly compared due to 1. different representations, 2. the use of 3D poses, and 3. window sizes. We outperform them in a similar setting (R1 uud2-Tab. a) and our ablation study (Tab. 5). We kindly refer the reviewer to check our response to R1 uud2-Question 2 in detail. We will also make the point clearer in the revision.

[Design. 3] Cross-frame novelty.

A: Thanks for your recommendation and comments. Our contribution lies in inspecting and effectively solving the challenges of 2D misalignment and temporal inconsistency of current ViT human pose estimation models with a simple and elegant solution. We devise a unified 2D pose representation with attention extended across the modality and frame to derive a robust video ViT model. The attention is thus implemented “simply and straightforwardly”, which aligns with techniques applied in other fields such as Multiview 3D generation, as the reviewer kindly mentioned. We will also include it in the reference.

We believe your contribution will definitely help enhance our work. Please feel free to let us know if there are still concerns not addressed in our feedback, and we are more than happy to communicate.

We are grateful for the reviewers' valuable efforts and their recognition of our work! If you have any other questions, we are always here, and you are welcome to discuss with us.

We sincerely appreciate the reviewer's constructive comments and address the concerns below:

[Strengths & Weaknesses 1/Exp. 1] The effect of noise in 2D pose detection?

A: Thanks for your great insight in studying the effect of 2D pose quality on performance!

• Firstly, in Fig. 6, we qualitatively highlight when erroneous 2D poses occur (e.g. left middle and right ring fingers), the original image branch assists in maintaining accurate predictions, i.e. correcting the 2D pose inaccuracy as the review rightly inferred. But indeed, there are failures where the prediction still adheres to the wrong 2D pose guess, particularly when the image information is hard to perceive in challenging cases, e.g. occlusion and blur in Fig. 7.
• Secondly, to quantitatively assess the effect of 2D pose noise, we introduce Gaussian noise with increasing standard deviations to the 2D pose while following Gu et al. (2024) to adjust the confidence score accordingly to reflect the increasing prediction uncertainty. It is worth mentioning that not only the coordinate prediction but also confidence as additional information are drawn in the 2D pose image. Table b shows the benefit from auxiliary 2D poses diminishes progressively with the increase in noise levels. When the confidence becomes low and the keypoint is displayed as transparent in the 2D pose image, the model does not use much of 2D poses and degenerates to the original ViT model (instead of collapsing), indicating the robustness to 2D pose noise of varying levels.

Table b. The effect of synthetic 2D pose noise on the 3DPW image dataset. ExtPose makes use of high-quality 2D pose detection while maintaining robustness to pose noise.

References:

• Gu et al. On the calibration of human pose estimation. ICML 2024.

[S & W 2/Sup. 1] Explanation for the failure in the Sup. video?

A: Thanks for the insightful observations. Yes, although our method shows significant enhancement in robustly aligned and temporally coherent pose estimation, challenges remain. For instance, as mentioned earlier, Fig. 7 showcases some really challenging and long-tail scenarios like ambiguous occlusion and blur. Analogously, the hand misalignment in the Sup. video frame, we speculate, likely stems from 2D pose estimation difficulties and, therefore, are not confident to provide any useful cues (not depicted in the pose image). Such out-of-distribution cases can lead to diverse error predictions across different models. A thought of future work to mitigate the issue may be to leverage more 2D prior knowledge, e.g. SAM2 segmentation.

Other writing suggestions will be fixed accordingly. Thanks!

We are confident that your input will greatly improve our manuscript. Should there be any unresolved issues in our feedback, please do not hesitate to inform us, as we are committed to continuous improvement and welcome further dialogue.

We sincerely thank the reviewer for the valuable feedback and address the concerns below:

[Method 1] Pose-pose attention mask in Eq. (11)?

A: Really thanks for your careful inspection! There is indeed a typo in Eq. (11). As described in the text Ls. 223-229 right above Eq. (11), the lower right quadrant pose-pose attention should also be a full 0 matrix. This is to focus on temporal modeling only within the respective modal branch.

[Question 1.1/Exp. 1] In Fig. 4, the highlight in the left corner indicates positional bias?

A: This is really an insightful observation! The phenomenon is also observed in other ViT-based methods like HMR2.0. Indeed, we speculate this is expected as the image corner and boundary may serve as helpful references for global/absolute positioning and final SMPL parameter regression. Thus, there are highlights indicating spatial awareness and being crucial for accurate pose estimation.

[Question 1.2/Exp. 2] How to interpret mutual attention of two branches from Fig. 4?

A: Thank you for highlighting the unclear points!

Column meanings: In the $3^{rd}$ column “Attention weights,” row $x$ inside the red box shows the flattened attention between point $x$ in the image and the entire 2D pose image. The red box in the $2^{nd}$ column “Attn maps” averages across the row (i.e., all points in the image branch) and reshapes to display the image branch's attention to the 2D pose for information gathering. The green boxes depict the reverse process: where the 2D pose focuses on the image.

Thus, the claim is supported by keypoint emphasis in the upper figure and extensive attention across the lower figure. Specifically, the upper figure in the middle column highlights localized 2D keypoint locations, explaining improved alignment; the lower figure reveals that 2D pose tokens focus on not only the central foreground hand but also the background (e.g., wrist in the green box, in Ls. 399-401 right), collecting RGB info and aiding 3D lifting. This broader focus helps estimate depth using the additional contextual info.

We will supply Fig. 4 with more details in our revision.

[Question 2] Comparison to lifting-based methods, specifically FinePOSE the only one better?

A: Thanks for raising this point! Indeed, lifting and our work share similarities of supplementing models with 2D poses; both are evaluated with JPE-type errors. However, some critical protocol differences make the two lines of work difficult to compare directly.

• Firstly, the target outputs in lifting are only skeleton joint coordinates. In model-based human mesh reconstruction (HMR) which also estimates shape, the output is SMPL parameters or joint rotations. The paradigm of direct coordinate regression has more advantage in body position prediction than SMPL-based methods due to the bottleneck of SMPL representation (Yu et al., 2023), thus tending to have a lower MPJPE (Tab. 8). The MPJPE metric has different preferences of different pose representations.
• Secondly, FinePOSE (and other SOTA lifting methods like D3DP) tend to use GT 3D keypoints for final aggregation of its multiple predictions; this is the case of their SOTA performance of 25.0mm in PA-MPJPE. Without GT 3D keypoints, their performance drops to 32.8mm (vs. our 27.2mm shown in Tab. a $2^{nd}$ row). Please find more details of the J-Best and J-Agg metric in the D3DP paper.
• Thirdly, lifting-based methods significantly benefit from using a larger window size (number of frames), specifically 243, due to their operation in a lower-dimensional space. This advantage is evident as shown in the 3$^{rd}$ row of Table a, where a reduction in window size to 16, typical of model-based methods constrained by available GPU memory, results in a noticeable increase in PA-MPJPE (Pavllo et al., 2019; Zhang et al., 2022).

To make a more equal comparison, we add ablation studies in Tab. 5 and Ls. 382-384 right & 412-414 left: our ExtPose using both the image and 2D pose outperforms either the image or 2D pose branch alone (i.e. 2D-to-SMPL lifting) in our setting. We will add the context and description in the revision.

Table a. Comparisons with the SOTA lifting-based method on the Human3.6M dataset. “Agg.” stands for aggregation. With 2) no GT poses and 3) a same window size of 16, ours outperforms FinePOSE.

References:

• Yu et al. Overcoming the trade-off between accuracy and plausibility in 3D hand shape reconstruction. CVPR 2023.

Other writing suggestions will be revised accordingly, and thanks for bringing these to our attention.

Please feel free to point out any unclear issues, as we value your feedback highly and are always ready to discuss them further.