Semantics-aware Test-time Adaptation for 3D Human Pose Estimation

We express our sincere appreciation for the helpful reviews and tackle the concerns below:

[Weakness 1] Failure cases from the VLM

R: Thank you for your suggestions. We manually examined 5,000 frames from the 3DPW dataset, where 96.4% is accurate, as the actions are relatively simple and VLM is reliable. For the remaining 3.6%, the VLM hallucinates actions based on background, e.g. a person picking up foil is mistaken as "golfing" as the foil resembles a golf club, or "eating something using both hands" and "cast fishing pole" in scenes with fruit stands and rivers, respectively. Visualizations of these VLM's failure cases can be found in Fig.I (anonymous GitHub link).

Despite these VLM failures, our method maintains robustness. As our semantics-aware alignment is a regularizer, its impact is limited if it conflicts with 2D projection loss. The table below shows that the adaptation with the wrong VLM label still reduces the MPJPE, but not as much as when the (oracle) correct action labels were applied.

[Weakness 2] No GT mesh in Figure 1

R: Thanks for pointing this out. As shown in Fig.IV (anonymous GitHub link), we include GT mesh for a better comparison, where our predictions preserve the same motion semantics as the GT.

[Weakness 3] Runtime analysis

R: Thanks for pointing this out. Yes, our runtime (107.6ms) is longer than CycleAdapt (74.1ms); the extra overhead is primarily from the VLM (85.4%).

Details are given in our response to [Question 1] (Runtime analysis) for the reviewer qP6M.

[Weakness 4] Include challenging poses

R: Thank you for your suggestions. We provide more challenging poses, i.e., "open a door and sit", "hugging", "squats series" and "spin around with right foot". Please refer to Fig.II and Fig.III in the anonymous GitHub link.

[Question 1] Direct optimization without fine-tuning

R: Yes, it is faster since it does not require model fine-tuning. However, the success of direct optimization is linked heavily to the accuracy of the initial estimate.

We compare with ScoreHMR [3], a state-of-the-art direct optimization approach on the 3DPW dataset using OpenPose 2D poses. As shown in Fig.V (anonymous GitHub link), our method is robust across different initial prediction qualities. ScoreHMR is more sensitive and when the initial MPJPE exceeds 200mm, ScoreHMR reduces error only by 26.6%, whereas we can reduce it by 64.5%.

References

[3] Stathopoulos et al. Score-guided diffusion for 3D human recovery. CVPR'2024.

We express our sincere appreciation for the helpful reviews and welcome further discussion!

We express our sincere appreciation for the helpful reviews and tackle the concerns below:

[Method 1] Fairness in Table 1

R: Our paper highlights the problem of motion semantics and proposes a method to incorporate semantics as the core contribution. No existing works use semantics, so it is difficult to compare directly. We verify our method by comparing it with the closest components from the literature, as shown in Table 3. Specifically, we use CycleAdapt as a baseline (top row) and then add semantics through a motion discriminator (row 2) or unpaired local poses (row 3). In this way, we enforce the predicted motion to be aligned with the generated motion from [1] by sharing the same semantics as the video segment. Table 3 in the paper shows that our method better integrates semantics without additional motion data generation.

[References 1] Diffusion models

R: Thank you for your suggestion. We will add these references and also discuss diffusion models in our related works section.

[Weakness 1] Method alternatives

R: Thanks for pointing this out. We explore different strategies of integrating semantic information; two alternatives to our method are shown in Table 3. We will adjust the labels and captions of Table 3 and Sec. 5.4 (‘Analysis on semantics-incorporated techniques’) in the text to clarify this point.

[Weakness 2] Long periods of occlusion or jitters

R: Thank you for pointing this out.

For long periods of full-body occlusion, motion can be generated to maintain semantic consistency with the predicted activity. However, this aligns more with motion generation, which is evaluated by plausibility (e.g., FID [2]), unlike reconstruction, which relies on ground truth. We plan to merge reconstruction and generation into a single framework and develop suitable evaluation metrics in future work.

We evaluate jitter on the 3DPW dataset, comparing our method with ground truth and CycleAdapt. As shown in the table below, our jitter is significantly lower than the image-based DynaBOA but 11.1% higher than CycleAdapt. However, CycleAdapt often predicts static motion, even in dynamic motion scenes, resulting in low jitter but deviating from the ground truth. Take the walking motion from Fig. 1 of the paper as an example, we report the jitter values for this case in the last row of the table below, where ours is higher than CycleAdapt but closely aligns with the ground truth.

[Other Comments 1] Typo

R: Thank you for pointing this out. We will correct it in the revision.

[Question 1] Runtime analysis

R: Thank you for your suggestion. We perform runtime analysis following the environment settings described in Table 7 in CycleAdapt. The table below shows that compared to CycleAdapt, our method introduces an additional 33.5ms runtime per frame, 85.4% of which is caused by VLM, and 14.6% by our proposed framework components. However, it is still much faster than other TTA methods. We will add runtime analysis in the revision.

[Question 2] Hyperparameters and EMA

R: We choose hyperparameters based on a grid search. The EMA update factor ($\alpha$) values used for the search are provided in the table below. For the fill-in threshold ($\sigma$) of missing 2D detection, please refer to Table 5 in the paper.

The EMA and $\alpha$ do not cause jitters. This is because the EMA is performed over 2D poses for each frame and each frame has one fixed action label.

[Question 3] GT 2D

R: Thank you for your suggestion. We add results in the table below. Our method still performs better than CycleAdapt. The improvement becomes smaller because GT 2D keypoints provide very strong indications for occluded keypoints, which is unrealistic in real-world scenarios. Our method especially takes effect when 2D keypoints are noisy or missing.

References

[1] Jiang et al. MotionGPT: Human motion as a foreign language. NeurIPS'2023.

[2] Guo et al. Generating diverse and natural 3d human motions from text. CVPR'2022.