SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training

Q3: Instead of using weights, could one instead not use a more appropriate loss that will automatically lead to larger effects depending on the samples weights? For example, MSE will automatically weight the contributions of more distant samples stronger.

A3: Based on the reviewer’s suggestion, we implemented MSE as a loss function to minimize the distance between samples. We pre-train the model on the Ego4D-100K dataset and fine-tune on FreiHAND*. The experimental results below:

(AW: adaptive weighting)

Q4: L143-144: Why balance the number of left and right hand if they all end up being converted to right-handed images?

A4: Performing a flip on left-hand images to convert to right-hand images is a standard approach in 3D hand pose estimation. Since our hands are symmetrical, this approach is widely used to reduce the complexity of the input space and make it easier to learn the hand pose. For the downstream use cases, the converted images and predicted poses are flipped again to align with the original images.

Q5: Eq 1: Why not use the cosine similarity which is more popular for distances in feature space?

A5: We use the Euclidean distance in Eq 1 because the pose embedding originates from 2D keypoints rather than general image features. Analogy to measuring the distance in pose with the Euclidean metric (e.g., MPJPE in the evaluation of pose estimation), we select the Euclidean metric.

Q6: Fig3: The colored boxes at the end of the model pipeline seem to be in the wrong order. E.g the figure shows positive samples minimizing alignment. -L238-239: rough -> noisy

A6: Thank you for pointing out the ordering issue in Figure 3 regarding "Minimize alignment" and "maximizing alignment," which could lead to ambiguity in the original version. This issue has been addressed and clarified in the latest uploaded revision.

Q7: Table 1: What is "baseline"? This needs to be explained in the image caption.

A7: The baseline model used in Tab.1 and Tab.2 follows our fine-tuning algorithm of ResNet50 + heatmap regression [4], which are trained from scratch, i.e., the baseline without pre-training. To improve the clarity, we have replaced the term "baseline model" with "w/o pre-training" in the revised manuscript.

Q8: Table 1: Why are the worst results of SimCLR in bold? Shouldn't the most performant number be in bold?

A8: Thank you for the reviewer's observation. Based on the reviewer's suggestion, we have adjusted the table formatting in the newly uploaded version.

Q9: Not all figures and tables are referred to in text.

A9: In the newly uploaded version, we revise all figures and tables to be properly referenced. Thank you for your suggestion.

Q10: Table 3: inconsistent capitalization of simclr etc. This also occurs occasionally in the text.

A10: We have adjusted for case inconsistencies in the newly uploaded version. Thank you.

Q2 (W2): How does this work perform compared to other related work on the tested datasets?

A2: Since our work primarily focuses on achieving more efficient contrastive learning-based pre-training on large-scale wild hand data, we have dedicated more space and effort to presenting experiments related to the pre-training method. To address the reviewer's question, we have included more comparative results on the test datasets. We present some more comparisons below:

For DexYCB:

For AssemblyHands:

Reference:

[5] Fu Xiong et al. “A2J: Anchor-to-joint regression network for 3D articulated pose estimation from a single depth image.”

[6] Adrian Spurr et al. “Weakly supervised 3D hand pose estimation via biomechanical constraints.”

[7] Tze Ho Elden Tse et al. “Collaborative learning for hand and object reconstruction with attention-guided graph convolution.”

[8] Shangchen Han et al. "UmeTrack: Unified multi-view end-to-end hand tracking for VR"

[9] Takehiko Ohkawa et al. "AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation"

We deeply appreciate the time and effort reviewer PnQd (R3) took to review our paper, and especially the valuable suggestion to compare with the weakly-supervised approach via training of noisy annotations. Below are our responses to all the questions (W: Weakness; Q: Question; A: Answer):

Q1 (W1): How does this work compare to a weakly-supervised approach with noisy annotations?

A1: We appreciate the reviewer’s recommendation for another comparison method to improve our work, ie., a weakly-supervised baseline using 2D noisy keypoints. We have added a comparison with the experimental results of a weakly-supervised setting. We find that a naive joint training across the labeled and unlabeled data rather worsens the performance due to the noisiness and unreliability of the 2D keypoints. We believe that additional keypoints filtering (to remove highly noisy labels) or a scheme that corrects 2D keypoints would be necessary to effectively utilize the noisy labels on unlabeled data. Notably, our pre-training method has superiority as it performs well without such additional filtering and keypoint correction methods. Based on the same experimental setup, the results on FreiHand* for the two different settings are as follows:

As reviewer PnQd mentioned, this baseline demonstrates that the approach of directly adding noisy 2D annotations during pre-training or supervised training results in noticeably worse performance, especially when there are highly noisy labels and significant differences from the officially provided labeled samples, as shown in the example above.

Before designing our pre-training method, we identified the following issues with weakly-supervised setting applied to larger-scale, in-the-wild hand data based on past experience:

• When the amount of noisy hand data is significantly smaller than the Official training data size, it can provide some improvement in a weakly-supervised setting, but it's hard to apply to large-scale, in-the-wild hand data (e.g., 2 million in-the-wild hand images in our work);
• Introducing a larger-scale noisy hand data in a weakly-supervised setting extends training time and slows convergence;
• The weakly-supervised setting lacks cross-dataset generalizability, as a model trained on dataset A using a weakly-supervised setting performs poorly on datasets B or C.

I thank the authors for their extensive response to my review. As I believe these results are extremely important to underly the value of the paper, will the authors be including the results as well as the code to these experiments that i requested upon acceptance?

We appreciate the reviewer 4kod (R2) insightful and positive feedback. We have summarized the questions raised in the reviewer 4kod, as well as the areas of weakness suggested for improvement, as follows (W: Weakness; Q: Question; A: Answer):

W1: Some presentation issues need improvement.

A1: Thank you very much for reviewer 4kod suggestions regarding the presentation of our work. We have addressed the issues in the newly uploaded version. Specifically, all images in the paper have been replaced with vector versions to ensure clarity and eliminate blurry text.

W2: The article lacks references and discussions on self-supervised methods. The recent two works, S2Hand[2] and HaMuCo[3] , although not pre-training methods, also attempt to use unlabeled images and 2D off-the-shelf detectors to train 3D hand pose estimation models.

A2: Thank you for suggesting additional related work for self-supervised methods. Although they are not pre-training methods, we find some relevance to our work. S2Hand attempts to learn 3D pose only from noisy 2D keypoints on a single-view image, while HaMuCo extends such self-supervised learning to multi-view setups. Based on the reviewer’s recommendation, we have updated the first section of "Related Work" for discussions. Where we have highlighted the additions in blue text between lines L110 and L114.

[2] Yujin Chen et al. "Model-based 3D Hand Reconstruction via Self-Supervised Learning."

[3] Xiaozheng Zheng et al. "HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning."

We sincerely appreciate the careful and thoughtful comments and the time reviewer vbov (R1) spent on them, especially the suggestion to compare with the TempCLR method. We have provided responses to all the questions as follows (W: Weakness; Q: Question; A: Answer):

Q1 (W1): TempCLR [1] proposes a pre-train framework for 3D hand reconstruction with time-coherent contrastive learning, and shows better performance compared with PeCLR. Although TempCLR focuses on reconstruction tasks, the used parametric model can output 3D pose results. Therefore, more comparisons with TempCLR would be helpful.

A1: Thank you for your suggestion. We have added a comparison with the experimental results of TempCLR [1]. As shown in the Table below, we evaluate our comparison methods pre-trained from the 50K and 100K sets of Ego4D and fine-tuned on FreiHands*. The TempCLR’s performance surpasses PeCLR, which is consistent with the results reported in the original paper.

We find the following limitations of TempCLR compared to our method:

• 1. inefficiency in data collection
• 2. limited gains in contrastive learning from neighbor hand images.

While TempCLR treats neighbor hand frames as positives, the tracklets of hands in such dynamic egocentric videos are often truncated due to hand occlusion or hand detection failures. This makes it difficult to collect hand crops in adjacent frames. Indeed, its collection requires x4 more images of detected hand crops to construct 100K samples of neighbor hands from Ego4D, which suggests that detected hand images are not fully utilized.

Furthermore, as shown in Figure 2, the sampled neighbor frames often have limited diversity in backgrounds. Thus, the improvement over PeCLR, which makes positive pairs from a single image, is marginal. In contrast, our HandCLR leverages similar hands that provide diverse characteristics, including various types of

• 1. hand-object interactions,
• 2. backgrounds, and
• 3. appearances.

That's the reason HandCLR exhibits significant gains over PeCLR and TempCLR.

Q2 (W2): In the second column of Figure 6, HandCLR demonstrates advanced performance in hand-object occlusion. Does the proposed method exhibit robustness in similar severe occlusion scenarios involving hand-object interactions? More qualitative analysis in datasets like DexYCB or in-the-wild scenarios would be helpful.

A2: Yes, our method indeed demonstrates enhanced robustness in hand-object occlusion scenarios. Our HandCLR method benefits from pre-training on large-scale in-the-wild videos, including complex and various hand-object interactions. In addition, our pre-training from non-identical similar hands effectively handles scenarios where the query image contains partial occlusion, while the corresponding similar hand image does not, and vice versa. Such examples can be found in both Figure 2 and Figure 7. We appreciate the reviewer’s recommendation to include more qualitative analyses. In the appendix of the revised paper, we will add additional visualizations in hand-object interaction scenarios, such as DexYCB.

Q3 (W3): The proposed adaptive weighting mechanism is a straightforward approach that has proven effective; however, it lacks a clear articulation of its motivation, particularly regarding the challenges faced in 3D hand pose estimation tasks as mentioned in the introduction.

A3: The motivation behind our adaptive weighting mechanism lies in effectively utilizing similar hands in the contrastive learning framework. A naive approach to using sampled similar hands in the contrastive learning is to simply replace original positive pairs with the similar hands, but it fails to account for the degree of similarity between pairs. Our adaptive weighting scheme overcomes this limitation by dynamically assigning higher weights to more similar pairs, enabling the model to better capture the proximity of samples and enhance the contrastive learning.

In response to reviewer vbov’s comments, we have updated the introduction in the newly uploaded version. Please refer to the L81-L87 in the newly uploaded version, where the additions are highlighted in blue.

Reference:

[1] Andrea Ziani et al. "Tempclr: Reconstructing hands via time-coherent contrastive learning."

Dear Reviewer vbov (R1),

We would like to sincerely thank you for reviewer vbov's thoughtful feedback and for acknowledging the improvements in our work, particularly in addressing the concerns raised by previous reviewers. Your recognition of our efforts means a great deal to us.

We would like to kindly remind you that ICLR allows for the modification of ratings directly within the reviewer's system. This ensures that the rating changes are effectively recorded and considered. We would greatly appreciate your assistance in updating the rating directly in the reviewer's system to reflect the improvement of your rating.

Once again, thank you for reviewer vbov's time and valuable comments.

Best Regards,

All Anonymous Authors