Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba

Q1. Efficiency of the model

R: We did not claim the "efficiency" of the model in terms of Inference time or GPU memory in our manuscript. From the line "GSS block uses 88.5% less tokens", we implied that compared to transformer-based models that utilize a large number of tokens for 3D hand reconstruction, the proposed GSS block uses fewer tokens and is token-efficient. As requested by the reviewer, we provide the ablation for the method's efficiency in terms of inference time, FLOPs, and GPU memory usage, which shows our model is also more lightweight compared with Transformer-based models.

Table: Comparison of the Model efficiency

Q2. Transfer to full body human reconstruction

R: We adapted our proposed model to the body mesh recovery task. Our model achieved comparable performance with 4D-humans (called HMR2.0b) (ICCV 2023). We trained our model on the same mixing datasets as 4D-humans. Furthermore, we compare the two models as shown in the table below. We only trained on a single A100 GPU for 300K steps due to the rebuttal time constraint. The metrics and results on various datasets are shown in the Table below. Hamba showed improvements on LSP-Extended and COCO datasets and achieved comparable results on the 3DPW dataset, even though trained for fewer steps. The performance of our model may be further improved by training more iterations as HMR2.0b did. This confirms our proposed module is capable of serving as a plug-and-play component to solve similar or downstream tasks. We have also included the visual results for in-the-wild scenarios in Figure 1 of the rebuttal PDF.

Table: Transfer Results of full-body mesh recovery task compared to HMR2.0b (ICCV 2023). It confirms that the GSS Block acts as a plug-and-play for 3D body reconstruction.

Q3. Typographical error

R: We thank the reviewer for pointing out the typographical error in the citation. For the final version, we will again proofread the manuscript for typographical and grammatical errors.

Q4. Social Impact of the Paper

R: We have already discussed the Broader Impacts in line 334. We will provide more discussions about Social Impacts in the revision. The proposed Hamba framework for 3D hand reconstruction from a single RGB image can significantly enhance human-computer interaction, medical diagnostics, and rehabilitation by providing more accurate 3D hand estimation. It holds promise for improving sign language recognition and robotic dexterity. The technology can also help for economic growth in various tech industries. However, it raises potential privacy concerns that need to be addressed to ensure ethical use.

Q1. Ablation study with Transformer + GCN.

R: As requested by the reviewer, we replaced the state space block with GCN + Attention (from Graformer [R2]) in our Hamba model and evaluated it on the Freihand benchmark dataset. Further we compared it with our GCN + SS2D (see Table below). Both models are trained on the same dataset setting with a single A6000 GPU for 60K steps. Our GCN + SS2D model shows improvement on all metrics compared to the “Graformer-like Transformer + GCN” architecture. This confirms that our state-space model has better capability to learn the relationship between hand joints.

Table: Additional Ablation w GCN+Attention. Illustrates that GCN + state space modeling outperforms GCN + Attention.

Q2. Transfer to full body mesh recovery.

R: We adapted our proposed model to the body mesh recovery task. Our model achieved comparable performance with 4D-humans (as called HMR2.0b) (ICCV 2023). We trained our model on the same mixing datasets as 4D-humans. Furthermore, we compare the two models as shown in the table below. We only trained on a single A100 GPU for 300K steps due to the rebuttal time constraint. The metrics and results on various datasets are shown in the Table below. Hamba showed improvements on LSP-Extended and COCO datasets and achieved comparable results on 3DPW dataset, even though trained for fewer steps. The performance of our model may be further improved by training more iterations as HMR2.0b did. This confirms that our proposed model is capable of serving as plug-and-play components to solve similar or downstream tasks. We have also included the visual results for in-the-wild scenarios in Figure 1 of the rebuttal PDF.

Table: Transfer Results of full-body mesh recovery task compared to HMR2.0b (ICCV 2023). It confirms that the GSS Block acts as plug-and-play for 3D body reconstruction.

Q3. Justification for the intermediate 3D reconstruction

R: In Hamba, the Joints Regressor (JR) performs the intermediate 3D reconstruction, and the 2D joints (from this 3D re-projection) serves as the input to the token sampler (TS). This particularly helps in effective token selection that encodes the strong local context. The JR is necessary else the GSS block might learn from irrelevant tokens, especially during the early training stage and may be influenced by the background and continue to make random guesses.

• For the joint regressor, we use stacked SS2D layers with one layer MLP head. Compared to this simple architecture, employing heavy off-the-shelf joint detectors like MediaPipe [49] or OpenPose [4] would increase the model complexity.

• Instead of regressing 2D hand joints, we do an intermediate 3D reconstruction to get the initial MANO parameters. This serves as an effective initialization for the Hamba model. Using an off-the-shelf 2D Joints Estimator could not provide this MANO parameters initialization, and we aren’t able to obtain a good hand reconstruction.

• Popular existing off-the-shelf hand 2D detectors are not trainable (e.g. Mediapipe), and could not take advantage of the features extracted by the backbone. Our JR design is simple yet effective, thus providing robust results.

Q1.Clarification of motivation?

R: Our main motivation is to improve SOTA methods (e.g., HaMeR [57]) by modeling the structural relation in the hand skeleton that leads to the model's performance improvement. HaMeR [57] designed a ViT-based model, using ViTPose weights and large datasets to achieve good performance. However, HaMeR requires a large number of tokens per image for reconstruction, and applying attention to all image tokens does not fully utilize the joint spatial sequences, which often results in an inaccurate 3D hand mesh in real-world scenarios.

This raises concerns about whether such model structures can effectively capture the relationships between joints. To address this limitation of the previous model, we utilize the emerging Mamba model, known for its sequence processing capabilities. However, since the Mamba model is primarily designed for vector-sequence inputs, such as those used in NLP or time-series tasks, we adapted it by integrating GCN layers. This adjustment aims to enhance the model's ability to learn the fixed skeletal structure of hands. Our key idea is to reformulate the Mamba scanning into graph-guided bidirectional scanning for 3D reconstruction using a few effective tokens.

All of them are verified in our ablation study, as shown in Table 5 of the manuscript. When we remove the Mamba blocks or GCN layers, performance drops significantly.

Q2. Results on the Ego4D?

R: We were unable to download the EGO-4D dataset before the NeurIPS deadline because it required signed consent and agreement from the official dataset maintainers. Additionally, we encountered technical challenges due to the dataset's large size (1 TB) and the additional 4 TB of storage needed for frame extraction from the Ego4D clips. Now that we have successfully downloaded the dataset, we present our results in Table 3 (included in the submitted PDF). Our Hamba achieves the SOTA performance, surpassing other models on the Ego4D dataset.

Q3. The effectiveness of GSS block with 88.5% less tokens.

R: We provide additional ablation to show the effectiveness of the GSS Block. For this ablation, we replaced the GSS Block with a self-attention transformer taking all 192 image tokens. The comparison is shown in the Table below. The ablation confirms the effectiveness of our proposed GSS Block in reconstructing hands in 3D while utilizing fewer tokens.

Table: Ablation comparing the Tokens, Parameters, FLOPs and the Runtime demonstrating the effectiveness of the SS2D (GSS Block compared to Transformers).

Q4. Finetuned for 170k iterations on Hamer.

R: We want to clarify that our Hamba does not use the HaMeR (CVPR 2024) checkpoint as a starting point. our Hamba and HaMeR have significantly different decoders and overall model architectures, making it impossible to directly load or start from HaMeR's checkpoint. What we intended to convey is that Hamba used the same encoder (the ViTPose backbone) as HaMeR, and we only loaded the backbone's weights. We appreciate the reviewer pointing out this confusion, as it could be confusing to other readers. We will revise the sentence in the supplementary material to make this distinction clearer.

As requested, we fine-tuned the official HaMeR checkpoint for 170K more steps and compared it with Hamba (see Tables 1, 2, and 3 for comparison on FreiHAND, HO3Dv2 and HInt datasets in the submitted pdf). This confirms that merely fine-tuning HaMeR for 170K more steps does not improve performance. We will provide more discussion about it in the revision.

Q5. About the relative minor on the HInt dataset.

R: We would like to emphasize that the HInt dataset is used exclusively as an 'in-the-wild' test set, meaning that none of the models have been trained or fine-tuned on the HInt training set. This makes the evaluation particularly challenging. Despite this, our model significantly outperforms other popular models such as MeshGraphormer (ICCV 2021), METRO (CVPR 2021), and HandOccNet (CVPR 2022). Additionally, when compared to HaMeR (CVPR 2024), Hamba demonstrates consistent improvements. Specifically, it achieves a notable 3% to 6% increase in performance over HaMeR on the Ego4D-VIS@0.05, Ego4D-ALL@0.05, and VISOR-VIS@0.05 subsets.

Q6. More visual comparisons

R: Due to the submission length constraints (9 pages for NeurIPS and 1 page for the rebuttal), we limited the number of visual comparisons. We put 4 more comparison images in the rebuttal pdf, and we plan to include additional visual comparisons in the revision's appendix.

Q7. Response to Limitations

R: We agree with the reviewer’s suggestion that like other SOTA models, like HaMeR (CVPR 2024), MeshGraphormer (ICCV 2021) and METRO (CVPR 2021), etc., our model also relies on hand-detector to crop the hand image. We will include more failure cases with detailed discussion in our final manuscript.