Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars

We sincerely thank the reviewer’s confirmation of sufficient technical novelty. To address the concerns of the reviewer, we conduct several ablation studies and provide extra visual examples in the RF. Our responses are listed as follows:

W1 Effects of off-the-shelf MANO regressor and mask detector: Thank you for your advice. We add experiments accordingly as follows:

(i) We experiment to evaluate the effects of the off-the-shelf MANO regressor (ACR). The qualitative results are shown in RF-Fig. 2 Ablaiton Mesh and the quantitative results are listed as follows:

The performance of our methods with parameters estimated by ACR is still satisfactory. This is because we have added regulation to texture map bias to prevent our model from over-fitting the incorrect alignment between hand appearance and geometry, which addresses minor estimation errors effectively.

(ii) To validate the impact of using masks produced by SAM, we conduct ablation experiments between using the SAM masks and the mesh rendering masks. The qualitative results are shown in RF-Fig. 2 Ablaiton Mask and the quantitative results are listed as follows:

We can see that using the masks produced by SAM improves the model performance, due to the better alignment with hand boundaries, which helps to mitigate background clutters.

W2 Non-smooth hand boundaries: Thank you for pointing this out. Compared to NeRF-based methods, GS-based methods usually use fewer rendering points, which may lead to less smooth hand boundaries. To confirm this, we further lower the number of Gaussian points to 24k in our method. The experimental results are shown in RF-Fig. 2 (denoted as Low G. Num. in Ablation S1). We find that hand boundaries indeed deteriorate due to the fewer number of Gaussian points.

Q1 The advantage of GS compared to high-resolution hand meshes: The advantage of using GS is that GS is more flexible compared to parametric high-resolution hand meshes. The geometry of parametric high-resolution hand meshes is relatively fixed and thus it's hard for them to present fine-grained geometric deformation of interacting hands. To alleviate such limitation of high-resolution hand mesh, we devise a Gaussian refinement module that adaptively adjusts the number and positions of 3D Gaussian according to different hand poses and identities, which results in better hand geometry in the rendered images.

Q2 Comparing the Reconstruction Time with OHTA: The one-shot avatar reconstruction of OHTA takes 56 minutes on an A100 GPU (according to the OHTA paper) while our method takes 2.5 minutes on an A6000 GPU. For the fairness of comparison, we conduct an experiment to test the fine-tuning cost of our method and OHTA, of which the results are listed below:

where OHTA* means that it uses the same number of training steps as in our method, while that of OHTA** is times by five to ensure sufficient fitting. From these results, we can see that the computational complexity of our method is significantly lower than that of OHTA. The reduction of fine-tuning consumption comes from three aspects: (i) We choose to fine-tune the texture map bias instead of the whole network, which accelerates the process and lowers the calculation cost. (ii) OHTA designs a two-stage fine-tuning method while we only have one fine-tuning stage as texture map bias can be easily regularized. (iii) Thanks to the design of Gaussian Splatting, we lower the rendering points and reduce the rendering cost compared to NeRF-based methods.

Q3 The impact of the accuracy of mesh reconstruction: We conduct an experiment to estimate the effects of different meshes. In this experiment, we compare the performance of our method with the ground-truth meshes provided by the Interhand2.6M dataset, and that with meshes predicted by ACR. The qualitative results are shown in RF-Fig. 2 Ablaiton Mesh and the quantitative results are listed as follows:

The performance of our methods with parameters estimated by ACR is still satisfactory. This is because we have added regulation to texture map bias to prevent our model from over-fitting the incorrect alignment between hand appearance and geometry, which addresses minor estimation errors effectively.

We sincerely thank you for your suggestive comments. We have performed multiple experiments and presented additional real-world results to further validate the proposed method. Our point-by-point responses are listed as follows:

W1 Shadow modeling: Thank you for your suggestive advice. Our current framework indeed focuses on the disentanglement of poses and identities and ignores shadow modeling. However, as we do not impose any constraint on shadows, our framework is of great flexibility and hence is easy to incorporate existing shadow models. To demonstrate this, we follow OHTA to disentangle pose-related features and predict shadow coefficients, which allows our method to model shadows and albedos separately. As shown in RF-Fig. 2, our method with this enhancement successfully separates shadows caused by occlusions. Moreover, we also provide more in-the-wild examples regarding three challenging tasks in RF-Fig. 1. These examples indicate that even our current model can handle real-world scenarios. From these two aspects, we believe the proposed method is a feasible solution for various applications.

W2 In-the-wild results: Thank you for your advice. As shown in RF-Fig. 1, we demonstrate more visual examples of three scenarios, including in-the-wild reconstruction, text-to-avatar, and texture editing. In the in-the-wild results, we use ACR to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results. These results will be added to our revised paper.

W3 Failure cases: We add the demonstration of failure cases as shown in RF-Fig. 3. If severe hand mesh estimation errors occur, our method may fail to model the proper appearance. We notice previous methods also suffer from this issue (as mentioned in the OHTA and VANeRF paper), and a better mesh recovery method is necessary in this case.

Q1 The benefits of using interacting hands for avatar creation: The advantage of using two-hand images is that they provide more complementary information compared with single hands, especially when severe inter-hand occlusions occur. To validate this, we compare the performance of our method with that of the single-hand counterpart by masking one hand in reference images. The quantitative results are listed as follows:

The above results show that our method with two-hand images outperforms that with single-hand images significantly. Moreover, RF-Fig. 2 Ablation Hand Num. demonstrates the visual examples of this experiment. These examples also suggest that using both hands increases the quality of rendered images.

Compared to single-hand modeling, the intra- and inter-hand interaction exacerbates information loss and introduces complex geometric deformations. It is necessary to study the interacting hand reconstruction. Extensive efforts have been made by related research communities to reconstruct interacting hands from a single image, such as VANeRF, ACR, and IntagHand. To improve the performance of interacting hand reconstruction, we propose an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions.

Furthermore, our method can construct a hand avatar using images of both hands with interaction or single-hand images. Since there are situations in which we only have single-hand images or interacting hand images, our method has a wider application than single-hand reconstruction methods.

Q2 Implementation of OHTA*: Please note that OHTA is a single-hand reconstruction approach, and its setup and dataset completely differ from ours. Because its pre-trained model cannot be used in our scenario, we utilize our own pre-trained model and only compare the fine-tuning stage design. The code of OTHA is not published before the deadline for paper submission, hence we implement OHTA* with the texture inversion stage and texture fitting stage following its paper. In the texture-inversion stage, we keep the weights of the pre-trained network frozen and optimize the identity code along with per-channel color calibration coefficients to fit the input image. In the texture-fitting stage, we fine-tune the MLP for texture feature extraction and impose the constraint that the texture-fitting results of reference views should be close to the rendering results before texture-fitting.

Concerning the inferior performance of OHTA*, we consider that the previous number of training steps might be insufficient for OHTA* to fully fit reference images. To investigate this, we further prolong the training steps (denoted as OHTA**) and obtain better performance as follows:

Yet, our method still achieves a better balance between rendering quality and fine-tuning time.

We are grateful for your acknowledgment of the technical contributions and novelty of our method. We have considered your as well as other reviewers' suggestions carefully and tried to solve them as follows:

W1&W2 Presentation: Thank you for pointing this out. We will enhance our presentation in Figure 3 and revise the typo accordingly. The dotted line from the Interaction Points to the Interaction-Aware Attention means that we feed the information of interaction points into the Interaction-Aware Attention module. Considering the page limitation, the revised version of Figure 3 using different colored sub-boxes for geometric flow and texture flow will be presented in the future version.

W3 In-the-wild results: RF-Fig. 1 shows more real-world results and applications of our proposed method, affirming its effectiveness in real-world scenarios and its suitability for handling out-of-distribution data. RF-Fig. 1 comprises three challenging tasks, i.e., text-to-avatar, real image in-the-wild reconstruction, and texture editing. The proposed method handles these tasks effectively.

Q2 The cost of one-shot fitting: Yes, introducing texture map bias helps accelerate one-shot fitting due to the reduction of learnable parameters. Compared to directly fine-tuning the network (427 M), optimizing the texture map bias (0.65M) reduces the fine-tuning time significantly.

Q2 Interacting label: We adopt binarized interacting labels instead of utilizing probabilistic ones because we propose a simple and effective strategy to detect interacting points (P6, L193-197). Specifically, we calculate the difference between the neighboring point sets of posed hand meshes and a canonical mesh. If the difference is above the user-defined threshold, the corresponding interacting label is set to 1 otherwise 0. In our experiments, we find this strategy effective enough to detect self-interacting and cross-interacting points.

Q3 Implementation of OHTA: Because the code of OTHA is not published before the deadline for paper submission and the pre-trained model of OTHA cannot be used in our scenario, we utilize our own pre-trained model and only compare the fine-tuning stage design. Specifically, we implement OHTA* with the texture inversion stage and texture fitting stage following its paper. In the texture-inversion stage, we keep the weights of the pre-trained network frozen and optimize the identity code along with per-channel color calibration coefficients to fit the input image. In the texture-fitting stage, we fine-tune the MLP for texture feature extraction and impose the constraint that the texture-fitting results of reference views should be close to the rendering results before texture-fitting.

Concerning the inferior performance of OHTA*, we consider that the previous number of training steps might be insufficient for OHTA* to fully fit the reference image. To investigate this, we further prolong the training steps (denoted as OHTA**) and obtain better performance as follows:

Yet, our method still achieves a better balance between rendering quality and fine-tuning time.

I appreciate the authors' proactive response. The supplementary ablation experiments on camera parameters and the additional experiments using predicted meshes instead of GT meshes have addressed my concerns.

Regarding OHTA and the authors' reproduced versions OHTA* or OHTA**, I suggest that the authors clarify their descriptions in the revision. Based on my understanding, the actual OHTA is not equivalent to OHTA* (Inv+Calib), as there are other details such as the use of reference views for assistance during finetune.

I'd also like to gently remind the authors about their commitment in the "Open access to data and code" section, where they mentioned that "Code and models will be released upon acceptance." Following through on this would be greatly appreciated and could help address any lingering concerns about reproducibility that reviewers might have.

At this point, I am inclined to give this paper a score of Borderline Accept.

Q1 Implementation of OHTA*: Please note that OHTA is a single-hand reconstruction approach, and its setup and dataset completely differ from ours. Because its pre-trained model cannot be used in our scenario, we utilize our own pre-trained model and only compare the fine-tuning stage design. The code of OTHA is not published before the deadline for paper submission, hence we implement OHTA* with the texture inversion stage and texture fitting stage following its paper. In the texture-inversion stage, we keep the weights of the pre-trained network frozen and optimize the identity code along with per-channel color calibration coefficients to fit the input image. In the texture-fitting stage, we fine-tune the MLP for texture feature extraction and impose the constraint that the texture-fitting results of reference views are close to the rendering results before texture-fitting.

Since our fine-tuning process only contains one stage while OHTA contains two fine-tuning stages, we use twice the training steps for OHTA* (one for each stage). Concerning the inferior performance of OHTA*, we consider that the previous number of training steps might be insufficient for OHTA* to fully fit the reference image. To investigate this, we further prolong the training steps (denoted as OHTA**) and obtain better performance as follows:

Yet, our method still achieves a better balance between rendering quality and fine-tuning time.

Q2 The advantages of introducing camera parameters: Thank you for your advice. We conduct the related ablation experiments to analyze the impact of the camera parameters. The qualitative results are shown in RF-Fig. 2 (denoted as w/o Cam. in Ablaiton S1) and the quantitative results (denoted as w/o Cam.) are listed as follows:

The performance of the baseline without camera parameters drops obviously. Considering the anisotropy properties of Gaussian points (e.g. scale and opacity), it is necessary to utilize camera parameters to encode the view direction information for rendering.

Q3 Mesh estimation method: We use the ground-truth hand parameters provided by the dataset. We conduct the experiment using hand pose parameters estimated by ACR. The qualitative results are shown in RF-Fig. 2 Ablation Mesh and the quantitative results are listed as follows:

The performance of our methods with parameters estimated by ACR is still satisfactory. This is because we have added regulation to the texture map bias in order to prevent the model from over-fitting to the incorrect alignment relation between hand appearance and geometry, which addresses minor estimation errors effectively. However, as demonstrated in the failure cases in RF-Fig. 3, serious estimation errors make it difficult for our model to learn the correct appearance. Previous works like OHTA also mention this issue, which might be alleviated by more reliable hand regressors.

We sincerely thank the reviewer’s valuable comments and suggestions. We have conducted extensive ablation studies and provide more in-the-wild results to further verify the proposed method. Our responses are listed as follows:

W1 Contribution and novelty: We feel it is necessary to clarify the technical contributions of the proposed method from the following perspectives:

First, constructing two-hand avatars via simple extensions of one-hand baselines faces certain limitations. Particularly, it is challenging for single-hand methods to handle severe occlusions and inter-hand interference, which results in inferior performance. This can be validated by our new quantitative (the table in our following response to W2) and qualitative comparisons (RF-Fig. 2, Ablation Hand Num.) provided in the rebuttal. Moreover, single-hand methods do not include modules to leverage the structure and texture similarity of hands, which are informative in various scenarios. Therefore, extensive methods (VANeRF[1], ACR[2], and Im2hands[3]) are tailored specially for the two-hand task.

Second, three novel components have been introduced in our method, including (i) the disentangled 3D presentation to improve flexibility and exploit data priors, (ii) the interaction-aware attention module to handle complex interactions, and (iii) a Gaussian refinement module to improve rendered image quality. The effectiveness of these modules has been validated by the ablation studies in our paper.

Last but not least, we are glad that all other reviewers found the proposed method novel and effective, such as ``novel and intuitive" (Reviewer #gWjw), technically sound (Reviewer #eWJx and #mjNr), and one of the first works" (Reviewer #eWJx and #mjNr). We believe such appreciation can indicate that the proposed method is not incremental.

Hence, we sincerely hope the novel and contributions of our method can be reconsidered. We are open to further discussion and will try our best efforts to address your concern.

[1]Xuan Huang et al., 3d visibility-aware generalizable neural radiance fields for interacting hands. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2400–2408, 2024.

[2]Zhengdi Yu et al., Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12955–12964, 2023.

[3]Jihyun Lee et al., Im2hands: Learning attentive implicit representation of interacting two-hand shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21169–21178, 2023.

W2 The advantages of using images of both hands: The advantage of using two-hand images is that they provide more complementary information compared with those of single hands, especially when severe inter-hand occlusions occur. To validate this, we compare the performance of our method with that of the single-hand counterpart by masking one hand in reference images. The quantitative results are listed as follows:

The above results show that our method with two-hand images outperforms that with single-hand images significantly. Moreover, RF-Fig. 2 Ablation Hand Num. demonstrates the visual examples of this experiment. These examples also suggest that using both hands increases the quality of rendered images.

Compared to single-hand modeling, the intra- and inter-hand interaction exacerbates information loss and introduces complex geometric deformations. It is necessary to study the interacting hand reconstruction. Extensive efforts have been made by related research communities to reconstruct interacting hands from a single image, such as VANeRF, ACR, and IntagHand. To improve the performance of interacting hand reconstruction, we propose an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions.

Furthermore, our method can construct a hand avatar using images of both hands with interaction or single-hand images. Since there are situations in which we only have single-hand images or interacting hand images, our method has a wider application than single-hand reconstruction methods.

W3 In-the-wild results: As shown in RF-Fig. 1, We demonstrate more visual results of various applications, including in-the-wild results from real-captured images, text-to-avatar, and texture editing. For text-to-avatar, we utilize ControlNet with depth maps as condition information for hand image generation. For in-the-wild results, we use ACR to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results.

W4 Shadow modeling: Handavatar and OHTA disentangle the pose-related information and further predict the shadow coefficient so that they can visualize the shadow and albedo separately. We disentangle pose-related information from identity-related information to ensure generalization ability. Instead of predicting the shadow coefficient, we render the shadow and albedo simultaneously. However, our method is flexible in choosing shadow modeling strategies. To demonstrate this, we conduct an experiment that uses disentangled pose-related features to predict shadow coefficients as in OHTA, allowing for the separate modeling of shadows and albedos. The visual results are shown in RF-Fig. 2 Shadow. From these results, it is apparent that our method successfully separates the shadows (e.g. the area beneath the palm). The qualitative results are shown in RF-Fig. 2 w/ Shadow of Ablation S1.