3D Interacting Hands Diffusion Model

Dear Reviewer MWq4,

Thanks for giving valuable comments. Let us address the concerns.

1. Sampling speed

Like many diffusion-based methods, we agree that sampling speed could be one of our weaknesses. We will clarify this in the final version. We observed that even with such a high number of iterations, thanks to our low-dimensional representation of two-hand (201 dimension), our sampling speed is affordable. We observed that it takes about 25 seconds for 1000 iterations, shorter than 1 minute of Tevet et. al. (MDM).

2. Novelty

We think our biggest novelty is using image-independent generative model to address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images. As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images. We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff. We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images. Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.

3. Predicting the original clean sample

We agree and we do not argue this is our main contribution. Please check out the above rebuttal (2. Novelty) for our main contribution.

4. What benefits does the method offer?

Please refer the the above rebuttal (2. Novelty).

5. Why is generating interacting hands meshes considered an important task?

There could be two benefits. First, being able to generate means we successfully modeled the prior distribution of 3D interacting hands. By successfully modeling the prior distribution, we could bridge the domain gap between in-the-lab and in-the-wild environment, as described in (2. Novelty). Second, it can be used for many action decision problems, including human-like robots. Let's say human-like robots are doing some jobs or having social interaction with humans. For every timestep, robots should decide what to do next based on current/past observations and status. Robots should not do any unexpected or unnatural behaviors. In this case, we can regularize the next 3D hand pose of robots be included in our prior distribution so that robots do some plausible and meaningful interaction between two hands. Without such a regularizer, robots can do very unnatural hands, for example, collisions between two hands, which can destroy the robots.

6. More comparison to diffusion-based methods

Unfortunately, there are no diffusion models for 3D interacting hands. Instead, in Table 2, we compared ours with IHDiff (SA only). In Section 3.2 and page 8, we described that the IHDiff (SA only) has almost the same network architecture as Tevet et. al. (MDM). Table 2 shows that our IHDiff produces much better results than IHDiff (SA only). We will clarify more about this in the final version.

Please let us know if the reviewer has more concerns or more things to discuss. We are happy to have any discussion.

Best, Authors

Hi Reviewer MWq4,

Thanks for raising valuable concerns. Let us address your concerns.

1. Following Reviewer MWq4's suggestion, we decreased the number of iteration to half, which reduces the fitting time to half. It actually improves the vertex error of Table 3 to 19.63, while achieving almost the same collision (0.88) and contact metrics (12.91). We tried to reduce the iteration even more, but the overall numbers become worse. We agree that the fitting time is one of our limitations and will clarify this in the manuscript. Despite the limitation on the fitting time, it would be greatly appreciated if the reviewer recognizes that our IHDiff is the first unconditional generative model for the 3D interacting hands, and achieves clearly better results than existing regressors and other generative baseline models, such as VAE and diffusion-based MDM-style one (IHDIff (SA only)).

2. The reviewer's concern on the dataset is actually applicable to all data-driven learning-based systems, not only for ours. We are not clear why the reviewer thinks such a limitation is evident in our pipeline, but all experimental results, done on unseen data, show how our IHDiff generalizes well to unseen data. For example, Table 2 is performed on InterHand2.6M test set (30 fps), unseen during the training of IHDiff, and Fig. 7 is from unseen in-the-wild video, captured by a mobile phone. Last, Table 3 is from HIC dataset, not used for the training of IHDiff. In particular, InterHand2.6M test set (30 fps) consists of highly diverse 3D interacting hands, while different from those of the training set. We think this makes our evaluation on the generalizability reliable.

Thanks for recognizing the significance of our work.

Although there is a little technical innovation in the network architecture (cross attention-based Transformer (Ours) vs. self attention-based Transformer (MDM)), we think our approach of unconditional generative model has strong novelty. Existing generative models for 3D interacting hands recovery mainly belong to conditional generative model like Zuo et al. ICCV 2023. Although their conditional generative model can perform well on studio dataset, such as InterHand2.6M, by being fully supervised by 3D GT, their system fail to produce robust results from in-the-wild images. This is because in-the-wild images do not provide 3D GT, while their system requires paired (image, 3D GT) for the training; hence, their system suffers from the domain gap between studio and in-the-wild datasets. Such domain gap is necessary to be addressed, but greatly challenging due to the 1) big image appearance domain gap between studio and in-the-wild images (studio images mainly have black backgrounds with artificial illuminations. Please see images of InterHand2.6M) and 2) difficulty of capturing 3D GT from in-the-wild environment as setting the multi-camera setup in in-the-wild environment requires huge amount of manual efforts and expert-level techniques of camera calibration and synchronization.

On the other hand, we smartly addressed the domain gap by choosing the unconditional generative model. Although our unconditional generative model is trained only on studio datasets with 3D GT, it can generalize well to in-the-wild images as no image appearances are conditioned.

Hence, designing unconditional generative model itself is a research problem and we firstly tried it to address the domain gap using it. It would be great if the reviewer see the motivation of our approach for the real-world applications instead of focusing on technical details, such as network architectures.

Thanks, Authors

Dear Reviewer uZ7W,

Thanks for giving valuable comments. Let us address the concerns.

1. More concrete description of future work

Once our IHDiff is trained, we can design a regressor that takes a single image and estimates the latent code. Then, the latent code can be passed to our pre-trained IHDiff to decode it to 3D interacting hands. By estimating additional 3D global orientation and translation of the right hand, we can put the decoded 3D interacting hands in the camera-centered coordinate system. Then, we can supervise the 3D interacting hands with various targets, such as 2D joint coordinates. While training the regressor, we think there could be two options: fixing the pre-trained IHDiff or fine-tuning it. We think this is a research problem. If we fix IHDiff, it can preserve useful prior knowledge, learned in the studio data; however, there could be a small domain gap to the in-the-wild datasets. On the other hand, if we fine-tune IHDiff, we can further reduce the domain gap; however useful prior knowledge could be diminished. We think both directions make sense, and different research contributions should be made for those two directions.

2. Generalization on the varying hand sizes and shapes

Thanks for pointing out such a valuable comment. We actually agree with the reviewer. In the current manuscript, what we did for the hand shape is regularizing it with our collision avoidance loss. This should encourage our IHDiff to give hand shape parameters that do not introduce collisions given 3D poses; however, it is not fully investigated in terms of diverse hand shapes. For example, the extremely thick hand could still introduce collision between two hands. For this research direction, we think we first need to collect a 3D hand dataset with diverse hand shapes as currently available ones do not have enough diversity. We'd like to push that research direction in the future.

We will include and clarify all valuable comments and suggestions from the reviewer. If the reviewer has more things to discuss, please let us know. We are happy to have more discussion.

Best, Authors

Dear Reviewer Z2YY,

Thanks for giving valuable comments. Let us address the concerns.

1. Vertex error

Please note that we achieved almost the same vertex error (0.7 mm difference) compared to the state-of-the-art method while significantly improving the physics-based metrics, such as contact and collisions. In particular, contact between two hands is critical to determine semantic meaning of interaction between two hands. Ours is the first work that explicitly address the contact metric between two hands.

2. Contribution

We think our biggest novelty is using image-independent generative model to address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images. As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images. We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff. We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images. Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.

3. Comparison to NN

Please note that NN retrieves the nearest GT from the training set for each sample of the test set. The ratio of collision is computed only using the output of each method (retrieved GT for NN's case), not against GT from the testing set. Hence, by the nature of GT, it gives low ratio of collisions. In other word, the ratio of collision checks the quality of the output (retrieved GT for NN's case) and say nothing about the similarity to the GT of the test set. On the other hand, it gives low vertex error and contact accuracy as these two metrics measure similarity to GT from the testing set.

4. Left hand-relative space

We think the model can also learn left hand-relative space. Actually, the left hand-relative space can be easily modeled by appying the inverse transformation of the root joint of the left hand to our representation.

5. Generalization to unseen data

All the numbers in Table 2 and Table 3 are from unseen test data. In addition, all results of Figure 7, Figure 10, Figure 11, and Figure 12 are from unseen test data.

Please let us know if the reviewer has more concerns or more things to discuss. We are happy to have any discussion.

Best, Authors

Hi Reviewer Z2YY,

We think there is a misunderstanding. The results of IntagHand are not from our own reproduced ones, but as we said, they are from their official code and pre-trained checkpoint. In other words, we used code and pre-trained checkpoint from here: https://github.com/Dw1010/IntagHand. The different numbers are only from a different evaluation protocol because previous MPVPE calculates the vertex error for each right and left hand separately by normalizing the translation of each hand with each root joint. This is exactly the same way of calculating the error for the single hand case, which cannot consider interactions between two hands. On the other hand, we calculate the vertex error by normalizing the translation of the two hands with the right hand root joint, which means the relative translation between two hands is not canceled during our evaluation. We think ours is a better metric because the relative translation between two hands is an important signal to define interactions between two hands, such as contact.

In terms of collision and contact between two hands, our results are clearly state-of-the-art. We improved the contact accuracy 100% compared to InterWild, and 400% compared to IntagHand. Figure 7 shows that collision and contact between two hands are necessary metrics. In particular, compared to InterWild, the positions of two hands (vertex error) is similar, but ours 'looks' much better in Figure 7 because of the less collision and better contact.

In addition, we decreased the number of iteration to half to reduce the fitting time to half. It actually improves the vertex error of Table 3 to 19.63, while achieving almost the same collision (0.88) and contact metrics (12.91), which achieves state-of-the-art results on all metrics.

Dear Reviewer h3k4,

Thanks for giving valuable comments. Let us address the concerns.

1. Novelty

We think our biggest novelty is using image-independent generative model to address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images. As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images. We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff. We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images. Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.

2. Why not use InterHand2.6M for Table 3?

As described in the paragraph right above Section 6, networks trained on InterHand2.6M already perform well on test images of InterHand2.6M as training and testing sets of InterHand2.6M look almost the same. As we described in the above rebuttal (1. Novelty), we are focusing on more real-world applications by addressing the domain gap between in-the-lab datasets and in-the-wild images. Hence, we used the HIC dataset (Table 3) and Figure 7, which consists of images captured in a daily environment, not from a studio.

Please let us know if the reviewer has more concerns or more things to discuss. We are happy to have any discussion.

Best, Authors