SIGHT: Single-Image Conditioned Generation of Hand Trajectories for Hand-Object Interaction

We thank Reviewer ZM35 for the valuable feedback and provide detailed answers to the points raised as weaknesses and questions.

Use of left hand: We note that extending the method to work with both hands is mostly a limitation of the data: the FPHAB dataset features exclusively right-hand interactions, while the HOI4D dataset features right-hand interactions only for all but 2 objects. The referenced work [1] uses H2O, ARCTIC and GRAB, which record trajectories for both hands. An extension to both hands would simply consist of doubling the dimensionality of our model's output to account for another hand.

Validity of generating hand motions from a static image: The ambiguity of human motion generation is inherent in the problem setting. In fact, the generated motions are expected to exhibit a certain diversity, which is measured by the DIV metric. The user of the model must hence expect diverse outputs if insufficient disambiguating information is provided in the input image (or text prompt in text-based motion generation methods such as [2], [3]). We further consider the study of our setting, where explicit action labels are not available and the action to synthesize has to be inferred from the image, to be a valid research question in itself.

Retargeting to simulator: The simulator recordings in the Supp. Mat. are slowed down to allow a more detailed examination by humans. Our generated trajectories include 200 steps, and a linear interpolation of the corresponding actions is further performed to ensure a smooth execution in the simulator. Please refer to section A.6.1 in the Supp. Mat. for more details on the retargeting procedure.

Learning of manipulation prior: The FPHAB dataset features 26 objects from a diverse range of everyday situations. Furthermore, all recordings in FPHAB start with the object already grasped, making learning the actual interaction part of the trajectory the prerequisite for good performance. We believe that our method's strong results on FPHAB provide a reasonable proof that a manipulation prior has been learned. The trajectories of the HOI4D dataset, in contrast, include the grasping of the object in addition to the manipulation itself. Here, too, our method outperforms the baselines, showing its ability to learn a useful manipulation prior.

Performance on deformable objects: We extract a subset from the test set of our FPHAB subject split, and report the performance of our $\mathcal{P}$ (object+part-conditioned) and $\mathcal{O}$ (object-conditioned) models on the letter, book, paper, cloth and sponge classes of FPHAB in the table below. We further report the performance on an equally-sized subset of the remaining data, to account for the sensitivity of the FID metric to the dataset size. The diversity of the respective ground-truth data is reported for each subset.

Deformable objects present a bigger challenge to identify, as their appearance may vary significantly depending on the manipulating hand's grasp. This is reflected in the lower accuracy and inferior FID compared to tests on non-deformable objects. Only in terms of diversity do we observe a better performance on the deformable subset, possibly attributable to the method learning diverse ways to manipulate a deformable object.

We would be grateful if the reviewer would kindly consider an improved assessment of the paper. We thank you very much for your time and feedback.

[1] Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction, CVPR 2024.
[2] Generating Diverse and Natural 3D Human Motions From Text, CVPR 2022.
[3] Human Motion Diffusion Model, ICLR 2023.

Dear Reviewer ZM35,

The discussion phase is coming to an end soon, and we thus kindly request you to let us know if our response has addressed your concerns. We will be happy to answer if there are additional issues or questions, and if not, we would be grateful if you would consider a revised assessment of the review score to reflect that the issues have been addressed.

Best regards,
Authors of SIGHT (Submission 8107)

Dear Reviewer ZM35,

We thank you for your insightful review and for taking the time to address our response. We further thank you for revising your rating of our work, and are delighted that you found our response addressing your concerns satisfactory. We appreciate and have taken note of your final suggestions.

Best regards,
Authors of SIGHT (Submission 8107)

We thank Reviewer 5791 for the valuable feedback and provide detailed answers to the points raised as weaknesses and questions.

Absence of object's trajectory/object-related metrics: We recognize the limitations posed by a missing reasoning about the object's trajectory. However, we believe the other contributions of our work, namely the introduction of the SIGHT task as well as a simulation-based evaluation framework for evaluating the physical plausibility of hand trajectories, together with our method's strong performance against baselines, present useful advances for the research community.

Hand's global translation: Indeed, our method generates the global translation of the hand. In l. 255, for $J=17$, 16 joints (5 fingers x 3 joints per finger + 1 joint for the wrist) describe the pose of the hand, and one joint represents the global translation of the wrist. We merely fixed the hand's translation in the videos included in the Supplementary Material for visualization purposes.

Questions:

1. How can we confirm that the generated hand trajectory properly interacts with the object corresponding to the image?
   Please consult Sec. 4.6, where we perform evaluations in a simulator to verify the physical plausibility of our trajectories.

2. Does the generated hand motion vary significantly even when the same image is used as the condition image, as shown in Figure 1?
   The variation of generated motions is captured by the DIV metric. Please see Section 4.2 for a detailed description. As the motion sequences are initialized with random vectors before being denoised into the output sequences, there is room for variation in the final motions. Generally speaking, we find minor alternations when performing side-by-side visualizations of motions generated for the same image, verifying the stability yet diversity of the outputs of our models.

3. a) What is the P model? Is it different from M? It seems to be a model not described in the Method section. From the caption of Table 3, it seems to be an other version M that uses the part features.
   The $\mathcal{P}$-conditioned model is the variation of our model conditioned on both object and part features. Please refer to Section 3.2 (l. 224) for a more detailed explanation. We would ask for the line where the "M" model is mentioned, as such a model or model variant is not introduced in our work.

   b) Additionally, O is described as the interacting object in L219, but in the caption of Table 3, it is referred to as a model.
   In l. 219, we refer to the $O$-conditioned model, i.e. the model conditioned on object features without part features.

4. Is there a specific reason for not using object features and part features together?
   Please note that part features are always used together with object features. We indicated this in line 224 with "additional conditioning information", yet this may not have been clear enough. We will add a statement indicating this more explicitly in the final version of the paper.

5. Does the model perform well even in situations where there are only a few part features before averaging due to a small object in the image? In other words, I am curious if the model's performance varies significantly depending on the size of the object in the image and the number of part features.
   Please see our reply in the comment below.

6. Is there a reason why ours, part experiments were not conducted for unseen objects in Table 2? It would be helpful to test ours, part as well to understand hand interactions with unseen objects.
   The instance split of HOI4D features images of objects by themselves, i.e. not interacted with by a hand, as described in l. 252 ff. Please consider Table 2 for results on unseen objects.

7. In the simulation examples, are the objects shown in the result images the same as those in the condition image? It would be clearer if the condition (input) images were also displayed.
   The condition images for the simulation runs are 50 randomly selected images for the given action and object in the test set of the FPHAB subject split, corresponding to 50 generated trajectories to evaluate. We average over the hit rates of these trajectories to obtain the final performances in Table 4. Please consider the Supplementary Material (in Trajectory_Visualizations/*fphab*) for examples of the condition images for the pour juice and put salt actions evaluated in the simulator.

We would be grateful if the reviewer would kindly consider an improved assessment of the paper. We thank you very much for your time and feedback.

Our answer to question 5 was extracted to comply with comment length limits.

5. Does the model perform well even in situations where there are only a few part features before averaging due to a small object in the image? In other words, I am curious if the model's performance varies significantly depending on the size of the object in the image and the number of part features.
   We extract a subset from the test set of our FPHAB subject split, and report the performance of our P-conditioned (part-conditioned) model on images with less than 5 part patches (as defined in Sec. 3.2 and visualized in Fig. 2), approximately corresponding to the 10th quantile of the part patch count distribution over all samples in the test set. We further report the performance on an equally-sized subset of the remaining data, to account for the sensitivity of the FID metric to the dataset size. The diversity of the respective ground-truth data is reported for each subset.

Interestingly, the accuracy of the few-patch setting is better than that of the many-patch ("Rest of dataset") one. We attribute this to a loss of disambiguating capabilities when too many patches are averaged over, some of which might not be related to the contacted object part. In terms of the diversity, the many-patch setting performs slightly better, yet both settings yield roughly comparable numbers. The FID for the few-part setting is markedly worse than that for the many-part setting. We hypothesize that averaging over more patches gives features that are more consistent across similar samples, leading to the model being able to learn to use the conditional input more effectively.

Dear Reviewer 5791,

The discussion phase is coming to an end soon, and we thus kindly request you to let us know if our response has addressed your concerns. We will be happy to answer if there are additional issues or questions, and if not, we would be grateful if you would consider a revised assessment of the review score to reflect that the issues have been addressed.

Best regards,
Authors of SIGHT (Submission 8107)

We thank Reviewer Sxj2 for the valuable feedback and provide detailed answers to the points raised as weaknesses and questions.

Diversity of objects in FPHAB and HOI4D dataset: FPHAB includes 26 different objects used in everyday kitchen (fork, juice, liquid soap, milk, mug, peanut butter, salt, soda, spoon), office (calculator, charger, glasses, letter, match, paper, pen, tea bag), and social (card, coin, human hand itself, wallet, wine bottle, wine glass) scenarios, of which we train and evaluate on 25, excluding only the case where the hand itself is treated as an object in the handshake and high-five actions. We further train and evaluate on 9 objects (bottle, bucket, knife, laptop, mug, pliers, scissors, stapler) from the HOI4D dataset.

Contrary to reconstructing hand-object interaction trajectories, as is the focus in HOLD [1], we do not believe it is possible to generate action-appropriate hand-object interaction trajectories without at least seeing an interaction with an object of a similar object category first.

We do not see a connection of our image-based hand motion generation method to text-based 3D mesh synthesis methods such as the referenced [2]. Contrary to the text-to-image setting, where ample data is available online for pretraining, there is a shortage of large-scale datasets with hand mesh data and a sufficient diversity in object categories, making it nontrivial to pretrain a hand motion generation method on large corpora that would allow it to approach category-agnostic synthesis capabilities. However, we demonstrate in Table 2 that our method is well-capable of synthesizing appropriate and realistic trajectories for images of objects from object categories seen during training.

Video comparisons with baseline: Please consult the videos in the supplementary material for examples of trajectories generated by our method. Video comparisons of our method with the text baseline shall be included shortly.

Object-specific metrics: Our method does not depend on/consider the object mesh when synthesizing interactions, as we focus solely on the motion of the human hand. Hence, object-specific metrics cannot be calculated.

Discussion of previous literature: For a discussion of prior hand-motion generation literature, please consult the "Hand motion generation" paragraph of Section 2 where multiple methods are analyzed. Contrary to our method, the referenced work [3] depends on the object mesh as input. Further, it is unable to make use of datasets without 6D object pose annotations and/or action annotations, such as most of FPHAB apart from 4 kitchen objects, while our method only requires hand trajectories as ground-truth supervision during training.

We would be grateful if the reviewer would kindly consider an improved assessment of the paper. We thank you very much for your time and feedback.

[1] HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video, CVPR 2024.
[2] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023.
[3] Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction, CVPR 2024.

Dear Reviewer Sxj2,

The discussion phase is coming to an end soon, and we thus kindly request you to let us know if our response has addressed your concerns. We will be happy to answer if there are additional issues or questions, and if not, we would be grateful if you would consider a revised assessment of the review score to reflect that the issues have been addressed.

Best regards,
Authors of SIGHT (Submission 8107)

We thank Reviewer Yoeh for the valuable feedback and appreciate their comments regarding the importance of the investigated problem, the meaningfulness of our simulation-based evaluation protocol, and the effectiveness of our proposed SIGHT-Fusion method. We provide detailed answers to the points raised as weaknesses and questions.

Handling of articulated objects: We concur with the reviewer that learning to interact with articulated objects is a difficult task, and provide experiments to support this assumption. Articulated objects are indeed found in the datasets used in our work, namely in the open milk/juice bottle/peanut butter/liquid soap, close milk/juice bottle/peanut butter/liquid soap, flip book pages, unfold/clean glasses, open letter and open wallet actions of FPHAB, as well as the clamp with pliers, cut with scissors, open/close laptop, and use stapler actions of HOI4D.

We evaluate our P-conditioned and O-conditioned models on a subset of the FPHAB subject split's test set including the aforementioned articulated objects, and evaluate on an equally-sized, randomly selected subset of non-articulated objects as well. An equally-sized subset is used to account for the sensitivity of the FID metric to the dataset size.

We repeat the same experiment on the HOI4D dataset:

As evident from the results, the accuracy of all models is lower on the articulated subsets of both datasets, confirming the difficulty of generating realistic interaction trajectories for articulated objects. Furthermore, the $\mathcal{P}$ (object+part-conditioned) model outperforms the accuracy of the $\mathcal{O}$ (object-conditioned) model by a remarkable margin on the articulated settings while also yielding trajectories with a better diversity, further validating the usefulness of our part-aware design.

Please refer to the Supplementary Material for visualizations of trajectories generated for the pliers, scissors, stapler and laptop objects of HOI4D, all featuring articulation.

Details regarding conditional injection: The conditional injection is performed by prepending object and/or part feature tokens to the input motion sequence tokens forwarded to the denoising Transformer, as explained in Sec. 3.3. The conditioning tokens are extracted as described in Sec. 3.2. One single token is used for the object and part features respectively. Please see the right side of Fig. 2 for a visualization, where we visualize a single denoising step of the diffusion model. The conditioning information is illustrated using a single blue token, while the motion sequence is illustrated using multiple red tokens.

Dear Reviewer Yoeh,

The discussion phase is coming to an end soon, and we thus kindly request you to let us know if our response has addressed your concerns. We will be happy to answer if there are additional issues or questions, and if not, we would be grateful if you would consider a revised assessment of the review score to reflect that the issues have been addressed.

Best regards,
Authors of SIGHT (Submission 8107)