Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation

1. Why does the paper compare its method with single image-based hand-object reconstruction method (e.g., gSDF) while the paper addresses problem of 4D tracking with ViTaM-D. Maybe comparison with video-based methods like HOLD (Fan et al., CVPR 2024) may be more proper?
2. What is the reason that the method predicts flow from time t-1 to time t? If the model already possesses point cloud for time t, wouldn't it be more intuitive to directly predict 3D geometry based on the time t point input?
3. If the paper does not take any RGB input, wouldn't it be unfair comparison with gSDF? As reconstructing 3D hand and object directly from RGB data is much more difficult than reconstructing 3D hand and object from point cloud input, I believe further explanation on fairness of comparison should be discussed.
4. If gSDF re-implementation takes point cloud as input, how is the implementation done? As gSDF is based on RGB image input, it seems non-trivial to implement gSDF with point cloud input.
5. How is the memory requirement of the proposed method compared with gSDF and HOTrack? Curious because the correspondence prediction seems to be memory-heavy.
6. Is it fair to fine-tune VDT-Net on each object category (from L382) while (as far as I know) other methods (e.g., gSDF) did not take such fine-tuning approach on each object category but rather trained the training set as a whole?
7. Is there any reason why the numbers for gSDF from this paper does not match that of Table 6 from gSDF's manuscript?
8. Is there a specific reason why implementing tactile sensor concept for contact representation is superior than modeling contact representation directly but naively based on distance between all vertices of hand mesh and all vertices of object mesh?
9. If previous contact-based hand-object optimization strategies are implemented instead of force-aware optimization (FO) process, would they underperform compared to FO process?
10. Current approach seems to exploit synthetic tactile data (HOT dataset) built upon ZeMa. Could this approach be extended to real tactile sensor data?

1. Question about force-aware hand-pose optimization. As presented in section 4.3, the adopted optimization loss formula (9) penalizes excessive rotation or large angular changes of joints using Lr, and keeps the optimized hand pose stay close to the original prediction using Lo. Both Lr and Lo reflect the geometric constraints. Only the loss E relates to tactile. According to formula (1) to (4), if the distance between the hand keypoint and the object vertex lij is larger than the threshold, it will result in Bij=0, therefore only the relative potential energy Eij will be preserved in the overall energy E. While Eij~Mlij, when the lij is large enough, M will equal to 0. In this situation, the hand pose optimization seems to have lost its driving force. That is to say, if the initial prediction of one joint wrongly stays away from the surface of an object(deep penetration or untouched), the proposed optimization method PO would not have the ability to correct it. So how to solve this problem? If the tactile readings are fixed, E will only proportional to geometric distance, irrelevant to tactile. How to explain? In addition, the commonly used penetration loss and attraction loss based on geometric distance of paired points have been proved effective to make hand-object contact better. What are the advantages of the proposed PO method compared to the commonly used method?

2. About the tactile annotations. As illustrated in Figure 3, the collected tactile information is annotated on the hand surface. In section 5.1, line 339, it says that the training data sequence includes depth images, point clouds, and tactile arrays for each frame. However, in Figure 5, the description is ‘… and the contact map is closer to the ground truth, making the contact more reasonable.’ It seems that there are ground truth contact maps annotated on the object surface. How to get these ground truth contact maps? How to use these ground truth contact maps during training the proposed model? What are the differences between the tactile annotations and the contact maps? Why not examine if the predicted tactile distribution on the hand regions is consistent with the ground truth tactile annotations?

• Improvement of the presentation of the paper is needed (detailed above).
• What's the relationship between DA-Field and FO? From L066-L-074, it seems that DA-Field and FO are differnt things. Is FO a characteristic of DA-Field?
• Can you provide more ablations for each component? For example, Tab. 2 only ablates simple fusion of tactile features. The role of DA field is missing.
• SInce both DexYCB and HOT datasets are possible - would training on both datasets improve performance? Is sim2sim transfer possible?

In summary, my current evaluation leans towards the negative side primarily due to concerns on the presentation of the paper and unclear ablations.

Q1: In Table 1, could you provide more details on the baseline settings? Specifically, how were the baseline results obtained—did you retrain these models, or were pre-existing checkpoints used? Additionally, the Chamfer Distance (CD) values in Table 1 differ from those in Table 6 of the gSDF paper. Could you clarify the reasons for this discrepancy?

Q2: In Table 1, since object reconstructions are not optimized in your approach, could poor-quality object reconstructions lead to worse results after optimization? To help clarify this impact, could you present per-category results to show if object reconstruction quality would affect the final hand pose and contact performance after optimization?

Q3: In Figure 4, the reconstructed object mesh appears coarser than those in gSDF and HOT, even though you report using the same SDF resolution as gSDF and achieve a lower Chamfer Distance. Could you explain this discrepancy in visual quality?

Q4: The ablation study setup in Table 2 is somewhat unclear. I would expect the comparison of incorporating tactile information directly within the network (VDT with Tactile) versus using it in post-optimization (VDT+FO). Ideally, this setup would allow for a comparison of hand pose and contact outcomes between these approaches. However, only object reconstruction results are shown in Table 2. Could you include metrics such as MPJPE, PD, and CIoU to provide a more comprehensive evaluation and enable direct comparison with Table 1?

Q5: In Table 3, this ablation study is also unclear. From Figure 4, it appears that most contact occurs on the fingers, while the palm typically remains non-contact. Assigning a uniform force reading $M_j = 0.5$ across all regions is obviously wrong. Given that your initial result from VDT is refined using this inaccurate information yet still shows improvement, it’s difficult to derive meaningful insights from this ablation, as your experimental premise is wrong. Since Table 1 already demonstrates the effectiveness of force-aware optimization, could you clarify the motivation for this particular ablation study?