Learning Dense Hand Contact Estimation from Imbalanced Data

We thank the reviewer xnje for reviewing our paper.

We sincerely appreciate the reviewer’s positive assessments of our work. The recognition that the paper is clearly written and accessible is especially encouraging. We also thank the reviewer for highlighting the comprehensiveness of our ablation studies, which we believe strengthens the credibility of our contributions. Below, we address the specific points raised in the review.

Q: Limited novelty of the proposed methods

A: While Balanced Contact Sampling (BCS) and VCB loss build on known principles, both are carefully adapted to address the specific challenges of dense hand contact estimation, which requires precise vertex-level prediction under severe class and spatial imbalance, common in real-world hand interaction datasets.

BCS is not a simple resampling heuristic, but a dataset-driven strategy targeting class imbalance. It groups training samples by a contact balance score, exposing the model to both contact-sparse and contact-heavy hands. Since most hand vertices are non-contact, BCS is essential to avoid overfitting to dominant patterns of non-contact. As it uses dataset-level statistics, it remains effective across varying datasets.

VCB loss addresses spatial imbalance by applying vertex-specific reweighting based on contact frequency. This prevents overfitting to frequently contacted regions (e.g., fingertips) and promotes learning from underrepresented regions. Unlike generic balancing techniques, VCB directly resolves the spatial imbalance nature of dense hand contact datasets and is broadly applicable through its reliance on label statistics.

Our method yields strong gains over prior work and supports downstream tasks like 3D grasp optimization and hand-object reconstruction. By explicitly addressing class and spatial imbalance, it provides both practical benefits and conceptual contributions to dense hand contact estimation.

We will clarify these motivations and contributions more explicitly in the final version.

Q: Since the proposed method is to solve the imbalanced data issue, I think experiments on the data with different classes might be helpful to verify the effectiveness of the method

A: We appreciate the reviewer’s insightful suggestion. To directly evaluate the effectiveness of our method under imbalance issue, we constructed a test set composed of highly imbalanced samples. Specifically, we selected the top 500 samples with the highest contact balance scores (as defined in Eq. 1 of the main paper) from a combined pool of MOW, HIC, RICH, and Hi4D datasets. This subset explicitly reflects extreme imbalance, making it well-suited to assess the robustness of imbalance-aware methods.

Table R1. SOTA comparison for dense hand contact estimation on assorted dataset with high imbalance

As shown above, HACO achieves the best performance across all metrics, demonstrating strong robustness even on samples with severe imbalance. This directly confirms the impact of our method in handling imbalance issue. We will include these results and the process of dataset acquisition in the final version.

Q: Can you further explain the rationale of network structure design? Why self-sttention+cross attention? Why adding contact token? Is it learnable?

A: Our model architecture closely follows the HaMeR framework, which has demonstrated strong performance in hand mesh recovery task. This design decision reflects our focus not on proposing a novel network architecture, but rather on advancing contact estimation through: (1) training on a large-scale composite dataset spanning 14 independent datasets, (2) Balanced Contact Sampling (BCS) to address class imbalance, and (3) Vertex-Level Class-Balanced (VCB) loss to address spatial imbalance in hand contact.

Regarding the self-attention and cross-attention modules, we adopted the structure used in the officially released HaMeR implementation. This combination has proven effective for fusing image features with mesh queries, and we retained it to ensure fair comparison and isolate the effects of our proposed training strategies.

The contact token used in our architecture serves as a query token for cross-attention Transformer (as mentioned in L154), which is also analogous to the query token used in HaMeR's official implementation. Similar to the class token in Vision Transformers, this learnable contact token serves as a global summary vector for predicting vertex-level contact. By learning this token end-to-end, the model is encouraged to develop a contact-specific representation that guides the Transformer’s attention during feature aggregation. This design is widely adopted in Transformer-based frameworks where there needs effective feature aggregation for regression or classification task.

We will clarify these architectural choices more explicitly in the final version.

Q: I found that the comparison with SoTA methods is only conducted on MOW dataset. Testing them on more other datasets will be better.

A: We appreciate the reviewer’s suggestion and agree that broader evaluation across multiple datasets provides a more comprehensive assessment. In response, we conducted additional comparisons on three more benchmark datasets: HIC, RICH, and Hi4D. These datasets represent diverse hand interaction scenarios and include hand-hand, hand-scene, and hand-body contacts.

Table R2. SOTA comparison for dense hand contact estimation on HIC dataset

Table R3. SOTA comparison for dense hand contact estimation on RICH dataset

Table R4. SOTA comparison for dense hand contact estimation on Hi4D dataset

Across all three additional datasets, HACO significantly outperforms prior SOTA methods, confirming its robustness and accuracy in diverse interaction settings. These additional results will be included in the final version of the manuscript.

Q: Can you give more details about how to compare with DeepContact since your model only outputs contact map?

A: Our comparison with DeepContact is conducted using the ContactOpt framework, which optimizes coarse 3D hand and object predictions using contact information. We utilize ContactOpt’s Differentiable Contact Optimization stage to refine the coarse 3D hand and object poses (from HFL-Net) for both DeepContact and HACO experiment.

The key difference lies in how contact is estimated: DeepContact predicts contact from the coarse 3D hand and object meshes (extracted using HFL-Net), while HACO predicts contact directly from the RGB image. Note that DeepContact does not use image information in any manner. Rather, it relies solely on the geometry of the coarse 3D hand and object meshes to estimate contact. As 3D hand grasp optimization inevitably requires initial 3D hand and object poses, we use the same HFL-Net predictions for both methods during ContactOpt’s Differentiable Contact Optimization stage. In HACO’s case, we supply the HACO's estimated contact from image alongside these coarse 3D poses to ContactOpt’s optimization process.

As shown in Section 5.4 and Table 6 in main paper, HACO achieves comparable or superior results across various metrics. These results demonstrate that HACO's contact prediction enables robust and effective contact-guided 3D grasp optimization, despite relying only on image input during contact prediction.

We will clarify this protocol more explicitly in the final version.

Dear reviewer xnje,

We want to thank you for reviewing our paper and spending time reading and analyzing our paper. Although we are notified that the reviewer's decision has been sent, we still wish to follow-up and organize some of the discussion points that we have made during the rebuttal. In the previous response, we made the following updates/clarifications:

• Clarified that the contributions of BCS and VCB loss are addressing challenges specific to dense hand contact estimation. Added new experiments on a highly imbalanced subset (Table R1) and additional datasets (Tables R2–R4) to further validate our method.

• Explained the rationale behind the network design and clarified that the comparison protocol with DeepContact is within the ContactOpt framework.

This expanded discussion will be included in the final version. We are happy to answer further questions.

We thank the reviewer BqXF for reviewing our paper.

We appreciate the recognition of our method’s contributions, including the use of Balanced Contact Sampling (BCS) and Vertex-Level Class-Balanced (VCB) loss to address data imbalance in dense hand contact estimation. We are also grateful for highlighting the clarity & originality of our formulation, the planned code release, and the significance of our work for hand interaction tasks. Below are our responses to the questions mentioned in the review.

Q: Flawed baseline comparisons where POSA, BSTRO, DECO are designed for full-body HS but evaluated on HO tasks

A: We thank the reviewer for this important concern. Our goal in comparing HACO with full-body contact models (POSA, BSTRO, DECO) was to demonstrate HACO’s capability in the broader context of hand contact estimation, a task lacking dedicated baselines.

In real-world use cases, full-body models (e.g., DECO) are often the only available tools for hand contact estimation from a single RGB image. To address this practical gap, we compared HACO with these methods.

However, we acknowledge the need for broader evaluation. Hence, we implement and compare hand-specific versions of POSA, BSTRO, and DECO by replacing their SMPL/SMPL-X modules with MANO and training them on the same 14 datasets as HACO (Table R1). Additionally, we reproduced hand contact modules from DefConNet (Decaf), Stage 1 in CHOI, and InteractionNet (DICE) (Table R2). While not designed solely for dense hand contact task, these methods offer relevant baselines.

Table R1. SOTA comparison for dense hand contact estimation with hand version of POSA, BSTRO, DECO on MOW dataset

Table R2. SOTA comparison for dense hand contact estimation with hand contact estimation modules (†: reproduced, re-trained with 14 datasets, code not available) on MOW dataset

HACO consistently outperforms both hand-specific variants of full-body methods and dedicated hand contact modules. These results highlight the strength of HACO’s task-specific design. We will include them in the final version to provide a more comprehensive evaluation.

Q: Primary evaluation on MOW dataset despite GT problems

A: We thank the reviewer for this important observation. While we acknowledge that the MOW dataset still contains spatial imbalance, it is a key benchmark with relatively lower spatial imbalance compared to other HO datasets (e.g., DexYCB, HOI4D) and still serves as important baseline for HOI tasks (e.g., HORT [D]).

We selected MOW as our primary benchmark as it is closest to real-world hand contact scenarios among the 14 datasets. In contrast, most other datasets are collected in controlled settings, making them less suitable for evaluating dense contact under realistic, unconstrained conditions.

Nevertheless, we agree that including additional evaluation provides a broader perspective. To address this, we conducted additional SOTA comparison on the HIC, RICH, and Hi4D dataset as part of response for reviewer xnje in its Table R2, R3, and R4.

As shown in the results, HACO continues to outperform prior methods by a substantial margin. We will include these results in the final version.

Q: DeepContact, EasyHOI experiments show minimal improvements

A: We have added experiments in Table R1 and R2 to facilitate broader comparisons with significant improvements. We will add them to our final version.

Q: Table 4 ablation ordering is not motivated and it starts with non-target domains

A: The purpose of Table 4 in main manuscript is to show that diverse hand interaction datasets are critical for robust contact estimation. To highlight the impact of dataset diversity, we intentionally excluded the target interaction category for the first three rows (e.g., exclude HO when evaluating on MOW, or HS when evaluating on RICH).

The table is ordered to begin with the most generalizable interaction types (e.g., hand-object, hand-scene), which provide broader contact coverage than narrower categories like hand-hand. Starting with the same category as the evaluation set could lead to overfitting and would obscure the benefit of diverse training.

By progressively adding categories, we emphasize that diversity is key to accurate and robust dense contact estimation.

Q: Unclear contribution of HACO vs. massive 14-dataset training

A: We thank the reviewer for raising this point. To assess the contribution of large-scale training to HACO’s performance, we conducted an ablation study comparing different training dataset size. The results below show that while training on the full set of 14 datasets improves performance, HACO still achieves competitive results when trained on only 3 datasets (DexYCB, ObMan, MOW), demonstrating that the method’s effectiveness does not solely rely on large-scale data.

Table R3. Ablation for training dataset size on MOW dataset

We will include this ablation study in the final version of the paper to better clarify the individual contributions of the model design and training data scale.

Q: In Tables 2 & 3, BCS and VCB were not properly isolated

A: We thank the reviewer for pointing out this concern. Our ablation experiments in Tables 2 and 3 in main manuscript were intended to show the individual contributions of Balanced Contact Sampling (BCS) and Vertex-Level Class-Balanced (VCB) loss relative to the full HACO model, following standard ablation protocols adopted in many previous works. However, we acknowledge that the original tables did not include a baseline where both BCS and VCB were removed, which is helpful to fully isolate their effects.

To address this, we provide an extended ablation study in Table R4, which includes all combinations of BCS and VCB. This allows for a clearer interpretation of each component’s contribution and their combined effect on performance.

Table R4. Ablation for Balanced Contact Sampling and VCB loss on MOW dataset

We will include this full ablation in the final version of the paper to better clarify the individual and joint contributions of BCS and VCB loss.

Q. Fragmented contact predictions contradict claimed "smoothness loss" benefits mentioned in L238-239

A: Fragmented contact refers to spatially isolated predictions. As noted in L238–239, our smoothness loss penalizes such discontinuities, promoting more coherent and anatomically plausible results. We will clarify this in the final version.

Q. Missing GT visualizations in Figure 3

A: Thank you for your thoughtful feedback. We will include GT visualizations in the final version to improve interpretability.

Q. Some predictions inconsistent with visual evidence

A: We acknowledge that HACO, as the first dense hand contact estimator trained on large-scale datasets, still has limitations. We will provide a detailed discussion of failure cases to guide future improvements.

Q. Section 3.2 clarity can be improved

A: Thank you for your feedback. We will revise Section 3.2 to improve clarity on Balanced Contact Sampling.

Q. Unclear grouping of Decaf (hand-face) and Hi4D (whole-body)

A: Thank you for your detailed feedback. We will strictly categorize Decaf and Hi4D in different interaction category and fix any mis-categorization accordingly throughout the entire manuscript.

Q. Inflated performance claims: Dramatic improvements in Table 5 come from comparing incompatible methods

A: We now include stronger and more broader baselines that is re-trained on the same dataset with HACO. We will add them to final manuscript.

Q. Failure case analysis: Missing discussion of when/why the method fails or performs poorly

A: We will include detailed failure cases and limitations to help future researchers build upon HACO.

[A] Wu et al. DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image. In ICLR, 2025.

[B] Prakash et al. How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions. In CVPR, 2025.

[C] Ravi et al. SAM 2: Segment Anything in Images and Videos. In ICLR, 2025.

[D] Chen et al. HORT: Monocular Hand-held Objects Reconstruction with Transformers. In ArXiv, 2025.

We thank the reviewer kFWg for reviewing our paper.

We sincerely thank the reviewer for highlighting the strengths of our work. We appreciate the recognition of our paper’s clear motivation, especially the need for contact estimation models trained on diverse datasets and our approach to data imbalance. We are also grateful for the positive remarks on the clarity of our writing and the strong empirical performance across contact estimation, grasp optimization, and reconstruction tasks, as well as the comprehensive ablation and qualitative results. Below are our responses to the questions mentioned in the review.

Q: Lack of empirical support for resolving imbalance between contact and non-contact classes

A: We appreciate the reviewer’s thoughtful comment. Our Balanced Contact Sampling (BCS) is specifically designed to mitigate the class imbalance issue at the vertex-level, which is where the contact estimation prediction and supervision occur. Therefore, although we excluded fully non-contacting hand-level samples from evaluation due to limitations in existing contact metrics (e.g., recall, F1-score), our evaluation remains valid within the vertex-level prediction context, where contact and non-contact vertices co-exist in most samples.

Nevertheless, we acknowledge that excluding fully non-contacting hands during evaluation may overlook certain aspects of the contact estimation problem, particularly the model's ability to distinguish between entirely contacting and entirely non-contacting hands at the hand level. To provide a more comprehensive evaluation, we conducted additional experiments (Table R1, Table R2, Table R3) using binary classification metrics of Accuracy, Specificity, and MCC (Matthews Correlation Coefficient), which are better suited for this scenario. Additionally, since the MOW dataset primarily contains hand-level contacting cases, we evaluate on the RICH dataset with fully non-contacting hand-level samples included, which provides a broader range of contact and non-contact instances.

Table R1. SOTA comparison for hand-level contact estimation on RICH dataset inlcuding fully non-contacting hand samples

Table R2. SOTA comparison for dense hand contact estimation on RICH dataset inlcuding fully non-contacting hand samples

Table R3. Ablation study for Balanaced Contact Sampling (BCS) and VCB loss on RICH dataset inlcuding fully non-contacting hand samples

These results demonstrate that our method remains effective in addressing the contact vs non-contact imbalance, even when fully non-contacting hand-level samples are retained during evaluation.

Regarding the concern about a trivial strategy such as filtering out all non-contacting samples during training, we emphasize that this approach is not applicable in our setting. Our method explicitly targets vertex-level class imbalance, where hand-level contact is irrelevant in the training standpoint. Therefore, we do not need to discard entire hand-level non-contacting samples. In fact, doing so would reduce the diversity of non-contact patterns seen during training, which is counterproductive for learning a robust vertex-level contact estimator. By retaining all samples and addressing imbalance at the vertex-level through Balanced Contact Sampling and VCB loss, our method is able to learn from both contact and non-contact regions in a fine-grained manner, enabling more robust and accurate contact prediction.

We will revise the final version to better clarify this design choice and include evaluation results with fully non-contacting hand samples.

Q: Questionable design for balanced contact sampling

A: Thank you for the insightful question. Our goal with Balanced Contact Sampling (BCS) is to mitigate the severe imbalance in vertex-level contact labels, a common challenge in real-world datasets where most hand vertices are non-contact. Since the ground-truth contact labels are provided per-hand (rather than per vertex-image pair), sampling must be performed at the hand-level, making it essential to design a principled, hand-level proxy that reflects the underlying vertex-level imbalance.

To achieve this, we compute a dataset-wide average contact vector as a neutral reference and define a per-hand contact balance score based on the deviation of each hand’s contact vector from this mean. This scalar score provides a consistent and interpretable way to stratify hands by how contact-heavy, contact-sparse, or atypical their vertex-level contact patterns are. The mean is not used to enforce a unimodal assumption, but to offer a common reference point across thousands of heterogeneous contact configurations.

We emphasize that BCS is not designed to simply span the extremes of the contact spectrum (e.g., from fully contact to fully non-contact), but rather to increase exposure to diverse and underrepresented hand contact patterns. Real-world datasets tend to be skewed and sparse along this spectrum, making naive or uniform sampling approaches risk over-representing common patterns or producing arbitrary diversity without structure. In contrast, BCS applies non-linear binning and stratified sampling based on the contact balance score to construct a training subset that is structurally balanced and representative.

However, we believe that future extensions of HACO could incorporate clustering or interaction-aware multi-modality modeling if the presence of such multi-modality is well analyzed. We will clarify these points and the implications of our sampling strategy in the final version.

Regarding the concern about regularization, we acknowledge that the regularization term encourages predictions to not deviate excessively from the dataset-wide contact mean. While this may seem to contrast with the goal of BCS, which promotes diverse contact patterns, these two components serve different but complementary purposes. The regularization acts as a stabilizer to prevent overfitting to extreme or noisy contact configurations, ensuring that the model remains within a reasonable solution space. This is similar to many standard regularization strategies in deep learning, including adversarial training or entropy regularization, which temper the main loss without undermining its core objective. In our case, BCS enhances the diversity of training signals, while the regularization ensures stability and robustness.

We will revise the final version to clarify these design choices and explicitly address this point.

Q: Lack of justification for spatial imbalance

A: Thank you for the thoughtful feedback. We visualize spatial imbalance using heatmaps on the hand surface in Figure 1-b of the main manuscript for dataset-wide imbalance and Figures A2 and A3 in the supplementary material for per-dataset imbalance. These visualizations highlight the spatial imbalance present for all 14 datasets. In particular, they show that fingertips tend to exhibit higher contact probability (in red) compared to other regions of the hand in many datasets.

However, we agree that including raw data statistics, such as bar plots showing the contact to non contact ratio per vertex, would offer a more detailed and quantitative view of the spatial imbalance. Unfortunately, due to the limitations of the rebuttal format, we were unable to include these visualizations at this stage. We will incorporate this additional analysis in the final version to enhance clarity and depth.

We thank the reviewer a7WB for reviewing our paper.

We sincerely thank the reviewer for recognizing the strengths of our work. We appreciate the acknowledgment that our method addresses the underexplored problem of dense hand contact estimation from vision inputs. We are also glad that the effectiveness of our contact predictions in hand grasp optimization and hand-object reconstruction tasks was appreciated. Furthermore, we thank the reviewer for highlighting the value of our extensive ablation studies, especially the analysis of the VCB loss, which demonstrates clear advantages over standard and class imbalance aware losses. Lastly, we are grateful for the comment that the paper is clearly written. Below are our responses to the questions mentioned in the review.

Q: Is it possible to extend and apply the proposed method to continuous contact representations?

A: We thank the reviewer for raising this insightful and important question. Our current formulation focuses on discrete (binary) vertex-level contact estimation, which enables interpretability and direct integration with downstream tasks such as contact-driven hand-object optimization and reconstruction. While our representation is discrete, we recognize that continuous contact fields, such as offset vectors or signed distance functions (SDFs), represent an important and growing research direction in hand contact modeling.

From a training standpoint, both Balanced Contact Sampling (BCS) and Vertex-Level Class-Balanced (VCB) loss can be extended to regression settings. In the discrete case, BCS mitigates class imbalance between contact and non-contact labels by grouping hand samples based on their overall contact ratio and ensuring that the model sees a more balanced distribution of contact-related and non-contact-related vertices during training. In the continuous setting, a similar imbalance arises: the distribution of contact magnitudes (e.g., SDF values or offset norms) tends to be skewed, with values indicating distant (non-contact) configurations being much more frequent than those near contact. To address this distributional imbalance, BCS can group training samples based on statistics of the continuous contact signal, such as the distribution of offset vectors or SDF values from the hand surface, to ensure balanced exposure that does not skew toward either contact or non-contact cases.

The VCB loss, originally designed to address spatial imbalance across the hand surface in the discrete case, can also be extended to continuous contact regression. In our setting, spatial locations (e.g., mesh vertices or surface regions) exhibit different contact signal distributions. For example, fingertip regions tend to be more frequently in contact, while other areas such as the dorsum are rarely contacted. This imbalance can persist in continuous representations such as SDFs or offset vectors, where certain regions consistently exhibit lower absolute SDF values (i.e., closer proximity), while others predominantly produce large-distance signals. As a result, the regression model may become biased toward frequently occurring signals that is different by spatial region. To address this, VCB loss can be extended by reweighting the regression loss per spatial location using a continuous weighting strategy that assigns higher importance to underrepresented contact-relevant signals. This approach generalizes the vertex-wise effective number of samples strategy used in our original discrete VCB loss, which discretely re-weights contact and non-contact contributions at each vertex based on their contact frequency. In the continuous case, the weighting function would smoothly vary with the contact signal itself, enabling fine-grained correction of spatial imbalance without relying on discrete labels. This formulation would help the model avoid bias toward dominant signal and encourages learning that is more spatially balanced across the hand surface.

From an inference or integration standpoint, HACO’s binary contact predictions can serve as auxiliary supervision or initialization for continuous contact regression. For instance, HACO can be used to first localize probable contact regions, allowing a second-stage model to refine predictions into continuous fields such as SDFs or offset vectors. Conversely, a model trained to regress continuous contact fields can have its outputs thresholded or quantized to derive discrete contact labels, which may be useful for downstream tasks.

We will include this expanded discussion in the final version.