Generalizable Hand-Object Modeling from Monocular RGB Images via 3D Gaussians

We sincerely appreciate the reviewer’s insightful comments. To facilitate a more accurate interpretation of our work’s scope and contributions, we would like to provide some clarifications that may help address potential misunderstandings.

Clarification of Our Setting
HOGS focuses on hand-object assets generalizing to unseen scenes and motions rather than synthesizing entirely new object categories. In particular, prior methods, such as HOLD, are typically tailored to fixed motion trajectories and environments, spanning only short clips of a few hundred frames and requiring per‑sequence modeling. In contrast, HOGS generalizes hand‑object assets across novel motions and lighting conditions (e.g., diverse interactions, poses, and dynamic shadows) without any per‑sequence fine‑tuning. During inference, we run on monocular images from unseen sequences with novel viewpoints, environments, and motions, rather than on new object categories. Details follow.

• Object Pose Estimating Generalization
  While template-based monocular pose estimators may limit category generalization, this is not a concern in our setting, which focuses on cross-sequence generalization for trained hand and object assets. On the contrary, compared to prior methods that adopt per-sequence motion estimation (e.g., SfM), our monocular pose estimators significantly improve generalization by eliminating the need for sequence-specific optimization.

• Cross-scene and Cross-motion Generalization
  HOGS is trained end-to-end on a large collection of monocular interaction sequences that include mixed objects, motions, and environments. While the training data indeed encompasses multiple objects, it focuses primarily on variations in interaction behaviors and environmental context. Specifically, on DexYCB, we sample 147,526 monocular images from over 1,000 sequences covering 10 subjects and 20 objects. Through such training, HOGS acquires scene modulation capability and, when encountering unseen sequences with novel contacts, poses, and lighting conditions, can adaptively modulate hand-object assets from monocular images in a purely feed-forward manner without per-sequence optimization.

• Clarification on Evaluation Scope and Object Categories
  Our goal is not novel-object synthesis but cross-sequence generalization for trained hand and object assets. At inference time, we evaluate on unseen sequences featuring novel motions and viewpoints. Since our method is not intended for novel-category synthesis, we focus on benchmarks that share YCB object categories rather than those containing entirely new objects.

Transient Appearance Effects Visualization
As demonstrated in Figure 3 of the main paper, our visually-driven modulation effectively captures fine-grained transient phenomena such as hand-induced shadows on the can surface (columns 2–4) and dynamic reflections, which static baseline methods fail to reproduce. These effects are also observable in the supplementary video, where the can’s movement induces dynamic, transient surface reflections. Due to the transient nature of these changes, they may be overlooked; we will include more explicit annotations to better showcase them.

Qualitative Effects of V-PM and G-PM
As shown in Figures 3 (rows 2–4) and 5, we provide qualitative ablations of V-PM and G-PM: removing the vision-dependent component leads to degraded rendering quality and reduced robustness to color jitter, while removing G-PM affects geometric consistency. However, we only visualized the effect of G-PM on the rendered results. We appreciate the reviewer’s suggestion and will include explicit 6D pose visualizations (e.g., via 3D bounding boxes) in the final version to more clearly illustrate the benefits of geometry-driven pose refinement.

Details of ViT
We use a CLIP-pretrained ViT backbone and fine-tune it with an initial learning rate of 1e-6 during HOGS training.

Input Setup and Cross-Scene Definition
The input setup and cross-scene definition are detailed above in the ‘Cross-Scene and Cross-Motion Generalization’ subsection within ‘Clarification of Our Setting’. We appreciate the reviewer’s feedback and will incorporate clarifications in the final version to prevent any possible confusion.

Inference Efficiency
Training on DexYCB takes approximately 20 hours, as described in Sec. 4.3. '0.1s adaptation' means HOGS adapts to unseen scenes purely via feed-forward inference without fine-tuning, achieving 11.6 FPS rendering on NVIDIA 4090 GPU. Explicit mesh reconstruction requires additional post-processing, taking seconds. By contrast, HOLD requires hours of per‑sequence optimization and lacks real‑time rendering.

Clarification on Wording and Claims
We regret any confusion caused by the term 'unconstrained'. Our intention was to emphasize that, unlike prior sequence-tailored methods requiring per-sequence fine-tuning, our approach generalizes to unseen monocular images in a purely feed-forward manner. We will revise the wording and clarify the setting in the final version to avoid overstating the claims.

We thank the reviewer for recognizing the paper’s clear structure and the importance of our research direction. Below, we address the concerns about the current performance.

Clarification of Empirical Performance
We appreciate the reviewer’s insightful feedback on both quantitative and qualitative results. It is important to highlight that our method operates under a substantially different and more challenging setting. Unlike methods such as HOLD that rely on per-sequence optimization tailored to specific scenes, our model is purely feed-forward during inference, requiring no optimization. This presents a more demanding scenario where the model must generalize across diverse motions and lighting conditions without any scene-specific adaptation. Despite this, we achieve comparable or better generalization than sequence-tailored methods (e.g., Table 3), and significantly outperform other baselines under the same setting (e.g., MOHO), which we believe underscores the effectiveness of our approach.

Choice of Baselines
Our comparisons focus primarily on recent 2D-supervised methods, such as MOHO (CVPR 2024) and 3DGS-Avatar (CVPR 2024). These methods leverage only 2D annotations (e.g., 2D keypoints, photometric supervision) and share a similar evaluation setting with ours. Additionally, we compare against the latest 3D-supervised SOTA method, HORT (concurrent work, ICCV 2025 accepted) [1], and our method, without requiring dense 3D ground-truth supervision, still demonstrates superior object reconstruction performance on DexYCB.

Regarding G-HOP (CVPR 2024), which pre-trains diffusion-based implicit hand-object priors for reconstructing interaction sequences, designed for video-based interaction reconstruction and cannot be directly used as a comparable baseline.

Writing Quality
We appreciate the reviewer’s suggestions and will improve the paper’s clarity and accuracy by:

1. Revising descriptions of implicit representations (e.g., NeRF and SDF) versus explicit representations (3DGS) for correctness.
2. Fixing typographical errors and adding missing citations.

Reference
[1] Chen Z. et al., "HORT: Monocular Hand-held Objects Reconstruction with Transformers," ICCV, 2025.

I find the authors' response to the "(1) Clarification of Empirical Performance" question very confusing. In the main paper, "generalizability" indicated reconstruction performance on unseen sequences. However, in the above response, the arguments are primarily focused on the efficiency aspect of the proposed method—claiming that its "efficient, scalable feedforward inference" enables "the scalability of directly running on unseen sequences without any per-sequence tuning."

While I agree that the scalability and efficiency of this feedforward method are likely better than those of the optimization-based HOLD, if the "generalization" emphasized in the paper was actually referring to this scalability enabled by higher efficiency, I believe the overall narrative of the paper would require a significant revision.

As a similar concern about this "generalization" claim was also raised by another reviewer (fKDv), I will maintain my negative rating.

We appreciate your continued response. In our view, both the reconstruction quality and the efficient scalability for novel sequences together constitute the key aspects of generalization for high-fidelity hand-object reconstruction, rather than two unrelated points. In addition to reconstructing the hand-object pose and mesh, we also need to reconstruct flexible appearance elements such as texture, shadow, and lighting. Because existing methods such as HOLD lack generalization capabilities for unseen appearance details, they need to require a lot of post-optimization stepsand time to fit these appearance details (HOLD ~1 Hours Vs. Ours ~30 Seconds for a clip with 300 frames). Therefore, in the photorealistic hand-object reconstruction task, we argue that efficient scalability for novel sequences is also one of the important criteria for generalization ability.

Accordingly, in the main paper, our narrative addresses both the modeling quality on unseen sequences and scalability enhancement over sequence-specific methods (e.g., in Line 27, reducing novel-scene setup time from ~1 hour to ~0.1 seconds without any post-training adaptation). To support this claim, we design comprehensive experiments from two perspectives:
(1) Tables 2 and 4 evaluate the reconstruction and rendering performance on unseen sequences without test-time tuning; and
(2) in Table 3, we demonstrate the comparable performance in contrast to per-sequence optimization approaches.  

In Tables 2 and 4, we achieve leading performance on unseen sequences compared with monocular reconstruction/rendering baselines under the same generalization setting (i.e., tested on novel frames without test-time tuning). In particular, in Table 4, we re-implement sequence-specific methods to investigate their performance on unseen sequences. The results highlight their limited generalization capability without per-sequence tuning.  

The reason we emphasized efficiency more in our response is to clarify your earlier concern regarding the seemingly comparable performance to HOLD: HOLD operates on seen sequences and requires per-sequence optimization, whereas our method offers highly efficient scalability. Additionally, we have previously clarified a similar concern brought up by Reviewer fKDv, and we hope our earlier response was helpful in that context.

I thank the authors for their response, and some of my questions have been resolved. However, I believe the overall narrative in both the main paper and the rebuttal is not very clear, and I am concerned that the paper may require a major revision to improve its clarity.

In particular:

1. In the rebuttal, “generalization” is framed as a combination of reconstruction quality and efficiency, but this framing is not clearly presented in the main paper.

2. I remain unconvinced by the responses to my concern regarding the “seemingly comparable [or worse] performance to HOLD.” The authors addressed this by stating (i) that “HOLD lacks generalization capabilities for unseen appearance details,” yet the comparison in question was about shape modeling, not appearance, and (ii) by highlighting efficiency, which I agree is an advantage, but which does not address my specific question about comparable or worse reconstruction quality. I keep raising this point because I believe HOLD is the most competitive baseline among those in Tables 2 and 4, as most of the others were not originally designed for the same hand–object reconstruction task.

We appreciate the reviewer’s recognition of our method’s strong performance and technical soundness. We would like to clarify some misunderstandings regarding the intended scope and generalization of our work.

Clarification of Generalization Capability
HOGS focuses on generalizing hand-object assets to unseen scenes and motions rather than synthesizing entirely new object categories. Specifically, our goal is to improve over prior methods that are limited to fixed motion trajectories and environments (e.g., only hundreds of frames), by enabling assets to generalize across novel motions and lighting conditions. Thus, during inference, we evaluate on monocular images from novel sequences with unseen viewpoints, environments, and motions, while not on new object categories.

Symbol and Equation Clarity
We appreciate the reviewer’s suggestion to clarify the symbols, and will replace σ with sigmoid(·) to avoid confusion with the opacity symbol. The sigmoid function in Eq. (7) is used as an activation for the color component, following the design used for other attributes in 3D Gaussians.

Inference Efficiency
HOGS adapts to unseen scenes purely via feed-forward inference without any fine-tuning, achieving a rendering speed of 11.6 FPS on NVIDIA 4090 GPU. Explicit mesh reconstruction requires additional post-processing, which takes a few seconds. By contrast, HOLD requires hours of per‑sequence optimization and lacks real‑time rendering.

We thank the reviewer for recognizing the novelty, performance improvements, and the advantage of not requiring per-scene fine-tuning. Below, we provide detailed clarifications to address the concerns regarding the training and inference procedures.

Clarification of V-PM
The V-PM module is designed to disentangle transient, context-dependent visual effects (e.g., lighting variations, occlusions) from stable, geometry-related attributes. It achieves this via inferred offsets predicted from ViT-based visual features, which inject rich contextual cues into the parameter space. Rather than simple additive noise, these offsets enable conditional modulation of Gaussian attributes based on visual modality, promoting disentanglement.
Take color modeling as an example, SH coefficients handle angular reflectance but fail to capture transient effects like global illumination, cast shadows, or motion-induced highlights. To address this, we introduce a visual-dependent color component, adaptively blended with SH via a learnable gate. Table 7 (ID 4) demonstrates the render quality improvement with this adaptive appearance modeling; Figure 3 shows the visual-driven modulation captures fine-grained transient effects (e.g., hand-induced shadows, dynamic reflections) that static baselines cannot reproduce.

Details of the Training and Inference Process
HOGS is trained end-to-end on a large collection of monocular interaction sequences that include mixed objects, motions, and environments. Training data indeed covers multiple objects, but focuses more on variation in interaction and scene context. Specifically, on DexYCB, we sample 147,526 monocular images from over 1,000 sequences covering 10 subjects and 20 objects. Through end-to-end training on diverse sequences with varying motions and environments, HOGS learns scene modulation capabilities. In this way, when presented with unseen sequences, HOGS can adaptively modulate the hand-object assets from monocular images in a purely feed-forward manner, without per-sequence optimization.

For the inference procedure, we clarify that HOGS is intended to generalize to unseen scenes and motions rather than to synthesize entirely new object categories. Accordingly, during inference, HOGS takes monocular images from unseen sequences with novel motions and viewpoints, and directly produces scene‑adapted outputs in a purely feed‑forward manner, without any fine‑tuning.

Clarification of Pose Refinement
In G-PM, we focus solely on refining the object pose, as it critically impacts reconstruction quality. Hand poses are not updated within G-PM; instead, hand articulation is represented using a learnable LBS weight during hand Gaussian modeling, which avoids the complexity of jointly optimizing both hand and object poses.

---

Ablation for Loss Functions

We appreciate the reviewer’s suggestion to further analyze the loss terms. In response, we conducted ablation studies on the mask loss, perceptual loss, and contact/penetration losses. The results show that removing any of these losses leads to performance degradation, particularly in terms of structural and perceptual quality metrics. We will include the complete results in the final version.

---

Comparison with More Recent Methods

EasyHOI is a zero-shot hand-held mesh reconstruction method. It leverages the generative capability of LRMs to model arbitrary object categories. However, in this paper, we focus more on the complexity of scenes during interactions, involving irregular object orientations, frequent motion changes, and dynamic lighting effects. Such scene complexity poses significant challenges for accurate mesh generation in zero-shot models. In contrast, HOGS is explicitly trained on diverse interaction sequences, enabling hand-object assets to generalize across varied motions and environments. This fundamental difference leads to a significant performance gap, as shown below:

We follow the evaluation protocol of EasyHOI and test on the same 500 images from DexYCB. We report Chamfer Distance and F‑score with 5 mm and 10 mm thresholds.

Since a fair comparison is NOT applicable between these two settings, we omitted EasyHOI in the main paper. We appreciate the reviewer’s suggestion to include more recent methods. To this end, we additionally report a comparison with the latest 3D-supervised SOTA monocular hand-held object reconstruction approach, HORT (concurrent work, ICCV 2025 accepted) [1], where our method still demonstrates superior object reconstruction performance on DexYCB.

Results of HORT are cited from its original paper. Chamfer Distance is omitted due to inconsistent computation protocols.

---

References

[1] Chen Z. et al., "HORT: Monocular Hand-held Objects Reconstruction with Transformers," ICCV, 2025.

We sincerely appreciate the time and effort you have invested in reviewing our paper and providing insightful feedback. As the discussion period is drawing to a close, we want to ensure that our rebuttal has comprehensively addressed your concerns. We are keen to receive any further feedback you may have and are prepared to make additional clarifications or modifications as needed.
Additionally, we have supplemented the analysis with additional metrics to provide a more comprehensive comparison of SOTA 3D-supervised monocular hand-held object reconstruction methods. Thank you once again for your valuable insights. We look forward to your final thoughts.

Besides the median Chamfer Distances reported in the main paper, we also include the mean values. The mean Chamfer Distance is more sensitive to outliers. Our method achieves superior reconstruction results across multiple metrics without relying on dense 3D supervision.