OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model

We sincerely value Reviewer juyL's comments, which are helpful. A point-by-point reply to your comments is provided below.

Q1：Is the method using a single <Aff> token? It appears as if multiple are used in Fig 2, but only one is mentioned in the methodology section. If multiple are used, these need to be made explicit in the method section, and relevant experiments should be included in the ablation studies.

A1: Yes, our method uses multi <AFF> tokens. Lines 139-141 and Formula (6) show that we use S occurrences of the segmentation token <AFF>. The additional ablation study results are as follows:

Q2: Furthermore, all results presented would benefit from statistical significance testing, as mean and standard deviation are provided. This is important, as some results in the ablation (eg, MPJPE and Diversity) appear comparable, and the importance of each component may be undermined (eg, affordance loss).

A2: Thank you for your suggestion. For the ablation study, we performed paired two‑sample t‑tests, repeating each test five times and reporting the mean p‑value. As summarized in Table , the proposed method is significant at the 95% confidence level for the majority of metrics (P-value $\leqslant$ 0.05).

Q3: Failure cases, error analysis and limitations are missing

A3: Thanks for your question. We enumerate the failure cases, error analysis and limitations below.

Failure Cases and Error Analysis: Given a scenario, there is a row of cabinets, and the instruction "Open the cabinet", our model can open one but cannot target a specific one (e.g., "Open the second cabinet"). This limitation arises because the model has not been trained on a large-scale 3D QA dataset, resulting in reduced logical reasoning capabilities.

Limitations: As stated in the Conclusion section, we have outlined our limitations and future work (lines 292-297). We acknowledge that our model currently struggles with (1) highly complex, scene‑level, long‑horizon interactions (e.g., "cooking a meal") and (2) extremely fine‑grained dynamics on articulated objects (e.g., "fluid simulation in pouring tasks"). Addressing these challenges will be the focus of our future work.

Q4: How long is the horizon before performance deteriorates?

A4: Due to the accumulation of model errors, performance typically degrades after more than three consecutive actions.

Q5: Diversity arrows on tables seem incorrect.

A5: Our diversity arrows on tables are correct. In fact, in the HOI domain, sequence diversity is deemed better the more closely it matches the ground‑truth distribution of the original dataset; this is exactly how Text2HOI and HOIGPT report their results.

We sincerely thank Reviewer wJWx for his/her comments and questions. We offer point-by-point clarifications to the issues you raised.

Q1-1: The paper adopts ShapeLLM as the backbone of the 3D MLLM. What are the advantages and disadvantages of this choice compared to the other 3D MLLM models?

A1-1：Thank you for your question. We enumerate the advantages and disadvantages below.

Advantages: ShapeLLM is trained on a large amount of 3D embodied interaction data and achieves state-of-the-art performance across various downstream tasks. It possesses strong priors in 3D interaction and demonstrates impressive zero-shot 3D representation capabilities.

Disadvantages: ShapeLLM has not been trained on part‑level object annotations, which limits its reasoning capabilities for fine‑grained object understanding.

Q1-2: How does ShapeLLM's performance in the context of OpenHOI align with its original design goals and capabilities?

A1-2：First, ShapeLLM is a powerful 3D foundation model that performs exceptionally well on multiple downstream tasks (e.g., Embodied Visual Grounding, Visual Question Answering, and Scene Understanding), making it highly suitable for HOI tasks.

Second, we adopt a coarse-to-fine tuning strategy: we first pre-train the model on an object-centric affordance dataset to enable it to acquire strong affordance priors. Then, we fine-tune the model on HOI datasets to better align the semantics with the target domain. This allows the model to learn highly effective affordance representations.

Q2: What are the potential applications of the proposed OpenHOI framework in real-world scenarios, such as robotics and virtual reality? What challenges might arise in practical applications, and how can they be addressed?

A2: Thank you for your suggestion. Applying OpenHOI to real‑world scenarios is both a challenging and compelling endeavor.

Robotics Application: OpenHOI can be integrated into real‑world robotic manipulation systems, including industrial robot arms and service robots, to enable more flexible and human‑like interactions.

• Open-World Affordance Grounding as Powerful Guidance for Robots:Leveraging OpenHOI's 3D MLLM, our method performs open-world affordance grounding to facilitate the identification of feasible grasping, pushing, and tool-use regions on novel objects, thereby significantly improving success rates for pick-and-place, assembly, and tool-handling tasks.
• Realistic HOI sequences synthesis for Robot Manipulation: The HOI sequences generated by OpenHOI can be adapted into robotic manipulation sequences. First, we can use an extraction algorithm to obtain the object's 6-DoF pose. Then, inverse kinematics are applied to the wrist parameters to compute the robot arm's pose. Finally, a retargeting algorithm transfers the human hand motions onto various robotic hand configurations for manipulation. This approach ensures smooth, precise and robust manipulation behaviors in real‑world deployments.

Virtual Reality Application: By synthesizing realistic 3D hand–object interaction sequences, OpenHOI enables users to manipulate virtual objects naturally, for example by picking up, twisting, or pouring items, thereby enhancing immersion in training simulators, gaming, and virtual prototyping.

Challenges: Robotic manipulation tasks typically demand rapid inference. We plan to employ DPM‑Solver to accelerate the diffusion inference process important direction for our future work.

We appreciate Reviewer 7WnN's detailed feedback and the opportunity to clarify our work. We provide comprehensive responses to your concerns in the following.

Q1: Questionable generalizability of the diffusion-based HOI generation module.The proposed framework consists of two sequential stages: (1) 3D MLLM-based instruction decomposition and affordance reasoning, and (2) diffusion-based HOI generation conditioned on the outputs of the 3D MLLM. While the first stage may offer high generalizability by leveraging the pre-trained 3D MLLM, it remains unclear how the second, diffusion-based module achieves strong generalizability. In the experiments, the performance gap relative to the existing state-of-the-art baseline (Text2HOI) does not noticeably differ between seen and unseen objects. This raises the question of whether the improved performance is primarily due to stronger representation capacity rather than actual improvements in generalization, which is the main claim of the paper.

A1: Thanks for your question. Affordance is a powerful and explicit prior for interaction that can guide complex and fine-grained HOI synthesis. Our model achieves strong generalization by using affordance as a middleware layer.

• Open-World Affordance Grounding: We first employ a coarse-to-fine tuning strategy to equip the model with strong affordance reasoning capabilities, enabling it to generate open-world affordance grounding. This enables the synthesis of realistic HOI sequences, demonstrating strong template-free generalization capabilities.
• Affordance serves as a crucial condition: The open-world affordance grounding serves as a crucial condition for the affordance-driven HOI Diffusion.We incorporate affordance not only during training but also in the loss-guidance applied during inference.
• Affordance-based Refinement: we design an improved refinement strategy based on affordance: for the interaction between the hand and the object, we optimize based on affordance grounding rather than the conventional closest-surface-point approach.
• A Template-free Example: Our 3D MLLM has learned from a wide variety of cups, it can still generate accurate affordance grounding for a completely unseen mug. Accurate affordance grounding will guide the synthesis of realistic HOI sequences.

Q2: Missing evaluation for physical realism. One of the claimed technical contributions is the introduction of a physics-based refinement stage to enhance the physical realism of the generated HOI sequences. However, empirical validation for this component is missing, despite the fact that the closest baselines (e.g., Text2HOI, HOIGPT) report evaluation metrics specifically designed to assess physical realism (e.g., physical realism, interpenetration volume).

A2: We supplemented our experiments by evaluating Physical Realism and IV metrics against the closest baseline, Text2HOI (HOIGPT’s code is not publicly available), and conducted ablation studies on our Physical Refinement module. We will include these two metrics in the final version of the paper.

Compare with the closest baselines:

Ablation Studies on Physical Realism and IV:

Q3: Questionable performance on long-horizon generation. Another core contribution claimed by the paper is the ability to generate long-horizon HOI sequences. However, I have concerns about this claim due to the method design. Methodologically, the approach performs in-between motion refinement by enforcing smoothness only between adjacent frames (Eq. 11), which contrasts with existing long-horizon synthesis methods (e.g., FlowMDM) that consider much longer transition windows. Clarifying these points would be helpful.

A3: Thank you for your suggestion. Motivated by FlowMDM's longer transition windows, we conducted comparative experiments using transition windows of different sizes, and the results are shown below. We will include this experiment in the final version of the paper and revise our Motion In-between method accordingly. We sincerely appreciate your feedback, which has helped improve our approach.

We greatly appreciate Reviewer 7v2M's valuable comments and insightful questions. We will address your questions in detail below.

Q1:The visualizations provided for affordance prediction and hand-object interaction are somewhat limited, particularly for unseen objects. More qualitative examples would help better illustrate the model’s generalization capability.

A1: Thanks for your suggestion. We included additional visualizations of HOI sequences for unseen objects in the Appendix(Figure A2). More visualizations of the affordance results will be added in the final version, particularly highlighting comparisons on unseen objects.

Q2:In line 193, shouldn't the term "hand joints" actually refer to hand surface vertices? If so, how did you recover the hand's surface?

A2: Yes, you are right. The "hand joints" actually mean "hand surface vertices". We recover the hand's surface information to the MANO model using $L_{hoi\_train}$ (Eq. (6)).

Q3: In the affordance refinement, why is the distance computed to the nearest point on the entire object surface, rather than to the nearest point within the predicted contact region?

A3: Thank you for raising the concern. We are sorry for the typo error in our description and will revise the explanation accordingly in the final version of the paper. Yes, aligning the hand keypoints more closely with the affordance region yields superior performance. In our method, we explicitly guide the hand joints toward the affordance region, which is why we refer to this process as "Affordance Refinement".