InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

We thank the reviewer for thoughtful and insightful comments. We address your concerns below:

W1: handle more complex long interactions:

• As shown in Figure 4 and in the examples after 00:43 in demo_1.mp4 of the supplementary material, our approach is capable of handling complex and extended interactions, such as “A person holds a medium box up with their right hand, lowers their right arm, and pulls the box with left hand towards them.”
• However, we recognize the model's limitation on very long and highly complex sequences, particularly those involving multiple distinct phases, which we found to be difficult for existing work even with supervised training [21,50,71 in the paper]. To address this, future work could explore temporal compositionality in HOI, as well as incorporate more complex multi-phase interactions into the training data, to enhance the model's capability in these scenarios.

Q1: Accuracy of contact reasoning for novel objects:

• The handcrafted interaction retrieval works well with predefined objects but lacks generalizability, as it requires building a specific database. In contrast, the learning-based interaction retrieval (L671-681 in the supplementary material) can handle novel objects but may struggle in complex scenarios due to two main issues:

  1. Stable Diffusion sometimes generates low-quality images in complex human-object interactions, leading to unnatural humans, incorrect object states, or unreasonable contact patterns.
  2. These lower-quality images fall outside the distribution that LEMON (an off-the-shelf model that we used for estimating object affordance and human contact from images) is not able to predict accurate contact areas (affordance).

  Improving the text-to-image model for human-object interaction could enhance image quality, which we see as a promising direction for future work.

Q2: Efficiency of optimization:

• The optimization is expensive, and each step often takes a few seconds. Thus, for efficiency, as mentioned in the supplementary material (L693), optimization is only performed if the loss exceeds a certain threshold. This strategy prevents unnecessary computations, thus maintaining overall computational efficiency.

Limitation

• Thanks for your feedback. We agree with the reviewer that generating fine-grained human-object interactions (HOIs), such as dexterous hand manipulations, remains a challenging task. One potential solution would be to incorporate a dedicated hand model, trained specifically on the nuances of fine-grained hand interactions (e.g., OakInk 2[1]), and then integrate it with the body model.

  [1] Zhan, et al. OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion. CVPR 2024

We thank the reviewer for thoughtful and insightful comments. We address your concerns below:

W1: Novelty

We appreciate the reviewer's thoughtful suggestions. While our method incorporates elements adapted from existing approaches, we would like to clarify the novelty of these components, particularly in the context of human-object interaction (HOI) generation.

• High-Level Planning: Our approach introduces a simple yet effective method to bridge the distributional gap (Line 138) between pre-trained text-to-human generation and free-form text-to-interaction generation. This is a key distinction from most existing work [102], which primarily utilizes LLMs only for understanding contact parts. We evaluate the significance of our design in Figure 6 and L300-309.

• Low-Level Control: While we employ existing methods and pre-trained models for low-level control, which we do not claim as novel, our contribution lies in the seamless integration and evaluating the effectiveness of four established text-to-motion models within our framework, which show that our framework is general.

• World Model: Unlike existing work that typically encodes the full state of the interaction (as shown in references [43, 83, 108] in the paper), our method introduces a novel approach to modeling contact vertices. As discussed in the paper (Lines 64-68, 184-191), this vertex-based representation ensures that the world model focuses on the critical contact regions, preventing overfitting to specific details, such as particular object shapes. This leads to improved performance, as demonstrated in Table 1, Figure 8, and Figure 5 (generalize to unseen objects from the CHAIRS dataset).

Overall, HOI generation, as a challenging task with inherent data scarcity issues, requires a novel paradigm distinct from existing approaches. Our perspective focuses on how to effectively reuse and harness large models, which is an emerging and crucial research problem. While naive reuse can lead to poor performance (e.g., Figure 6), our approach fosters synergy among models, resulting in contributions tailored to this task. We welcome the reviewer's feedback on any specific related papers and would be happy to discuss further or clarify any similarities with these works.

W2.1: Database build for interaction retrieval:

• We apologize for the omission of details. To clarify, we used only the training set to build the database. We will revise in the revision.

W2.2: Generalization of low-level control:

The low-level control does not require additional training. We would like to clarify how the model generalizes across action types from the following perspectives:

• Generalization to Diverse Actions: The text-to-motion model inherently possesses a degree of generalization across actions, similar to CLIP, where the compositionality of natural language allows for generalization. Semantics in the language space can be composed to represent different actions. By aligning language space [74] or discrete tokens [128, 35] with motion space/tokens, the motion model benefits from the generalization capabilities of the text compositionality.
• Generalization to HOI Descriptions: While these models are not directly generalizable to free-form HOI descriptions, we bridge the distributional gap between free-form interaction descriptions and the text-to-motion model's distribution using chain-of-thought prompting, as demonstrated in Figure 6 and Lines 300-309, which constitutes one of our contributions.

W2.3: No need for pair data:

• To clarify, the proposed method does not require training on paired HOI data. However, it does require training on HOI data without paired text (Lines 62-63), and only the world model needs to be trained with this data (Lines 76-77). We will revise.

W2.4: size of SDF:

• If the size refers to the resolution of the SDF, our approach directly calculates the distance from points to the object’s surface, theoretically providing infinite precision. If a different aspect of size was intended, we would appreciate further clarification.

W2.5: number of vertices:

• It depends on the specific objects used. For example, for the BEHAVE dataset, we use their provided fine-grained objects, each containing over 10,000 vertices, to calculate the SDF.

W3: Motivation for control based on contact vertices

• The motivation for our quantitative experiment in Section 4.2 stems from the need to validate a key contribution of our work: as outlined in the paper (L64-68, L184-191), the vertex-based representation ensures that the world model focuses on critical contact regions. This enables the network to predict how object motion is influenced by interactions at these vertices, reducing the risk of overfitting to specific object shapes or detailed body part movements. By emphasizing high-level concepts–the fundamental principle that human-applied force on an object leads to acceleration (akin to Newton's law)--the network learns more generalizable patterns, enhancing its performance in HOI modeling.
• To substantiate this claim, we compared our method against various control conditions, including a raw control where all available information is fed into the network. The results clearly demonstrate that our approach is superior.

W4: vertex-based control

• Yes, we appreciate the reviewer for pointing this out. The vertex-based control refers to our method where only the contacting vertices are used for control. We will clarify this definition in the revision.

W2.6: object shape encoding:

• The object’s shape is inherently encoded using the human vertex-to-object surface distance, as described in L204-206. This information is then processed through dynamics blocks (MLPs) as outlined in L208 and L210, and further integrated with attention layers (Line 212) for interaction modeling.

We put the response to W5 in the global response of rebuttal because of the character limit.

Thanks to the authors for their responses, but there are still some concerns that remain unresolved.

W1: novelty

1. About High-Level Planning. Firstly, the description “[102], which primarily utilizes LLMs only for understanding contact parts” is incomplete, [102] also predicts object size. Secondly, Figure 6 is confusing. For Figure 6 (a), a fair comparison is the generation of different methods under the same input, and the current comparison is confusing. For Figure 6 (b), the difference between red and green dots is not significant. It cannot be observed that “reduces the distributional gap (L306)”.
2. About the world model. What is the full state of the interaction?

W2.3: no need for pair data: It is still confusing for L180. which dataset for training the world model, if it does not need a paired dataset?

W2.6: object shape encoding: Following the new description, it is still hard to follow to understand how to encode object shapes. It should improve the writing.

W5: handling complex and long sequences: For long sequences, why interpolate the sequences to 30fps? How to interpolate?

In short, the writing of the paper is poor, resulting in some confusing descriptions in the paper. In the rebuttal, the author still claims that there are descriptions in the paper, but the description is confusing. I hope the author will improve the writing to clarify the details of the method. Also, the novelty of the method is limited, so I keep my score, borderline reject.

Thank you to the reviewer for the additional feedback and suggestions. We will incorporate these suggestions and revise our manuscript accordingly. Below, we address the remaining concerns:

W1: [102]

In [102], the input to the LLM includes both HOI interaction descriptions (represented as action and object category labels) and human and object sizes measured in the image space; the output includes both contact reasoning and the real scale of the human and object. In contrast, we only provide the LLM with textual interaction descriptions (free-form text instead of category labels) as input – thus, in the rebuttal, we focused solely on the LLM’s reasoning capability based on the HOI textual descriptions. In comparing the LLM’s reasoning on interaction descriptions, we address not only contact reasoning (L136) but also object categorization (L134) and, importantly, distributional shift, where the free-form HOI description falls outside the distribution of the text-to-motion model (L137). We will provide a more detailed discussion of our differences with [102] in the revision, following the reviewer’s suggestion.

W1: Figure 6(a)

• We would like to clarify that Figure 6(a) ablates the effectiveness of high-level planning. Therefore, in this setup, we use a single model – the text-to-motion model, MotionGPT. We compare two types of inputs: the raw text description ("w/o planning") and the rephrased text generated by the LLM ("w/ planning"), which is stated in L300-302.
• Figure 6(a) provides an example for comparison. As noted in the figure caption, the text on the left side, “Someone can be seen sitting on a yogaball,” represents the raw description. Below this text is the motion sequence synthesized by MotionGPT based on this raw description. On the right side, the text “A person is seated on an object” represents the text rephrased by high-level planning from the original description (“Someone can be seen sitting on a yogaball”). Below this rephrased text is the motion sequence synthesized by MotionGPT using the rephrased input. For both mesh sequences, the transition in color from gray to blue indicates the progression of the time series.
• The comparison was fair: the rephrased text, generated through LLM high-level planning, does not need to be exactly the same as the raw description. Instead, it aligns more closely with the style of text descriptions used to train MotionGPT. Consequently, the motion sequence synthesized by MotionGPT from the rephrased text tends to better match the intent of the raw description (L302-304).

W1: Figure 6(b) & L306

• As stated in the rebuttal, “we evaluate the significance of our design in Figure 6 and L300-309.” Therefore, the claim that our method “reduces the distributional gap (L306)” should be interpreted as being supported by the collective evaluations in Figure 6(a), Figure 6(b), and L309, which include both visualizations and quantitative measurements.
• Specifically, Figure 6(b) qualitatively visualizes the CLIP features, while L309 provides quantitative evidence by comparing the CLIP differences between the raw descriptions and in-distribution text, as well as between the rephrased text and in-distribution text. As stated in L309: “The text processed by the planning shows greater similarity to the in-distribution text from HumanML3D, with an average cosine similarity of 0.932 compared to 0.913 from the raw annotation.” This evidence supports the claim that our approach “reduces the distributional gap (L306).”
• To provide further evidence, we tested our method on out-of-distribution text. We selected examples with an average cosine similarity to in-distribution text of less than 0.85, resulting in an overall average of 0.838. Our high-level planning successfully rephrased these texts, increasing their average similarity to 0.927. For instance, in Figure 6(a), the text 'Someone can be seen sitting on a yoga ball' has a cosine similarity of 0.874 to the closest in-distribution text, whereas the rephrased text by high-level planning, “A person is seated on an object,” achieves a similarity of 0.958 to the closest in-distribution text. We will incorporate this additional evidence to update L309 and the caption of Figure 6.

W1: Full state

• In the rebuttal, we clarified that “unlike existing work that typically encodes the full state of the interaction (as shown in references [43, 83, 108] in the paper)...” The “full state” in existing work refers to the encoding of human joint and object motion with object geometry encoding.
• In contrast, our method introduces a novel approach by focusing on modeling contact vertices. As mentioned in our rebuttal to Reviewer Jz5F, our inputs to the world model include vertex-based control signals provided to the conditional block $\mathcal{G}$ (L208), with the object’s past motion directly input to the unconditional dynamics block $\mathcal{F}$ (L206). We believe this strategy is novel in the field. More specifically, the control signals includes the human vertex motion (including past and future, in L193) and its features: 1) vertex coordinates in T-pose; 2) vertex-to-object surface vectors; and 3) the vertex’s velocity relative to its nearest object vertex as described in L203-L206. Importantly, we only use contacting vertices instead of all vertices.

W2.3: which dataset is used

We mentioned in L68 that the BEHAVE dataset is used to train the world model. We will revise L180 to make it clearer. We would like to clarify that no text-HOI pair data is used; but BEHAVE, as a pure HOI dataset, is used.

W2.6: object shape encoding

• We would like to clarify that the object shape is encoded implicitly in our approach. As detailed earlier in “W1: Full state”, the input to the world model includes the trajectories of human vertices (depicted as small red spheres in the top-right of Figure 2) along with vertex-to-object surface vectors. By adding the vertex-to-object surface vectors to human vertices, the object vertices (shown as small blue spheres in the top-right of Figure 2) can be inferred. This is why we describe the object geometry information as being implicitly encoded. The network of the world model does not receive this information directly, but it can learn to combine these features to derive the object geometry as needed.
• In addition, our insight is that providing partial information with locality and sparsity is more effective than using the complete object geometry encoding. By focusing on critical contact regions through our vertex-based representation, the world model can more accurately predict how object motion is influenced by interactions at these key vertices. This approach minimizes the risk of overfitting to specific object geometries. We welcome further discussion with the reviewer if there are any additional questions or if further clarification is needed.

W5: handling complex and long sequences

As mentioned in L236, the BEHAVE dataset operates at 30 Hz, and our world model, trained on this dataset, is designed to handle this framerate. However, our text-to-motion model runs at 20 Hz as it is trained on the HumanML3D dataset. To ensure compatibility, we use spherical linear interpolation (Slerp) after converting HumanML3D representation to SMPL and then calculate the vertices, allowing the vertex motion to align with the speed that the world model can handle.

Writing and Novelty

We appreciate the reviewer’s thoughtful engagement and will revise unclear descriptions based on the suggestions.

However, we respectfully disagree with the reviewer’s assessment of the novelty of our work. We have clearly outlined the novelty in our proposed components, such as the locality and sparsity design in the world model, which we believe is not presented in existing work before, and the decoupling of semantics from dynamics, which enables the entire pipeline to operate without requiring paired text-HOI data for training, as well as our overall motivation to effectively reuse and leverage knowledge from large models without extensive fine-tuning, which have been recognized positively by other reviewers, acknowledging our approach as insightful (Reviewer Jz5F), promising (Reviewer R7nv), and sound (Reviewer Kjsy). In addition, our evaluations demonstrate that using existing methods naively (e.g., the naive application of text-to-motion models as seen in Figure 6) is ineffective, whereas our synergistic approach provides a robust solution. We welcome the reviewer to give more insight or explanation regarding the perceived limited novelty and are willing to address any specific concerns or questions.

We thank the reviewer for the reply. We are glad that our response has clarified some of the reviewer’s concerns. Below, we address the remaining concerns:

Figure 6(b): We would like to clarify that Figure 6(b) is informative to show the distribution differences between raw text descriptions (“w/o planning”, denoted as green dots) and annotations processed through our high-level planning framework (“w/ planning”, denoted as red dots). Our aim is for the distribution of the red dots to be closer to the blue dots (which represent the in-distribution descriptions from the HumanML3D dataset) compared to the green dots. Upon zooming in on Figure 6(b) and examining the cluster in the middle, the reviewer will observe that the red dots largely overlap with the blue dots, while the green dots show minimal overlap with the blue cluster.

W5: handling complex and long sequences: We believe there is a misunderstanding from the reviewer. As clarified in the previous general rebuttal, “the capability for managing longer sequences arises from autoregressive generation, where the length of the sequence depends on the capacity of the text-to-motion model.” Also, as clarified in the previous rebuttal, the interpretation is introduced in this process to ensure compatibility between the world model (operating at 30 Hz) and text-to-motion model (operating at 20 Hz).

Novelty:

• Our innovations, including the locality and sparsity design in our world model and the decoupling of semantics from dynamics, enable our framework to operate without paired text-HOI data. Additionally, our overarching motivation is to effectively reuse and leverage knowledge from large models without extensive fine-tuning. We believe this approach represents an important paradigm for future research, extending beyond the HOI synthesis task. Other reviewers have recognized these contributions as insightful (Reviewer Jz5F), promising (Reviewer R7nv), and sound (Reviewer Kjsy).
• While using LLMs to reparse for text-to-motion can be seen as a form of preprocessing, no existing work leverages LLMs to address distribution shifts in text-to-motion as we do. We have discussed the detailed difference with the most relevant work [102] in the previous response.

Writing: We sincerely thank the reviewer for the invaluable comments on improving our writing and presentation. We will incorporate all the comments into the revision. Meanwhile, we also found that Reviewer Jz5F rates the presentation as 3 (good) and Reviewer Kjsy rates the presentation as 4 (excellent). We are fully committed to refining our submission to ensure the highest quality in both content and presentation, and we are confident that any writing issues can be effectively resolved during the revision process.

Again, we thank the reviewer for the constructive feedback throughout the review.

We thank the reviewer for thoughtful and insightful comments. We address your concerns below:

W1: full template of the query

• We appreciate the reviewer’s suggestion and have included our detailed query log in Fig. 1 of the rebuttal PDF file. We will discuss related work on this and add the log to the revision.

W2: Contribution formulation

• We believe that learning text-guided HOI generation from data without direct text supervision is a contribution to our work. We would like to mention that Reviewer Kjsy recognized the novelty of this task, noting that "The authors introduce a novel task of synthesizing whole-body interactions with dynamic objects guided by textual commands, without relying on text-interaction pair data." We appreciate the reviewer’s suggestions and apologize for any confusion. To clarify our statements, we will revise them as follows:
  • For Lines 71-72: We address the task of synthesizing whole-body interactions with dynamic objects guided by textual commands, achieving this without the need for paired text-interaction data—a novel approach to the best of our knowledge.
  • For Line 319: We focus on the task of text-guided 3D human-object interaction generation, aiming to accomplish this without relying on paired text-interaction data.

W3&Q3: evaluation on OMOMO

• Our method is designed to be dataset-agnostic. We chose the BEHAVE dataset for training the world model because it provides sufficient HOI dynamics to develop an effective dynamics model. We then tested our method on both the BEHAVE, as well as CHAIRS datasets for novel objects, which we believe is a sufficient evaluation.
• We appreciate the reviewer's suggestions. In response to it, we expanded our interaction planning and retrieval database to include the OMOMO dataset, similar to our approach with CHAIRS.
• Table 1 in the rebuttal PDF demonstrates that our interaction planning effectively bridges the distributional gap between the text-to-motion model and OMOMO text. Additionally, qualitative results of the full pipeline on OMOMO, provided in Fig. 2 of the rebuttal PDF, further illustrate the effectiveness of our entire pipeline on novel objects. We will incorporate more experiments into the revision.

Q1: two approaches to interaction retrieval

• For our primary experiments, we primarily utilized the handcrafted approach to implement the full pipeline, as it is straightforward and does not require training. To explore the feasibility of retrieval without relying on handcrafted rules, we also investigated a learning-based method for interaction retrieval. As shown in Figure C of the supplementary material, our qualitative evaluation demonstrates that the learning-based retrieval effectively captures diverse and realistic interactions, producing results comparable to the handcrafted method. Notably, the learning-based approach does not require constructing a database during inference, offering a more flexible solution.

Q2: text annotation release

• Yes, we plan to release the text annotations for the BEHAVE dataset upon acceptance.

Q4: typo revision

• We appreciate the reviewer for catching this typo. Yes, the final result should indeed be denoted as $a^*$. We will correct this in the revision.

We are grateful for the reviewer’s acknowledgment of our response and the task we’ve addressed. Regarding the new concern about visual quality, we agree that our generated results contain minor levels of artifacts. However, as noted by Reviewer Jz5F, “the proposed framework can generate realistic HOI motions that align with the input text conditions.”

It is important to highlight that many of the observed artifacts in fact originate from the dataset itself rather than our method. Specifically, the BEHAVE dataset lacks hand motion, and the text-to-motion models we employed do not account for hand dexterity. As a result, the hand from the average MANO pose may penetrate the object or appear to be floating above it.

Because of this inherent dataset issue, such artifacts are also present in existing works [1,2,3,4,5]. Note that our setting handles free-form text input without paired text-HOI data, which is inherently more challenging compared to the supervised methods used in these works [1,2,3,4,5] that rely on paired text-HOI data. Given these differences in setting difficulty, our achieved results, which exhibit comparable or even fewer artifacts, are particularly pronounced.

We provide examples below that demonstrate how similar artifacts are frequently observed in existing works, even in relatively simpler settings based on supervised training:

• [1] Floating and jittering are visible in the right example from 0:00-0:02, and penetration is noticeable at 2:18-2:23 in this video.
• [2] Penetration can be seen in the middle right of Figure 1 and the top right of Figure 6 in this paper, as well as in Figure 2 of the supplemental material.
• [3] Penetration can be found at 3:04-3:06 and 2:47-2:51 in this video.
• [4] Floating is evident in the first example, and penetration is present in the second example on this website.
• [5] Penetration can be seen in the top right of Figure 4, and floating is visible in the middle left of Figure 6 in this paper.

Note that the works referenced in [1,2,3] are published after the NeurIPS submission deadline (May 22, with CVPR being in June). References [4,5] are preprints available on arXiv.

Clarity: We are grateful to the reviewer for the suggestion on improving the manuscript’s clarity, as well as pointing out the confusion of contribution formulation in the review. We are confident that the writing improvements can be effectively managed in the revision process. We are fully committed to revising unclear descriptions and including any missing details based on all the reviewers’ suggestions to ensure the highest quality in writing and presentation.

We hope our clarifications have addressed the reviewer’s remaining concerns. Once again, we appreciate the reviewer’s engagement and thoughtful discussion.

[1] Diller et al. "Cg-hoi: Contact-guided 3d human-object interaction generation." CVPR 2024.

[2] Song et al. "HOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models." CVPR 2024.

[3] Li et al. "Controllable human-object interaction synthesis." ECCV 2024.

[4] Peng et al. "Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models." arXiv 2023.

[5] Wu et al. "Thor: Text to human-object interaction diffusion via relation intervention." arXiv 2024.

We thank the reviewer for thoughtful and insightful comments. We address the concerns below:

W1: How the world model acts in novel objects

• The world model employs "contact vertices" as input, which includes features derived from the object distance field. These features encompass the human vertex-to-object surface distance and the human vertex velocity relative to the nearest object vertex (L205-206), inherently including information related to the object's shape. This encoding is consistently applied to both training objects from the BEHAVE dataset and novel objects from the CHAIRS dataset.

• As discussed in (L64-68, L184-191), this vertex-based representation ensures that the world model concentrates on modeling the critical contact regions, and based on such contact modeling, the network learns to predict how object motion will be affected by these interactions. This approach prevents the model from overfitting to specific details, such as particular object shapes or body part motions. By focusing on high-level concepts—the principle that human-applied force on an object results in object acceleration, akin to Newton's law—the network can learn more generalizable patterns that apply across various contexts, e.g. different human actions and objects.

• Empirically, our approach is more effective than encoding the entire object shape to the world model. The improved performance is demonstrated by the results in Table 1 and Figure 8, comparing vertex control (our proposed) v.s. raw control (full geometry and full human motion). Figure 5 further highlights the model's ability to generalize to unseen objects from the CHAIRS dataset.

W3: World model for non-contacting objects

• How the network accepts the input without contact condition: Our network can process inputs without contact conditions by adopting an approach similar to ControlNet (Zhang et al., Adding conditional control to text-to-image diffusion models, ICCV 2023). The network comprises two components: $\mathcal{G}$ (L208) that operates without contact vertex conditions, applicable in scenarios where no contact occurs, and $\mathcal{F}$ (L210), akin to the control components in ControlNet, which incorporates contact vertex conditions into the object trajectory when contact is present. When there is no contact, only the unconditional network is utilized.

• Why the network can learn to respond to the input without contact conditions: The model is aware of past object motion and thus needs to learn how human interaction affects the object’s state. This includes understanding how objects follow contact positions or normals by $\mathcal{F}$ and how they move without contact by $\mathcal{G}$. With the no-contact object motion data provided by BEHAVE, the world model (more specifically, $\mathcal{G}$) learns to infer whether the object should free-fall based on its previous velocity or remain on the ground based on its height, as illustrated by the example at 02:27 in demo_2.mp4 (“a person throws a yoga ball towards the ground”) in the supplementary material.

W2: Efficiency of the autoregressive optimization and reason for long-term prediction

• As mentioned in the supplementary material (L693), optimization during autoregressive generation is performed selectively and only when the loss exceeds a certain threshold. In the Fig. 2 caption, we will clarify that while the overall framework is autoregressive, optimization is applied sparingly. This approach minimizes unnecessary computations, thereby maintaining computational efficiency.

• The rationale behind training the world model for longer sequences is for effectiveness in capturing temporal dependencies. As noted in L197 of the main paper, we reference "Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023,” which similarly found that predicting over a longer horizon promotes smoother and more coherent sequences for autoregressive generation, by planning actions ahead of time. However, longer predictions can increase the risk of error accumulation. To address this, we use only the early stage of long-horizon predictions, optimize them (while leaving some later parts unused), and use these optimized predictions in the next generation round. This strategy balances the advantages of long-term prediction with the need for accuracy and computational efficiency.

W4: Comparison with the baselines

• We would like to emphasize that our model is trained without direct text supervision, unlike the baselines that rely on it. Despite this, our model outperforms the baselines in text-interaction alignment to some extent, generating interactions that more accurately reflect the text instructions, even for long and complex descriptions, as demonstrated in the demo videos. Furthermore, our method achieves better realism to the baselines, with less penetration and jittering due to the optimization.

• To further enhance realism, we acknowledge that integrating a hand model trained on hand-specific datasets (as full-body datasets like BEHAVE do not include hand poses) could address some of the limitations observed in our results. We agree with the reviewer that this addition could improve the fidelity of human-object interactions, which we leave as interesting future work.

W5: Major bottlenecks presented in the current HOI generation

• As discussed in the limitations section of our paper, one of the major challenges in human-object interaction (HOI) understanding is achieving physics realism, which requires a more advanced dynamics model or incorporating simulation. Another significant bottleneck is the lack of detailed hand pose representation in conjunction with full-body interaction in many datasets. This limitation hinders the accuracy of modeling interactions that involve fine motor skills and detailed object manipulation. We welcome the opportunity to discuss these challenges further with the reviewer.

We greatly appreciate the reviewer’s thoughtful comments and acknowledgment of our work. We hope that our contributions, particularly the decoupling strategy that eliminates the need for paired text-HOI datasets in training and the integration of dynamics models with optimization, will inspire future research, potentially extending to tasks beyond HOI synthesis.

We are also encouraged that the reviewer recognizes our efforts to enhance the realism of HOI, especially given the challenges posed by object motion not being directly correlated with text, which complicates maintaining realistic interactions in the dynamics model. We believe that integrating a stronger dynamics model, such as a physics simulator, in future research would further improve realism.

Moreover, we would like to emphasize several aspects related to the synthesis realism:

• It is important to highlight that many of the observed artifacts in fact originate from the dataset itself rather than our method. Specifically, the BEHAVE dataset lacks hand motion, and the text-to-motion models we employed do not account for hand dexterity. As a result, the hand from the average MANO pose may penetrate the object or appear to be floating above it.
• Because of this inherent dataset issue, such artifacts are also present in existing works [1,2,3,4,5]. Note that our setting handles free-form text input without paired text-HOI data, which is inherently more challenging compared to the supervised methods used in these works [1,2,3,4,5] that rely on paired text-HOI data. Given these differences in setting difficulty, our achieved results, which exhibit comparable or even fewer artifacts, are particularly pronounced. Note that the works referenced in [1,2,3] are published after the NeurIPS submission deadline (May 22, with CVPR being in June). References [4,5] are preprints available on arXiv.

Once again, we appreciate the reviewer’s engagement and thoughtful discussion.

[1] Diller et al. "Cg-hoi: Contact-guided 3d human-object interaction generation." CVPR 2024.

[2] Song et al. "HOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models." CVPR 2024.

[3] Li et al. "Controllable human-object interaction synthesis." ECCV 2024.

[4] Peng et al. "Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models." arXiv 2023.

[5] Wu et al. "Thor: Text to human-object interaction diffusion via relation intervention." arXiv 2024.