HOComp: Interaction-Aware Human-Object Composition

We thank this reviewer for the positive evaluation and appreciate the recognition in the paper’s clarity and the motivation for the customization of the human-object interaction task. We address each of the questions below. (Abbreviations used: Q/W refer to Question/Weakness.)

---

W1: Support for Multi-object and Multi-person Composition

To evaluate our model’s ability to handle multi-object and multi-person compositions, we have constructed a dedicated test set, since the current HOIBench does not include such cases. Our new benchmark covers two scenarios: (1) two persons with two objects and (2) three persons with three objects.

Test Set Construction:
We first collected 40 images from the internet containing multiple persons: 20 images with two persons, and 20 images with three persons. These images were selected to ensure diversity in human appearance, pose, and clothing.

For Two Persons – Two Objects: To broadly generate interaction-object pairs, we have adopted the 117 interaction types defined in HICO-DET. For each interaction type, we prompt GPT-4o to infer a plausible foreground object (e.g., “playing”→“guitar”). A concise textual description is then used to retrieve a representative object image from the internet, resulting in 117 interaction–object pairs. For the two-person scenario, each person in an image is randomly assigned a unique interaction–object pair from this pool, ensuring a wide variety of combinations. For each of the 20 two-person images, we repeat this assignment process 10 times with different pairs, resulting in 200 two-person, two-object test cases.

For Three Persons – Three Objects: Similarly, for three-person images, each person is assigned a unique interaction–object pair, randomly sampled from the 117 pairs described above. This process is repeated for each three-person image, resulting in 200 three-person, three-object cases.

Test Results:
We have evaluated our model using the new test set of 400 cases (200/scenario) and compared its performance with the top three methods from the main paper. We compute the HOI-Score and DINO-Score for each individual person-object interaction pair, and then take the average of all these scores across all interaction pairs.

As shown in the following table, our approach achieves the best results in both scenarios and on all metrics.

---

W2: The Impact of the Area of Interaction Region

We evaluated HOComp's performance on cases with exceptionally large interaction regions, such as those illustrated in Fig.2 (Supplementary) (e.g., "a man is playing a guitar," where the interaction covers over 70% of the image). We have identified 82 images in HOIBench where the interaction region occupies more than 70% of the total image area, and compared HOComp with previous methods on this subset. As shown in the following table, HOComp achieves substantially better results than other baselines across all metrics in large interaction region cases.

---

Q1 & Q2: MLLMs' Influence on Inference Performance and Comparison with Vision-Based Methods to Detect Interaction Regions

At MPRG's coarse level, it leverages MLLMs to automatically identify suitable interaction types/regions via multi-stage querying. In some rare cases, MLLMs may fail to accurately predict the interaction regions. We provide a detailed analysis of these failure modes and potential solutions below.

In our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in 100% of samples. The corresponding interaction region was predicted correctly in 91.33% of cases (548/600). For the remaining 8.67% (52/600), two main failure modes are observed:

1. Mismatch Between Interaction Regions and Types (6% of cases): When multiple plausible interaction types exist, MLLMs may misassign interaction regions. For example, if the object is sunglasses and the interaction type is "hold", the model may incorrectly assign the region around the eyes (i.e., "wear" action). While the generated image depicts a person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.
2. Incorrect Interaction Region Size (2.67% of cases): In these cases, the predicted interaction region does not fully cover the area, including the object and the human body part, for modification. As shown in Fig.6 (main paper), this can hinder generation of correct interactions.

To address these two issues, we explored the following solutions:

• Additional Input Conditions: To alleviate the mismatching issue, we can further incorporate human pose priors (from a pose estimator) as an additional prior to MLLMs. For GPT-4o, the region prediction accuracy increased from 91.33% to 96.5%. This improvement can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.
• Training with Noisy Data: To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training.
  We first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of a “soccer ball” with a “kick” action to cover the hand, which may correspond to a “hold” action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following tables, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.

Qualitative results and further experimental details will be included in the Supplemental.

Comparison using MLLM with other vision-based methods to detect interaction regions: Vision-based methods face significant challenges in this task, as they not only need to localize humans and objects but also infer plausible interaction types and predict specific body parts/regions involved in the interaction.

To the best of our knowledge, existing vision-based approaches [1-3] are limited: they typically operate on images where the interaction already exists (e.g., our composited results), and can only detect existing relationships or involved regions. They generally lack the ability to perform open-ended reasoning to predict plausible interaction types/regions for images where no interaction has yet occurred, given only an isolated person and a reference object.

We have also compared the performance of different MLLMs, as shown in the following table. The accuracy of interaction type/region predictions is computed through a user study involving 20 participants, who evaluate the correctness of MLLM predictions. The correct predictions are counted, and the average accuracy per participant is computed.

---

References

[1] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, ECCV'24
[2] Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models, NeurIPS'24.
[3] OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling, NeurIPS'24.

We thank this reviewer for the positive review and useful comments. We appreciate your recognition of our IHOC dataset. We address each of the questions below. (Abbreviations used: Q/W/L refer to Question/Weakness/Limitation.)

---

W1: Physical Constraints in Our Model

We would like to clarify that our approach utilizes pose-guided supervision to enforce physical constraints in the generated images. Specifically, during the training of HOComp, we impose human poses constraints within the interaction regions, using pose keypoints as explicit supervision. This helps the model learn physical priors that ensure realistic interactions between the foreground object and the background person.

While it is possible to employ reconstruction methods[2] to generate 3D hand meshes and apply stronger physical constraints on specific body parts (e.g., hands), our model is not limited to hand interactions alone. As shown in Fig.3(c) (Supplemental), samples involving hand interactions alone comprise 54.3% of our training data, with other body parts (e.g., legs) also playing a significant role in the interactions in the remaining samples.

If we were to impose 3D constraints on the entire body, it would have introduced additional modeling errors and substantially increased computational complexity during training. Instead, we use DWPose[1], a robust pose estimator to capture landmarks across the entire body. DWPose predicts multiple hand keypoints and key points for major joints (e.g., each finger contains four keypoints from fingertip to base, and each joint has one keypoint). This approach allows our model to accurately capture human body movements across a variety of interaction types while maintaining computational efficiency.

---

W2&L1: Lack of Discussion and Comparison of Recent Related Works

We thank the reviewer for pointing out relevant works on human-object interaction generation. We would like to clarify that the methods mentioned (CG-HOI, HOIAnimator, InterFusion, Controllable HOI Synthesis, and F-HOI) primarily focus on 3D human-object interaction generation, which aims to generate physically plausible 3D human-object interactions. These methods typically generate 3D models or sequences based on the interaction description, differing significantly from our approach, which focuses on generating 2D composited images.

Our work addresses a distinct task of generating 2D composited images with not only harmonious human-object interactions, but also consistent foreground object and the background person appearances. Existing methods fall short in addressing our challenges. Although human-object interaction image generation methods can generate images with good human-object interactions, which is discussed in Section2 (main paper). One key distinction is that most of the referenced methods do not specify particular reference humans and objects (except for DreamRelation[3] and PersonaHOI[4]). Further, these methods do not emphasize the need to maintain consistent appearances of both humans and objects, in addition to generating harmonious interactions, which is a core aspect of our approach. Because PersonaHOI has not made their official code available for direct comparison, we have included comparisons with DreamRelation in Tab.8 and Fig.10 of the Supplemental.

We will incorporate the above discussions in our revision.

---

W3: Incorporate a More Sophisticated Metric to Assess the Physical Plausibility of the Generated Motions

As suggested, we have thoroughly examined the six metrics, MPJPE and MPVPE (for hand pose/shape reconstruction accuracy), IV and PD (for physical plausibility and penetration degree), Contact IoU (for contact area accuracy), and Plausible Rate (for overall plausibility and manipulation credibility), used in ``Physics-aware Hand-object Interaction Denoising'' (CVPR'24).

However, all of these metrics require the generated 3D mesh for computation. Except for "Intersection Volume'', the other metrics additionally require access to the corresponding GT 3D mesh for evaluation. Our method produces only 2D composited images as output and does not include corresponding 3D GT during inference. While it is theoretically possible to reconstruct 3D meshes from 2D images and then compute Intersection Volume without the need for 3D GT, such reconstruction can introduce significant errors, especially given the diverse interaction types and occlusions present in our task. Nevertheless, we will explore if there are more appropriate metrics for such an evaluation.

---

Q1: Difference from Prior Human-object Interaction Datasets

The key distinctions and advantages of our dataset compared to BEHAVE and OMOMO are as follows:

Task and Modality: Our dataset is the first large-scale resource specifically tailored for the interaction-aware human-object composition task, providing paired pre-/post-interaction image data with comprehensive annotations. In contrast, BEHAVE and OMOMO focus primarily on 3D human-object interaction generation, providing 3D meshes of humans and objects. While BEHAVE contains some RGB images during interaction, it lacks the crucial pre-interaction data required for modeling before/after composition.

Scale and Diversity: Our dataset covers a significantly broader range of human-object interactions, actions, and scenes. Compared to BEHAVE (8 subjects, 20 objects) and OMOMO (17 subjects, 15 objects), our dataset includes 11,700 unique samples, each with a different human subject. We provide 117 types of interactions and 342 distinct foreground object categories, encompassing diverse human poses, viewpoints, both indoor and outdoor environments, and a wide variety of object categories, sizes, and interaction body parts. In addition, our dataset features a broad spectrum of image styles. Comprehensive statistics and visualizations can be found in Fig.3 (Supplemental).

In summary, our dataset uniquely enables research on image-based, interaction-aware human-object composition with a level of scale, diversity, and paired pre/post-interaction coverage not available in prior datasets. We will add this discussion in our revision.

---

Q2: "Is affordance estimation explicitly required in the proposed method?"

Our method does not explicitly involve affordance estimation. Instead, we rely on the MLLM to infer the appropriate human-object interaction based on the visual input of a background human image and a foreground object image. The MLLM performs reasoning on these inputs to predict the interaction type, without the need for explicit affordance estimation. In our HOIBench evaluation, MLLMs (e.g., GPT-4o) correctly predicted plausible interaction types in 100% of the tested cases.

---

L2: Generalization to Interactions with Out-of-Distribution Objects

Our method performs well on objects not included in the BEHAVE or OMOMO datasets. For example, all examples shown in Fig.1 in the main paper, as well as those in Fig.4 (rows 1, 2, and 4), involve object categories that are absent from both BEHAVE and OMOMO. Nevertheless, our method is able to generate satisfactory results on these cases.

We would like to clarify that, when constructing our dataset, we did not select object types based on those present in BEHAVE or OMOMO. Specifically, BEHAVE contains only 20 objects and OMOMO contains 15 objects. In contrast, the object types in our training set were determined using an MLLM, which inferred potential object categories according to 117 interaction types (such as ''hold'', ''play''). These pairings (e.g., ''hold-bag'', ''play-guitar'') were then manually verified for plausibility, resulting in a total of 342 distinct object categories in our dataset.

In our HOIBench evaluation, we cover all 117 interaction types and 108 object categories. While some categories may overlap with those in the training set, we ensured that both the human subjects and the object images in the test set are completely disjoint from those in the training set. We will provide more visual results in the supplementary materials.

---

Ethical Concerns on Data Privacy, Copyright, and Consent

All images in our training set are sourced from the public HICO-DET dataset, images generated by the T2I model FLUX.1 [dev], or openly licensed free image websites such as Unsplash and Pexels. The HOIBench also exclusively uses images from freely available public websites. Therefore, there are no ethical issues for our dataset. We will release our dataset upon acceptance of the paper.

---

References

[1] Effective Whole-body Pose Estimation with Two-stages Distillation, ICCV'23.
[2] Reconstructing Hands in 3D with Transformers, CVPR'24.
[3] Dreamrelation: Bridging customization and relation generation, CVPR'25.
[4] PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation, CVPR'25.

We thank this reviewer for the positive review and constructive comments. We appreciate the recognition of paper organization and clarity, as well as the sufficiency of our qualitative/quantitative results. We address each of the questions below. (Abbreviations used: Q/W refer to Question/Weakness.)

---

Q&W: Concerns about the Complexity of the HOComp Framework.

Although our framework contains multiple components, each component of HOComp fulfills a distinct, essential function: the framework builds on latest diffusion transformers, integrates MLLMs to enforce coarse-level interaction constraints and reduce user input, incorporates a pose estimator during training to optimize human pose generation for harmonious interactions, and employs a multi-view generator to ensure appearance consistency and semantic alignment across viewpoints.

---

Q&W: Generalizability of HOComp.

To address the generalizability concern of our model, we have evaluated HOComp under challenging and diverse conditions.

We first validated our method's performance under three extreme or special cases:
(1) multi-subject and multi-person compositions,
(2) extremely large interaction regions, and
(3) interactions involving completely occluded target objects.

---

1. Multi-subject and multi-person compositions:

To evaluate our model’s ability to handle multi-object and multi-person compositions, we have constructed a dedicated test set, since the current HOIBench does not include such cases. Our new benchmark covers two scenarios: (1) two persons with two objects and (2) three persons with three objects.

Test Set Construction:
We have collected 40 diverse images (20 with two persons, 20 with three persons) from the internet, and generated 117 interaction–foreground image pairs following the same procedure of constructing HOIBench described in Section 4 (main paper).

For Two persons, two objects: Each person in an image is randomly assigned a unique interaction–object pair, ensuring a wide variety of combinations. For each of the 20 two-person images, we repeat this assignment process 10 times with different pairs, resulting in 200 two-person, two-object test cases.

For Three persons, three objects: Similarly, each person is assigned a unique interaction–object pair, randomly sampled from the 117 pairs described above. This process is repeated for each three-person image, resulting in 200 three-person, three-object cases.

We have evaluated our model using the new test set of 400 cases (200/scenario) and compared its performance with the top three methods from the main paper. We compute the HOI-Score and DINO-Score for each individual person-object interaction pair, and then take the average of all these scores across all interaction pairs. As shown in the following table, our approach achieves the best results in both scenarios and on all metrics.

---

2. Extremely large interaction regions:

We have also evaluated HOComp's performance on cases with exceptionally large interaction regions, such as those illustrated in Fig.2 (Supplementary) (e.g., "a man is playing a guitar," where the interaction covers over 70% of the image). We have identified 82 images in HOIBench where the interaction region occupies more than 70% of the total image area, and compared HOComp with previous methods on this subset. As shown in the following table, HOComp achieves substantially better results than other baselines across all metrics in large interaction region cases.

---

3. Interactions involving completely occluded target objects:

Some actions may cause the interacting object to become entirely occluded in the resulting image, e.g., food being eaten (fully inside the mouth) or an object blocked by a hand. To evaluate the model’s generalization in such cases, we have curated 50 test instances where the target object is fully occluded after the interaction. We append “the object is occluded because of the human action” to the MLLM-generated descriptions, forming a dedicated evaluation subset.

Standard metrics requiring object visibility (e.g., DINO-Score, HOI-Score) are inapplicable here. Instead, we manually assess the plausibility of the depicted interactions (“Occlusion Accuracy”), focusing on whether the generated images realistically reflect the intended occlusion. The results in the table below clearly demonstrate the superiority of our method over existing works.

We will include these qualitative results in our paper.

---

Extension to the User Study:
To further demonstrate the generalization of our method to diverse inputs, we have expanded the user study in the main paper from 10 to 50 input cases, covering a broader range of human poses, viewpoints, and object categories. 45 participants have completed the evaluation. As shown below, our method consistently outperforms all baselines across all metrics, indicating strong user preferences for our approach.

---

Our framework can also be adapted to different foundation models.
In our original experiments, we used FLUX.1[dev] as the base DiT model. We additionally evaluated HOComp with FLUX.1 Kontext[dev] [1] as the base model. As shown below, HOComp attains superior performance on FLUX.1 Kontext[dev], further demonstrating the adaptability of our approach across base model variants.

---

Q&W: Reproducibility of our HOComp.

We will release all codes, the IHOC dataset, and HOIBench upon acceptance, to facilitate reproducibility.

---

References

[1] FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space, arXiv 2506.15742.

We thank this reviewer for the positive review and insightful questions. We appreciate the recognition of the novelty of our MRPG module, the value of the IHOC dataset and HOIBench benchmark, and the appreciation of our experimental results. We address each of the questions below. (Abbreviations used: Q/W refer to Question/Weakness.)

---

Q1&W1: Concerns Regarding MRPG's Reliance on MLLMs' Predictions

To address this concern, we provide a detailed analysis of the failure modes and potential solutions.

In our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in 100% of samples. The corresponding interaction region was predicted correctly in 91.33% of cases (548/600). For the remaining 8.67% (52/600), two main failure modes are observed:

1. Mismatch Between Interaction Regions and Types (6% of cases):
   When multiple plausible interaction types exist, MLLMs may misassign interaction regions. For example, if the object is sunglasses and the interaction type is "hold", the model may incorrectly assign the region around the eyes (i.e., "wear" action). While the generated image depicts a person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.

2. Incorrect Interaction Region Size (2.67% of cases):
   In these cases, the predicted interaction region does not fully cover the area, including the object and the human body part, for modification. As shown in Fig.6 (main paper), this can hinder generation of correct interactions.

We have explored the following potential solutions to mitigate these two issues:

• Additional Input Conditions:
  To alleviate the mismatching issue, we can further incorporate human pose priors (from a pose estimator) as an additional prior to MLLMs.

  For GPT-4o, the region prediction accuracy increased from 91.33% to 96.5%. This improvement can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.

• Training with Noisy Data:
  To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training.

  We first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of a "soccer ball" with a "kick" action to cover the hand, which may correspond to a "hold" action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following table, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.

Qualitative results and further experimental details will be included in the Supplemental.

---

W2: Concerns on Framework Complexity and Reproducibility

Although our framework comprises multiple components, each component of HOComp fulfills a distinct, essential function: Our framework builds on state-of-the-art diffusion transformers, introduces MLLMs for coarse-level interaction constraints, and reduces user input requirements. The pose estimator is incorporated during training to optimize the generated human poses, enabling more harmonious interactions. The multi-view generator ensures appearance consistency and semantic alignment across viewpoints.

We will release all codes, the IHOC dataset, and HOIBench upon acceptance, to facilitate reproducibility.

---

Q2&W3: Potential Data Bias in the IHOC Dataset

We understand and appreciate the reviewer’s concern about potential bias introduced by the use of inpainting models to generate pre-interaction images in IHOC dataset.

First, our IHOC dataset is a mixture of real-world data collected from the internet and AIGC-generated content. During training, inpainting is employed to generate pre-interaction images, similar to previous methods [1,2]. In contrast, the ground-truth post-interaction images depict natural human-object interactions sourced from both real-world images and AIGC-generated interaction images.

Second, the vast majority of input images used in our evaluation are real-world photographs, not inpainted data (e.g., rows 1–3 in Fig.1 (main paper), rows 2–3 in Fig.4 (main paper), and rows 1–3 in Fig.9 (supplementary)). The strong performance of our model on these real images demonstrates that it does not rely on or reproduce inpainting-specific biases.

Finally, to further alleviate this concern, we will provide additional results on more real-world test cases, and in future work, we plan to expand our dataset by extracting pre- and post-interaction pairs directly from video footage to ensure even greater realism and diversity.

---

References

1. Learning Object Placement by Inpainting for Compositional Data Augmentation. ECCV'20.
2. ObjectMate: A Recurrence Prior for Object Insertion and Subject‑Driven Generation, ICCV'25.

We thank this reviewer for valuable comments. We appreciate the recognition of our IHOC dataset. We address the main concerns and questions below. (Abbreviations used: Q/W/L refer to Question/Weakness/Limitation.)

Q1&W1: ''Ordinary and unremarkable writing''

We would like to briefly summarize the logical structure of our paper:

• Background and Motivation: We start with the practical need for compositing product images with humans in advertising scenarios (e.g., perfume ads), highlighting two essential objectives: (a) achieving natural and plausible human-object interactions, and (b) maintaining strict appearance consistency for both the person and the object.
• Task Definition: We define the problem as interaction-aware human-object composition and explicitly state the main challenges: existing approaches often fail to deliver either natural interactions (such as human pose and gesture) or visual consistency (identity and appearance preservation).
• Technical Contributions: We introduce HOComp to address these challenges, proposing two novel modules:
  • MLLMs-driven region-based pose guidance (MRPG): A coarse-to-fine mechanism for constraining and guiding human-object interactions.
  • Detail-consistent appearance preservation (DCAP): A method to preserve both the details and identities of the foreground object and the background person.
• Dataset Contribution: To enable training, we introduce the IHOC dataset, the first large-scale dataset specifically designed for interaction-aware human-object composition.
• Experiments: We conduct comprehensive experiments on HOIBench. Results demonstrate the advantages of HOComp in generating realistic interactions and visually consistent human-object compositions.

We would be happy to refine our presentation if this reviewer can provide more specific information of which part (e.g., section/discussion) that we may improve on.

---

Q2&W2: ''The novelty level is not too high''

Our work has three main categories of contribution.

Idea Contribution. In this paper, we introduce HOComp, the first interaction-aware human-object composition approach to seamlessly integrate a foreground object into a human-centric background while ensuring harmonious interactions and maintaining consistent appearance between the foreground object and the background person. Our method can be applied in areas such as automated advertising generation and digital content creation.

Compared to existing methods, HOComp addresses the following shortcomings:
(1) Failure in natural human-object interactions: Many image-guided composition methods fail to generate realistic human gestures/poses during human-object interaction, resulting in unnatural outcomes.
(2) Inconsistent appearance of humans or objects: Existing multi-concept customization or HOI-based image generation methods regenerate both the foreground object and the background human, leading to inconsistencies in the background human's appearance.

Technical Contributions. HOComp introduces two key technical contributions:

1. MLLMs-driven Region-based Pose Guidance (MRPG), which utilizes MLLMs to identify the interaction region and interaction type (e.g., holding and lifting) to provide coarse-to-fine constraints to the generated pose for the interaction while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints.
2. Detail-consistent Appearance Preservation (DCAP), which unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the foreground and faithful reproduction of the background human.

Dataset/Benchmark Contributions.

1. To facilitate research in interaction-aware human-object composition, we introduce the IHOC dataset, specifically designed for this task. It is the first dataset to contain paired pre-/post-interaction data, which is crucial for modeling realistic and coherent human-object compositions.
2. We also present HOIBench, a benchmark for comprehensive evaluation of human-object composition methods.

If this reviewer can provide more specific information regarding this novelty issue, we would be happy to further discuss.

---

W3&L1: ''Too few Bad Cases'' and More Analysis

We appreciate the reviewer’s suggestion to provide more failure cases and more analysis.

Our MRPG module adopts a coarse-to-fine strategy for constraining human-object interactions. At the coarse level, it leverages MLLMs to automatically identify suitable interaction types/regions via multi-stage querying. In some rare cases, MLLMs may fail to accurately predict the interaction regions. We provide a detailed analysis of these failure modes and potential solutions below.

In our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in 100% of samples. The corresponding interaction region was predicted correctly in 91.33% of cases (548/600). For the remaining 8.67% (52/600), two main failure modes are observed:

1. Mismatch Between Interaction Regions and Types (6% of cases): When multiple plausible interaction types exist, MLLMs may misassign interaction regions. For example, if the object is sunglasses and the interaction type is ''hold'', the model may incorrectly assign the region around the eyes (i.e., ''wear'' action). While the generated image depicts a person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.

2. Incorrect Interaction Region Size (2.67% of cases): In these cases, the predicted interaction region does not fully cover the area, including the object and the human body part, for modification. As shown in Fig.6 (main paper), this can hinder generation of correct interactions.

To address these two issues, we explored the following solutions:

• Additional Input Conditions: To alleviate the mismatching issue, we can further incorporate human pose priors (from a pose estimator) as an additional prior to MLLMs.
  For GPT-4o, the region prediction accuracy increased from 91.33% to 96.5%. This improvement can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.
• Training with Noisy Data: To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training.
  We first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of a ''soccer ball'' with a ''kick'' action to cover the hand, which may correspond to a ''hold'' action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following tables, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.

Qualitative results and further experimental details will be included in the supplemental materials.