Dear reviewer oDfg,

First and foremost, we would like to express our sincere gratitude for your detailed review and valuable feedback. We truly appreciate your recognition of our method, e.g., "novel method", "addresse a significant problem in the HOI task" and "innovative and important for the community".

Below, we provide a comprehensive response to each of your concerns (please note: the metrics for HICO-DET in the table below are reported under the "default full setting," while those for V-COCO are AP_{role}^{\\#1}; the references maintain their original numbering from the main paper):

---

Q1: Text embeddings model.

A1: Following prior work, we employ CLIP’s text encoder to embed both the generated text and the textual labels, and then compute their similarity for interaction prediction.

---

Q2: Necessity of Visual Interaction Decoder.

A2: We conduct an ablation study by using only the LLM’s outputs (S2) for interaction prediction on the HICO-DET. As shown in the table below, relying solely on S2 results in a 5.09 mAP drop in the default full setting compared to the combined "S1 + S2" approach (where S1 represents interaction scores generated by the Visual Interaction Decoder). These experimental results underscore the critical role of the Visual Interaction Decoder: by fusing the visual interaction encoder (S1) with MLLM reasoning (S2), InstructHOI could effectively preserves the generalization ability of the MLLM and mitigate hallucinations, leading to superior overall performance.

---

Q3: Directly use the predicted bounding boxes to filter the image token.

A3: We conducted an experiment in which we directly used the predicted bounding boxes to filter the image tokens (region-overlap strategy). As shown in the table below, this region-overlap strategy results in a 1.22 mAP performance drop compared to the Interest Token Selector (ITS). Directly using the detected HOI region as the interaction region may neglect discriminative features and introduce irrelevant information.

Although the connection between the visual features extracted from the object detector and the LFM is implicit, the instruction projector (Sec. 3.3) could aligns these two visual feature spaces via feature mapping. ITS evaluates the interaction relevance of each local patch for each human-object pair, thereby more effectively selecting the interaction-related region.

---

Q4: Removal of Data from Related Categories.

A4: To ensure a fair zero-shot evaluation, we excluded all training data related to the held-out HOIs from the fine-tuning dataset, thereby preventing any potential data leakage. For instance, when "talk on a cell phone" (in HICO-DET) is considered an unseen HOI, we also exclude related HOI categories from the fine-tuning dataset, e.g., "phone" in SWIG-HOI and "talk on phone" in V-COCO.

---

We truly value your insightful and constructive suggestions, and we hope our response could addresse your concerns. If you have any further questions or feedback, please do not hesitate to share them with us.

In the revised version of the paper, we will include a clearer explanation of the text embedding model, ablation experiments on the Visual Interaction Decoder, experiments on local image token filtering with detected HOI regions, and a more detailed explanation of data removal.

Dear reviewer jzDg,

We gratefully acknowledge your comprehensive review, insightful critiques, and affirming evaluation of our approach, e.g., "well-motivated", "reasonable" and "easy to follow".

Below, we provide a detailed response to each of the points you raised (note: the metrics for HICO-DET in the following table are reported under the "default full setting", while those for V-COCO are AP_{role}^{\\#1}; the references maintain their original numbering from the main paper):

---

Q1: Concern about additional training data or a stronger multi-modal LFM (InternVL2).

A1: For finetuning data, we have reported the results of ablation study in the supplementary materials (Table 2). As shown in the following table, the“pretrained" indicates pretrained LFM without fine-tuning; “fine-tune$^{†}$" indicates fine-tuning on a specific training dataset (i.e., HICO-DET or V-COCO); “fine-tune$^{‡}$" indicates fine-tuning on the collected dataset. Notably, even when fine-tuned exclusively on HICO-DET or V-COCO (fine-tune$^{†}$), our model still achieves SOTA performance. Moreover, the “fine-tune$^{‡}$” strategy yields an additional gain of 0.59 mAP on the HICO-DET benchmark and 0.40 mAP on the V-COCO, demonstrating that finetuning on the larger dataset could further enhance InstructHOI's generalization ability.

For LFM choice, we have reported the results of ablation study on LFMs in the supplementary materials (Table 1). As shown in the following table, we explore variants of InstructHOI that leverage different LFMs, including LLaVA-OV and InternVL2. Furthermore, for each LFM variant, we assess the impact of different foundation model scales. The experimental results demonstrate that InstructHOI achieves SOTA performance with both LLaVA-OV and InternVL2. InternVL2 demonstrates slightly better performance, which could be attributed to its strong foundational and visual perception capabilities. Additionally, as the scale of the underlying LFM increases, InstructHOI’s performance further improves, highlighting the enhanced reasoning capabilities at larger model scales.

---

Q2: The necessity of Spatial Context in CIG.

A2: The primary objective of integrating an offline object detector is not to extract Spatial Context (SC), but to achieve precise instance localization. Consistent with prior work [21], [58] demonstrating the importance of spatial relationships for interaction prediction, the experimental results in Table 4 (in the main paper) show that the introduction of SC token improves performance on the HICO-DET by 0.78 mAP (1.8% relative improvement). Moreover, when the LFM model is scaled up to 7B parameters (InternVL2-7b), the gains of SC further increase to 1.42mAP (3.1% relative improvement) on the HICO-DET. LFMs with greater foundational capacity demonstrate an enhanced ability to comprehend SC token and leverage it for interaction reasoning.

---

Q3: Clarify that the Backbone in Table 1.

A3: For VLM-based HOI detectors, the term "backbone" (e.g., R101+ViT-L) refers to the combination of the basic feature extractor (e.g., R101 for object detection) and the visual encoder backbone of the large foundation model (e.g., ViT-L). Additionally, to provide a clearer perspective, we also add the specific Large Foundation Model (LFMs) of different methods in Table 1, as following:

---

Q4: Inference and training cost.

A4: We evaluated the inference time of our model in comparison to high-performance HOI detectors, utilizing Tesla A800 GPU (or GPU with comparable performance). The results are presented in the table below, compared to existing methods, InstructHOI demonstrates notable performance improvements while maintaining comparable computational costs.

We evaluated the training time of our model in comparison to high-performance HOI detectors, utilizing 8 Tesla A800 GPUs (or GPUs with comparable performance). The results are presented in the table below, specifically, the 7.2 hour training time refers to the HOI-SFT time for the MLLMs, while the 11.1 hours represents the second-stage training time for the full model’s component.

---

We greatly appreciate your insightful and constructive feedback, and we hope our response could addresses your concerns. If you have any further questions, please don’t hesitate to share them with us.

In the revised version of the paper. we will provide a clearer explanation of the backbone component and the role of the spatial context token, as well as include more details on the inference and training time.

Dear reviewer hkno,

First and foremost, we extend our deepest gratitude for your thorough review and insightful feedback, as well as your positive recognition of our method, e.g., "clear motivation", "novel architecture design" and "SoTA performance".

Herein, we provide a detailed response to each of your concerns (note: the metrics for HICO-DET in the following table are reported under the "default full setting", while those for V-COCO are AP_{role}^{\\#1}; the references maintain their original numbering from the main paper):

---

Q1: Efficiency metrics in Table 1.

A1: Following prior work, we add the most commonly used metrics ("Large Foundation Model" (LFM), "Training Data" (TD), "Full Parameters" (FP), "Trainable Parameters" (TP), "Inference Time" (IT), and "Training Time" (TT)) in Table 1, to provide a clearer efficiency comparison, as shown below. All measurements are obtained using Tesla A800 GPUs (or GPUs with comparable performance).

Compared to existing methods, InstructHOI demonstrates notable performance improvements while maintaining comparable model parameters (1.13B full parameters and 42.3M trainable parameters) and computational costs (267ms inference time and a total training time of 18.3 hours). Specifically, the 7.2 hour training time refers to the HOI-SFT time for the MLLMs, while the 11.1 hours represents the second-stage training time for the full model’s component.

---

Q2: Lack of visualizations.

A2: Thank you for pointing this out. Due to rebuttal guidelines, we are unable to upload images here. As for Interest Token Selector (ITS) module, we ensure that additional visualizations will be introduced in the revised version to more clearly validate the effectiveness. Other visualizations—covering Interaction Prediction in Two Branches, Failure Case Analysis, and In-the-Wild HOI Detection—have been provided in the supplementary materials.

Additionally, to quantitatively validate the effectiveness of ITS module, we have conducted an ablation experiment in Table 3 (main paper). The results demonstrate a 0.97 mAP increase on HICO-DET (a relative gain of 2.2 %) and a 0.6 mAP increase on V-COCO (a relative gain of 0.9 %).

---

Q3: Directly use LoRA-finetuned MLLM for interaction prediction.

A3: We introduce an additional baseline in which the LoRA-finetuned MLLM is directly employed for HOI interaction recognition, referred to as Multi-Modal Reasoning (MMR). As shown in the table below, MMR alone results in a 5.09 mAP drop on the HICO-DET compared to InstructHOI. The standalone MMR may be prone to hallucinations in complex and ambiguous scenarios. In contrast, integrating the visual interaction decoder (S1) with MMR (S2) for HOI detection effectively maintains the generalization capability of the MLLM while mitigating hallucinations, leading to a significant improvement in overall performance.

---

We sincerely appreciate your valuable and constructive suggestions, and we hope that our response could addresses your concerns. Please feel free to provide any further feedback if you have additional questions.

In addition, we will incorporate your suggested revisions into the future version of the paper, including: adding efficiency metrics in Table 1, providing visual results for ITS, and including additional baseline experiments on directly using LoRA-finetuned MLLMs for interaction prediction. These revisions will significantly enhance the quality of the paper, allowing readers to gain a more comprehensive understanding and evaluation of our work.

Dear reviewer cgtz,

First and foremost, we sincerely appreciate your thorough review and valuable feedback. Your recognition of our method is greatly appreciated, e.g., "well-written", "reasonable design" and "ablation experiments given the multi-stage nature of the method".

Below, we provide detailed responses to each of your questions (note: the metrics for HICO-DET in the following table are reported under the "default full setting", while those for V-COCO are AP_{role}^{\\#1}; the references maintain their original numbering from the main paper):

---

Q1: Full parameter size comparison in Table 1.

A1: For a clearer comparison, we add the "Large Foundation Model” (LFM) and the “Full Parameters”(FP) and “Trainable Parameters” (TP) in Table 1, as shown below. Compared to existing methods, InstructHOI achieves substantial performance gains with comparable number of trainable parameters (42.3M) and full parameters (1.13B).

Thank you for your valuable suggestions, we will revise Table 1 in future versions for clearer comparison.

---

Q2: End-to-end prediction using VLM models.

A2: We develope an end-to-end HOI detector using Qwen2.5VL-7B for comparison. As shown in the table below, the pretrained Qwen2.5VL-7B achieves a 24.72 mAP on HICO-DET (with the default full setting), while the fine-tuned Qwen2.5VL-7B-SFT (fine-tuned on the HOI dataset) improves this performance to 32.64 mAP. However, both variants are significantly outperformed by InstructHOI (only equipped with a 1B-parameter MLLM). This performance gap can primarily be attributed to the limited grounding precision and reduced interaction-prediction capabilities of MLLMs models within ambiguous and complex scenarios.

---

Q3: Select the local split with the detected HOI regions.

A3: We conduct an experiment in which we select the local split that overlaps the most with the detected HOI regions (Overlap-based Strategy). As shown in the table below, this overlap-based strategy results in a 1.22 mAP performance drop compared to the Interest Token Selector (ITS). Directly using the detected HOI region as the interaction region may neglect discriminative features and introduce irrelevant information. In contrast, ITS evaluates the interaction relevance of local images for each human-object pair based on reasoning instructions, allowing it to more effectively select the region most closely associated with the interaction.

---

Q4: The shape of $T_l$.

A4: In Eq. (4), $T_l$ denotes the discretized local image token sequence generated with the dynamic image encoding strategy of InternVL2, with a shape of $N_l \times N_s \times d$ (where $N_l$ represents the number of local images and $N_s$ represents the number of tokens per local image). $\overline{{T}}_{in}$ refers to the representative instruction tokens, with a shape of $N_p \times n \times d$. Consequently, the concatenated feature has a shape of $N_p \times N_l \times (N_s + n) \times d$.

---

Q5: Ablation experiment using the LLM outputs (S2) in Eq. (8).

A5: We conduct an ablation study by using only the LLM’s outputs (S2) for interaction prediction (in E.q.(8)) on the HICO-DET. As shown in the table below, this configuration suffers a 5.09 mAP decline in the default full setting compared to the combined "S1 + S2" approach. In contrast, integrating the visual interaction decoder (S1) with MLLM reasoning (S2) for HOI detection effectively preserves the generalization ability of the MLLM while mitigating hallucinations, leading to a noticeable improvement in overall performance.

---

Q6: Full finetuning vs LoRA finetuning.

A6: We conduct a comparative analysis of full fine-tuning versus LoRA fine-tuning strategies for HOI detection (see the table below). The results show that LoRA fine-tuning outperforms full fine-tuning, yielding 45.95 mAP versus 43.54 mAP. As a lightweight adaptation method, LoRA more effectively preserves LFM’s knowledge learned in pre-trained stage and maintains the generalization capability, thus enhaning the interaction reasoning capabilities in complex scenarios.

---

Q7: Freeze or unfreeze the image encoder.

A7: We conduct an ablation experiment by jointly training the LLM, the image encoder and the V-L MLP. The experimental results, presented in the table below, show that training only the LLM yields a relative advantage, with a 1.33mAP improvement in mAP. In the InstructHOI, we fine-tune the language model of LFMs to better understand the reconstructed HOI-specific token sequences. For relatively small-scale datasets and single-domain fine-tuning, freezing the image encoder while training only the language model may help prevent catastrophic forgetting and preserve the generalization ability of pre-trained LFMs.

---

Q8: Task interaction reasoning to predict labels in L_t's taxonomy?

A8: Computing textual similarity provides a more flexible approach to open-world interaction recognition. For instance, detecting unseen or novel interaction categories can be achieved through cosine similarity matching between the generated texts from MLLMs and the textual labels of unseen or novel categories. In contrast, directly assigning interaction reasoning to predict labels within the taxonomy of L_t's requires the design of specific classification heads for predefined interaction categories. However, when new interaction categories are introduced, the model structure needs to be modified and retrained.

---

Overall, we sincerely appreciate your valuable and constructive suggestions, and hope our responses could addresse your concerns. Please don’t hesitate to share any further feedback or questions.

In addition, we will incorporate the suggested revisions into the future version of the paper, e.g., the parameter count in Table 1, end-to-end prediction using VLM, local image token filtering with overlap-based strategy, a clearer explanation of E.q. (4), the ablation experiment on S1 and S2 in E.q. (8), and so on. These revisions will significantly enhance the quality of the paper, enabling readers to gain a more comprehensive understanding and evaluation of our work.