Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models

Q1: Concerns about unfair comparisons as DIFFUSIONHOI has a larger total parameter count.

A1: Though DIFFUSIONHOI contains a larger count of parameters, this does not incur heavy consumption in computation resources compared to existing work. First, the training time ($*i.e.*$, 5.7 hours for relation embedding learning and 11.5 hours for HOI detection learning) is more efficient than most existing work, which can be evidenced by the comparison of training time below. Second, as the discussion provided in L321-330 and the results summarized in Table 4, the inference time of DIFFUSIONHOI is faster than the two-stage HOI detectors, and comparable to the one-stage HOI detectors without a significant gap. Based on the analysis above, we believe we render a fair comparison to existing work.

More importantly, as stated in L75-77, one valuable contribution of this work is the effective knowledge transfer from diffusion models without heavy fine-tuning. We address this challenge by learning task-oriented embeddings ($*i.e.*$, relational embeddings) as textual prompts, and further transfer the knowledge from relation-driven sample generation and feature extraction two aspects. Considering the current trend towards leveraging large-scale pre-trained models to assist in various downstream perception tasks, we believe this work could offer valuable insights to the border community, and thus the larger count of parameters cannot be considered as the drawback of our work.

---

Q2: Comparsion to [79].

A2: We provide the comparison between DIFFUSIONHOI with VQGAN and PViC with ResNet-50 as the backbone in Table 1. For PViC with Swin-L as the backbone, it leverages $\mathcal{H}$-Deform-DETR as the detector which achieves 48.7 mAP on MS COCO by running merely 12 epochs, significantly higher than DETR which achieves 36.2 mAP by running 50 epochs. For a fair comparison, we i) reimplement PCiV with ViT-L as the backbone and DETR as the detector, and ii) augment the two parallel decoders for instance and interaction detection of DIFFUSIONHOI with the technology used in $\mathcal{H}$-Deform-DETR, resulting in DIFFUSIONHOI with the decoder holds a similar detection ability to $\mathcal{H}$-Deform-DETR. The detailed results are summarized below.

As seen, for models with similar backbones, DIFFUSIONHOI achieves much better performance. For instance, 38.12 $*vs.*$ 34.69, and 42.54 $*vs.*$ 39.89. On this basis, as observed in the last two rows of the table above, though ViT-L is inferior to Swin-L in downstream tasks, DIFFUSIONHOI still surpasses PViC by 1.14 mAP on the Full set of HICO-DET and 3.8/1.3 mAP on the Scenario1/Scenario2 setup of V-COCO, respectively.

Table 1 and $\S$4.2 will be updated accordingly.

---

Q3: Quantitative results and analysis of failure cases.

A3: Thank you for pointing this out. We will include the quantitative results regarding HOI detection in the Appendix for detailed analysis. Additional quantitative results for relation-driven sample generation have already been provided in Figure S1.

Regarding failure case analysis, the visualized results are provided in the PDF attached with "global" response. After reviewing the predictions delivered by DIFFUSIONHOI, we found that failure cases primarily manifest in the following scenarios: i) scenes featuring only partial human bodies, such as arm or leg, which introduces challenges for person detection; and ii) chaotic scenes teeming with people, which causes occlusion and difficulties in identifying interactions. Despite these challenges, our algorithm has shown remarkable improvement over existing approaches. Additionally, the patterns of these failure cases provide valuable insights for future research.

---

Thank you for your valuable time and constructive feedback. We will revise the main text with discussions and experimental results according to your review. It is welcome if there are any further questions or suggestions.

Q1.1: Number of parameters in the VQGAN.

A1: We only utilize the encoder of VQGAN as the backbone which contains 39.2M parameters and is smaller than that of ResNet-101 (42.8M). To address your concern about unfair comparison with models using ResNet-50 as the backbone, we provide a detailed comparison with top-leading approaches using ResNet-101 below.

As seen, our DIFFUSIONHOI achieves SOTA performance on both HICO-DET and V-COCO. This verifies the superiority of our proposed method. Table 1 will be updated accordingly.

---

Q1.2: Comparison with large backbones.

A2: The implementation of DIFFUSIONHOI relies on pre-trained Diffusion models that do not provide small backbones such as ViT-B. Therefore, we provide a detailed comparison with models using large backbones like ViT-L below to ensure a fair comparison.

Currently, only a few models provide results on large backbones such as ViT-L, and DIFFUSIONHOI can achieve the best performance on two datasets.

---

Q2.1: Unfair comparison to previous work without using additional knowledge.

A2.1: Though prior work does not utilize diffusion models for knowledge transfer, they adopted other visual-linguistic models like CLIP ($*e.g.*$, CTAN [45], SSRT [42], DOQ [44], GEN-VLK [2], HOICLIP [18], CQL [80], AGER[82], RmLR[83], and ADA-CM[84]). We provide a detailed comparison with approaches using pre-training models such as CLIP or MobileBERT in Table 1. It can be seen that when provided with additional knowledge, DIFFUSIONHOI can achieve a clear gap in performance ($*e.g.*$, 38.12 mAP $*vs.*$ 33.75 mAP for GEN-VLK[2], 35.35 mAP for CQL[80], 36.75 mAP for AGER[82], and 36.93 mAP for RmLR[83]), which highlights the effectiveness of our design.

Q2.2: The difference from methods simply adding more data for training.

A2.2: The primary distinction lies in three aspects:

First, adding training data from other datasets is impractical in real-world applications, as it requires significant time and labor to gather annotated data, which include bounding boxes, object classes, and the interactions between humans and objects. This also imposes significant challenges in generating HOI samples using diffusion models due to the complexity of annotations.

Second, the training data in our work is generated with respect to the relations between humans and objects, resulting in images that depict specific human-object interactions more accurately ($\S$3.3 Relation-Driven Sample Generation). As outlined in L178-180, for a text prompt such as "The man at bat readies to swing at the pitch," the action word "swing" is replaced with the relation embedding learned through relation-centric inversion. This steers diffusion models to focus on complex relationships rather than single objects, without heavy retraining.

Third, as stated in L53-61, we achieve knowledge transfer from diffusion models in two aspects. In addition to data generation, features distinguished for each interaction can be derived with learned relational embeddings as prompts ($\S$3.4 Relation-Inspired HOI Detection). Therefore, we enhance HOI detection with diffusion models in both training data and hypothesis set ($*i.e.*$, feature extraction), while adding more data can only affect the training set.

Additional experiments on HICO-DET are given below to verify the effectiveness of our design. As seen, simply adding more data from SWiG-DET for training (row #2) delivers limited improvement compared to using additional data synthesized with relation-driven sample generation (row #3), or features extracted with relational embeddings as prompts (row #4).

Q2.3: More details about the data and parameter scale of the diffusion model used.

A2.3: As specified in L254, DIFFUSIONHOI is built upon Stable Diffusion v1.5 pre-trained on laion2B-en, a dataset containing 2.3B samples, and subsequently fine-tuned on high-quality subsets of laion2B-en. The total parameter size is 1066M. Despite the large parameter size, as summarized in Table 4, the number of trainable parameters is significantly smaller compared to existing works ($*e.g.*$, 27.6M $*vs.*$ 50.4M for STIP [77], 41.9M for QPIC [37], and 42.8M for GEN-VLKT [2]), and the inference speed is comparable to leading counterparts ($*e.g.*$, 9.49 FPS $*vs.*$ 6.23 for iCAN [74], 7.12 for STIP [77], 3.24 for ADA-CM [84]). This indicates that the usage of diffusion models does not impose a heavy burden on our approach.

Moreover, as stated in L76-77 and L365-366, this work presents an efficient way to transfer knowledge from diffusion models, with human-object relational embeddings learned via relation-centric inversion. This can offer valuable insights to the community, and further advance the exploration of large visual-linguistic pre-training models for downstream tasks.

---

Thank you for providing such a comprehensive and detailed review. We are committed to revising the experiment section in accordance with your valuable suggestions. Please do not hesitate to post comments if there are any further questions we can help.

Q1: Analysis on the training cost.

A1: For relation-centric inversion, unlike the original textual inversion technology that learns text embeddings within the image space, we optimize relation embeddings within the latent space by reconstructing interaction features. This results in reduced training costs. Consequently, the 117 relation embeddings in HICO-DET can be learned within 5.7 hours (23 minutes per relation embedding), using 8 Tesla A40 cards.

For the main training of HOI detection on HICO-DET, as shown in Table 4, our method utilizes significantly fewer trainable parameters compared to existing work ($*e.g.*$, 27.6M $*vs.*$ 50.4M for STIP [77], 41.9M for QPIC [37], and 42.8M for GEN-VLKT [2]), and therefore the training process is completed in just 11.5 hours. The comparison of the whole training time with some representative work is summarized below, with all experiments conducted on 8 Tesla A40 cards. It can be observed that our method is more efficient.

The discussion and results above will be integrated into $\S$4.2.

---

Q2: Comparison to SOTA work.

A2: To solidly address your concern, we provide a comprehensive comparison with PViC below. Here PViC with Swin-L leverages $\mathcal{H}$-Deform-DETR as the detector which achieves 48.7 mAP on MS COCO by running merely 12 epochs, significantly higher than DETR which achieves 36.2 mAP by running 50 epochs. For a fair comparison, we i) reimplement PViC with ViT-L as the backbone and DETR as the detector, and ii) augment the two parallel decoders for instance and interaction detection of DIFFUSIONHOI with the technology used in $\mathcal{H}$-Deform-DETR, resulting in DIFFUSIONHOI with the decoder holds a similar detection ability to $\mathcal{H}$-Deform-DETR. The detailed results are summarized below.

As seen, with a similar backbone ($*i.e.*$, ResNet-50 and VQGAN), our method surpasses PViC by 3.43 mAP on HICO-DET, 4.0 mAP on V-COCO. Next, after reimplementing PViC with DETR and ViT-L as the backbone, the performance is still inferior to DIFFUSIONHOI ($*i.e.*$, 42.54 $*vs.*$ 39.89 on HICO-DET). Finally, after augmenting DIFFUSIONHOI with tricks used in $\mathcal{H}$-Deform-DETR, the performance boosts to 45.46 mAP on HICO-DET. Please note that ViT-L (employed by DIFFUSIONHOI) is inferior to Swin-L (employed by PViC) for downstream tasks.

We will provide a discussion in the experiment section, to highlight methods not using additional knowledge ($*e.g.*$, PViC) during comparison.

---

Q3: Complicated technical design.

A3: This study represents the first attempt to utilize diffusion models for HOI detection, from the perspective of both data generation and conditioned feature extraction. This may bring a fundamental shift in architectural design and training strategy. However, we agree that simplicity in design is crucial for reimplementation and extension by the community. We will definitely consider this as our future work.

---

Thank you so much for your careful review and suggestive comments. We will update our manuscript with the training time analysis and a more detailed comparison with the SOTA approaches. Please feel free to post your feedback if you have any further questions.