Zero-shot Human-Object Interaction Detection via Conditional Multi-Modal Prompts

Dear Reviewer n8e2,

First and foremost, we extend our deepest gratitude for your insightful feedback and hope our clarifications address your concerns. We're eager to highlight the significance and potential of our work.

1. The differences between the proposed method and the ICCV2023 paper.

Thank you once again for your careful consideration of our work. We propose an approach for zero-shot HOI detection that addresses the issue of performance degradation between seen and unseen classes. While the concurrent ICCV2023 paper focuses on improving efficiency when adapting CLIP to HOI detection, our method prioritizes generalizability. Our approach differs from it in several key aspects:

• Conditional Vision Prompts for Region-Level Interactiveness Knowledge: While both the ICCV paper and our approach utilize prior knowledge from DETR, our method specifically focuses on leveraging instance-level visual priors to distinguish between non-interactive human-object pairs and interactive ones. This differentiation is crucial, especially in zero-shot HOID settings where unseen HOI concepts are encountered.

• Learnable Embeddings as Context Words: To augment the CLIP text encoder's representation capabilities for actions, we introduce learnable embeddings as context words, i.e., an additional input known as language prompts.

• Regularization Loss for Generalizability: Since the hand-engineered prompts outperform the learnable soft prompts for the zero-shot setting, then, in order to avoid overfitting to the seen classes and strengthen generalizability, we further propose that the learnable ones should be trained so that they can be correctly classified in language space where the class weights are given by the hand-engineered prompts. This is achieved through the constrastive loss in Equation (6) of the main text.

• Exclusive Reliance on Language for Interaction Classification: For interaction recognition, the ICCV paper proposes a key-value mechanism storing both visual and linguistic knowledge. In contrast, we argue that visual knowledge is challenging to generalize to unseen classes. In other words, the visual features of unseen classes are hard to acquire. As a result, we exclusively employ language for interaction classification.

2. Connecting the predictions with the GT labels.

To associate the detected human–object pairs with the ground truth, we calculate the intersection-over-union (IoU) between each detected pair and the ground-truth pair. The IoU is computed for human and object boxes separately, and the minimum of the two is taken. Detected pairs are considered to be positive when the IoU surpasses a predefined threshold.

3. Experimental results using the typical settings.

We are truly grateful for this insightful consideration. To further validate the effectiveness of our method, we conduct experiments on the typical experimental settings on both the HICO-DET and V-COCO datasets. Empirically, we find that our method achieves competitive results on the default setting on both datasets. Notably, our method achieves sota on rare classes on HICO-DET, aligning with our methodology's design since rare classes are typically challenging to detect, similar to the scenario of detecting unseen interactions.

4. Missing related works.

Thank you for bringing this to our attention. We will incorporate discussions about the literature on HOI in CVPR2023 [1-4] into the related work section, and revise the formats of references in the final version.

[1] Open-Category Human-Object Interaction Pre-Training via Language Modeling Framework

[2] Relational Context Learning for Human-Object Interaction Detection

[3] Category Query Learning for Human-Object Interaction Classification

[4] ViPLO: Vision Transformer Based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection

Dear Reviewer n8e2,

We sincerely appreciate the time you invested in reviewing our submission and your invaluable feedback. We have diligently addressed your comments and provided corresponding responses and results. We believe that these revisions have addressed the concerns you raised. We would be grateful for the opportunity to further discuss whether your concerns have been adequately addressed. If there are any aspects of our work that remain unclear, please do not hesitate to inform us.

Once again, thank you for your guidance and insights.

Warm regards,

Dear Reviewer c79G,

First and foremost, we extend our deepest gratitude for your thorough review and insightful feedback. Your recognition of our method is truly appreciated. We acknowledge the concerns you've raised and will attempt to address them point by point:

1. The novelty of our method.

Thank you once again for your careful consideration of our work. CoOp[1] first proposes to use context tokens as language prompts in the image classification task. Co-CoOp [2] proposes to explicitly condition language prompts on image instances. Recently, other approaches for adapting V-L models through prompting have been proposed. For example, ProGrad [3] utilizes gradient matching, while TTTuning [4] employs test time tuning.

Different from them, our innovations mainly lie in the following two aspects:

• Introducing Multi-Modal Prompts: Previous methods have mainly focused on unimodal solutions, which means adding prompts either in the vision or language branch, particularly in the language branch when adapting Vision-Language pretrained models. We have discovered that unimodal prompts are not effective for the HOID task. This is because the HOID task requires not only the generalizability of the text branch through language prompts but also the inclusion of vision prompts to enhance the vision encoder's understanding of region-level spatial relationships. In light of this, we first propose multi-modal prompts in zero-shot human-object interaction detection to improve visual-language feature alignment and zero-shot knowledge transfer.

• Leveraging Prior Knowledge for Vision and Language Prompts: Previous methods involving conditional prompts, like CoCoOp, generate an input-conditional token for each image and only apply it in the text branch. Our method differs from it in the following two aspects: (1) First, we propose a vision condition that incorporates spatial information for each image, making it more fine-grained and suitable for the HOID task in the vision branch. (2) Second, we introduce a language condition as a regularization term to prevent the model from deviating excessively from the CLIP text feature space.

[1] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models."

[2] Zhou, Kaiyang, et al. "Conditional prompt learning for vision-language models."

[3] Zhu, Beier, et al. "Prompt-aligned gradient for prompt tuning."

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models."

2. Discussion on the PD.

• PD is larger in RF-UC setting.

The PD could be attributed to the semantic gap between the seen and unseen classes. In RF-UC setting, the rare classes are filtered out during the training process. Given that rare classes in real-life scenarios are inherently infrequent and semantically challenging, they are not well-understood by the original CLIP text embedding space, leading to a larger PD.

• Ablation studies on which module is more important for reducing the PD.

We propose a constraint for each modality to alleviate the performance degradation. Following the reviewer's suggestion, we further conduct ablation studies to check which condition is more important for reducing the PD in RF-UC setting. As shown in the table below, we find that the language condition holds greater significance for reducing PD, showing the efficacy of utilizing hand-crafted prompts for regularization.

3. Verb-agnostic knowledge for interactiveness-aware feature extraction.

• Verb-agnostic knowledge is independent of the specific actions or verbs involved in a given situation. Similarly, interactiveness judgment is inherently binary, devoid of the necessity for specific verb-related information. As a result, we propose to utilize verb-agnostic knowledge to facilitate interactiveness-aware feature extraction.

• We would like to kindly remind you that the local spatial structure we utilize only involves the spatial information of the objects instead of the spatial arrangement of human-object pairs. Since an interaction may manifest at virtually any position within an image (taking into account the different shooting angles of the images), we argue that the local spatial structure is inherently verb-agnostic.

Dear Reviewer c79G,

We sincerely appreciate the time you devoted to reviewing our manuscript and the invaluable feedback you provided. We have diligently addressed your comments and provided corresponding responses. We believe that these responses adequately address the concerns you raised. If any ambiguity remains, we sincerely invite further inquiries. We genuinely appreciate your time and dedication to reviewing our research.

Once again, thank you for your constructive insights.

Warm regards,

Dear authors, Thanks for your response and sorry for getting you late.

novelty

your response does actually support my point of view: the novelty is limited and the proposed method mainly adopts the core idea from CoOp by extending it to multi-modal prompts.

We have discovered that unimodal prompts are not effective for the HOID task. According to the value in the Table above, this is clearly not right. The method with language prompt only can actually achieve an effective result, which is 27.97 on RF-UC. right?

verb-agnostic knowledge.

How do you represent the local spatial structure? two binary maps? If so, I disagree that the spatial structure is verb-agnostic though I think it is helpful for interactioness recognition.

Dear Reviewer c79G,

Thank you for providing further feedback.

1. Unimodal prompts are not effective for the HOID task.

• Regarding the first line "w/o Vision Condition" in the table, we apologize for the confusion caused by the formulation. It actually means that we utilize both unconditional vision prompts and conditional language prompts, which constitutes a multi-modal setup. To align with the formulation in Table 2 of the main text, we have revised the table as follows:

• We would also like to kindly remind you that we discuss the performance when only utilizing unimodal prompts in Section 4.4 of the main paper. The ablation in Table 2 of the main text demonstrates that when using unimodal prompts, the model exhibits inferior performance on unseen classes.

2. Regarding the verb-agnostic knowledge.

To represent the local spatial structure, we solely utilize the bounding box coordinates of the objects. Based on your feedback, we will rename the verb-agnostic knowledge in the final version.

Thank you once again for your valuable feedback.

Dear Reviewer YqBQ,

First and foremost, we would like to express our sincere appreciation for your comprehensive review and valuable feedback. We are truly grateful for your recognition of our approach. Herein, we provide a detailed response to each of your concerns:

1. The novelty and the motivation behind each module.

We concur with your perspective that leveraging vision foundation models for the HOI detection task is a recent trend. In this work, we aim to apply CLIP for the HOI detection task through decoupled multi-model prompts:

• Conditional Vision Prompts for Region-Level Interactiveness Knowledge: Since CLIP is originally designed for image-level recognition, we design vision prompts for extracting feature maps with region-level pair-wise interactiveness knowledge. Leveraging the instance-level prior knowledge from DETR, our encoder is guided to allocate heightened attention to regions with potential interactions.

• Learnable Embeddings as Context Words: To augment the CLIP text encoder's representation capabilities for actions, we introduce learnable embeddings as context words, i.e., an additional input known as language prompts.

• Regularization Loss for Generalizability: In order to utilize the original structure of CLIP's text embedding space and avoid overfitting to the seen classes, we further introduce a regularization loss. This loss prevents the model from becoming too specialized on the seen classes and strengthens its generalizability when dealing with unseen classes.

2. The data split used for fine-tuning DETR.

To clarify, in our approach, we fine-tune DETR only on the training set. The 'whole' in 4.2 means all the instance-level annotations, without distinction for instances engaged in unseen interactions, which is the same as previous works (including Bansal et al. 2020 and Hou et al. 2020). We apologize for the confusion caused by the earlier statement and will revise it in the final version of our work.

3. Experimental dataset.

We do not conduct experiments on v-coco dataset since the official evaluation utilities do not support a zero-shot setting.

To further validate the effectiveness of our method, we conduct experiments on the typical experimental settings on both the HICO-DET and V-COCO datasets. Empirically, we find that our method achieves competitive results on the default setting on both datasets. Notably, our method achieves sota on rare classes on HICO-DET, aligning with our methodology's design since rare classes are typically challenging to detect, similar to the scenario of detecting unseen interactions.

4. Reflections on the Study's Deficiencies and Future Directions.

Thank you for your constructive feedback. In light of your feedback, we will expand our conclusion section to incorporate the following points:

• Deficiencies: In this paper, we employ CLIP for HOI detection via conditional prompt learning. However, a limitation of this approach is its dependency on the input resolution of CLIP, specifically constrained to 224 $\times$ 224 pixels, which may impede the accurate detection of interactions involving small objects.

• Future Prospects: Considering the quick-evolving trends in Vision Language Models (VLM), and Large Language Models (LLM), in future, there are several promising directions: (1) We aim to employ VLMs for flexible input resolutions to deal with HOIs with different scales or distances. (2) We will explore to integrate the semantic spaces of VLMs and LLMs to improve the model's comprehension of interactions, especially for unseen interactions, aiming for a more generic HOI detection system in the real-world scenario. (3) We will explore efficient knowledge transfer methods in the future to facilitate the integration and collaboration of specialized HOI detectors powered by VLMs or LLMs, thereby reducing the inference costs of large models.

Dear Reviewer NGK6,

We greatly appreciate your thoughtful review and feedback. Herein, we provide a detailed response to each of your concerns:

1. More Analysis or Clarification.

a) Regarding the language prompts: We are truly grateful for this insightful consideration and conduct more ablation studies on the language prompts in the RF-UC setting as shown in the table below. We observe that:

• Directly using $C_L$ as language prompts results in inferior performance on all categories since the original text embedding space is not optimized or tailored for these specific interactions.

• By incorporating $U_L$, the overall performance improves about 1% mAP at the cost of increased performance degradation between seen and unseen classes. This shows that while $U_L$ enhances the semantic space for interaction recognition, it also leads to a larger discrepancy between known and unknown classes.

• In order to avoid the potential overfitting to seen classes and strengthen generalizability, $C_L$+$U_L$ should be trained so that they can be correctly classified in language space where the class weights are given by the hand-engineered prompts $C_L$.

b) The capability of our method: Following the previous experimental protocol, $\mathbb{A}$ contains both seen and unseen verbs. However, it is worth noting that since our method treats verb labels in a semantic manner, embedding them into a unified visual-and-text space as opposed to employing traditional one-hot labels, our method can detect any HOI in the wild given its name.

c) The instance-level visual priors: Yes. The instance-level visual priors come from the pre-trained DETR.

2. Discussion of the vision prompts.

In contrast to most previous two-stage HOI detectors, our approach integrates prior knowledge from the pretrained detector directly into the image encoder's architecture, in addition to applying it to the output of the image encoder through ROI-Align. This integration empowers the vision encoder with a heightened awareness of regions with greater potential for interactions, thereby guiding the image encoder to produce feature maps that are more sensitive to interactiveness, which is crucial for zero-shot HOI detection.

3. Explanation on how these prompts work.

We address the challenge of HOI detection by dividing the task into two subtasks: visual feature extraction and interaction classification. We introduce vision prompts and language prompts to guide each subtask, leveraging prior knowledge as constraints to mitigate the issue of overfitting. The key ideas can be illustrated in the following 3 aspects:

• Conditional Vision Prompts for Region-Level Interactiveness Knowledge: For visual feature extraction, we design conditional vision prompts that extract feature maps with region-level pair-wise interactiveness knowledge. Our encoder is guided to allocate heightened attention to regions with potential interactions, using instance-level prior knowledge from DETR.

• Learnable Embeddings as Context Words: To augment the CLIP text encoder's representation capabilities for actions, we introduce learnable embeddings as context words, i.e., an additional input known as language prompts.

• Regularization Loss for Generalizability: Since the hand-engineered prompts outperform the learnable soft prompts for the unseen classes, in order to utilize the original structure of CLIP's text embedding space, we further introduce a regularization loss. This loss prevents the model from becoming too specialized on the seen classes and strengthens its generalizability when dealing with unseen classes.

Dear Reviewer NGK6,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have carefully addressed your comments and provided corresponding responses and results. We believe that these responses and results adequately address your concerns. We would value an opportunity to discuss further whether your reservations have been resolved. Should there remain any aspects of our work that are unclear to you, please do not hesitate to inform us.

Once again, thank you for your invaluable feedback.

Best,