EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection

We appreciate the positive feedback and detailed reviews. The following is our response for your questions. Note all references are from the main paper’s citations.

1. Illustration for Fig. 4

We provide more illustration for Fig. 4 in the following. Fig. 4 shows the qualitative results of both our method and MaPLe. In particular, MaPLe struggles to detect unseen classes, either missing unseen HOI classes or predicting the wrong unseen HOI classes. For example, if an image only contains unseen classes, MaPLe tends to predict wrong seen classes and miss the correct ones. As shown in the bottom right of Fig. 4, this image contains unseen class only ("wear tie"), MaPLe predicts related wrong seen classes such as "pulling tie" and "adjusting tie", but fails to predict the ground-truth unseen HOI ("wear tie").

This shows the limited generalization ability of MaPLe to unseen classes. In contrast to MaPLe, our method can predict both seen and unseen classes more accurately. We will revise the discussion for Fig. 4 in the updated paper.

2. Avoid negative effect of misleading information from LLM in UTPL

We agree that LLM can provide misleading information. To deal with the problem, we utilize LLM description information with learnable attention. Specifically, the UTPL module integrates the LLM information into the unseen text learnable prompts, through multi-head cross-attention (MHCA), where learnable attention is optimized during the training. The training supervision for UTPL is from the class-relation loss in Eq. (15), to retain the relationship among each HOI class, indicated by cosine similarity between text class features. If a wrong information is provided by LLM, then the unseen text features, integrated with the wrong information, display improper relations to other seen class text features. This will be penalized by the class-relation loss, and then the attention to the wrong information will be reduced in UTPL for unseen prompt learning. Therefore, the design can mitigate the possible negative effect from inaccurate information provided by LLM

In the future, we will also explore other approaches such as chain-of-thought (CoT) and retrieval-augmented generation (RAG) to mitigate the negative effect from LLM-generated misleading information.

3. Control the length of individual statements in UTPL

The length control of individual statements is achieved by giving the prompt for LLM before the generation for the desired description: "Please reply with multiple short sentences in the following, instead of a long sentence."

4. IoU threshold value selection

We follow existing papers [19,28,38,23] to choose the threshold value, the IoU between predicted human and object bounding boxes and the ground-truth bounding boxes, to be 0.5. The threshold value is one criterion to judge whether a prediction is true positive or not. This is not only used in the evaluation during testing time, but also used in training to match predictions with ground-truth human-object pairs for model learning.

We conduct the ablation study for the threshold value selection in training time, as shown in the following table. We find that using 0.5 achieves the best performance for both seen and unseen classes.

Writing details

We will make sure the abbreviations are introduced with their full names the first time they appear in the revised version of our paper.

We appreciate the positive feedback. The following is our response for your questions. Note that references maintain their original numbering from the main paper, with new references denoted by letters and listed at the end.

1. Pre-definition of all HOI classes

Regarding the comparison with previous work, we follow the common practices in the zero-shot HOI detection setting, as introduced in the following. Zero-shot HOI setting involves predicting unseen HOI classes, where unseen class names are typically used in training [13,15,14,45,38]. In particular, VCL, FCL and ATL [13,15,14] "compose novel HOI samples" during training with the unseen (novel) HOI class names. EoID [45] distills CLIP "with predefined HOI prompts" including both seen and unseen class names. HOICLIP [38] introduces "verb class representation" during training, including both seen and unseen classes.

Beyond the zero-shot HOI setting, there are HOI unknown concept discovery [a] and open-vocabulary HOI detection [b], where unseen class names cannot be used in training. The open-vocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing. We will explore these directions in our future work.

2. Experiments without pre-definition of all HOI classes

We appreciate the suggestion to conduct experiments without pre-definition of all HOI classes. As explained in the above question, our work follows the established zero-shot HOI detection protocols, where the use of unseen class names is a common practice, as illustrated by the existing HOI detection methods [13,15,14,45,38]. Nonetheless, we recognize the value of exploring alternative settings where unseen class names are not pre-defined, which could further demonstrate the effectiveness of our approach. While this falls outside the scope of the current submission, we are willing to include this experiment in the final version of our paper, to comprehensively validate our method across varying settings.

References:

[a] Discovering Human-Object Interaction Concepts via Self-Compositional Learning, ECCV22

[b] Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection, CVPR24

We appreciate the insightful feedback. Here is our response. Note all references are from the main paper’s citations.

Weakness-1. Our novelty and technical contribution

(1) Novelty

We would like to clarify that our innovation lies in proposing a novel framework, rather than a new approach to prompt learning. Existing methods such as UniHOI [3], HOICLIP [38] and GEN-VLKT [28] align HOI model's visual features to text features from a frozen VLM. However, aligning with VLM features requires training transformer-based models, which is computationally expensive.

Unlike existing methods that utilize frozen VLM text features, we fine-tune both the visual and text encoders of the VLM with guidance from an LLM. This allows us to adapt both visual and text features to our HOI setting, rather than only adapting visual features. Consequently, our method, EZ-HOI, improves the alignment between visual and text representations, achieving SOTA performance in zero-shot HOI settings with significantly fewer trainable parameters.

A comparison between our method and UniHOI is shown in the table below. Like most existing HOI detection methods [38,28,36,45], we use CLIP as the VLM. UniHOI [3], on the other hand, uses BLIP2, a more powerful VLM than CLIP. As shown in [25], the performance of BLIP2 is significantly superior to CLIP in zero-shot image-text retrieval tasks.

Despite using CLIP, our method still achieves comparable performance to UniHOI under various zero-shot settings, which demonstrates the effectiveness of our novel framework, EZ-HOI.

(2) Technical contributions

Regarding modeling the relationship between unseen and seen classes, our novelty lies in the UTPL module. Our UTPL enhances unseen prompt learning by leveraging related seen classes with the help of LLM-generated descriptions, which to our knowledge is a novel idea. This is in contrast to existing methods, which either use LLM descriptions without seen class context [3] or rely on seen class information alone [13,15].

Our technical contribution is centered around mitigating the overfitting problem when using prompt learning to adapt a VLM for zero-shot HOI detection. While existing prompt learning typically suffers from the overfitting problem to seen classes [56,6,8,19], our method achieves 11.6 mAP improvement on unseen HOI classes compared the prompt learning baseline (MaPLe [19]) as shown in Fig. 1(d).

Weakness-2. Performance under fully-supervised setting

In our Appendix, Table 6 presents the quantitative comparison under fully-supervised settings. To ensure a fair comparison, we used CLIP as our VLM, the same as most existing methods [38,28,36,45]. As mentioned earlier in this rebuttal, UniHOI employs BLIP2, which is a superior VLM compared to CLIP.

To provide a fair comparison, we show the performances of UniHOI and our method using the same VLM (CLIP). As observed, our method outperforms UniHOI when both use CLIP.

The results for UniHOI using CLIP were obtained from UniHOI's OpenReview rebuttal (https://openreview.net/forum?id=pQvAL40Cdj&noteId=kI6HJnJ1KB).

We will follow the suggestion to move Table 6 of the appendix to the main text, and include our rebuttal discussion to the paper upon acceptance.

Weakness-3. Baseline method and performance gain

(1) Baseline and the strong seen performance

Our baseline is MaPLe [19], not the first row of Table 4 in our main text. The first row (Table 4) is part of our developed method, focusing on improving seen classes, which employs our intra-HOI fusion and visual adapter. Both modules are mentioned in Section 3.3: Intra-HOI is introduced in Appendix 6.4, and visual adapter is based on [23].

To clarify the baseline performance, we provide an extended ablation study table below. The first row shows the baseline method, MaPLe [19]. Our intra-HOI fusion module improves the seen HOI performance by 7.41 mAP, as shown in the second row. The third row is the result of adding the visual adapter, which is the first row of Table 4 in our main text. We will follow the suggestion to update the ablation studies in Table 4 of the main paper and include this discussion.

(2) Performance gain from each designed modules

Since our baseline is MaPLe [19], the performance gain over UniHOI is not solely due to the UTPL module but results from all the modules we designed (including LLM guidance, VLM guidance, and intra- and inter-HOI fusion). Each module's contribution is shown in the table above and in Table 4 of the main paper, illustrating how each module incrementally enhances the final performance. While it is true that UTPL contributes 2.42 mAP to unseen class performance, other modules, such as our LLM guidance, also significantly improve unseen performance by 1.52 mAP, and VLM guidance increases unseen performance by 1.33 mAP.

Question-1. Fully fine-tuned method and baseline

As discussed in Question 3, our baseline is MaPLe [19]. Our framework is based on prompt tuning to enable a VLM to adapt to HOI tasks, so we do not use full fine-tuning in our method. We acknowledge that full fine-tuning might be beneficial, and we plan to explore this in our future work.

Thank you for your detailed, helpful feedback. We address each of your concerns in the following.

1. Task-specific configurations

Our method is specifically designed for HOI settings with the following task-specific configurations.

First, our method is configured for HOI detection by extracting visual features for each human-object pair in an image, different from using visual features from an entire image like general prompt learning methods [19,56,6,8]. We extract HOI visual features from each human-object pair to compare with text features, which is introduced in Appendix Section 6.4.

Second, the UTPL module is designed for HOI settings to recognize interactions with strong connections and subtle differences (i.e., "straddle bike" and "ride bike"). If "ride bike" is unseen, UTPL leverages related seen HOIs like "straddle bike" to help with recognizing "ride bike".

Third, our proposed LLM-generated description is another task-specific configuration. In particular, the LLM-generated descriptions provide the nuances between unseen and related seen HOI classes, which further enhance the understanding of the unseen class.

2. Comparison with CLIP4HOI

We acknowledge our method’s slightly lower performance when compared to CLIP4HOI under the unseen verb setting, where our method's unseen performance is 0.92 mAP lower. While it is true that there is a minor drop in accuracy, it is important to consider the significant reduction in trainable parameters that our method achieves—87.9% fewer than CLIP4HOI. Moreover, under the non-rare first unseen composition setting, we outperform CLIP4HOI by 2.22 mAP for unseen HOI classes (Table 2 second column from right to left, of the main paper).

This substantial decrease in model complexity not only lowers the computational cost but also makes it more accessible to researchers in the field with limited resources. Achieving a balanced trade-off between performance and efficiency is essential, which aligns with the growing need for more efficient models in the field. Our method provides a practical solution by offering competitive performance while drastically reducing the resource requirements. Thus, the minor compromise in accuracy is counterbalanced by significant gains in model efficiency and accessibility.

3. Clarification and training details for Eqs. (3),(5)

$W_{down}$ in Eqs. (3) and (5) refer to the down projection layers, but they are not the same projection layers with different weights. We provide more training details for Eq. (5) in the following.

Although there is no annotated image for the unseen/unknown categories in training, we have two ways to optimize Eq. (5). First, we design a class-relation loss (Eq. (15) in the Appendix) to keep the relationship between seen and unseen classes, measured by cosine similarity between text features. This way, unseen prompts can also be refined based on their relation to seen classes. Second, while there are no annotated images for unseen/unknown classes, the annotated training data serves as negative samples. If the prediction score for an unseen class is too high, the model is penalized (Eq. (16) in the Appendix).

Additionally, in Eq. (5), each unseen learnable prompt is linked to a similar seen learnable prompt. If the seen prompts are optimized after each training step, the updated seen prompts will be used to refine the corresponding unseen prompts in UTPL. We will include the above discussion in the updated paper.

4. Benefits from action descriptors

We thank the reviewers for pointing out the related LLM-based works such as action descriptors, which are potentially useful for HOI detection. We conducted the following preliminary experiment given the limited time in this rebuttal.

Descriptors mentioned in CuPL[a] and DVDet [b] refer to attribute decomposition to benefit category recognition. We leverage the attribute decomposition idea from DVDet and tailor it to our method. We follow DVDet [b] to generate action descriptors for each class and integrate them into HOI class text features. This process enhances the detail and distinctiveness of the HOI class representations. Descriptors with low cosine similarity to the HOI class text features are discarded to avoid noisy information.

The following table presents our preliminary result under the unseen verb setting, showing improvement with our simple and direct adoption of the action descriptors. These results demonstrate that LLM-generated descriptions, such as action descriptors, are potentially useful for HOI detection. We will include a discussion of the related LLM-based works in our updated paper.

5. Visualization results

We acknowledge the reviewer’s request for additional visualizations. While we have made every effort to understand and address this, with all due respect, we are still unsure what visualization results we need to add. Based on our understanding, in this rebuttal, we provide the qualitative results to illustrate the impact of LLM guidance (using general description) and UTPL (using discriminative description) shown in Fig. 1 of our attached PDF.

The LLM guidance provides detailed class information, improving the performance over the baseline. Distinctive descriptions help to distinguish unseen classes from related seen HOIs, enhancing unseen performance and challenging case predictions.

References

[a] Visual classification via description from large language models, ICLR23

[b] LLMs meet VLMs: Boost open vocabulary object detection with fine-grained descriptors, ICLR24

We appreciate the positive feedback and insightful comments. The following is a detailed response to each of your questions. Note that references maintain their original numbering from the main paper, with new references denoted by letters and listed at the end.

Weakness-1. Use of unseen class names

We follow the common practices in the zero-shot HOI detection setting. Zero-shot HOI setting involves predicting unseen HOI classes, where unseen class names are typically used in training [13,15,14,45,38]. In particular, VCL, FCL and ATL [13,15,14] "compose novel HOI samples" during training with the unseen (novel) HOI class names. EoID [45] distills CLIP "with predefined HOI prompts" including both seen and unseen class names. HOICLIP [38] introduces "verb class representation" during training, including both seen and unseen classes.

Apart from using unseen class names, the prior knowledge of unseen class names is also used in existing work. UniHOI [3] utilizes LLM-generated descriptions, the prior knowledge of the unseen class names, to recognize unseen classes better with "rich and detailed representation". Thus, our method follows the common practices in the zero-shot HOI detection setting for conducting experiments.

Beyond zero-shot HOI settings, there are HOI unknown concept discovery [a] and open-vocabulary HOI detection [b], where unseen class names cannot be used in training. The open-vocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing. We will explore these directions in our future work.

Weakness-2. Comparison with CLIP4HOI

We acknowledge our method’s slightly lower performance when compared to CLIP4HOI under the unseen verb setting, where our method's unseen performance is 0.92 mAP lower. While it is true that there is a minor drop in accuracy, it is important to consider the significant reduction in trainable parameters that our method achieves—87.9% fewer than CLIP4HOI. Moreover, under the non-rare first unseen composition setting, we outperform CLIP4HOI by 2.22 mAP for unseen HOI classes (Table 2 second column from right to left, of the main paper).

This substantial decrease in model complexity not only lowers the computational cost but also makes it more accessible to researchers in the field with limited resources. Achieving a balanced trade-off between performance and efficiency is essential, which aligns with the growing need for more efficient models in the field. Our method provides a practical solution by offering competitive performance while drastically reducing the resource requirements. Thus, the minor compromise in accuracy is counterbalanced by significant gains in model efficiency and accessibility.

Question-1. Figure 2 visual encoder

In Figure 2, the "VLM visual" is different from the "visual encoder". The "VLM visual" is the frozen CLIP visual encoder, while the "visual encoder" shown in Figure 2 is fine-tuned during training and is based on our design, including visual learnable prompts $\hat{\mathcal{H}}_V$ and the HOI feature fusion module.

Question-2. Language model selection

Pure language models can also be applied to our methods for text description generation. We leverage LLaVA because it demonstrates quite strong reasoning results with GPT-4 level capabilities [32]. As shown in the following, outputs from LLaVA and Chatgpt 3.5 are both informative and reasonable, benefiting the model for learning detailed representation.

LLaVA output: "Swinging a baseball bat" describes a person using a baseball bat to hit a ball. This action typically involves the person holding the bat with both hands, standing in a stance with their feet shoulder-width apart, and using their body rotation to contact the ball.

Chatgpt3.5 output: "Swing a baseball bat" involves standing with feet apart, gripping the bat, and stepping forward as the pitch approaches. The batter rotates the torso, bringing the bat through the hitting zone to hit the ball.

References:

[a] Discovering Human-Object Interaction Concepts via Self-Compositional Learning, ECCV22

[b] Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection, CVPR24

We appreciate the positive feedback and detailed reviews. The following is our response for your questions. Note that we use the same reference number for related papers as the main paper.

Weakness-1. Structure of the paper

We agree with the suggestion to introduce the "encoder layer" in Section 3.3 rather than at the beginning of Section 3, and will revise our paper.

Weakness-2. Design intuition for Eqs. (2), (3), (5)

Here we provide detailed design intuition for Eq. (2) about LLM guidance, Eq. (3) about VLM guidance and Eq. (5) about UTPL.

In Eq. (2), the text learnable prompts $\mathcal H_T$ are integrated with text embeddings $\mathcal F_{txt}$ from LLM-generated class description. Through MHCA, $\mathcal H_T$ only aggregate useful information from $\mathcal F_{txt}$ with learnable attention. This is important because the information provided by LLM may not be equally crucial for our task.

Moreover, specific design is introduced and used in Eqs. (3) and (5). To keep trainable parameters small, we apply a down-projection layer $W_{down}$ before MHCA to reduce the feature dimension, and an up-projection layer $W_{up}$ afterward. In addition, initially Eq. (2) output equals to input $\mathcal H_T$, by initializing $W_{up}$ to 0, which stabilizes training by gradually fine-tuning $\mathcal H_T$.

In Eq. (3), visual features $f_{vis}^{\mathcal I}$ from the VLM are integrated with visual learnable prompts $\mathcal H_V$ by MHCA. Since the frozen VLM visual encoder can extract features for unseen HOIs, $\mathcal H_V$ aggregates information from these visual features, improving performance on unseen HOIs.

In Eq. (5), the unseen learnable prompts $\hat{h_{T_u}}$ are combined with disparity information $f_{ txt_u}^{\rm disp}$, related seen class prompts $\hat{h_{T_s}}$, and the unseen class prompts themselves $\hat{h_{T_u}}$, in MHCA. The disparity information $f_{ txt_u}^{\rm disp}$ provides distinctive attributes $\hat{h_{T_u}}$. The related seen class prompts $\hat{h_{T_s}}$ enhance $\hat{h_{T_u}}$ by transferring shared features to unseen classes. The unseen class prompts $\hat{h_{T_u}}$ retain self-information and are emphasized through MHCA's processing.

Weakness-3. Design intuition for Eqs. (6) and (8)

Here we provide detailed design intuition for Eq. (6) for deep text prompt learning and Eq. (8) for deep visual prompt learning.

In deep text prompt learning, we put the learnable text prompts $\tilde{h_T^i}$ at the end of the original text prompts $W_i$. $W_1$, in the first layer, is generated from "a photo of a person <acting> a/an <object>". Since we design $\tilde{h_T^i}$ to be class-specific with HOI class information, its semantic information is very different from "a photo of", related to the beginning position of $W_1$. Therefore, the end position of $W_1$ is better suited for $\tilde{h_T^i}$, where the semantic information is more connected to $W_1$.

Moreover, after N layers, we stop introducing new learnable prompts. Basically, when introducing new learnable prompts to each layer, we observe the VLM deals with seen classes better with decreased unseen performance. Moreover, inserting learnable prompts into deeper layers of VLM, makes the performance sensitive, because the feature space is mature in those layers [19].

Similar to deep text prompt learning, deep visual prompt learning also only introduces new learnable prompts $\hat{h_V^i}$ until layer N. The position of $\hat{h_V^i}$ does not significantly influence the outcome, as original frozen visual prompts $E_i$ correspond to different regions of the input image, but $\hat{h_V^i}$ contains global image information, equally enhancing $E_i$.

Question-1. Ablation study for hyperparameter $N$

The following table shows the ablation study for the hyperparameter $N$, where $N$ means we introduce learnable prompts from the first layer until layer $N$.

We use N=9 in our main paper because it shows the best unseen performance.

Question-2. Ablation study for different positions of learnable prompts

The following table shows the ablation study for the different positions of learnable prompts. We find that the position of learnable prompts does not affect the outcome too much. Thus, we follow [19] and fine-tune layers 1-12, where fine-tuning layers 1-9 shows slightly better unseen performance.

Question-3. Intuition for components tackling generalization, other than UTPL

We introduce components including LLM guidance, VLM guidance and inter-HOI fusion, other than the UTPL module, which can enhance generalization ability to unseen classes. We discuss the design intuition one by one.

The LLM guidance, mentioned in Section 3.1 "Text Prompt Design" of the main paper, integrates learnable text prompts with detailed HOI class descriptions. This approach enhances the model's understanding of unseen classes, which lack training data, by providing detailed information rather than simple class names.

The VLM guidance, detailed in Section 3.1 "Visual Prompt Design," combines learnable visual prompts with image features from the frozen VLM visual encoder. Since the VLM includes unseen HOI information, the frozen encoder can extract unseen HOI representations during testing. The learnable visual prompts then aggregate these representations, enhancing unseen class prediction.

Inter-HOI fusion, as mentioned in Appendix Section 6.4, refines HOI visual features by considering surrounding HOI features in the image. For instance, to detect "cut a cake," the model uses the surrounding visual context like "cut with a knife," making recognition easier.

Thanks the authors for the rebuttal, and reviewers' for the comments. We are in the period of author-reviewer discussion.

• Reviewers, please read through all the reviews and rebuttal and see if authors' responses have addressed your and others' important questions.
• Authors, please follow up with any reviewers' further comments.

Cheers,

AC