We thank the reviewer for the constructive feedback. We appreciate the recognition of our motivation and its effectiveness in addressing intra-class diversity and inter-class entanglement in HOI detection. Below, we clarify the novelty of VDP and the robustness of RAP with supporting analyses.

---

1. On the novelty and contributions of VDP

We thank the reviewer for pointing out connections to prior distribution-based prompt learning methods such as ProDA and DAPT. While all three methods adopt a distributional approach, VDP introduces several key innovations that make it uniquely suited for zero-shot HOI detection.

Importantly, as noted by Reviewer FDPj, our design offers “novel technical contributions”, and both variance injection and Sparsemax-based concept retrieval are considered innovative.

(1) Difference in distribution modeling strategy

Unlike ProDA and DAPT, which model prompt distributions through non-parametric or regularization-based methods on text embeddings, VDP directly leverages visual modality to guide the prompt distribution. Specifically, we extract union-region visual features per verb and inject their group-wise variance into the context vectors of the prompt learner. This method enables more faithful modeling of intra-class diversity in HOI representations.

(2) Zero-shot verb generalization

Both ProDA and DAPT focus on seen-class classification tasks. In contrast, VDP is explicitly designed for zero-shot HOI detection, with variance modeling applied to unseen verb classes via semantically grouped visual variance transfer, as shown in Table 5 (Group-5 > Global). This structure-aware grouping stabilizes learning and improves generalization to rare or novel classes.

(3) Region-aware extension

VDP is also integrated into a dual-module architecture (VDP + RAP), where RAP introduces region-specific augmentation via Sparsemax-based concept selection from human, object, and union regions. This region-aware design is absent in prior works and further improves verb disambiguation by capturing localized concepts.

(4) Structured prompt distribution without explicit regularization

Unlike methods such as CMMP that rely on explicit loss functions to shape the prompt space, VDP naturally induces a structured and balanced distribution through variance-guided training. Based on the analysis we conducted in Supplementary Section 6 and Fig. 1, prompt embeddings evolve from a collapsed space (0.105 ± 0.031) to a more expressive and semantically structured representation (0.550 ± 0.103), closely reflecting the diversity of visual features (0.518 ± 0.105). In contrast, CMMP over-expands the prompt space (0.937 ± 0.133), resulting in loss of semantic structure. These results highlight that VDP produces well-regularized, diversity-aware prompts without requiring additional regularization losses.

---

2. On the robustness and sensitivity of RAP to concept quality

While we acknowledge that RAP’s performance can be affected by the quality of LLM-generated concepts, our analysis shows that the Sparsemax strategy is generally robust. The slight drop in UO performance is not due to Sparsemax pruning itself, but rather to semantic noise arising from imperfect concept generation and region-feature alignment.

2.1 Qualitative diagnosis of noisy object concepts

The real source of noise lies in semantic leakage from the human region during concept generation. Since our prompts follow the template “a photo of a person [verb]-ing the object”, LLMs often generate object concepts with unintended human descriptions.

For example, object concept pool for “lift” includes:

"Person's legs are crossed or positioned comfortably.", "Person's feet are on the ground or resting on the object.", "Person's arms are folded or resting on their lap.", etc.

These human-centric concepts are particularly problematic in object concepts retrieval, because object region crops frequently include parts of the human (e.g., hands, arms) due to physical proximity. This causes the region feature to match human-like concepts, leading Sparsemax to assign them higher weights—pruning away actual object-relevant concepts.

2.3 Retrieval strategy sensitivity: τ-Sparsemax

We further test the effect of adaptive sparsity by applying τ-Sparsemax (τ ∈ {0.0, 0.05, 0.1}), where weights below τ are zeroed. As shown below, our model remains robust across all τ values, with only minor performance fluctuations.

RF-UC

UO

UV

These results demonstrate that RAP is robust to sparsity thresholds, and that performance variations are more likely caused by concept generation noise, not the retrieval strategy itself.

---

3. Clarification on the positioning relative to prior work

We acknowledge that the challenges of intra-class diversity and inter-class entanglement have been previously recognized (e.g., Bongard-HOI, Open-World HOI). However, our contribution lies in:

• Structuring these challenges into a unified dual-module framework (VDP + RAP),
• Providing a visually grounded solution that addresses both variation and confusion explicitly through prompt learning,
• And validating it under four zero-shot settings on HICO-DET with consistent SOTA performance and low parameter count (4.5M).

Thus, while the problem is not entirely new, our solution is novel in formulation, integration, and empirical strength.

---

4. Citations and acknowledgements

We thank the reviewer for suggesting citations to ProDA and DAPT. We will incorporate them in the revised version and clarify how our formulation builds upon and differs from these prior works.

---

We sincerely thank the reviewer for pointing out the sensitivity of RAP to concept quality and the connection between VDP and prior prompt distribution methods. We hope our clarifications and analyses help address these concerns.

Dear Reviewer,

Thank you for acknowledging our rebuttal. We hope it addressed your main concerns—particularly regarding contributions of VDP, sensitivity of RAP.

If there are any remaining questions or suggestions, we would be more than happy to clarify them during the discussion phase.

We sincerely appreciate your time and thoughtful review.

Best regards,
Authors

We thank the reviewer for the thoughtful assessment and helpful suggestions. We address the reviewer’s concerns regarding novelty, ablation, and comparative evaluation below.

---

1. On the novelty of VDRP

As noted by reviewer FDPj, who highlighted “novel technical contributions” including our dual-module design and the variance injection and Sparsemax-based concept retrieval, our method introduces several key advances.

(1) Visual Diversity-aware Prompt (VDP)

VDP provides a new perspective beyond prior distributional prompt methods such as ProDA and DAPT. While existing approaches operate in the text embedding space using non-parametric or regularization-based strategies, VDP directly models prompt distributions from the visual modality by extracting union-region features per verb and injecting their group-wise variance into the prompt embedding. This enables accurate modeling of intra-class diversity. Furthermore, unlike prior works that focus on seen class, VDP explicitly targets zero-shot HOI detection by transferring visual variance from seen verbs to unseen ones via semantic grouping, improving generalization (Tab.5).

(2) Region-Aware Prompt (RAP)

Although both RAP and SICHOI explore multi-view alignment, their formulations differ in both scope and implementation:

• SICHOI aligns linguistic triplets (subject, verb, object) with global image features using dependency parsing.
• RAP, in contrast, aligns visual regions (human, object, union) with region concepts, refining prompts via Sparsemax-based selection.

Our RAP introduces region-specific, retrieval-based prompt augmentation, offering a complementary and orthogonal design to SICHOI’s text-driven attention alignment.

---

2. On the ablation study and robustness of group-wise variance

We respectfully disagree with the reviewer’s concern. We have provided extensive analysis of our group-wise visual variance mechanism through ablations in both the main paper and supplementary. Also, additional results are presented below for this rebuttal:

2.1 Analyses in the main paper and the supplement

• The main paper (Tab. 5) compares Global vs. Group_3 vs. Group_5 grouping strategies.
• The supplement (Tab. 3) evaluates the contribution of mean + variance (μ, σ²) vs. variance-only (σ²) to assess which statistics are most impactful.
• These results confirm that group-wise variance improves robustness to visual diversity while being more stable than per-class estimation.

2.2 (New) Robustness to semantically isolated verbs

To evaluate robustness under reduced semantic support, we removed the top-3 semantic neighbors for each unseen verb during variance computation:

→ Minimal performance drop (−0.67 mAP in unseen) confirms that our grouping retains generalization capacity even with limited support.

2.3 (New) Robustness to few-shot variance sampling

We simulate low-data scenarios by limiting each verb class to 5 samples ($N_v$ = 5) when estimating visual variance.

→ We observe only minor drops (≤ 0.2 full mAP), confirming that our grouping strategy produces stable variance estimates even with few samples.

---

4. On comparisons with SoTA (e.g., EZ-HOI, SICHOI)

We thank the reviewer for pointing out the importance of comparison with recent SoTA methods such as EZ-HOI (NeurIPS 2024) and SICHOI (CVPR 2024). We provide a comprehensive and quantitative response below.

4.1 Comparison with EZ-HOI under all zero-shot settings

In the main paper and the supplement, we already compared our method with EZ-HOI (NeurIPS 2024) across all four zero-shot HOI settings.

Our method consistently outperform EZ-HOI across all settings. Also, note that the reviewer’s reference to 36.73 corresponds to Full mAP in the RF-UC, which is already reported in our paper (Main: Table 1–3, Supplementary: Table 2). Under the same setting, our method achieves a higher Full mAP of 38.13, while also improving other key metrics such as harmonic mean and unseen performance.

(1) ResNet50+ViT-B

(2) ResNet50 + CLIP ViT-L

These results demonstrate that our method achieves state-of-the-art performance across all four zero-shot settings and two backbones, consistently improving generalization capability to unseen interactions.

We also clarify that EZ-HOI is a NeurIPS 2024 paper (not ECCV). Our comparisons are consistent with the official ViT-B and ViT-L configurations reported in their NeurIPS version.

---

4.2 Clarification on SICHOI and its limited comparability

Thank you for sharing a recent and interesting work, SICHOI (CVPR 2024). We will include this work in the final version. We acknowledge the recent advancement of SICHOI (CVPR 2024) performance. However, direct and fair comparison with our framework is problematic for several key reasons: 1) No code or model release, 2) No results on UO or UV settings, and more importantly 3) Unfairly high computational cost induced by a stronger backbone, additional foundation models (e.g., BLIP, SAM) and multiple decoders/branches.

For reference, the reviewer’s cited score of 36.27 corresponds to Unseen mAP under RF-UC, while SICHOI also reports 36.44 under NF-UC. We include both below for comparison.

Also, supervised comparison is already provided in Supplementary Tab.1, showing that our method performs competitively under the supervised setting.

---

We hope these address the reviewer’s concern regarding comparison fairness and highlight the superior generalization capability of our approach across multiple zero-shot settings.

5. On the use of fixed prompt templates

As reviewer requested, we experimented with three formulations:

• BASE template: "A photo of a person is [verb]-ing an object."
• Simplified template: "Person [verb] the object."
• LLM-style template: "A example of the person [verb]-ing the object."

Results show only marginal differences, indicating that our framework is robust to the template design:

NF-UC

RF-UC

UO

UV

Template variation shows minimal performance impact, indicating that our prompt learning framework is robust to sentence structure.

---

We truly appreciate the reviewer’s thoughtful comments and critical insights. We hope our responses and new analyses help resolve the concerns.

Dear Reviewer,

We sincerely appreciate your careful review and the time you invested in evaluating our work. Your candid feedback has been valuable in helping us better understand how our contributions are perceived within the community. We are grateful that you considered our rebuttal and acknowledged the effectiveness of our approach, and we will take your suggestions into account to our paper.

Best regards, Authors

We thank the reviewer for the thoughtful feedback and the recognition of our contributions to modeling intra-class visual diversity and inter-class entanglement in zero-shot HOI detection. Below, we address all the questions/concerns raised by the reviewer.

---

1. Analysis of hyperparameters (α, β, Gaussian perturbation)

As requested, we provide additional analyses of hyperparameters across four zero-shot settings:

• α (Eq. 8): scaling factor for the delta vector derived from group-wise variance.
• β (Eq. 13): scaling factor for region-aware concepts augmentation.
• Contribution for Gaussian perturbation

Our analyses show that the proposed method is robust to the choice of hyperparameters in all settings and Gaussian perturbation added prompt embedding improves generalization capability in “unseen” classes.

1.1 Impact of variance injection scale (α)

We conduct ablations on α ∈ {0.02, 0.01, 0.1} to assess the effect of delta scaling in Eq. (8). The results are summarized below:

NF-UC

RF-UC

UO

UV

We find that α = 0.02 offers the best trade-off across most settings. This value (0.02) was chosen to match the initialization scale of CLIP’s context embeddings, which ensures stability as large delta scales can lead to over-perturbed context tokens and unstable training. Since context tokens are known to be sensitive to initialization, keeping the scale aligned enhances stability while capturing diversity.

1.2 Concepts scaling factor (β)

We evaluate β ∈ {0.2, 0.5, 1.0} in Eq. (13) to study how strongly region-derived concepts should contribute to the prompt augmentation.

The results are summarized below:

NF-UC

RF-UC

UO

UV

Moderate augmentation scaling (β = 0.2) achieves the best balance overall. Excessive weighting (β = 1.0) slightly degrades performance across multiple settings, due to overconfident region concepts.

1.3 Effect of Gaussian perturbation (ε)

We evaluate the effect of adding stochasticity to prompt embeddings via ε ∈ {0, 0.1}. The results below demonstrate that the Gaussian perturbation consistently improves generalization capability in “unseen” classes.

NF-UC

RF-UC

UO

UV

Perturbation improves generalization in most unseen settings, confirming that stochastic sampling helps model diverse visual realizations for each verb.

---

2. Robustness of visual variance under few-shot sampling

As requested, we provide additional analysis on the robustness of our model across four zero-shot settings. We estimate group-wise variance using only 5 visual features per verb ($N_v = 5$) and compare it to the default setting. Performance remains stable, indicating that our method generalizes well even under few-sample conditions.

NF-UC

RF-UC

UO

UV

We observe only minor drops (≤ 0.2 full mAP), confirming that our grouping strategy produces stable variance estimates even with few samples.

---

3. Distribution of diversity-aware prompt

As requested, we analyze the distribution of prompt embeddings after training using the inter-class cosine distance metric. This analysis was conducted and reported in Section 6 of the Supplementary, along with visualizations in Supplementary Fig.1. Prompt embeddings begin in a collapsed state (0.105 ± 0.031) and become significantly more diverse after training (0.550 ± 0.103), closely matching the visual space (0.518 ± 0.105). Unlike CMMP (0.937 ± 0.133), which over-separates classes, our method yields structured and aligned distributions that reflect visual diversity. These results confirm that our prompts evolve into meaningful representations without needing additional alignment losses.

---

We sincerely thank the reviewer for raising insightful questions regarding the effect of group-wise variance (Eq. 8, 13) and the role of Gaussian perturbation in our framework. We hope our additional analyses and clarifications help address these concerns.

Dear Reviewer,

Thank you for your detailed review. We wanted to kindly follow up and check whether our rebuttal addressed your concerns.

If any points remain unclear or if you have further suggestions, we would be happy to provide additional clarification during the discussion period.

We sincerely appreciate your time and consideration.

Best regards, Authors

We thank the reviewer for the constructive feedback. We are encouraged by the positive assessment of our technical contributions, evaluation rigor, and clarity. Below, we address the main concerns regarding zero-shot verb generalization, Sparsemax sensitivity, and phrase encoding bias.

---

1. Robustness to semantically isolated verbs

We thank the reviewer for highlighting a potential limitation of our group-wise variance estimation when applied to semantically isolated unseen verbs. To simulate this extreme case, we conduct an additional experiment under the UV setting, where we remove the top-3 semantic neighbors (based on CLIP text embedding similarity) for each unseen verb prior to computing group-wise visual variance. This setting emulates scenarios where unseen verbs lack nearby counterparts in the training set. As shown below, the unseen performance drops only slightly (−0.67 Unseen mAP), indicating that our method maintains generalization ability even under reduced semantic support:

Also, our main paper also includes a related analysis using global variance estimation (Main paper: Tab.4), further supporting the robustness of our approach when class-specific support is limited.

---

2. Sparsemax failure diagnosis

We confirm that our method is robust to concept overlap and sparsity thresholding, as validated through additional quantitative and ablation analyses.

2.1 Object concept overlap analysis

To evaluate the reviewer’s hypothesis on concept noise due to shared object semantics (e.g., “table” in both “sit” and “work”), we compute the percentage of exact string overlaps in object concepts across verb pairs. Using cosine similarity ≥ 0.90, we find:

• UO setting: 2.98% shared concepts
• UV setting: 2.94% shared concepts

These results suggest that both settings exhibit similarly low overlap, and the slightly lower UO performance appears to stem from stochastic variation rather than concept redundancy.

2.3 Qualitative diagnosis of noisy object concepts

While concept overlap remains low, we observe that some object concepts contain human-centric descriptions due to the prompt template (“a photo of a person [verb]-ing the object”) and lack of explicit object grounding. For example, object concepts for "lift" include:

"Person's legs are crossed or positioned comfortably.", "Person's feet are on the ground or resting on the object.", "Person's arms are folded or resting on their lap.", etc.

This suggests that semantic leakage from the human region may introduce localized noise, particularly in the UO setting. However, this appears to be a minor stochastic effect rather than a structural flaw, as overall performance remains stable.

2.3 τ-Sparsemax analysis

We further test the effect of adaptive sparsity by applying τ-Sparsemax (τ ∈ {0.0, 0.05, 0.1}), where weights below τ are zeroed. As shown below, our model remains robust across all τ values, with only minor performance fluctuations.

NF-UC

RF-UC

UO

UV

These results demonstrate that RAP is robust to sparsity thresholds, and that performance variations are more likely caused by concept generation noise, not the retrieval strategy itself.

---

3. CLIP phrase encoding bias

We thank the reviewer for pointing out the potential bias in CLIP’s phrase encoding, especially for compound expressions like “holding+baseball+glove.” While this is a known limitation of CLIP, our prompt template—“a photo of a person [verb]-ing the object”—is a standard formulation widely used in HOI literature. To assess its effect, we performed an ablation study using alternative templates (e.g., “person [verb] the object”).

Results show minimal performance difference, indicating that our method is robust to such phrasing variations. Specifically, we compare:

• BASE template: "A photo of a person is [verb]-ing the object."
• Simplified template: "Person [verb] the object."
• LLM-style template: "A example of the person [verb]-ing the object."

Results are summarized below across all four zero-shot settings:

NF-UC

RF-UC

UO

UV

Template variation shows minimal performance impact, indicating that our prompt learning framework is robust to sentence structure—even with CLIP’s known phrase encoding bias. This suggests that such bias does not significantly affect our method’s effectiveness.

---

5. Minor points

• Computational overhead: All LLM concepts are generated offline, once per (verb, region) pair. The total size of the concept pool is fixed (10 concepts × 3 regions × 117 verbs). Variance computation is also performed once and cached. Thus, our method adds no runtime cost during inference.
• Bias propagation: We acknowledge that LLM-generated concepts may contain bias (e.g., “woman cooking”). In practice, such cases were rare and did not result in consistent prediction shifts. Future work may incorporate human-in-the-loop filtering or bias-aware prompting.
• Failure cases: We will add qualitative failure examples in the supplementary file, highlighting the generated noisy concepts in the final version.

---

We again thank the reviewer for the encouraging evaluation and for the insightful suggestions on verb variance robustness and Sparsemax sensitivity. We hope our additional analyses and clarifications help address these concerns.

Dear Reviewer,

We sincerely appreciate your thorough and constructive review, as well as your thoughtful evaluation of our work. Your detailed feedback has been invaluable in refining our understanding and has provided clear directions for future improvement. We are grateful for the time and effort you dedicated to assessing our paper and for recognizing the strengths of our approach.

Best regards, Authors