Learning Human-Object Interaction as Groups

We thank reviewer 8R1p for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: Insufficient Evidence for Addressing HOI Challenges

A1: Respectively disagree. We would like to clarify that our work have clearly delineated the limitations of existing HOI methods and demonstrated how our proposed approach specifically addresses these challenges:

• Challenge. We explicitly define the challenge addressed in this paper as how to model the complex relationships between entities and propagate contextual information across them? (see L32-33).

• Limitation. ​​For this challenge, current methods remain confined to the modeling of predefined pairwise relations, leaving the inherent collective patterns (i.e., multiple entities engaging in joint activities) between entities unexplored (see L34-40).

• Motivation. To tackle this limitation，we revist HOI detection from a group perspective, i.e., structuring and organizing visual entities by finding groups of similar ones, formulating the challenge as two foundamental issues: (1) how is a group formed? (2) how does a group function? (see L41-50).

• Technical Solution. For group formulation, we propose two grouping strategies based on geometric proximity and semantic similarity principles (see L51-60). For group functioning, we introduce a new architecture for HOI detection involving these two grouping strategies (see L61-72).

• Experiment. ​​The superior performance of GroupHOI on V-COCO and HICO-DET demonstrates its ability to address the core challenge targeted by this study (see L76-78).

Overall, we argue that our geometric proximity and semantic similarity groups can effectively address the challenge raised by this study.

Q2: Whether these groups can cover all possible HOI scenarios, and whether the method is effective in handling cases that do not fit the situations described in the paper.

A2: It seems a little unfair that requires our method to cover all possible HOI scenarios, as no existing dataset provides comprehensive coverage for such validation. However, in response to your concern, we introduce additional validation on NVI dataset [ref1] which incorporates complex interactions involving arbitrary numbers of humans ranging from 2 to 12. Following [ref1], we report mR@25, mR@50, and mR@100 under different IoU thresholds. The experimental results on NVI val are as follows:

As seen, GroupHOI surpasses all these methods, reaching new sota for NVI-DET task. This result demonstrates that our model is applicable to more complex interaction scenarios.

[ref1] Nonverbal Interaction Detection. ECCV 2024.

Q3: Why is the group size fixed?

A3: Sorry for this confusion. Both the geometry-aware local transformer and the semantic-aware interaction decoder adopt a three-layer Transformer decoder architecture. In each decoder layer, a distinct geometric or semantic grouping is established to model contextual relationships. This strategy enables the total number of neighbors for each proposal (i.e., human, object, and interaction proposal) ranging from $K_g$ to $K_g\times3$ (4 to 12) for the geometric group and from $K_s$ to $K_s\times3$ (2 to 6) for the semantic group. Notably, over 96% of samples in V-COCO and over 94% in HICO-DET contain no more than 12 entities, so geometric group of 4 to 12 can cover the majority of scenarios. Meanwhile, the size of semantic group aligns with [ref2]'s finding, which underscores that the majority of the interactive dialogue involving 10 or more participants is produced by only the top 4-5 contributors. The above discussion will be incorporated into Sec. 3.5 and 4.3.

[ref2] Group discussion as interactive dialogue or as serial monologue: The influence of group size. Psychological science, 2000.

Q4: Source of baseline's performance gain

A4: We apologize for the omission of implementation details. Our baseline is built upon HOICLIP [ref3], with its performance primarily attributed to the following modifications:

• Training strategy. We maintain the same training duration (​​90 epochs​​) as HOICLIP. However, our initial learning rate is set to $5\times10^{-5}$​​, decaying by a factor of ​​10 after every 30 epochs​​. In contrast, HOICLIP uses an initial learning rate of $1\times10^{-4}$​, reducing it by a factor of ​​10 only after 60 epochs​​.
• HOI/object classifier. Unlike HOICLIP, we do not freeze the parameters of the HOI/object classifiers initialized from CLIP, allowing for further optimization during training.
• Interaction decoder. Due to the dimension mismatch, HOICLIP omit positional embeddings (256) in the interaction decoder (768). Instead, we expand the positional embeddings to 768 by stacking them three times for the interaction decoder.
• HOI loss. To better address the long-tail distribution issue, we replace the original Focal loss with Asymmetric Loss [ref4] for HOI loss computation.

The aforementioned information will be included in the Appendix Sec. A.

[ref3] Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. CVPR 2023.

[ref4] Asymmetric loss for multi-label classification. ICCV 2021.

Q5: Examples in Fig. 4.

A5: Apologies for the confusion. As stated in Q3, we construct a new geometric/semantic group at each interaction decoder layer, yielding geometric group sizes of 4-12 and semantic group sizes of 2-6. Thus, to facilitate presentation, we select three entities from the group for visualization. We apologize for the ambiguity in our previous statement. To clarify, we will update the visualization in Fig. 4 to illustrate the evolving group composition at each layer in the revised manuscript.

We thank reviewer PSyr for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: Does the learnable proximity estimator require supervision? If not, how does it distinguish groups?

A1: Apologies for the confusion. The linear layer, used to compute the proximity score by fusing the centroid distance and IoU between two bounding boxes, is not explicitly supervised but is trained in an end-to-end manner based on the final HOI loss. The proximity score measures the spatial closeness between two detection boxes, where a smaller value indicates closer proximity. We construct the neighbor set by selecting the $K_g$ detection boxes with the smallest proximity scores. To clarify, the manuscript will be revised in Sec. 3.3 as follows:

A simple linear layer is then employed to compute the proximity score $s_{i,j}$ based on the central distance $dis_{i,j}$ and the Intersection-over-Union $IoU_{i,j}$ between bounding boxes, without requiring additional supervision. This results in a proximity score matrix, where lower scores indicate closer spatial proximity. For each entity, we select the $K_g$ neighbors with the lowest scores to construct its geometric neighbor set $N^g_i$.

Q2: What is the performance gain of MLPs over distance-based methods?

A2: In response to your suggestion, we have added the ablation experiments, as:

where (IoU only) and (CD only) refer to using the IoU and center distance for group construction, respectively. Results demonstrate the linear fusion layer employed in GroupHOI provides a significant performance advantage.

Q3: Are distance and IoU sufficient for grouping, or should image features be included?

A3: Thank you for the insightful suggestion. We agree that incorporating depth maps [ref1] could be a promising direction to address the limitation you identified. However, it would introduce additional supervision and computational overhead. Moreover, it could lead to an unfair comparison with existing methods that rely solely on 2D visual cues.

Following your suggestion, we also conducted additional experiments utilizing image features:

For GroupHOI (image features), we incorporate image features into the computation of the proximity score by concatenating them with two geometric metrics. However, its results showed no significant improvements. We hypothesize that the image features in the HOI detector are also insensitive to depth, as the labels do not provide such supervisory signals.

[ref1] Improving visual relation detection using depth maps. ICPR, 2021.

Q4: Fixed group size.

A4: We acknowledge the confusion and apologize. Both our geometry-aware local transformer and semantic-aware interaction decoder employ a three-layer Transformer decoder architecture. For each decoder layer, we construct a new geometric/semantic group. Through this strategy, each proposal (i.e., human, object, and interaction proposal) can aggregate contextual information from $K_g$ to $K_g\times3$ (4 to 12) for the geometric group and from $K_s$ to $K_s\times3$ (2 to 6) for the semantic group. Empirical statistics show that over 96% of samples in V-COCO and over 94% in HICO-DET contain no more than 12 entities, ensuring that our geometric grouping strategy sufficiently covers the majority of scenarios. The semantic group design is motivated by [ref2], which observes that even in large-scale interactive settings, meaningful exchanges are often dominated by a small subset of participants (4–5 key contributors), aligning with our design. The above discussion will be incorporated into Sec. 3.5 and 4.3.

[ref2] Group discussion as interactive dialogue or as serial monologue: The influence of group size. Psychological science, 2000.

Q5: Discussion about related prior group-based methods.

A5: Thank you for your valuable suggestion. We agree that these works share similarities with our work and will add a discussion of them in Sec. 2 as follows:

Recent studies in related domains have also explored group-wise relational modeling based on proximity and similarity. For example, [ref3] addresses 3D human mesh recovery in crowded scenes using a multiscale hypergraph to capture both individual and group-level relations. Similarly, [ref4] focuses on multi-agent trajectory prediction by learning multiscale hypergraphs to represent pairwise and group-wise interactions. These works share the idea of modeling intra-group relations with spatial or semantic cues. However, they simply combine proximity and similarity information for relation reasoning (e.g., [ref3] introduces human spatial information through box coordinates). In contrast, our method explicitly constructs and learns relationships from both geometric and semantic perspectives through two distinct graphs: a geometric graph based on the spatial proximity of entities, and a semantic graph built on the semantic similarity of interactions. This design enables more fine-grained relation structuring tailored to HOI scenarios.

[ref3] Reconstructing groups of people with hypergraph relational reasoning. ICCV, 2023.

[ref4] Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. CVPR, 2022.

Q6: Semantic Group for human and object features.

A6: We apologize for the confusion. We construct geometric groups exclusively for human and object proposals (see L162-172), and semantic groups exclusively for interaction proposals (see L192-197) initialized by computing the mean of human and object proposals. Additionally, in Sec. 4.3, we empirically observed that distinguishing human and object proposals is necessary for constructing geometric groups (see Lines 278–281, Table 5c).

We thank reviewer H8t4 for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: HOI-DET and V-COCO may not be ideal for group-based relationship modeling.

A1: Thanks for your thoughtful suggestion! Although HICO-DET and V-COCO contain abundant single human-object interaction instances (e.g., a photo of a person sitting on a chair), we find that there is still a significant proportion of samples that contain multiple human-object pairs via statistical analysis. Specifically, HICO-DET demonstrates an average of four interactions per image, while V-COCO contains three interactions per image. However, to address your concern, we have supplemented our experiments with results on NVI datasets [ref1], which encompasses complex interactions involving multiple humans (see Q2). Additionally, we strongly agree with your perspective and intend to develop a new visual relationship benchmark for group-based relationship modeling in our future work.

[ref1] Nonverbal Interaction Detection. ECCV, 2024.

Q2: Design or adopt a benchmark for group-level reasoning.

A2: We appreciate the insight! We conduct additional experiments on NVI datasets [ref2], which encompasses complex interactions involving multiple humans. These experiments incorporate the Nonverbal Interaction Detection (NVI-DET) task, designed to localize individual entities and their corresponding social groups while identifying the category of its nonverbal interaction. The experimental results on NVI val are as follows:

These results demonstrate that our method also generalizes well to more complex interaction scenarios. We will add Sec. 4.4 for the analysis of results on NVI dataset [ref2] in the revised manuscript.

[ref2] Nonverbal Interaction Detection. ECCV, 2024.

Q3: How often do self-matches occur, and should they be addressed?

A3: As you noted, self-match can occur in both geometric and semantic groups (see Fig. 4). However, it is difficult to quantify the frequency of self-matches in the groups formulated by GroupHOI: with a fixed number of proposals (see L217) in our method, self-matches increase as detectable entities in an image decrease, and decrease as detectable entities increase. This scenario is not considered necessary to address in our method for the following reasons:

• Detection boxes typically exhibit deviations from ground truth ones; therefore, the overlapping detection boxes corresponding to the same entity can mutually complement their information by self-matches.
• For inputs with limited detectable entities (e.g., images only containing single human-object interaction), the groups formulated by GroupHOI are dominated by self-matches, which enables our method to maintain certain performance (albeit weaker) under such constraints.

Q4: Why adopt fixed neighbors？Is it possible to train an estimator for neighbor count assessment?

A4: My apologies for the misunderstanding. Both our geometry-aware local transformer and semantic-aware interaction decoder employ a three-layer Transformer decoder architecture. For each decoder layer, we construct a new geometric/semantic group. This strategy enables the total number of neighbors for each proposal (i.e., human, object, and interaction proposal) ranging from $K_g$ to $K_g\times3$ (4 to 12) for the geometric group and from $K_s$ to $K_s\times3$ (2 to 6) for the semantic group. Notably, over 96% of samples in V-COCO and over 94% in HICO-DET contain no more than 12 entities, so geometric group of 4 to 12 can cover the majority of scenarios. Meanwhile, the size of semantic group used in the paper also aligns with [ref3]'s finding, which underscores that the majority of the interactive dialogue involving 10 or more participants is produced by only the top 4-5 contributors. Hence, we deem it unnecessary to train an estimator for neighbor count assessment, as this could introduce instability or optimization challenges. The above discussion will be incorporated into Sec. 3.5 and 4.3.

[ref3] Group discussion as interactive dialogue or as serial monologue: The influence of group size. Psychological science, 2000.

Q5: How is the proximity score trained and supervised?

A5: Our apologies for the lack of clarity. The linear layer, used to compute the proximity score by fusing the centroid distance and IoU between two bounding boxes, is not explicitly supervised but is trained in an end-to-end manner based on the final HOI loss. To clarify, the manuscript will be revised in Sec. 3.3 as follows:

A simple linear layer is then employed to compute the proximity score $s_{i,j}$ based on the central distance $dis_{i,j}$ and the Intersection-over-Union $IoU_{i,j}$ between bounding boxes, without requiring additional supervision. This results in a proximity score matrix, where lower scores indicate closer spatial proximity. For each entity, we select the $K_g$ neighbors with the lowest scores to construct its geometric neighbor set $N^g_i$.

We thank reviewer oLM7 for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: The novelty needs further justification.

A1：We apologize for the confusion. In fact, our method is significantly different from previous two-stage graph-based HOI approaches in the following aspects:

1) Motivation

Previous two-stage HOI detection methods, such as DRG [ref1] , SCG [ref2] and SKGHOI [ref3], primarily construct graphs to perform the interaction reasoning between humans and objects. In contrast, our method constructs groups to explore the latent social attributes based on two visual attraction principles, enhancing feature representation of each proposal by aggregating information from its neighbors.

2) Formulation

Prior work constructs a complete bipartite graph between objects and humans with size $N_h \times N_o$, where $N_h$ and $N_o$ denote the number of object and human proposals, respectively. Our method constructs localized geometric and semantic graphs with sizes only $(N_h + N_o) \times K_g$ and $N_i \times K_s$, respectively, with $N_h$, $N_o$, and $N_i$ denoting the number of human, object, and interaction proposals, and $K_g$, $K_s$ representing the geometric and semantic group sizes. Notably, $K_g \ll N_h/N_o$ and $K_s \ll N_i$, highlighting the efficiency of our method.

3) Implementation

Prior two-stage methods relied on Graph Neural Network (GNN) architectures, employing iterative message passing via message functions to perform interaction reasoning. Our method can be directly integrated into one-stage frameworks. It defines neighbor sets defined by geometric proximity and semantic similarity, and employs simple yet effective operators (i.e., self-attention and max-pooling) to enhance the feature representations of human/object and interaction proposals.

In summary, our method fundamentally differs from prior graph-based methods in motivation, formulation, and implementation, which underscores the novelty of our work. We will revise the Sec. 2 of the manuscript to highlight our novelty.

[ref1] DRG: Dual Relation Graph for Human-Object Interaction Detection. ECCV, 2020.

[ref2] Spatially Conditioned Graphs for Detecting Human-Object Interactions. ICCV, 2021.

[ref3] SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection. ICDMW, 2023.

Q2: Weak link to the motivation of "social attributes in a group of people".

A2：Thank you for pointing this out. We acknowledge the potential ambiguity in our use of the term "social attributes", which we intended to denote collective behaviors observed in real-world interactions (i.e., multiple humans engaging in joint activities). Inspired by this, we approach relation modeling from a group perspective, conceptually grounded in the Gestalt principles of Proximity and Similarity [ref4], which describe the mechanisms underlying human visual organization. To operationalize these principles, we formulate geometric proximity groups and semantic similarity groups, and propose two modules (i.e., geometric-aware local transformer and semantic-aware interaction decoder) to integrate these groups into the HOI detection pipeline. To enhance clarity, the Abstract and Sec. 1 of the manuscript will be revised as follows:

Revision for Abstract:

However, current HOI methods primarily focus on pairwise relationships, overlooking the social attributes in a group of humans (i.e., multiple humans engaging in joint activities).

Revision for Sec. 1:

Despite achieving strong performance, these methods are still confined to predefined pairwise relations, leaving the inherent social attributes which exhibit collective patterns (i.e., multiple entities engaging in joint activities) unexplored.

we construct social groups guided by the Gestalt grouping principles of Proximity and Similarity: First, the geometric proximity principle, i.e., the tendency for individuals to form groups with those physically close to, is implemented by constructing geometric groups based on the central distance and Intersection over Union (IoU) of bounding boxes. Second, the semantic similarity principle, i.e., the tendency for individuals to affiliate with those sharing visual characteristics or behavioral patterns, is operationalized through building semantic groups of interaction proposals based on similarity of their feature embeddings.

[ref4] Ellis, W. D. (2013). A source book of Gestalt psychology. Routledge.

Q3: More recent sota works for comparison.

A3：Thanks for your suggestion. Per your request, we compare our model with more recent works published in the past three years.

For V-COCO, GroupHOI exhibits competitive performance but remains inferior to STIP [ref5], VIL [ref6], and CQL [ref7]. However, on HOI-DET, our method surpasses these approaches by a solid margin, especially for rare interactions. We attribute this to the enhanced capabilities of GroupHOI in more complex interaction scenarios, as evidenced by: HOI-DET contains 600 HOI interactions (vs. 293 in V-COCO), an average of 4 interactions (vs. 3 in V-COCO) and 6 entities per sample (vs. 4 in V-COCO). Furthermore, we also provide additional validation on NVI dataset [ref8], which contains an average of 5 social interactions involving a group of humans (from 2 to 12). The experimental results on NVI val also support this claim:

The results and discussion on V-COCO will be updated in Sec. 4.2. We will also add Sec. 4.4 for the analysis of results on NVI dataset [ref8] in the revised manuscript.

[ref5] Exploring structure-aware transformer over interaction proposals for human-object interaction detection. CVPR, 2022.

[ref6] Improving Human-Object Interaction Detection via Virtual Image Learning. ACMMM, 2023.

[ref7] Category Query Learning for Human-Object Interaction Classification. CVPR, 2023.

[ref8] Nonverbal Interaction Detection. ECCV, 2024.

Q4: More implementation details.

A4：Thanks for your suggestion. More implementation details and pseudo code can be found in Sec. A and Sec. D of the Supplementary Material, respectively. The Sec. A has been expanded to include additional details as follows:

To address the dimensional mismatch between positional embeddings (256) and the interaction decoder (768), we expand the positional embeddings by stacking them three times. During training, we replace the Focal Loss used in HOICLIP [ref9] with Asymmetric Loss [ref10] to better handle the long-tail distribution of HOI categories.

Futhermore, as stated in L17 and L245, we commit to making our code publicly available upon acceptance.

[ref9] Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. CVPR, 2023.

[ref10] Asymmetric loss for multi-label classification. ICCV, 2021.

Q5: Model efficiency analysis.

A5：Following your suggestion, we conducted a detailed comparison of GroupHOI with other HOI methods in terms of Parameters (Params), Floating Point Operations (FLOPs), and Frames Per Second (FPS) as follows:

where Full (DF) represents the results on Default evaluation setting of HICO-DET. Compared to HOICLIP [ref11], which shares the same network architecture as GroupHOI except for the GEO and SEM modules, GroupHOI introduces a slight increase in parameter count and a marginal reduction in FPS, yet achieves a substantial performance improvement, surpassing HOICLIP by +2.11 mAP on the Full setting of HICO-DET. Moreover, when compared with ViPLO [ref12], a graph-based method, under the same conditions, GroupHOI delivers superior accuracy while using fewer parameters and achieving faster inference, highlighting the efficiency of our model design (see Q1 Implementation). We will supplement Sec. 4.3 with this analysis in the revised manuscript.​

[ref11] Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. CVPR, 2023.

[ref12] Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. CVPR, 2023.

Q6: The results for V-COCO and HICO-Det are confused.

A6：We apologize for this mistake. It has been corrected in the revision.