Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection

How NVI-DET differs from HOI-DET task?

The task studied in this paper, NVI-DET, is very similar to HOI-detection, except that latter detects <human, object, interaction> while this paper studies <human, group, interaction>. Although the authors discussed in this paper that the NVI-DET requires subtle human clues for interaction recognition, the same also applies to HOI-DET. To me, it is unclear how this task significantly differs from the existing HOI-DET task, which has been studied for several years in the previous literature.

We agree that NVI-DET shares structural similarities with human-object interaction detection (HOI-DET). However, Wei et al. [44] state that “Unlike HOI-DET that mainly focuses on recognizing actions between human-object pairs, NVI-DET aspires to comprehensively interpret the full spectrum of communicative nonverbal signals, whose patterns are generally more subtle, ambiguous, and involving multiple persons”. In this regard, NVI-DET requires hierarchical reasoning over multiple individuals and their group-level interactions, making it fundamentally more complex than HOI-DET.

Limited novelty

The proposed achitecture is also very similar to a previous work on HOI-DET, i.e., GEN-VLKT [28]. Both models applies the idea of DETR, and adopt two decoders, where one decoder for instance detection while the other for interaction detection&recognition. Although the utilization of human pose seems to be a new contribution to the NVI-DET task, the same idea has been extensively utilized in HOI-DET tasks [R2, 46]. As such, I think the novelty of this paper is quite limited. The authors could further elaborate and discuss on the difference between this work and existing HOI literacture, like why the human information in HVI-DET is different from that in HOI-DET.

As pointed out by the reviewer, DETR-based architectures with separate decoders for instance and interaction prediction have been widely used in human interaction understanding [22, 28, 44]. For instance, GEN-VLKT [28] and NVI-DEHR [44] form interaction queries by fusing human-object embeddings. In contrast, our method first detects individuals and then introduces separate group queries that attend to individual embeddings and dynamically aggregate relevant individuals into a group representation. This design enables joint reasoning of group composition and interaction recognition, allowing social groups to emerge based on inter-personal relations rather than directly predicted from image features. Furthermore, we enhance individual representations through a part-aware representation learning module supervised by pose guidance, which enables more fine-grained reasoning over nonverbal cues. Unlike HOI-DET methods using human pose as an explicit input during both training and inference [R2, 46], our work leverages human pose only for training as a privileged information to learn part-aware representations. This design improves both generalizability and efficiency, as the model does not depend on pose estimators at inference. We appreciate the reviewer's suggestion and will cite Wan et al. [R2] in addition to the already-cited Wu et al. [46].

Limitation of the closed-world setting and the use of LLMs

The proposed method can only handle closed-world settings, where interactions are unseen during training and not recognizable during inference. However, this is already achievable in a previous work [28]. In the era of various pretrained and large models (e.g., CLIP, various MLLMs), it constitutes a deficiency of the model to be able to handle seen interactions only. This can be somehow solved by combining a visual understanding model with modern LLMs, but can not be handled with the model proposed in this paper.

We appreciate the reviewer’s insight on extending our work towards open-world settings and the potential of leveraging MLLMs. As noted by the reviewer, VLMs and MLLMs have shown impressive performance across various tasks. However, for complex tasks such as NVI-DET and HOI-DET—which require pair-wise detection and interaction recognition—the performance of MLLMs without fine-tuning remains limited. To empirically examine this, we utilized a recent MLLM, LLaVA-1.6-vicuna-7B [R4], on the NVI test set. We first prompted the model to directly output <individual, group, interaction> triplets, but this did not yield meaningful results. To support LLaVA, we provided ground-truth group bounding boxes and cropped the image accordingly before querying the model to identify fine-grained social interactions. Even under this favorable setup, LLaVA achieved only 31.88 AR, which is far lower than our method (Table R2). This suggests that even in a closed-world setting, MLLMs struggle with fine-grained reasoning without task-specific tuning. We acknowledge that our work is limited to the closed-world setting, and we agree that extending it to the open-world setting is an important future direction. We chose to focus on the closed-world setting because we believe that fine-grained social interaction detection still remains challenging and underexplored even in this setup. In the future work, we plan to explore incorporating VLMs to enable more generalizable social interaction understanding.

Table R2: Comparison with LLMs.

Ambiguity of the interactions

The ambiguity of the interactions, which may stem from the inherent difference between the image and text modalities.

We thank the reviewer for highlighting this important point. We agree that ambiguity is inherent in social interactions. This is also why NVI-DET adopted recall as the primary evaluation metric, since precision is less appropriate given the possibility of multiple plausible interactions that may not be exhaustively annotated in the dataset.

Privacy issue

Image with sufficient human information should be given to the model for recognition, which requires the human information.

We appreciate the reviewer’s concern regarding privacy. Recognizing fine-grained social interactions inevitably requires detailed human-centric visual cues, many of which can involve sensitive personal information. The creators of the NVI dataset [44] have already addressed this concern explicitly in their paper. As they stated: “There is a risk that someone could use it for malicious purposes, e.g., widespread surveillance, invasion of privacy, and potential abuse of personal information. Therefore, we strongly advocate for the well-intended application of the proposed method, while simultaneously underscoring the importance of employing the dataset in a responsible and ethical manner”. We will include a discussion on the ethical considerations in the revised manuscript.

[R2] Wan et al., Pose-aware Multi-level Feature Network for Human Object Interaction Detection, ICCV 2019.

[R4] Liu et al., Visual Instruction Tuning, NeurIPS 2023.

Thanks for the response.

• Under the question "How NVI-DET differs from HOI-DET task?", the authors claim that HVI focuses more on group-level activity, but this has already been studied in the literature of HOI-DET like [1].

• I think one merit of the proposed method is that it only requires human part during training, making it less cumbersome for inference. I suggest the authors to further hightlight this in the revision.

• I like the experiments which test LLaVA on the NVI test set. A more interesting experiments would be to perform SFT on LLaVA on the NVI training set and test it on the NVI test split. This may be out of the scope of this work since it may require proper feature engineering and remains unexplored. But I think it could be a promising future direction.

[1] Liu et al. Interactiveness Field of Human-Object Interactions. In CVPR 2022.

Clarify of Fig. 1

The visual representation of the embedding process, especially the part involving pose-based interaction, is not clearly aligned with the textual description in the manuscript.

Thank you for the constructive comment. As noted by the reviewer, the current figure incorrectly depicts the concatenation of the part-specific query $Q_P$ and the part embedding $E_P$, which may lead to confusion. In accordance with the textual explanation in the manuscript, the part embedding $E_P$ should be concatenated with the corresponding individual embedding $E_I$—not with the part-specific query $Q_P$. We will correct this misrepresentation in the revision to ensure consistency between the figure and text, thereby improving the overall clarity of the proposed bottom-up group reasoning framework.

Limitation of prior work

While the paragraph describes the strengths of prior work such as [44], it does not provide a critical analysis of its limitations.

We will revise the paragraph to more clearly contrast the proposed method with the limitations of prior work. Specifically, we will clarify that prior work [44] detects social groups directly without explicitly modeling inter-person relations—a limitation that becomes particularly problematic when detecting interactions like gaze, which often occur between spatially distant individuals. We will also point out that prior work relies on holistic person representations, thereby overlooking body part-level cues that are essential for distinguishing visually similar but semantically distinct interactions (e.g., wave vs. point, or mutual-gaze vs. gaze-following), as noted in the introduction (Line 40-52).

Handling co-occurring multiple interactions

It would be helpful if the authors could clarify how such cases are handled in the dataset annotation or during model inference. Specifically, how are multiple co-occurring interaction types defined, labeled, or distinguished—either in the data or by the algorithm?

Both the dataset annotation and our model are designed to handle the co-occurrence of multiple interactions. Following NVI-DEHR [44], we formulate interaction recognition as a multi-label classification task: for each detected group, the model outputs a probability vector over 22 interaction classes, indicating the presence of each interaction.
For instance, in the fourth row of Figure S3 in the supplementary material, one group simultaneously exhibits both gaze-aversion and handshake interactions. Moreover, the dataset allows one person to belong to multiple groups simultaneously, allowing overlapping social groups.

Impact of accuracy of the pose-guided mask

The potential issue is that the model does not clarify the expected range of interacting group members. If too many people are involved, pose estimation may suffer from occlusions, which could reduce the accuracy of the pose-guided mask.

We agree that the accuracy of the pose-guided mask could be degraded when many people are present in the scene, due to occlusions. We found that, in the training set, the average number of individuals involved in a group interaction is $2.02$, with some groups involving as many as $16$ people. However, our method is robust to such errors because the pose-guided mask provides soft supervision that guides the model to attend to the expected regions of each body part rather than enforcing exact localization. To verify the robustness of our method to pose errors, we perturb the outputs of the off-the-shelf pose estimator (as ground-truth pose annotations are not available). Specifically, for each keypoint, we apply random displacements $\Delta x$ and $\Delta y$ sampled from a uniform distribution in the range $[-\epsilon \cdot s, \epsilon \cdot s]$, where $\epsilon$ controls the magnitude of the perturbation and $s$ is the window size of the pose-guided mask (Line 214). Table R5 summarizes the results across varying perturbation levels. We observe that even with substantial noise, e.g., $\epsilon=2.0$, the performance drop is minor. These results indicate that our pose-guided supervision is robust to moderate keypoint localization errors, and does not require highly accurate keypoint estimation to remain effective.

Table R5: Impact of the pose-guided mask error.

Applicability to group activity understanding

In some tasks related to group activity recognition, there are several classic datasets, such as the Volleyball [R6] and Collective Activity Dataset [R7]. These datasets focus more on group actions—can they be considered as representations of social interactions?

We thank the reviewer for this insightful question. We agree that group activity can be considered as a form of social interaction. However, most group activity benchmarks—such as Volleyball [R6], CAD [R7], or Café [22]—focus on coarse-grained interactions (e.g., walking, queuing, fighting) rather than the fine-grained social interaction (e.g., gesture, gaze, facial expression) that our work primarily targets. Nevertheless, to further validate the applicability of our method beyond fine-grained interaction detection, we conducted additional experiments on Café [22], a recent benchmark that involves multiple group activities in complex scenes (Table R3). The results demonstrate that the proposed part-aware representation and bottom-up group reasoning are beneficial even in understanding group activities, demonstrating the broader applicability of our method.

Table R3: Comparison with the previous methods on Café detection-based setting.

Definition of the social interaction

We thank the reviewer for the helpful suggestion. We will include a formal definition of social interaction and cite the paper [R8] when it is first introduced in Section 1 as recommended.

Comparison of the training and/or inference time

We thank the reviewer for useful comments. We compare the inference time of our model and NVI-DEHR using $224 \times 224$ resolution inputs. Our method achieves an inference speed of 26.62ms, which is slightly faster than NVI-DEHR’s 27.07ms.

Discussion of the other loss weights

It would be helpful to also include a discussion of the other loss weights, such as $\lambda_\text{i}$, $\lambda_\text{c}$, and $\lambda_\text{l}$.

We investigated the effect of the other loss coefficients $\lambda_\text{i}$, $\lambda_\text{c}$ and $\lambda_\text{l}$ as suggested. Table R6 reports the performance under various combinations of these weights. First, for $\lambda_\text{i}$, comparing (1), (2), and (3), we find that increasing $\lambda_\text{i}$ to $2.0$ leads the model to overly emphasize person detection, resulting in a noticeable performance drop. Next, varying $\lambda_\text{c}$, a comparison of (1), (4), and (5) shows that too small a value degrades performance, with the best result at $2.0$, while $5.0$ causes a slight drop. Finally, for $\lambda_\text{l}$, comparing (1), (6), and (7) reveals that the performance remains relatively stable across its values, but $\lambda_\text{l}=1.0$ leads to the best overall results. As stated in Section 4.2 and Table S1, we select configuration (1) as our final setting.

Table R6: Impact of the other loss coefficients.

[R6] Ibrahim et al., A hierarchical deep temporal model for group activity recognition, CVPR 2016.

[R7] Choi et al., What are they doing?: Collective activity classification using spatio-temporal relationship among people, ICCVW 2009.

[R8] Cristani et al., Social interaction discovery by statistical analysis of F-formations, BMVC 2011.

Dear reviewer iWA3,

Thank you again for the time and effort you have dedicated to reviewing our submission. We have carefully addressed your comments in our rebuttal, and we would greatly appreciate it if you could share any additional thoughts or let us know if there are any remaining concerns. Your feedback would be invaluable in helping us improve our work. Thank you again for your thoughtful review and consideration.

Best regards,

Authors

Thanks for your response. My concerns have been well addressed.

Contribution to the domain of machine learning (technical novelty)

The technical novelty is limited. The proposed architecture is a very typical architecture, feature extraction, individual and group modelling, and association module. It is not clear the main contribution to the machine learning domain. Part-aware representation seem to be only contribution, which is incremental.

We agree that our framework is based on a commonly used architecture. However, we argue that the significance of our work stems from how we leverage these standard components to address the unique challenges of fine-grained social interaction detection, which remains underexplored despite its importance. Our key contribution lies in introducing part-aware representations that are learned using human pose labels as privileged information [R1]. In other words, our model leverages human pose only for training, unlike HOI-DET methods [R2, 46] that explicitly use human pose as an additional input for both training and inference. This design allows the model to benefit from fine-grained supervision while maintaining efficient inference without requiring additional inputs. To the best of our knowledge, such an approach using pose-guided supervision has not been studied in the literature of fine-grained social interaction detection.

Why are focusing on images?

We focus on images in this work because existing research on interaction detection—such as NVI-DET and HOI-DET—is predominantly conducted in the image domain, with widely used benchmarks providing image-level annotations. We however agree that extending this line of work to the video domain remains an important future direction and could further enhance the applicability of these methods.

Comparison with VLMs or LLMs

Today LLMs are pretty good at reasoning about images, but they are failing on videos. How do the authors compare their method with LLM-based reasoning methods?

We thank the reviewer for pointing out this interesting direction. Recent studies in HOI-DET [25,28] have indeed explored the use of VLMs or LLMs for interaction understanding. However, these models are rarely used for direct reasoning about visual inputs, and are instead often employed in an auxiliary manner. For example, Lei et al. [25] leverage LLMs to describe the status of human body parts given an HOI label, and utilize VLMs to enhance interaction recognition. To verify the effectiveness of VLMs and LLMs, we conducted two additional experiments. First, we employ CLIP [R3] to learn specific body-parts using text embeddings derived from body-part names. To this end, text prompts are constructed in the form of "A photo of a person [body part]". As summarized in Table R1, our method consistently outperformed this CLIP-guided variant, particularly in mR@25 and average recall (AR), suggesting that CLIP guidance is less effective in providing fine-grained spatial cues than pose estimators. We attribute this performance gap to the relatively weak spatial reasoning capabilities of current VLMs, which are primarily trained via the image-text contrastive learning. Second, we evaluated LLaVA-1.6-vicuna-7B [R4], a recent MLLMs, on the NVI test set. To support LLaVA, we provided ground-truth group bounding boxes and cropped the image accordingly before querying the model to identify interactions. Despite this favorable setup, LLaVA achieved only 31.88 AR, which is far lower than our method (Table R2). We believe this is because MLLMs are not trained specifically for fine-grained spatial reasoning and may not possess the ability to model group-level interactions. Nevertheless, we agree that the zero-shot capability of such models offer promising directions for future research.

Table R1: Comparison with the pose supervision and VLM supervision.

Table R2: Comparison with LLMs.

Can this method be applied to other datasets and downstream tasks? (+stronger baselines)

We thank the reviewer for the constructive suggestions. We initially considered GP-Static [R5] as suggested, but found that it is not publicly available. As an alternative, we have conducted experiments on Caf'e [22], a more recent and challenging benchmark for group activity detection that emphasizes multi-group scenarios, similar to JRDB-Act [10]. As shown in Table R3, our model outperforms stronger baselines such as Caf'e-base [22], JRDB-base [10], and HGC [39] in terms of both Group mAP and Outlier mIoU. These results demonstrate the effectiveness of our method in the related task and further suggest that part-aware representations and bottom-up group reasoning not only benefit fine-grained social interaction detection, but also contribute to group activity understanding tasks.

Table R3: Comparison with the previous methods on Café detection-based setting.

Analysis on the impact of data imbalance

One of the problems with the group activity datasets is the data imbalance problem. It would be good to see the results over different classes to see if the method is performing equally well and where are the limitations.

We appreciate the reviewer’s suggestion to further analyze the impact of data imbalance. Table R4 compares our method and NVI-DEHR [44] in a class-wise manner. Among the 22 interaction classes, the two most underrepresented categories, beckon and palmout, exhibit notably low performance for both methods due to their rarity. Nonetheless, our method achieves notably higher recall (66.67% and 22.22%) compared to NVI-DEHR (16.67% and 11.11%), suggesting improved robustness to data imbalance. We attribute this improvement to the use of part-aware representations, which allow the model to explicitly focus on the specific body parts. Unlike holistic representations that may overlook infrequent combinations of body parts, part-aware modeling enables better generalization to individual body parts and consequently leads to more reliable detection of fine-grained interactions such as beckon and palmout, even under limited training examples for these classes.

Table R4: Class-wise recall (%) comparison between Ours and NVI-DEHR.

Unclear output of the method

It is not clear, until the experimental results, the output of the method. In other words, what is the goal of social reasoning? However, if the problem definition can be introduced clearly in the beginning, it would be helpful for the readers.

We thank the reviewer for this helpful comment. To improve clarity, we will include a figure in the introduction section that illustrates the target task and its expected output. In addition, we will revise the description in Line 31–39 and Section 3.1 to make the task setup and the structure of the output more transparent and accessible to future readers.

[R1] Lopez-Paz et al., Unifying distillation and privileged information, ICLR 2016.

[R2] Wan et al., Pose-aware Multi-level Feature Network for Human Object Interaction Detection, ICCV 2019.

[R3] Radford et al., Learning Transferable Visual Models from Natural Language Supervision, ICML 2021.

[R4] Liu et al., Visual Instruction Tuning, NeurIPS 2023.

[R5] Chang et al., Gaze pattern recognition in dyadic communication, ETRA 2023.