Interaction-centric Hypersphere Reasoning for Multi-person Video HOI Recognition

• Capturing global context information. All of the CNN-based works listed Section 2 "Related Works-HOI detection" lacks the ability to capture global context information due to the inherent local nature of the CNN structure.

• Representational accuracy. Representational accuracy indicate the quality of learned representations of entities in the scence. Euclidean distance do not introduce structure bias of HOI into the model as hypersphere does (illustrated in the next question), ignoring valuable structure information of HOI for guiding predictions.

• The reason for choosing hyperspher.

  (1) The interaction-centric hypersphere is crafted to introduce a structure bias of HOI into our model. Specifically, this bias asserts that human-object pairs adhere to their respective interaction classes. Consequently, the model is compelled to make predictions within the confines of this structure bias. It is essential to clarify that the hypersphere serves merely as a means of visualizing this bias. Alternately, various other techniques may be employed for visualizing the structure bias. Hence, our paper's fundamental proposition is to instill the desired structure bias, emphasizing that hypersphere is just one illustrative manifestation of this broader concept.

  (2) The advantage of hypersphere over Euclidean distance is that the latter does not introduce such priors, ignoring valuable structure information of HOI for guiding predictions. In the revised paper, we conduct experiments of utilizing Euclidean distance by setting $\lambda=0$ in Equation (6). Results in Table 1, 2 and 3 (highlighed in red in the revised paper) show that utilizing Euclidean distance results in at least 7%, 10% and 3% $F_1$ score drop in MPHOI dataset, CAD-120 dataset and Bimanual Actions dataset, respectively. Thus, the effectiveness of the structure bias introduced by hypersphere have been validated. We also show the results below for better understanding:

  MPHOI	$F_1 @10$	$F_1 @25$	$F_1 @50$
  Ours	91.6$\pm$0.9	84.5$\pm$2.6	67.7$\pm$2.2
  Ours ($\lambda=0$)	81.2$\pm$0.7	74.8$\pm$4.2	53.2$\pm$0.3

  CAD-120	$F_1 @10$	$F_1 @25$	$F_1 @50$
  Ours	90.7$\pm$2.9	88.1$\pm$2.8	77.6$\pm$4.7
  Ours ($\lambda=0$)	72.0$\pm$4.4	65.0$\pm$6.9	48.6$\pm$6.3

  Bimanual Actions	$F_1 @10$	$F_1 @25$	$F_1 @50$
  Ours	85.0$\pm$2.5	82.9$\pm$2.9	74.2$\pm$4.3
  Ours ($\lambda=0$)	76.7$\pm$5.2	74.3$\pm$6.0	65.2$\pm$6.3

• Relations between classes. The relations between interaction classes are modeled by Interaction State Reasoner (ISR) module. ISR explicitly empowers the model to determine whether the current interaction should persist or transit to another interaction with a state interpolation module and a reasoner block shown in Figure 3.

• Multi-label and long-tail.

  (1) At each time step, there is only one sub-activity for each person. Therefore there is no multi-label problem in HOI recognition task.

  (2) We deal with long-tail problem with focal loss $\mathcal{L}_{cls}$, which is illustrated in Section 4.3.

• Temporal action segmentation. Temporal action localization or segmentation works usually focus on a single action of interest happening in the video, which is a different task compared to video HOI recognition, where multiple actions could happen in the input video.

Thanks for your comments. I would be glad to answer any additional questions if you have.

• Figure 2 and 3. We have added annotations for the letters in the figures.

• Follow 2G-GCN to extract feature. {${v_t}$}$_{t=1}^T$ denote video frames, whose features are extracted from 2G-GCN model. We use CLIP model to extract text features, not video frame features. The input of 2G-GCN is the same with our model, which is a video. The output is the human sub-activity class at each time step. $z_H$ and $z_O$ are extracted features of human and object from 2G-GCN, respectively.

• Experiment on VidHOI dataset. VidHOI is used for video HOI detection task, while our paper works on video HOI recognition, which is a different task. Consequently, the VidHOI dataset is not suitable for comparison in our study.

• Ablating hypersphere module. In the revised paper, we conduct an ablative analysis on the hypersphere module, substituting it with a conventional classifier implemented through a Multilayer Perceptron (MLP). The outcomes, as presented in Table 1, 2, and 3 (highlighted in red in the revised paper), reveal a notable decline of over 7% in the $F_1$ score within the MPHOI dataset when employing the traditional classifier. Likewise, in the CAD-120 dataset, the traditional classifier results in a substantial decrease of more than 11% in the $F_1$ score. Additionally, within the Bimanual Actions dataset, the traditional classifier induces a decline exceeding 2% in the $F_1$ score. These findings unanimously underscore the efficacy of the structural bias introduced by the hypersphere module. We also show the results below for better understanding:

  MPHOI	$F_1 @10$	$F_1 @25$	$F_1 @50$
  Ours	91.6$\pm$0.9	84.5$\pm$2.6	67.7$\pm$2.2
  Ours (Traditional Classifier)	84.6$\pm$7.2	74.7$\pm$10.5	54.6$\pm$13.7

  CAD-120	$F_1 @10$	$F_1 @25$	$F_1 @50$
  Ours	90.7$\pm$2.9	88.1$\pm$2.8	77.6$\pm$4.7
  Ours (Traditional Classifier)	79.5$\pm$11.0	73.9$\pm$11.4	56.6$\pm$12.5

  Bimanual Actions	$F_1 @10$	$F_1 @25$	$F_1 @50$
  Ours	85.0$\pm$2.5	82.9$\pm$2.9	74.2$\pm$4.3
  Ours (Traditional Classifier)	82.0$\pm$3.6	79.8$\pm$4.1	71.0$\pm$5.6

• Task definition.

  (1) As we mentioned in Section 4.1 ''Problem Formulation'' part in the paper, multi-person video HOI recognition means there are multiple persons in the scene and we aim to predict the temporal segmentation of each peron's sub-activity in that scene, indicating the person's interaction with the object.

  (2) Note that we do not aim to predict multi-person interaction. Rather, we predict human-object inteaction represented by human sub-activity classes.

  (3) It is important to underscore that multi-person HOI interaction stands apart from group action recognition [1][2]. Unlike the latter, which yields a singular action class for all individuals in the scene, multi-person HOI interaction goes a step further by prognosticating each person's distinct sub-activity in the given scene. This added granularity introduces a heightened level of complexity, surpassing the challenges inherent in group action recognition.

  [1] Azar, Sina Mokhtarzadeh, et al. "Convolutional relational machine for group activity recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

  [2] Gavrilyuk, Kirill, et al. "Actor-transformers for group activity recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

• Hypersphere.

  (1) Hypersphere indicates a high-dimensional sphere, indicating the boundary of a high-dimensional ball. We call "hypersphere" in this work because the feature dimensions in Equation (6) are high-dimensional vectors (larger than 3).

  (2) The interaction-centric hypersphere is crafted to introduce a structured bias of HOI into our model. Specifically, this bias asserts that human-object pairs adhere to their respective interaction classes. Consequently, the model is compelled to make predictions within the confines of this structured bias. It is essential to clarify that the hypersphere serves merely as a means of visualizing this bias. Alternately, various other techniques may be employed for visualizing the structure bias. Hence, our paper's fundamental proposition is to instill the desired structure bias, emphasizing that hypersphere is just one illustrative manifestation of this broader concept.

• Differnece of hypersphere from traditional classifier. As we explained in the above section, hypersphere is a way to visualize the introduced structured bias underlying HOI. This structure bias forces the model to make predictions within the confines of this structured bias. Compared to tranditional classifiers which usually employ multi-layer perceptron (MLP) to generate prediction logits, hypersphere introduce a strong prior into model prediction, which is abscent in traditional classifiers. Moreover, hypersphere do not introduce any training parameters like traditional classifiers, featuring higher computational efficiency.

• HOI manifold structure. In mathematics and machine learning, a "manifold structure" refers to a topological space that locally resembles Euclidean space. In other words, in the vicinity of any point on the manifold, the space behaves like a flat, n-dimensional Euclidean space. Manifolds can have various dimensions and shapes, and they are often used to represent complex data structures or patterns in high-dimensional spaces. In this work, HOI manifold structure specifically describes the topological relationship between human-object paris and interaction class, which is a structure bias that we introduce into our model.

• Complex HOI. Complex HOI incorporates multi-person HOI, meaning there would be interdependencies between human and object in single-person cases, or interdependencies between human and human under multi-person scenarios.

• Entities representations. Human and object are called "entity", so entity representations denote representation of human-object pair.

Thanks for your comments. I would be glad to answer any additional questions if you have.

• Experiment datasets. In our multi-person video HOI recognition experiment, we exclusively leverage the MPHOI dataset, as it stands as the sole dataset tailored for multi-person video HOI recognition at present. Aligning with the SOTA methodology, specifically 2G-GCN, which also centers on multi-person video HOI recognition, our experimental design mirrors theirs by utilizing the MPHOI dataset. Furthermore, 2G-GCN augments its investigation with two single-person datasets. Hence, our experimental settings maintain congruence with 2G-GCN to facilitate a meaningful comparative analysis.

• Why interaction-centric hypersphere helps? The interaction-centric hypersphere is crafted to introduce a structure bias of HOI into our model. Specifically, this bias asserts that human-object pairs adhere to their respective interaction classes. Consequently, the model is compelled to make predictions within the confines of this structured bias. It is essential to clarify that the hypersphere serves merely as a means of visualizing this bias. Alternately, various other techniques may be employed for visualizing the structure bias. Hence, our paper's fundamental proposition is to instill the desired structure bias, emphasizing that hypersphere is just one illustrative manifestation of this broader concept.

• Ablating CLIP features. We conduct ablation studies in the revised peper by utilizing random initialization for interaction features and context features, instead of using CLIP representations. Ablation study results in Table 1 (highlighted in red in the revised paper) show that random initialization results in around 10% in the $F_1$ scores in MPHOI dataset, while still higher than SOTA method 2G-GCN around 15%. Furthermore, Tables 2 and 3 showcase ablation study outcomes on the CAD-120 and Bimanual Actions datasets, indicating a drop of less than 3% in $F_1$ scores with random initialization. Notably, these scores remain competitive, if not superior, to the 2G-GCN model. Thus, our findings suggest that extracting CLIP features is not imperative for achieving strong performance. We also show the results below for better understanding:

  MPHOI	$F_1 @10$	$F_1 @25$	$F_1 @50$
  2G-GCN	68.6$\pm$10.4	60.8$\pm$10.3	45.2$\pm$6.5
  Ours	91.6$\pm$0.9	84.5$\pm$2.6	67.7$\pm$2.2
  Ours ($\Delta$ CLIP+BLIP)	80.0$\pm$6.7	73.0$\pm$9.9	55.6$\pm$7.4

  CAD-120	$F_1 @10$	$F_1 @25$	$F_1 @50$
  2G-GCN	89.5$\pm$1.6	87.1$\pm$1.8	76.2$\pm$2.8
  Ours	90.7$\pm$2.9	88.1$\pm$2.8	77.6$\pm$4.7
  Ours ($\Delta$ CLIP+BLIP)	89.4$\pm$2.3	85.5$\pm$3.9	74.9$\pm$5.7

  Bimanual Actions	$F_1 @10$	$F_1 @25$	$F_1 @50$
  2G-GCN	85.0$\pm$2.2	82.0$\pm$2.6	69.2$\pm$3.1
  Ours	85.0$\pm$2.5	82.9$\pm$2.9	74.2$\pm$4.3
  Ours ($\Delta$ CLIP+BLIP)	84.3$\pm$1.4	81.8$\pm$1.8	73.2$\pm$2.7

• Interaction prediction for each person. In the case of multi-person video HOI recognition, our model is able to predict interaction class for each person, as qualitative examples shown in Figure 5 and Figure 7 in the paper.

• Ablation studies with methods of HOI detecion. Our emphasis lies specifically on the HOI recognition task rather than HOI detection. These two tasks differs in both problem formulation and technical challenges. As articulated in Section 4.1 "Problem Formulation," our focus is on predicting the HOI class present in a given video frame. It is essential to clarify that our paper does not address HOI detection, which involves generating bounding box predictions for each individual and object in the scene, a distinct task from the scope of our work.

• Performacne on single-person videos. The relatively modest improvement of our model's performance on single-person videos compared to the SOTA method in contrast to its notable gains in multi-person videos can likely be attributed to the absence of a dedicated context fuser module for single-person scenarios, as illustrated in Figure 2. As evident from the outcomes presented in Table 1, the contextual fusion module plays a pivotal role in enhancing performance in multi-person settings. Consequently, we hypothesize that the less remarkable results observed in single-person instances are attributable to the lack of a context fuser module.

Thanks for your comments. I would be glad to answer any additional questions if you have.

• Comparison with other methods and ablation studies.: In our revised paper, we employ ablation studies to elucidate the impact of excluding prominent Vision-Language Models (VLMs) CLIP and BLIP (highlighted in red in the revised paper). In the ablation studies, we adopt random initialization for the interaction feature $z_I$ and context feature $z_C$. The results, detailed in Table 1, indicate approximately 10% in $F_1$ scores within the MPHOI dataset upon removing CLIP and BLIP. Despite this reduction, our scores remain superior to the SOTA method 2G-GCN by approximately 15% $F_1$ score. Moreover, Tables 2 and 3 reveal that in the CAD-120 and Bimanual Actions datasets, the omission of CLIP and BLIP yields a drop of less than 3% in the $F_1$ scores, still comparable to or surpassing the performance of the 2G-GCN model. These findings underscore that the comparison between our model and SOTA methods is equitable, as the reliance on large VLMs proves non-essential for achieving commendable performance. Rather, it is the intricately designed structure of our model that substantiates the substantial enhancement in performance. We also show the results below for better understanding:

  MPHOI	$F_1 @10$	$F_1 @25$	$F_1 @50$
  2G-GCN	68.6$\pm$10.4	60.8$\pm$10.3	45.2$\pm$6.5
  Ours	91.6$\pm$0.9	84.5$\pm$2.6	67.7$\pm$2.2
  Ours ($\Delta$ CLIP+BLIP)	80.0$\pm$6.7	73.0$\pm$9.9	55.6$\pm$7.4

  CAD-120	$F_1 @10$	$F_1 @25$	$F_1 @50$
  2G-GCN	89.5$\pm$1.6	87.1$\pm$1.8	76.2$\pm$2.8
  Ours	90.7$\pm$2.9	88.1$\pm$2.8	77.6$\pm$4.7
  Ours ($\Delta$ CLIP+BLIP)	89.4$\pm$2.3	85.5$\pm$3.9	74.9$\pm$5.7

  Bimanual Actions	$F_1 @10$	$F_1 @25$	$F_1 @50$
  2G-GCN	85.0$\pm$2.2	82.0$\pm$2.6	69.2$\pm$3.1
  Ours	85.0$\pm$2.5	82.9$\pm$2.9	74.2$\pm$4.3
  Ours ($\Delta$ CLIP+BLIP)	84.3$\pm$1.4	81.8$\pm$1.8	73.2$\pm$2.7

• Reference. Thanks for the suggestion, we have added the reference in the revised version.

Thanks for your comments. I would be glad to answer any additional questions if you have.