MaskCLR: Multi-Level Contrastive Learning for Robust Skeletal Action Recognition

Thank you very much for these informative and constructive comments. We are delighted that you found our work valuable to the community in terms of making skeletal action recognition models more robust and more deployable.

We would like to clarify some points regarding our method.

P1: How much performance gain can we get if we simply use the upper path and make the ST-MSHA block larger?

R: Making ST-MSHA blocks larger could have multiple interpretations. This includes increasing the number of blocks (depth), increasing the input and output feature dimensions, increasing the number of heads, MLP ratio, etc. Such experimentation has already been conducted by the authors of MotionBERT [1] and is provided in their ablation and appendix sections. The reported accuracy is the one obtained after optimizing all of these parameters to obtain the highest accuracies on the different datasets. We recognize the importance of this point because optimizing these parameters does indeed significantly improve the model performance. Yet, we further improve the model performance using our training approach. Moreover, we boost the model robustness to perturbations such as occlusion and noisy and we improve generalization to pose estimators of different levels of quality.

P2: Fairer comparison should be added under the scenario of similar computational cost.

R: Thank you for raising this point. As we mention in the paper, our method does not change the number of parameters of the backbone transformer model and does not change the number of FLOPs. Therefore, our comparisons are by default under the same computational cost at test time. However, at training time, our approach requires an extra amount of computation which varies from a model to the other. For MotionBERT, we train for twice the number of epochs of the original model, first training using cross entropy loss then with the combined contrastive approach discussed in the paper. We note that, generally speaking, the training time depends on the backbone model, and is less important than the inference cost, which does not change under our approach.

P3: I will be very willing to raise the score if the authors can address my concerns.

R: Thank you very much for the kind gesture. We hope we addressed your concerns.

P4: Similar methods can also be applied to non-skeleton based methods. We know the skeleton-based methods are more robust than non-skeleton-based methods. But will that conclusion still hold when we add masking as mentioned in the paper?

R: Indeed, this is a very interesting line of thought. Especially, for RGB-based method transformer-based methods work similarity, which could inspire future work in this domain. We hope that our work would be valuable for the community in other domains as well such as image classification, object detection, and trajectory prediction.

[1] MotionBERT: A Unified Perspective on Learning Human Motion Representations. ICCV'23

We appreciate your feedback and comments which are paramount in the process of refining and polishing this work. We find your reviews very insightful and speak to the core of this work. Thank you very much for the time and effort you put into reading this paper carefully.

Below we discuss your concerns on our proposed framework.

P1: (a) The paper primarily centers its analysis and experiments on MotionBERT, leading to doubts about whether the identified issues are prevalent in various backbone architectures rooted in self-attention mechanisms.

R: Thank you for this important observation. The reason why we chose MotionBERT is that it’s the best-performing transformer-based model whose implementation is ‘publicly available’ at the time of the start of this work. MotionBERT core design is very similar to other transformer-based methods in the sense that it uses spatial and temporal MHSA blocks. How these blocks are ordered and fused together is different from method to another, but virtually in all of them you can conveniently compute the attention scores given to every keypoint and hence, you can apply the idea of targeted masking and MLCL quite easily. In support of this statement, and to address your concern about the applicability of our method to other transformer-based backbones, we conduct further experiments on two new transformers: the vanilla transformer [1] and STTFormer [2]. On the vanilla transformer, we apply SkeletonMAE framework (originally proposed for GCN-based methods) using the vanilla transformer backbone. We provide the results below:

Table 1: Performance improvement (%) compared to transformer-based backbones. Parentheses indicate improvement over the second best method.

As shown, MaskCLR consistently improves the performance of the other transformer-based methods, showing the generalization and efficacy of our approach. We conclude the repones to this point by emphasizing that our focus is not on MotionBERT as an architecture, but on the idea of self-attention which is used in virtually all transformer-based methods.

P2: (b) It is noteworthy that the proposed method can exclusively be applied to models built upon self-attention. This constraint, in turn, restricts the method's applicability and generalizability to non-transformer backbones.

R: That’s exactly correct. We present this approach for transformer-based methods as discussed in the paper. The reason is that we take advantage of the attention weights computed by the self-attention mechanism to empirically find the most important joints for targeted masking. However, it’s important to note that the same idea could be applied to other backbones with a slightly different approach. For example, for convolution-based methods such as GCNs and 3D-CNNs, different other techniques could be employed to find the most important joints. CAM [7] and Grad-CAM [8] are some quick and easy-to-apply methods that are proposed to find the saliency regions in images but could be slightly tweaked to be used in the skeleton domain. In doing so, the rest of our approach, i.e., TM and MLCL, could be conveniently applied for convolution-based methods. We leave this for future work.

[1] An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv'20

[2] Spatiotemporal tuples transformer for skeleton-based action recognition. arXiv'22

[3] Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. AAAI'22

[4] CrossCLR: Cross-modal contrastive learning for multi-modal video representations. ICCV'21

[5] SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training, ICCV'23

[6] Masked motion predictors are strong 3d action representation learners. ICCV'23

[7] Learning Deep Features for Discriminative Localization. CVPR'16

[8] Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. ICCV'17

We appreciate the time you invested and reading our paper and providing valuable insights that further contribute to the strength of our work. We are glad that you found our work good in terms of soundness, presentation, and contribution.

Below we address some of your concerns, which we take into consideration to update our paper.

P1: There is potential room for improvement in the writing skills. For example, in the first sentence of the Abstract, it may be more suitable to use "tend to focus on" instead of "focus on" when discussing existing methods.

R: Thank you very much. Indeed "tend to focus on" is a better expression in this case. We will update the paper accordingly.

P2: Given that the critique of previous methods largely stems from qualitative investigations presented in Fig. 1, it would be advantageous to acknowledge the potential influence of individual bias on these findings. Addressing this issue has the potential to enhance the overall quality of the paper.

R: We appreciate you raising this point. Indeed, it’s an important aspect that need to be discussed and clarified. In the introduction section, we provide some visualizations and observations to motivate our idea. However, ‘individual bias’ is not the basis on which we develop our work. In the appendix section, particularly in sections A2 and A4, we offer quantitative examination that further validates our observations by reporting the insertion and deletion scores, proposed in RISE [1]. More specifically, we conduct an in-depth experimentation to study the effect of having big dependency on a few set of joints on the performance of the model on the overall NTU60-XSub dataset. For example, in Figure 9 in the appendix, we can see that starting from standard accuracy of 92.8%, removing just 3/17 of the most important joints results in dropping the accuracy to 75.4% and removing 5/17 damages the performance further to 42.3%, which reflects the huge dependency on a few number of joints. In comparison, we see more stable performance in MaskCLR (figure 9 in the appendix), where removing 3/17 and 5/17 joints results in 90.2% and 85.4% in accuracy consecutively. Our motivation is therefore, not just based on qualitative inspection but a quantitative examination of the model using different attention layers and performance evaluations on the entire dataset.

P3: b. Activation-guided augmentation is a concept relatively familiar within the vision community. However, it would be valuable to provide a more direct and explicit explanation of this point. Moreover, it may be better to probe some related works in this field in Sec.2, such as [1-2].

R: Thank you for this valuable addition. We will take this into consideration in updating our manuscript.

P4: Consider replacing the term 'multi-level' in the title with 'activation-guided' or similar terminology to better align with the core theme of this manuscript.

R: This an interesting suggestion. The reason why we termed our technique ‘muli-level’ is because we use contrastive learning at different levels (sample- and class- levels) as described in the paper. We think that 'activation-guided' would be an accurate term to describe our targeted masking strategy since it is indeed ‘activation-guided.’

P5: it's important to point out that despite the reviewer's familiarity with supervised contrastive learning, there was still some confusion experienced until the end of Section 2.

R: Your feedback is quite valuable to us. We will revise the paper with more focus on section 2 and before.

[1] Rise: Randomized input sampling for explanation of black-box models. arXiv, 2018.

P4: (iii) The reviewer noticed that [B] also uses a very similar two pathways setup, the pipeline looks quite similar? (why it is not discussed and it seems that they also use the same dataset for evaluation, and the differences are (1) they use skeletal hypergraph whereas this work uses skeletal graph (2) they use standard classification loss whereas this work applies contrastive learning, so the novelty is the introduced targeted masking strategy?) Also it is a transformer-based model that uses hypergraph? The reviewer noticed that [B] also considers unactivated but informative joints?

R: The 3Mformer paper is a very interesting contribution and offers an excellent opportunity to be combined with our approach. We would like to highlight the differences between the contributions of 3Mformer and MaskCLR:

1. We would like to note that the contribution of 3Mformer is basically a new transformer architecture, unlike our proposed method which is a new framework for training existing transformer-based methods. Consequently, while they use a two-pathway approach in one part of their model, it is quite different from the one we propose in our work. In our framework, the two pathways share the same weights (basically the same model) with different inputs that effectively enrich the learning of information from the input skeletons. In 3Mformer, the two pathways have different weights but the same input (which is the opposite of what we propose.)
2. We propose to use the attention weights to guide the masking strategy. 3Mformer is employing a significantly different approach with no masking.
3. We propose a contrastive learning approach, as you indicated, which is quite different from 3Mformer.
4. It is not clear to us how 3Mformer considers the unactivated joints. While it is reasonable to assume that the usage of hypergraphs would cover more joints and higher order correlations, our work is different in the explicit targeting of the unactivated joints. To the best of our knowledge, our work is the first to take advantage of this technique for skeletal action recognition.

P5: (iv) Furthermore, [B] is based on a transformer architecture that utilizes hypergraphs. The question arises as to whether the proposed strategy can be seamlessly integrated into their transformer-based model.

R: The 3Mformer is based on HoT (Higher Order Transformers) which essentially adopts higher-order self-attention to compute attentions weights of joints and hyper-edges at different orders 1,2, …, r. Since the higher-order attention maps reflect the importance of each joint (or group of joints in a hyperedge), our framework can be incorporated on top of 3Mformer in specific and hypergraph-based transformers in general by the targeted masking of the most important joints. Although the implementation of 3Mformer is not available, we will make sure to include hypergraph-based methods as part of our future research and experiments.

P6: The justifications of design choice/principle and rationale are not properly illustrated. Why the model are arranged/designed in this way, and its links/relationships to related works are not provided and discussed.

R: We would like to reiterate that the design of the model itself is not the contribution of our work. Our contribution is the proposed framework of training the existing transformer-based models. For further details about the design choices of the model, please refer to the original paper of MotionBERT: https://arxiv.org/pdf/2210.06551.pdf

P7: (i) It is suggested to have a notation section for the maths symbols. Why the b in Eq (1) and (2) sometimes in bold face, it is quite confusing.

R: Thank you for highlighting this. We will take your feedback into consideration and fix some of the notations for easier readability.

P8: (ii) What is the relationship among, $\mathbf{A}$, $\mathbf{A}_{b}$, $\mathbf{A}_{\mathcal{S}}$, and $\mathbf{A}_{\mathcal{T}}$? Why $\mathbf{A} = \frac{1}{2} \cdot (\mathbf{A}_{\mathcal{S}} + \mathbf{A}_{\mathcal{T}})$, and '*' denotes multiplication? The design choice/principle and rationale are not properly illustrated.

R: Here, the idea is to average the spatial and temporal attention weights from the spatial and temporal MHSA blocks. $\mathbf{A}_{\mathcal{S}}$ is the spatial attention scores, $\mathbf{A}_{\mathcal{T}}$ is temporal attention scores, $\mathbf{A}_{b}$ refers to both spatial and temporal scores, $\mathbf{A}$ is the average of spatial and temporal scores. Thank you for pointing this out. While such definitions are all in the paper next to corresponding notations, we will consider having a notations section to further avoid any confusion.

P3: (ii) Why only choose MotionBERT? The reviewer noticed that recent works, e.g., [B] and [C], are not discussed in the paper. There are only 3 works referenced from 2023, why?

R: Thank you for this important observation. The reason why we chose MotionBERT is that it’s the best-performing transformer-based model whose implementation is ‘publicly available’ at the time. MotionBERT core design is very similar to other transformer-based methods in the sense that it uses spatial and temporal MHSA blocks. How these blocks are ordered and fused together is different from method to another, but virtually in all of them you can conveniently compute the attention scores given to every keypoint and hence, you can apply the idea of targeted masking and MLCL quite easily. In support of this statement, and to address your concern about the applicability of our method to other transformer-based backbones, we conduct further experiments on two other transformer backbones: the vanilla transformer [1] and STTFormer [2]. On the vanilla transformer, we apply SkeletonMAE framework (originally proposed for GCN-based methods). We could not, however, experiment with 3Mformer because it does not have an open-source implementation. We provide the results below:

Table 1: Performance improvement (%) compared to transformer-based backbones. Parentheses indicate improvement over the second best method sharing the same backbone.

As shown, MaskCLR consistently improves the performance of the other transformer-based methods, showing the generalization and efficacy of our approach. We conclude the repones to this point by emphasizing that our focus is not on MotionBERT as an architecture, but on the idea of self-attention which is used in virtually all transformer-based methods.

[1] An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv'20

[2] Spatiotemporal tuples transformer for skeleton-based action recognition. arXiv'22

[3] Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. AAAI'22

[4] CrossCLR: Cross-modal contrastive learning for multi-modal video representations. ICCV'21

[5] SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training, ICCV'23

[6] Masked motion predictors are strong 3d action representation learners. ICCV'23

Thank you for your precious time that you put in reading our paper carefully. We are delighted that you understood the core of our work, and you found our contributions technically sound and the experiments and ablations extensive and demonstrative of performance improvements.

We find your feedback deep and constructive, and below we address your concerns on our work:

P1: The novelty of this work is limited.

R: We respectfully disagree with you in this point. To the best of our knowledge, MaskCLR is the first work to tackle the issue of robustness in transformer-based methods, and the first to address the issue of generalization to pose estimators in all skeletal action recognition methods in general. Our work proposes a new masking technique that is guided by the attention weights from transformer models, which has empirically been shown to result in performance improvements on all benchmarks. This is the first work to highlight this observation and effectively take advantage of it with the MLCL approach to learn the high-level actions semantics instead of low-level joint variations. Finally, we validate our approach with plenty of experiments and ablation studies on three different benchmarks, with three different pose estimators, and under different forms of skeleton perturbations. Our method shows consistent improvements under almost all of the aforementioned cases, which demonstrates and efficacy and generalization of our idea.

P2: Transformer-based models are not thoroughly discussed and analyzed (which are closely related to the argument 'seamlessly incorporated into transformer-based models' as mentioned in abstract section. For example, [A].

R: We appreciate providing the reference for FG-STFormer. We believe it’s a relevant and recent transformer-based work that will further contribute to the comprehensiveness of our related works section. We do provide a discussion of the core idea of transformer-based methods in the methods section of our paper, which mainly targets Multi-Head Self-Attention (MHSA) that is used in virtually all transformer methods. The reason why we did not want to delve deeper in the specific design choices of different transformers is that this is not the main focus of our contribution. While our method mainly targets transformer-based methods, it is agnostic to the specific design choices of such method.

P7: The paper mentions a "class-dissociated feature space" as a result of the multi-level contrastive learning framework. Could the authors delve deeper into the tangible benefits of this feature space in comparison to more traditional feature spaces, especially in terms of model interpretability and robustness?

R: Thank you very much, indeed that’s a very good question. We summarize the benefits of a class-dissociated feature space as follows: 1) a better-clustered feature space contributes to having a better decision boundary between different classes. Hence, this leads to faster convergence by the classification head and better classification performance in terms of top-1 accuracy. 2) Improved Model Interpretability: The "class-dissociated feature space" resulting from the multi-level contrastive learning framework signifies a representation where features are disentangled across different classes. This disentanglement can lead to enhanced model interpretability as the learned features may be more semantically meaningful and class-specific. Traditional feature spaces may lack such disentanglement, making it challenging to attribute specific features to particular classes. 3) Enhanced Robustness: The disentanglement of features in the class-dissociated space can contribute to increased robustness. Traditional feature spaces might encode information in a more intertwined manner, making models susceptible to perturbations or noise in the input data. In contrast, a class-dissociated feature space may offer a more resilient representation, where changes in one class do not unduly affect the features relevant to other classes. This characteristic can improve the model's robustness, especially when dealing with noisy or ambiguous data.

P8: Given the paper's focus on robustness to perturbations, could the authors provide insights into specific types of perturbations where the model excels and where it might face challenges?

R: According to our experiments, our model performs well under spatial noise (Fig. 3), spatiotemporal noise (Fig. 3), joint masking (Fig. 4), part masking (Fig. 4), Targeted masking (Fig. 9), and shifted joints (Fig. 10). Semi-optimal performance is achieved when evaluating under less than 30% of last frames from the input skeleton sequence.

P9: 3.The robustness of MaskCLR, especially against varying degrees of noise, is paramount for real-world applicability. How does the model's performance degrade or vary with incremental noise levels in the input data?

R: As shown in Figures 3, 4, 9, and 10, we evaluate MaskCLR as well as previous SOTA methods under varying degrees of noise. While the performance of all methods rapidly drops with more noise, MaskCLR shows the lowest drop in accuracy. We extend these evaluations with perturbing skeletons from different pose estimators, on which MaskCLR consistently outperforms the previous methods (see Table 4)

Thank you for taking the time to read our paper carefully, and for providing valuable feedback. We deeply appreciate your time and efforts, which will greatly contribute to improving this paper.

P1: The paper's comparative analysis with existing random masking methodologies seems limited in depth, particularly when referencing works like Zhu et al., 2023 and Lin et al., 2023; A more detailed juxtaposition detailing operational and performance nuances would be insightful

R: We are not sure what you mean in this point. It would be greatly appreciated if you can clarify specific analysis that can contribute to evaluating the masking strategies. We do not only compare against random masking. In the appendix section A4 we study the effect on performance vs the targeted masking of different numbers of joints. The results confirm the validity of our approach in diluting the focus on a small number of joints, exploiting the information carried in other joints.

P2: Another constraining factor is the exclusive reliance on attention weights from the final ST blocks, with Chefer et al., 2021 suggesting potential benefits from multi-layer attention scores.

R: Indeed, we mention this aspect as part of the limitations section of the paper. Certainly, more sophisticated saliency methods could be adopted in our framework. For simplicity, and for validating the efficacy of the approach, we experiment with taking the attention weights directly from the final MHSA block. In the Appendix section A2, we study the effect of using the attention weights from different layers and we find that the last MHSA block attention weights are indeed the best to use. We leave the more sophisticated saliency techniques to future work.

P3: Furthermore, while the paper claims robustness and generalization, expanded empirical validation across diverse datasets might solidify these assertions.

R: We appreciate your feedback on this point. We want to clarify that we conduct robustness and generalization experiments under three different cases: 1) perturbed skeletons 2) skeletons from different pose estimator between training and testing and 3) perturbed skeletons from different pose estimators. In each case, we report the results under the most popular action recognition datasets, NTU60 and NTU120, under varying degrees of noise, masking, and pose estimator quality levels. Our approach consistently improves the robustness of the backbone model and skeleton perturbations as well as generalization to different pose estimators. We provide even more experiments in the appendix. Kindly let us know if there are specific datasets or different validations that would further confirm the efficacy of our approach.

P4: The current exploration of hyperparameters, especially the effects of Lsc weight α and Lcc weight β, could be augmented by examining their combined effects.

R: Thank you for raising this point. The current model is tested with different values for α and β while applying the two losses separately. The search for better hyperparameters could indeed be improved by combining the losses under different weights and other parameters. We will conduct a more exhaustive search in our future work.

P5: A deeper theoretical exposition on choices made, especially regarding contrastive losses, could enhance comprehension.

R: Thank you for pointing this out. Our proposed contrastive learning approach is an extension on the previous works on this field, especially on the self-supervised methods that are described in related works section. Due to the space limitation, we cannot delve deep into the details of contrastive learning. However, given your feedback, we will consider extending our work with more discussion on the theoretical background as part of the appendix of our paper.

P6: Lastly, a rigorous analysis detailing the model's performance under varying noise conditions would strengthen claims about its robustness against noise.

R: We test our model under different types and degrees of noise from different pose estimators. As shown in section 4.3, our experiments include spatial Gaussian noise, spatiotemporal Gaussian noise, joint masking, part masking, and frame masking. Further, in the appendix section A4, we further evaluate our approach on targeted masking and shifted joints. The results empirically verify the efficacy of our method in improving robustness against different types of noise. We are open to providing further rigorous analysis under varying degrees of methods. We would greatly appreciate it if you point our specific studies or analysis that would further strengthen this work.