STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences

Please provide a comprehensive description about training process of each model (in Fig. 1) or adopt more metrics, as detailed in the Weakness 2.

In Fig. 1, we used the codes provided by each work to train for a single epoch and estimated the training time by multiplying it by the number of epochs provided by each work. In general:

• MAMP although uses a Transformer backbone and it is more costly to be trained, 90% of tokens are masked so the heavy encoder only receives 10% of the data and then a lightweight decoder has to reconstruct the motion for the remaining 90%, leading to a fast training time.
• STARS adds a bit of training time in the second stage (first stage is MAMP) by tuning the encoder by only 20 epochs. That’s the main reason why it is faster than other contrastive learning based methods since they train from scratch. More specifically, AimCLR and ActCLR train for 300 epochs, and CMD trains for 450 epochs.

To provide a thorough evaluation of generalizability, it is suggested to add more empirical comparison ...

&

How is the performance of other MAE-based approaches when applied to the proposed framework?

Since weights of a naive MAE model from MAMP work is not shared, we retrained the model from scratch using a naive MAE objective. We also trained STARS using MAE in the first stage. The tables below show their K-NN evaluation (K=1) and in few-shot settings.

K-NN evaluation:

Few-shot:

We also added these two tables in the revised supplementary material.

It is suggested to quantify the required number of epochs instead of using "a few epochs" in the contribution part.

We have mentioned the number of epochs on section 4.2 - pretraining. It needs a maximum of 20 epochs and the best representation is chosen based on K-NN (K=10) on validation data. We added this detail in the revised version. Thank you for mentioning it.

What is the reason for "use layer-wise learning rate decay Clark et al. (2020) to tune the second-half of the encoder parameters"?

Please refer to the answer provided to the third question of reviewer iumL.

The result "52.0*" of MAMP evaluated on PKU-II (Table 1) is not the same as the original paper, do different GPUs influence the performance?

Yes. The checkpoints from the PKU-II dataset is not provided by the MAMP authors. So we had to reproduce it by training using their provided code. However, probably due to differences in the GPUs used, we couldn’t reproduce the results reported in their paper.

The authors should provide comprehensive results in the ablation study.

Thank you for mentioning the missed ablations. We updated the Figure 4 of the paper to include the mentioned results. For the choice of K in K-NN accuracy, typically only K=1 is used in the literature. We also provided K=10 in our ablation as we used that for tuning the hyperparameters.

Thanks for the authors' response. However, the answer is not convincing for the following reasons:

"Backbone: The STARS framework focuses on refining the representations of a transformer-based model that has been pretrained using a masked autoencoder (MAE) approach. Unlike traditional backbones like ST-GCN, which lack a global receptive field, transformers are better suited for this purpose. Therefore, it wouldn’t make sense to use backbones like ST-GCN for STARS."

It seems that the STARS framework can only be used on transformer-based models with limited generality for other paradigms. There should be some backbones besides multi-scale GCN and its variants [1][2] that can utilize multi-scale receptive fields or global receptive fields to learn action representations. I suggest the authors to carefully check the literature and evaluate the framework on these models for a more thorough validation of method generality.

"Benchmarks: Most self-supervised learning methods in this field evaluate their performance on datasets like NTU-60, NTU-120, and PKU. While some methods skip certain evaluations, such as semi-supervised comparisons, STARS has been tested comprehensively, including these and a few-shot evaluation. This few-shot evaluation had been previously overlooked but reveals that MAE approaches underperform compared to contrastive learning methods. STARS, however, delivers a significant improvement in this scenario."

I acknowledge that few-shot evaluation is a relatively new evaluation setting in this work. However, adding new evaluation protocols does NOT mean improving the novelty of the proposed method. I agree with Reviewer qu9p and have concerns about the insufficient novelty of this work. For the generalizability of this method, I still think a more comprehensive evaluation should be conducted, including more action recognition benchmark datasets such as NW-UCLA and more up-to-date MAE-based approaches (as the authors seem to claim that all MAE approaches underperform).

I am not satisfied with the response of the authors to each weakness point (such as point 7), and I hope the authors can carefully read and revise the current paper point by point.

I provide a final rating of 5, which can not be improved considering the overall contributions, novelty, and generality. I’ll leave the final decision of this matter to AC.

Thanks.

References:

[1] Zhang et al. Context Aware Graph Convolution for Skeleton-Based Action Recognition. CVPR, 2020

[2] Chen et al. Multi-scale spatial temporal graph convolutional network for skeleton-based action recognition. AAAI, 2021.

What is the difference between the core idea in this paper and the MAE-CT [1]?

Similar to MAE-CT, our approach is also a sequential approach that uses contrastive learning after MAE with the following differences:

• The MAE-CT model is trained in three stages. In the second stage, the encoder is frozen, and only the NNCLR head is fine-tuned. In the third stage, both the backbone and NNCLR head are trained together. One motivation for this staged approach is the idea that the NNCLR head’s random weights could interfere with the representation quality by mapping the features into a random space, disrupting the learned structure. However, our findings challenge this assumption. The t-SNE visualization below demonstrates that even with random NNCLR head weights, the cluster structure in the encoder’s output space remains intact in the new representation space. Furthermore, we observed with STARS-3stage that replicating this three-stage process not only increases training time but also leads to a drop in final accuracy. Thank you for your question. We added the explanation in supplementary material. Please refer to Figure 6 in supplementary material to see the comparison.
• MAE-CT applies data augmentation during pretraining, which works well for image data where transformations like color jittering or small rotations do not interfere with recognition. However, for action recognition, data augmentation can be risky, depending on the actions being distinguished in the dataset. For example, time flipping could make the model unable to distinguish between actions like standing up and sitting down, while spatial flipping could make it unable to tell the difference between raising the right hand and the left. To avoid these issues, STARS does not use any data augmentation.

Another key contribution of this paper is highlighting a limitation in masked autoencoder methods like MAMP. While these methods often outperform contrastive learning approaches on many benchmarks, they fall short in few-shot settings, a scenario previously overlooked. We expose this gap and address it with a hybrid method that includes a few additional tuning steps during pretraining.

Some state-of-the-art methods are not compared, e.g., PCM3 [2], UmURL [3]. The author should supply them for a comprehensive comparison.

Thank you for mentioning them. We included them in the revised manuscript.

Compared to the previous studies, the training time is decreased. However, the memory requirements look like they are increasing; 4 A40 GPUs are needed. Meanwhile, the queue size can influence this method's computation overhead. The author should supply the overhead comparison or other metrics to demonstrate the effectiveness of their method.

Thank you for raising this concern. We used 4 A40 GPUs not because of the memory constraint but to leverage the PyTorch DDP for a faster pretraining in a limited time period as there are a lot of evaluations required to be done to prove the efficiency of the proposed method. Memory constraint as a computation overhead is a good concern that is not discussed in our submission . Here is the table we provided for the comparison, evaluated by measuring the memory consumed with a single input (Batch size = 1). Generally, MAMP requires more memory since it’s a heavy transformer-based approach. However they reduce this cost by passing 10% of tokens to the encoder and reconstruct the mask with the lightweight decoder. STARS use all the tokens and adds queues for contrastive learning, resulting in an increase in memory usage. AimCLR and ActCLR GCNs instead require much less memory. CMD uses three encoders in joint, motion, and bone at the same time and is GRU based, making it using the most memory usage. We also included this table in the revised supplementary material.

Compared to some other contrastive learning methods (such as AimCLR, CMD), the STARS method only relies on single-view sequences for operation and does not use two different augmented views....

In Tables 1 and 3, STARS operates as a single-stream model, meaning it uses only joint coordinates as input throughout training and inference. In contrast, other models employ a three-stream approach. This involves: (1) training on joint coordinates, (2) training on bone information (the difference between adjacent joints like the wrist and elbow), and (3) training on velocity (the difference in joint positions between consecutive frames). At inference, these models take a skeleton sequence as input, compute the bone and velocity representations, and feed each into the three pretrained models. The outputs from these models are then combined using a weighted average, with weights chosen heuristically for optimal accuracy. STARS, however, simplifies this process by using a single encoder at inference that directly processes the joint coordinates, producing the final output without the need for multiple models or averaged outputs.

However, in cross-view evaluation, the term "view" has a distinct meaning. It indicates that the same subject has repeated the same action in another recording from a camera positioned differently within the room. This setup presents challenges for video-based action recognition methods, which are sensitive to changes in viewpoint. For skeleton-based action recognition, however, all approaches apply a preprocessing step called canonicalization, aligning the torso with one of the XYZ axes in the first frame. This alignment minimizes the impact of viewpoint changes, so the model output remains consistent, and all models generally achieve higher scores in cross-view evaluations compared to cross-subject evaluations

As shown in the experimental results of Table 1, when the pre-trained and fine-tuned encoders are on a limited dataset, the difference in effect between the STARS method and other mask prediction methods (such as Skeleton MAE, MAMP) is not obvious, indicating that its generalization capability on small datasets needs further improvement.

In the STARS framework, contrastive learning is used to enhance model generalization by pulling the representations of an anchor closer to its corresponding positive sample to reduce the distance while pushing other samples away. When the dataset size is limited, there are fewer points (sequence samples) in the model's representational space, making it more likely for the model to select a sample from a different action as the closest sample and applying contrastive learning. This can negatively affect the model’s performance, preventing it from improving much in these cases. Although leveraging approaches like data augmentation could help improve performance, we chose not to use it, as it can negatively impact some scenarios (see the first answer to reviewer zBeL for further details). As a result, the proposed STARS refinement method is most effective when the dataset size is not limited.

The method uses some tricks in the ablation experiments, such as hierarchical learning rate decay (Formula 7); in addition, the Backbone is not aligned with other methods (Table 3), making it difficult to determine whether these tricks have brought benefits.

The idea behind hierarchical learning rate decay comes from the partial tuning approach described in the MAE paper (see Figure 9 in [3]). The MAE findings show that the early layers of a vision transformer already learn robust, general features during pretraining. When fine-tuning with labeled data, adjusting these early layers does not add much benefit. To make the most of the pretrained knowledge in these layers, we applied hierarchical learning rate decay, which prioritizes tuning the final layers. This reduces the risk of overwriting the foundational features learned in the initial training stage. Thank you for pointing this out; we will be sure to clarify this in the final manuscript. The choice of backbone is essential to the success of our method. Without the second training stage, the initial stage alone (the MAMP baseline) already outperforms other methods that use a different backbone. Our approach builds on this by introducing a refinement stage with a few additional fine-tuning epochs to further enhance MAMP’s performance. We also found that while MAMP exceeds the performance of other backbones, it falls short in few-shot scenarios that actions are not seen in pretraining (see Table 5 in the paper). Our refinement approach combines the strengths of both methods, leading to a substantial improvement in few-shot performance.

[3] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).

Dear Reviewer iumL,

Thank you once again for your valuable time and thoughtful feedback on our paper! As the discussion period comes to an end, we would greatly appreciate your thoughts on our responses. If there are any remaining concerns or unclear points, we’d be happy to address them further.

Best regards,

The Authors

Prior work: While simplicity is the strength for this paper. It proposes a combination of two existing approaches. The authors must make it clear if the two stages of training have any differences from the original approaches....

In the first stage of training, we follow the exact approach used in MAMP, using the pretrained model checkpoints provided by the authors. However, our second stage diverges from traditional contrastive learning methods: Typically, contrastive learning relies heavily on data augmentation, which involves creating multiple augmented versions of a sequence as positive samples, with other samples in the batch or queue serving as negatives. While effective, this augmentation approach can inadvertently harm action recognition. For instance, certain augmentations, like time flipping, may be beneficial in some benchmarks but could hinder the model’s ability to differentiate between actions such as standing up versus sitting down, as it becomes invariant to temporal changes. Similarly, spatial flipping can reduce the model’s sensitivity to directional actions like raising the left hand versus the right. However, our contrastive tuning uses nearest-neighbor contrastive learning instead. Given that the MAMP-pretrained model already captures reliable features, we simply select the closest sample in the queue as the positive sample and bring it closer in the representational space, while pushing other samples away. This tuning step has three key benefits:

1. For actions where MAMP features are dispersed, tuning helps form well-defined clusters, as shown with the “hand-waving” example in Figure 3.
2. It eliminates the need for hand-crafted augmentations, making it adaptable across diverse datasets without the risk of augmentations that may harm performance (e.g., see the shearing and axis masking results in the table 7 of the paper).
3. It requires minimal additional training time, making it a practical refinement step for any MAE-based method, as illustrated in Figure 1. As for the reason why we rely on this sequential approach instead of similar hybrid approaches proposed in Section 2.2, it is mainly because of the high training costs associated with contrastive learning methods. Running these methods in parallel would require: 1) Data augmentation, because when training from scratch, relying on nearest samples as positive pairs is not effective—they are randomly distributed and do not always yield good results. 2) A significant increase in training time. As Figure 1 illustrates, contrastive learning methods take much longer to train than MAMP. This is partly due to the need for multiple forward passes on each sequence, each with different augmentations for positive samples. In contrast, with the sequential approach, we only need to fine-tune the second stage with a minimal number of epochs, making it a lightweight and practical way to refine the learned representation.

How does training with NNCLR loss together with MAMP compare with the proposed approach ? What happens if the final stage employs both MAMP and NNCLR ?

That is a good question. The table below shows the K-NN evaluation (K=1) of STARS vs the variant where the second stage uses both MAMP and NNCLR. STARS outperforms this approach. The table is also added into supplementary material of the paper.

Experiments; Table 6 - why k=10 ? Table 2 uses k=1 making it difficult to compare. It would be interesting to have the same number of clusters (or both settings) to make comparisons easier.

We originally used k=10 since we used that to evaluate how effective our model is throughout the training, and used that for hyperparameter tuning. But later noticed that other methods used k=1 so we used that for comparison. Here we also provide the updated result for table 6 where k=1 is used.

Updated table 6:

Since NNCLR was shown to be very effective post MAE training, do you have any baselines which uses NNCLR alone and compare it with MAMP and the proposed approach?

That is again a wonderful idea for an ablation study. In general, since the used NNCLR method does not rely on data augmentation and uses the sample with the least distance in the encoder’s representational space as the positive sample, it is tricky to tune an NNCLR from scratch since initially in the training because of the random weights the encoder has, the model maps the input sequence into a vector in output that nearest samples are not necessarily the samples with the same action. As a result, using the nearest samples as positive samples in contrastive loss is not a good heuristic. Table below shows that on K-NN evaluation, once we train the model from scratch using 300 epochs, it cannot distinguish the actions well. The table is also added into supplementary material of the paper.

Table 5: How sensitive is the approach to different runs. How many times was this experiment repeated? Why do we see a drop from 1-shot to 2-shot ?

Following the approach in MotionBERT, we perform k-shot evaluation by measuring the similarity between each test sequence and all sequences in a set of unseen actions (i.e., actions that the model has not encountered during pretraining but for which we know the labels). Here is how it works: For each test sequence, we compute its distance from all other samples in the unseen set and select the k closest samples. Then, we assign the test sequence the label that appears most frequently among these k nearest neighbors (using majority voting). For example, let’s consider a 5-shot evaluation with three unseen actions, A, B, and C, each having 100 samples. For each test sample among these 300 samples, we compare it with the remaining 299 samples, identify the 5 nearest samples, and assign the label based on the majority action among these 5. In 2-shot evaluation, however, there can be cases where the two closest samples belong to different actions. When this happens, the test sample’s label is assigned randomly between the two, which can lead to a drop in accuracy. This accuracy drop is a useful indicator of how well-separated the clusters are in the representational space: a more substantial drop means that action clusters are less distinct, causing samples from different actions to be grouped together. Therefore, 2-shot evaluation is a good measure of the cluster separation quality for the 60 new unseen actions.

Do you use the same feature for linear evaluation, fine-tuning and kNN experiments?

In both linear evaluation and k-NN experiments, we use the same features, which are extracted from the model backbone after pretraining without any access to actual labels. However, in fine-tuning, the features differ because the model backbone is further trained with supervised learning, where it sees the actual labels. This access to label information in fine-tuning provides a stronger signal for updating the backbone weights, resulting in improved features. As shown in Table 3, there is not a significant difference in performance between STARS and MAMP (the baseline). This suggests that while the proposed refinement technique in STARS enhances the representational space for unsupervised evaluations like linear probing and k-NN (where labels remain unseen by the backbone), the choice of STARS or MAMP as the starting point matters less in fully supervised fine-tuning. Nevertheless, both STARS and MAMP lead to much better results than starting with randomly initialized weights.

Typo: L310 ViT?

That is a good point. Although skeleton sequences differ from visual perceptions (i.e. RGB images), we used the term ViT following the actual MAMP paper (See section 4.2 of [2]). That is because it follows the same design principle similar to Vanilla Vision Transformer. Instead of dividing the patch of images, now we have patches of joints in successive timeframes.

L231: "our method". Any changes from NNCLR must be clearly stated...

Thank you for this suggestion. We will clarify this in the final version to ensure the differences are delineated.

[1] Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., & Wang, Y. (2023). MotionBERT: A unified perspective on learning human motion representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15085-15099). [2] Mao, Y., Deng, J., Zhou, W., Fang, Y., Ouyang, W., & Li, H. (2023). Masked motion predictors are strong 3d action representation learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10181-10191).