Kronecker Mask and Interpretive Prompts are Language-Action Video Learners

Thanks for your encouraging comments and constructive feedback.

1) Concern about the task-agnostic information in LLM’s output:

Thank you for raising this thoughtful question. Indeed, LLM outputs can sometimes include task-agnostic content (referred to as noise), which we have encountered in our experiments. To mitigate this issue, we implemented certain measures, such as the format prompt mentioned in Sec. 3.3 and illustrated in Fig. 4. This format prompt guides LLaMA in generating interpretive prompts by structuring the format of Commands+Examples+Action Concept, providing a certain degree of control over the generated text. The Examples field, crafted through human-LLM interaction, aligns the text more closely with our expectations. The Commands field instructs LLaMA to adhere to the style of the given Examples. However, while this operation reduces bias, it does not entirely eliminate noise. To further filter low-quality or unexpected outputs, we employed simple approaches like human selection and asking LLMs to self-evaluate whether the generated text aligns with the expected Examples style, retaining only outputs with positive (Yes) confirmations.

Additionally, we observed that while some interpretive prompts improve performance, others may degrade it. Noise may have negative effects, often influenced by the design of interpretive prompts themselves. Taking the action ‘abseiling’ as an example, three perspectives of interpretive prompts used in Fig. 4 include:

Description of action decomposition Abseiling combines several actions to descend a vertical surface with a rope. Climbers secure themselves with a harness and utilize a descender device for controlled descent. Simple actions, like maintaining a straight body position and regulating rope tension, form the basis

Synonyms conversion Descending, rappelling, downclimbing, descending a rock face, climbing down

Bringing into body parts Abseiling involves the following actions. Feet planted securely on rock face, hands gripping rope for balance and support, arms stretched overhead, torso leaning backward, body fluidly transitioning between maintaining balance and swinging downward, legs straight or bent for control

Furthermore, we also explored some other designs, such as describing a scene corresponding to an action concept:

Describe a scene involved an action A scene involves abseiling. A climber slowly makes their way down the sheer rock face, their footing steady and deliberate. Hands grip the rope for support as they rappel, the wind whipping past their face, a myriad of emotions flitting across their face. With each drop, they gain control, their confidence in the safety line growing with every passing moment

However, the text in bold italics is not related to the action content but rather to the emotions of the subject performing the action. Introducing such interpretive prompts can negatively affect model performance.

Our motivation for designing three interpretive prompts is that action decomposition: helping distinguish complex and basic actions, and similar actions, as not all action concepts are of equal status, synonym conversion: improving zero-shot generalization, and introducing body parts: helping localize the region where action occurs. Their effectiveness was verified through extensive experiments. However, describing a scene involved an action may be easier to introduce potential noise because the concept corresponding to an action scene may be broader and more unconstraint.

Interpretive prompt (IP) is a promising general module for action recognition and understanding. We integrate it with previous state-of-the-art methods and observe the improvements, as shown in the following table:

Thanks for your feedback and for highlighting the strengths of our work. Below, we address your concerns in detail.

1) Regarding the performance:

We appreciate your concern regarding the potential influence of hyper-parameter settings. In Appendix D, Tab. 14, we provide a detailed hyper-parameter setting. Compared to the previous state-of-the-art methods, we slightly adjust the learning rate and epochs to observe a suitable convergence trend of the model, without any other modifications, also no additional tricks are employed. Moreover, CLAVER enhances not only the classification accuracy but also the interpretability of the model’s predictions, as evidenced by the spatial-temporal attention visualizations in Fig. 7 and Appendix Fig. 12-14. Beyond fully-supervised performance metrics, CLAVER demonstrates superior generalization, particularly in zero-shot settings where domain shifts and unseen classes pose significant challenges. We would like to stress that small yet consistent improvements on highly competitive benchmarks such as Kinetics-400, Kinetics-600, UCF-101, and HMDB-51 are non-trivial, given the saturation of these datasets.

To further validate the effectiveness of each module in our method, we have conducted ablation studies (see Tab. 6) that isolate the contributions of KMTA and interpretive prompts. These studies confirm that the improvements stem primarily from our proposed components rather than external factors. In addition, as shown in the table below, compared with the previous state-of-the-art ILA, our method has smaller parameter size and FLOPs, while the performance is better, which also indicates the efficiency of our method.

2) About the computational cost and efficiency:

We provide a detailed comparison of FLOPs, parameter size, and inference time for CLIP, ILA, XCLIP and our CLAVER in the table below. In addition, our model is trained on 4 NVIDIA A100 GPUs (CLAVER-B-32, CLAVER-B-16) or 8 NVIDIA A100 GPUs (CLAVER-L-14) for 3-5 days (on Kinetics-400 dataset). We test the inference time using a NVIDIA GeForce RTX 4090 GPU.

The parameter size of CLAVER (B/32, B/16, L/14) are 1.3x that of CLIP (B/32, B/16, L/14), the values of FLOPs increase by 1.2x, and the increment of inference time are (1x, 2x, 3.6x) respectively. The parameter size of CLAVER (B/32, B/16, L/14) are very similar to X-CLIP (B/32, B/16, L/14), with FLOPs being 1.3x and inference time increments of (0.75x, 1.3x, 1.6x), respectively. In terms of performance, CLAVER is much better than X-CLIP. The parameter size of CLAVER (B/32, B/16, L/14) are 0.77x~0.8x that of ILA (B/32, B/16, L/14), with FLOPs being 0.8x~0.86x and inference time increments of (0.2x, 0.4x, 0.7x), respectively. CLAVER is better than ILA in both terms of performance and computational cost.

* Note that the FLOPs here are calculated with a batch size of 1, 8 frames and a maximum sentence length of 77. The tools used to calculate these metrics are thop.profile and torch.utils.benchmark, where the inference time is the average of 100 runs.

Thanks for your encouraging comments and thoughtful suggestions. Below, we address your concerns in detail.

1) Insufficient performance difference between KMTA and KMCTA and how KMCTA solved the bottleneck:

Thanks for your attention to this issue. Tab. 8 reflects the impact of low-rank bottleneck that may arise from different patch sizes and frame lengths, comparing joint attention (JA), KMTA, and KMCTA. Perhaps due to the font size being too small, it may not be as noticeable, and we have fixed this issue in the rebuttal revision. Tab. 8 indicates that reducing patch size increases the number of tokens but does not highlight the low-rank bottleneck for KMTA and JA, as smaller patch sizes result in finer-grained features. JA, KMTA and KMCTA all achieve better performance. In contrast, when we increase the frame length, only the performance of KMCTA can steadily improve, and the performance increment of KMCTA is the largest. You can refer to related analysis in Sec. 4.3 (line 459-468) in main text.

However, this phenomenon is not so obvious in Tab. 1, likely because the Kinetics-400 is large and diverse enough, while the size of HMDB-51 and UCF101 are relatively smaller. Thus, in situations where data is limited, the low-rank bottleneck becomes more pronounced. The limited data is insufficient to support the model's full learning, which cannot effectively learn the diversity features in the data distribution, leading to the degradation of the attention matrix.

Thanks also for your reminder. We evaluate the performance of zero- and few-shot experiments on Kinetics-600, HMDB-51, and UCF-101 with CLAVER-B/16 (KMCT) configuration. The comparison of KMT and KMCT as shown in the following tables:

In the zero-shot scenario, the performance of KMCTA and KMTA has its own wins and losses, which is due to the lack of further fine-tuning, and the model's ability is relatively dependent on previous training. In the few-shot scenario, we can observe that in most cases, the performance of KMCTA is better, which further indicates that when the scale of training data is limited, the low-rank bottleneck is more significant. There are also some related studies that have been involved in this phenomenon:

On the Efficiency of Transformers: The Effect of Attention Rank

Efficient Transformers: A Survey

2) The pipeline of interpretive prompts:

The Commands field and Examples field are prepared in advance. Assuming we have 400 action classes (taking K400 as an example), we denote the original text of each class as Action Concept field (The colors of Commands, Examples, and Action Concepts fields correspond to the texts in the box). Then, we concatenate these three fields in the order of Commands + Examples + Action Concept to form a format prompt. We input it into LLaMA and obtain the corresponding interpretive prompt for that action class, which is used for training CLAVER. Three different interpretive prompt perspectives correspond to different Commands and Examples field contents, and the output descriptions for each interpretive prompt perspective are stored in a file for loading during training. Of course, the entire interpretive prompt generation process and training can also be conducted online. In order to improve training efficiency, we create and store the interpretive prompts before conducting training.

3) Further analysis for word importance:

We provide some representative visualizations in the Appendix (Fig. 15) to illustrate. We use sentence-level descriptions, and it can be observed that CLIP tends to prefer nouns in sentences, while CLAVER tends to prefer verbs or phrases that combine verbs and nouns.

4) Regarding why not use the CLIP-L/14 setting for all experiments:

Because many previous CLIP-based action video recognition works, like Action-CLIP [1], X-CLIP [2], and ASU[3], etc., employ the CLIP-B/16 configuration on Kinetic-600, HMDB-51, UCF-101 few-shot, zero-shot experiments. In order to make a fair comparison with these methods, we also follow the same configuration. Thus, we use the CLAVER-B/16.

[1] ActionClip: A New Paradigm for Video Action Recognition

[2] Expanding Language-Image Pretrained Models for General Video Recognition

[3] Video Action Recognition with Attentive Semantic Units

In the rebuttal revision, we recalibrate Fig. 4 and Fig. 6. Additionally, we add a new representative word importance visualization, which is Fig. 15 placed at the end of the Appendix, to show the transition of word importance from CLIP's attention on nouns to CLAVER's preference on verbs.

Thanks to the authors for taking great efforts on the rebuttal. Regarding my questions, I think: The pipeline of interpretive prompts becomes clear to me and new examples for qualitative analysis of word importance are great. Also, the explanation for Table 1 remains using CLAVER-B/16 is meaningful. However, I still feel like the claim for KMCTA aims to alleviate the low-rank bottleneck remains marginal for me and the current table cannot fully explain the performance. Overall, I think it's a relatively great paper with a Contrastive Language-Action Video Learner (CLAVER) to efficiently adapt both the visual and textual branches. So I will keep my positive rating.

We appreciate the reviewer’s thoughtful comments and constructive feedback. Below, we address each of the concerns raised.

1) The motivation and advantages of KMT, KMCT and the difference from previous method:

As illustrated in Fig. 2, our Kronecker Mask Temporal Attention (KMTA) and Kronecker Mask Causal Temporal Attention (KMCTA) are designed to overcome the limitations of existing temporal modeling approaches. Common methods such as pipeline temporal attention, alignment-guided temporal attention, and class-token-only temporal attention have limited temporal receptive fields, limiting their ability to capture dynamic information. Joint attention exhibits global receptive fields, on the other hand, faces the issue of spatiotemporal homogenization. Our KMTA leverages a Kronecker mask to simultaneously expand the temporal receptive field and mitigate spatiotemporal homogenization.

Spatiotemporal Homogenization: We define this as the phenomenon where random token shuffling (i.e., destroying the spatiotemporal structure of video clips) has limited impact on word importance and similarity, which is counterintuitive. If there is a significant change in the visual branch's semantics, it will correspondingly cause changes in the word importance and similarity. This reflects whether spatiotemporal modeling captures interpretable video representations. Specifically, word importance refers to the correlation between each word in a sentence and the semantics of the video content, while similarity refers to the similarity between visual and textual representations.

Fig. 6 illustrates the effects of token shuffling. Joint attention is minimally impacted, indicating spatiotemporal homogeneity. Conversely, KMT and KMCT are significantly affected, demonstrating their ability to capture interpretable spatiotemporal semantics. This highlights a key issue: learnable positional encoding has limited ability to mitigate spatiotemporal homogenization. Their optimization is driven by losses that aim to maximize accuracy, rather than explicitly learning spatiotemporal structures. In contrast, our Kronecker masks provide a natural inductive bias for spatiotemporal heterogeneity, leading to better interpretability. As shown in Appendix Fig. 8, previous state-of-the-art methods such as ILA and X-CLIP also encounter spatiotemporal homogenization in our experiments. Surprisingly, in some cases, token shuffling increases similarity, which is illogical.

In Fig. 2 (Right), the spatial and temporal attention we mentioned both can be derived by combining a tailored Kronecker mask with joint attention. Therefore, we collectively refer to them as Kronecker Mask Attention. In addition, we also illustrate Fig. 10 as an example to show that Kronecker mask has certain generality, such as for spatiotemporal graph data modeling.

KMCTA’s Contribution: KMCTA alleviates the low-rank bottleneck faced by KMTA. As the number of tokens increases (e.g., via higher frame lengths or reduced patch sizes), the expressive power of attention mechanisms will deteriorate, because the token count ($n$) exceed the dimension ($d$) of the token features ($n\gg d$). This leads to a low-rank self-attention matrix and feature homogenization. KMCTA ensures a full-rank self-attention matrix, as proven in the Appendix A. Tab. 8 demonstrates that JA and KMTA show limited improvements even degradations with increased frame lengths, while KMCTA consistently improves, underscoring its robustness in long frame scenarios.