EZ-CLIP: EFFICIENT ZERO-SHOT VIDEO ACTION RECOGNITION

W.3.4: Frame selection strategy.

Answer: Thank you for your observation regarding the frame count difference between EZ-CLIP and other methods. We appreciate your attention to detail.

In Table 6, throughput is calculated based on the number of frames processed per second. Therefore, we think that the efficiency comparison in Table 6 is appropriate, irrespective of whether the model employs 8 frames per video or 32 frames per video. Throughput provides a standardized measure that allows for a fair comparison of efficiency across different methods, accounting for the processing speed regardless of the specific frame count used by each method.

If you have further questions or if there are additional aspects you would like us to address, please feel free to let us know. We value your input and are committed to ensuring clarity in our methodology and results.

W.3.5: Evaluation of K-600 dataset

Answer: Thank you for highlighting the overlap between K-600 and K-400. We agree with the reviewer and acknowledge the importance of addressing this factor in zero-shot learning evaluations.

We use the same setup used by existing works where we performed the zero-shot learning evaluation on K-600 using a disjoint set of classes that are not shared with K-400. This approach ensures a more robust assessment of the models' performance on unseen classes, enhancing the reliability of our findings.

If you have any further suggestions or inquiries, we appreciate your input and are open to addressing any additional concerns you may have.

We thank the reviewer for their valuable time, helpful suggestions, and questions. We are grateful to provide our responses as follows:

W3.1: Clarification on overstatement.

Answer:
Thank you for bringing this to our attention. We appreciate your insightful observation and agree with the reviewer. It's important to note that our statement regarding the undesirability of fine-tuning the whole model for preserving generalization capability primarily pertains to zero-shot approaches, which was our main focus. We acknowledge the relevance of AIM and ST-Adapter in employing a similar learning approach but would like to clarify that these methods are fully supervised, whereas our emphasis is on zero-shot techniques. However, we value your feedback, and in the revised version, we have updated our statement to ensure accuracy and clarity. If you have any further questions or suggestions, please feel free to let us know.

W3.2: Spatial and Temporal Adapter.

Answer:
Thank you for your thoughtful comments and suggestions. Your observation regarding the comparison of the baseline models is valid, and we would like to provide further insights into our choices.

1) AIM as the reviewer rightly pointed out, is a fully supervised approach that excels in supervised classification. In our preliminary experiments, we conducted experiments including both spatial and temporal adapters in the baseline model. However, we observed that this approach tends to overfit to the source dataset, resulting in poor performance on the target dataset, particularly in zero-shot scenarios.

2) Apart from this performance issue, temporal adapters also suffer from scaling challenges. For a more detailed understanding, we have provided the breakdown of trainable parameters for different configurations, highlighting the scalability challenges associated with temporal adapters, especially with larger models like ViT-14.

Additionally, we explored the impact of incorporating temporal visual prompts alongside spatial adapters, and the results indicated challenges in scalability, especially with larger models.

We hope this clarifies our rationale and the considerations we took into account during the experimentation. If you have further questions or suggestions, please feel free to let us know. Your feedback is invaluable in enhancing the quality and clarity of our work.

W3.3 Impact of the LLM-generated action class descriptions.

Answer:

---

Thank you for your insightful observation. We appreciate your careful examination of our results. The role of LLM-generated action class descriptions is indeed crucial in our approach, and we acknowledge their positive impact on action recognition, as demonstrated in Table 5.

While the use of LLM-generated descriptions contributes to the improved performance of EZ-CLIP, it is not the sole factor. Although the use of LLM-generated descriptions in this context is a novel idea, we would like to emphasize that our approach involves a joint training strategy that incorporates both temporal visual prompting and motion loss. The ablation studies in our paper demonstrate that the introduction of temporal visual prompts and motion loss leads to significant performance gains across various datasets. The use of LLM-generated descriptions without temporal visual prompting and motion loss is not as strong as it is with these components.

Zero-shot Ablation on LLM-generated Action Class Descriptions:

                                                   |  HMDB-51                      |  UCF-101                      |    SSv2                            |

For instance, when training the model jointly with both temporal visual prompts and motion loss, we observed substantial improvements, particularly in the case of the SSv2 dataset. This indicates that our proposed techniques play a complementary role in enhancing temporal modeling in videos.

Regarding the comparison with ViFi CLIP, we would like to clarify that our goal is to improve efficiency with a focus on zero-shot action recognition. The use of LLM is indeed a novel idea, and we acknowledge its application in other contexts, as seen in "VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS". However, even without LLM, our approach achieves comparable performance to existing works while utilizing very few parameters.

We also appreciate your attention to the breakdown of how motion loss and temporal prompts contribute to our results. The attention visualizations in Figure 4 illustrate the shift in focus from appearance to motion in EZ-CLIP, particularly evident in datasets like UCF-101 and SSv2. This shift aligns with our goal of capturing complex relative motion features, especially beneficial in scenarios where motion cues are critical, as observed in the SSv2 dataset.

We hope this clarifies the contributions of our proposed approach, and we are open to further discussions or clarifications.

Thank you for your valuable feedback.

W/Q 2.2: Experiments on other CLIP models

Answer: Thank you for your suggestion. We have indeed conducted experiments on two additional CLIP architectures: CLIP ViT-32 and CLIP ViT-14. These experiments cover three scenarios: (a) Zero Shot, (b) Base to Novel, and (c) Few Shot. Here are the details:

CLIP ViT-32:

• Number of Tunable Parameters:
  • TVP: 0.07M
  • Visual Adapters: 3.5M
  • Text Adapters: 1.5M

CLIP ViT-14:

• Number of Tunable Parameters:
  • TVP: 0.19M
  • Visual Adapters: 12.6M
  • Text Adapters: 3.5M

Zero-Shot experiment

Base to Novel experiment | Kinetics-400 | HMDB-51 | UCF-101 | SSv2 | |------------------------|---------|----------|-------|---------|----------|------|---------|----------|------|---------|----------|------|

Few shot experiment

                         |HMDB-51                       |         UCF-101               |        SSv2                |

|-----------------------|---------------------------------|--------------------------------|----------------------------|

These experiments provide valuable insights into the effectiveness of our proposed approach across various CLIP model scales. If you have any further inquiries or suggestions, please feel free to let us know.

W2.3: Few-shot vs base-to-novel comparison on SSv2.

Answer: Thank you for your careful examination of our results. To clarify, we employed the same protocol as in existing works for the base-to-novel setting. Specifically, during base-to-novel training, we use all samples from the base classes (not just 16), creating a split between base and novel categories. In the few-shot learning setting, a fixed number of samples (e.g., 16-shot) is utilized.

The observed differences in accuracy between the base-to-novel and few-shot settings, even with the same number of samples (16 shots), are expected due to the distinct training strategies and the resulting differences in the number of training classes between these two setups. We appreciate your attention to detail and hope this explanation clarifies the discrepancy you noticed.

If you have any further questions or suggestions, please feel free to let us know.

We thank the reviewer for their valuable comments and detailed questions. Please refer to our responses below:

W/Q 2.1: Detail about spatial and text adapters?

Answer: We appreciate your inquiry regarding the details of additional training modules and components in our proposed framework. We understand the importance of providing a comprehensive understanding of these elements.

To address your concerns:

1. Text and Spatial Adapters:

   • The details on the placement of text and spatial adapters can be found in Figure 6, specifically in the Lth block BL, as highlighted in Section A.3 of the revised paper. This illustrates the incorporation of these adapters into our model.

2. LLM-Based Prompt Ensembling:

   • The usage of prompts from LLM is an integral part of our approach. In Table 7, available in Section A.2, we provide examples of action class descriptions generated by LLM using a template of "describe [category] as an action performed by humans." This demonstrates the effectiveness of LLM-based prompt ensembling in generating diverse and informative descriptions.

We recognize the importance of clarity in visual representation. In response to your suggestion, we will consider incorporating a high-level figure diagram in the revised version of the paper to provide a more intuitive overview of the overall framework, including the usage of these additional components.

Thank you for your understanding, and we welcome any further questions or suggestions.

We thank the reviewer for their time, insightful comments, and questions. We have provided our responses below.

W.1.1 Comparison with Visual Prompt Multi-Modal Tracking [A]

Answer: Thank you for sharing this work with us. It is indeed relevant, and we have added this to our related work discussion in the revised version.

We believe ViPT [A] and our proposed temporal visual prompting are significantly different and cannot be directly compared.

Conceptual Difference (Temporal communication and feature integration): Our temporal visual prompting adapts via communicating across several video frames with the help of prompts, enabling the efficient learning of motion aspects within the video. It is not just combining spatial information but learning temporal dependencies via prompting. This unique aspect of our approach allows us to capture and utilize information from the current frame, significantly enhancing our model's ability to understand the temporal dynamics within videos.

ViPT, on the other hand, is focused on integrating multiple modalities with an MCP module which combines features (addition to be exact, Figure 3 from ViPT) from two different modalities. There is no communication across modalities via prompting; it is rather an aggregation/fusion of features (due to prompts after fusion, not during fusion). We believe that the introduction of TVP represents a substantial contribution to the field, addressing the need for more effective methods in handling temporal aspects in video understanding.

Practical Difference: Even if we want to use ViPT for videos, it cannot be used for more than 2 frames without any significant modifications to the MCP module. Also, even after this modification, it will be closer to spatial prompting with pooling/fusion, as there is no way to communicate across frames, which is our main technical novelty.

W.1.3 Ablation experiments in Table 4 and Table 5.

Answer: Thank you for your insightful observation regarding the ablation experiments and the observed performance of our base model in comparison to existing methods.

The base model consists of a spatial adapter for the visual encoder and text encoder with text adapters, where only these adapters are trained, keeping all the CLIP weights frozen. We acknowledge that previous state-of-the-art (SOTA) methods often did not leverage the full potential of pre-trained weights and instead opted for fully training their models. This is a significant difference between our approach and existing methods, and therefore, for some cases, our base model outperforms their performance. This is also evident from fully supervised approaches as well (such as AIM), where freezing the CLIP weights has shown benefits. Even with this strong base model, we do demonstrate the effectiveness of both Temporal Visual Prompting and Motion loss on top of this with a comprehensive set of ablations.

We appreciate your diligence in scrutinizing the experimental setup, and we believe that this clarification provides a more comprehensive understanding of the choices made in our methodology.

Q1.1 Ablation on LLM in zero shot, base-to-novel, and few-shot.

Answer: Thank you for your thoughtful inquiry. We value your meticulous examination of our work and the chance to elaborate on the application of LLM in various settings. Here are the outcomes of ablation experiments conducted using LLM in both zero-shot and few-shot scenarios:

Zero-shot Ablation on LLM-generated Action Class Descriptions:

Few-shot Ablation on LLM-generated Action Class Descriptions: |-----------------------| HMDB-51 | UCF-101 | K-600 |

These additional ablation experiments further demonstrate the impact of LLM-generated action class descriptions as prompts in both zero-shot and few-shot scenarios. We hope this clarification addresses your concern, and we welcome any further questions or suggestions.

W.1.2 Additional ablation experiments. Answer: Thank you for your valuable feedback regarding the need for additional ablation experiments. In response to your suggestion, we have conducted comprehensive ablation experiments and have included the results in the revised manuscript. We present the impact of LLM-generated action class descriptions as prompts in various settings, including zero-shot, few-shot, and few-shot ablation with a focus on TVP and motion loss. The tables below provide a comprehensive overview of our ablation experiments: 1) Impact of LLM-generated action class descriptions as prompts on a Zero-shot setting:

3) Few-shot ablation on TVP and Motion loss: We evaluate the effectiveness of temporal visual prompting (TVP) and motion loss while using a Few-shot setup on UCF-101, HMDB-51, and SSv2.

                             | HMDB-51                   | UCF-101                    |    SSv2                             |

|---------------------------|------------------------------|-----------------------------|-----------------------------------|

Few-shot Ablation LLM: Impact of LLM-generated action class descriptions as prompts on Few-shot settings.

                         |  HMDB-51                       |      UCF-101                   |           SSv2                   |

|-----------------------|----------------------------------|---------------------------------|---------------------------------|

We hope that these additional ablation experiments address the concern.

Q.1.2 Detail of spatial and text adapters and placement.

Answer: Thank you for bringing this to our attention. We appreciate your inquiry and apologize for any confusion. Since the incorporation of the text adapter and spatial adapter was not the primary focus of our contribution, this detail was provided in the supplementary (Section A.3).

However, to address this concern, we have updated the figure in the revised version of the paper. The updated details on the placement of the adapters are included in the Lth layer in block BL. The transformer block structure BL in Figure 6 illustrates the integration of the Adapter after the self-attention layer, adapting the CLIP architecture.

We hope that these updates provide more clarity into our approach.

Q.1.3 Tunable parameter clarification.

Answer: We appreciate your attention to detail and the chance to clarify the tunable parameters in Table 6. There seems to be a misunderstanding regarding the cited AIM's spatial adapter, as it does not have 3.7M tunable parameters. The accurate figures are 14.3M for SSv2 and 11M for K-400.

It's crucial to note that AIM's method focuses on video classification in a fully supervised manner, while our proposed method addresses the more challenging task of zero-shot classification. For a detailed breakdown of our tunable parameters:

• Temporal Visual prompts: 73K
• Text adapters: 1.5M
• Image spatial adapters: 3.5M

These individual components collectively contribute to the overall efficiency and effectiveness of our model in achieving superior zero-shot action recognition. We hope this clarification addresses your concerns, and we remain open to any further questions or feedback.

Q.1.4 Prompt only added to the first layer.

Answer: Thank you for your thoughtful question. We did conduct experiments with a shallow Temporal Visual Prompt (TVP) in the context of zero-shot ablation. Specifically, in this experiment, TVP was applied only to the first layer of the CLIP ViT transformer.

The results of the zero-shot ablation on TVP and Motion loss are presented in the table below:

These results demonstrate the impact of applying TVP exclusively to the first layer of the transformer, where we observe some drop in performance. We hope this clarification addresses your concerns, and we remain open to any further questions or feedback.

Q.1.5 Motion loss explanation.

Answer:

We appreciate the opportunity to provide clarity on the role of motion loss and address your concerns.

In Equation (5), where the central difference C is computed using the embeddings of the previous and next frames, it's important to note that we deliberately avoid using consecutive frames for motion loss calculation. Instead, we select 8 frames with equal intervals from the video. This intentional choice is made to mitigate the common issue in action recognition tasks where adjacent frames are often very similar. This strategy allows us to capture meaningful temporal dynamics while avoiding redundancy caused by the similarity of consecutive frames. In essence, this also allows us to capture a broader context by sampling frames with some skip rate, enhancing the effectiveness of our motion loss calculation.

Please note that this approach aligns with strategies employed by other state-of-the-art methods in the field for frame sampling from videos.

We appreciate your thoughtful review and the opportunity to provide clarifications. If you have any further suggestions or inquiries, we appreciate your input and are open to addressing any additional concerns you may have.

I have carefully read the authors' responses and the comments from other reviewers.

1. I appreciate the additional experiments on zero-shot and few-shot settings. However, what I would like to see more is some design-based ablation studies regarding TVP. Since the design of TVP shares many similarities with existing methods, it necessitates further experiments to demonstrate that this design is superior to current prompts. It should be shown that some functionalities of this design are unachievable by existing prompts.

2. Although the authors explained how their baseline differs from the compared methods, they did not conduct a fair comparison under the same setting.

3. After reading the authors' response, I fail to discern the fundamental difference between this paper's prompt and ViPT. It seems that the temporal relation between frames can be modeled after passing through the self-attention layer, and it is not a capability unique to having a prompt.