MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer

We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.

---

1. Expanding the semantic space with large-scale generative models.

Thanks for your constructive suggestion! Following your suggestion, we replace the category names with rephrased detailed descriptions, which are generated by a large generative language model GPT3.5 as used in FROSTER [1]. The generated descriptions expand the semantic space by providing more details about the action and the scene. For example, "Ice climbing" is rephrased to "A person is seen climbing up a wall of ice using specialized equipment like crampons and ice axes". The results are presented in the table below.

We can see that the enriched description leads to better generalization performance. We will actively explore this direction in future work.

[1] FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition. ICLR 2024.

---

2. Extending the approach to other forms could improve generality and versatility.

Thanks for your constructive suggestion! To further demonstrate the generality and versatility of MoTE, we apply our method to ST-Adapter [2]. For simplicity, we remove the depth-wise Conv3D operation in our adapter. In this case, our baseline is slightly lower than ST-Adapter, but it doesn't affect justifying the effectiveness of our method. The results are shown in the table below.

Our method delivers a notable generalization performance improvement while maintaining the close-set result, indicating the effectiveness of MoTE applied to alternative networks.

[2] ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. NeurIPS 2022.

---

3. The model's performance on more diverse and challenging datasets needs further validation.

Thanks for your suggestion! We have evaluated our method on the Something-Something V2 (SSv2) dataset under the few-shot setting in Table 5, which requires intensive temporal understanding capability of the model. We find an inappropriate hyperparameter setting during fine-tuning and present the fixed results in the table below.

We also evaluate our method on SSv2 under the zero-shot setting, as shown in the table below

In both settings, our method outperforms the previous works by a notable margin, demonstrating the effectiveness of our method in facing more challenging datasets.

---

4. The additional complexity from Weight Merging Regularization and other components can slightly increase training time.

Thanks for your thoughtful comment! We show the training time costs (Table 8 of the supplementary material) and the corresponding GFLOPs in the table below. GPU days are calculated by the number of GPUs multiplied by the training time in days.

Applying our method on the baseline introduces slight computational and training time costs, and this cost can be effectively reduced when training with the Distributed Data Parallel framework of PyTorch, demonstrating the efficiency of our method.

---

5. Extensive fine-tuning required for different tasks can be computationally expensive and time-consuming.

Thanks for your comment! We would like to clarify that our method is evaluated on various datasets under zero-shot and close-set settings using one unified model, avoiding the need for multiple fine-tuning processes. Our method can also be applied to parameter-efficient training frameworks, such as adapters, which exhibit impressive training efficiency. The details of this part please refer to the response of the second question. We will actively explore this direction in future work.

---

6. How does the performance of MoTE vary with different numbers of temporal layers and experts? Are there optimal configurations for specific tasks?

Thanks for your comment! We have ablated the number of temporal layers and experts in Figure 1 (b) and Table 2 (b) in the main paper. Please refer to the figure and table in the main paper for the results.

According to the results, we observe that 4 temporal experts per layer are sufficient to learn various knowledge for both close-set and zero-shot tasks. More experts do not lead to further performance gains. As for the number of the temporal layers, our method consistently outperforms the vanilla Transformer layer and 6-layer MoTE achieves the best trade-off between close-set and zero-shot performance. Therefore, we conclude that the 6 layers MoTE with 4 experts per layer are generally the optimal configuration for various tasks.

---

7. What measures can be taken to reduce the computational overhead introduced by the additional components?

Thanks for your thoughtful question! Our work is applied to on the standard Transformer structure in the paper. Replacing it with some efficient structure (e.g. flash attention) can effectively reduce the computational overhead of the additional modules. Besides, applying our method to efficient structures (e.g. adapters) can also reduce the computational cost of the additional components.

---

8. Relevant works

Thanks for your suggestion! We find the recommended papers relevant to our work. We'll cite and discuss them in the revised manuscript.

---

We sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.

We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.

---

1. Ambiguity in the use of certain symbols.

Thanks for your kind reminder, and sorry for the confusion. We double-checked the usage of all symbols and will correct the mistakes in the revised manuscript.

---

2. Missing the notation of the operation exp(). The probability of selecting an expert increases with i, which seems to contradict the intended randomness of stochastic.

Thanks for your comments! The randomness of expert selection can be described and realized under different probability distributions. For example, the vanilla stochastic routing algorithm follows the uniform probability distribution, while ours follows the multinomial probability distribution. Compared to the uniform distribution, the multinomial probability distribution is less random but more controllable (by assigning different activation probabilities to each expert as in Equation (5)). This allows experts with a greater index to be activated more likely and therefore receive a larger volume of data during training. As a result, each expert can learn knowledge with different degrees of generalization and specialization.

We will include the explanation of the notation exp() and the above discussion in the revised manuscript.

---

3. It would be beneficial to add experiments validating the plug-and-play effect or adjust the relevant descriptions in the paper.

Thanks for your thoughtful suggestion! We agree that the plug effect should be further validated. Actually, Temporal Feature Modulation can be applied when the temporal module is separate from the CLIP encoder. Considering this constraint, we decided to adjust the relevant descriptions to make them more accurate and rigorous:

Original:

"We propose a plug-and-play module that measures the confidence of temporal features by means of CLIP’s text space."

Revised:

"We propose a test-time adaptation strategy for networks where the temporal module is separated from the CLIP encoder, to measure the confidence of temporal features by means of CLIP’s text space."

---

4. What is the design basis for the candidate set of the temperature hyperparameter in weight merging? Missing comparison with the continuous space selection schemes.

Thanks for your valuable question! The candidate set of the temperature hyperparameter {$±2^n · β$} $_{n=0}^4$ ∪ {∞} is designed by ourselves. We set the candidate temperature parameters to grow exponentially so that we can explore a larger range in the weight space (i.e. the coefficients calculated from Equation 7 with different temperatures vary more). {∞} is a special case where all the experts are averagely merged according to Equation 7.

The comparison with the continuous space selection schemes is presented below. We implement two continuous space selection schemes. (1) Sampling from a continuous standard normal distribution (mean=0, variance=1). (2) Sampling from a continuous uniform distribution. The continuous space sampling strategy results in a notable performance degradation.

---

5. The connection between the modulation method, the issue of constrained semantic space, and the paper's overall motivation.

Thanks for your constructive suggestion! This part is correlated with the second objective of our paper: How can generalization and specialization coexist in one unified model (line 45-46)? The additional parameters can model the temporal information quite well when dealing with the fine-tuning categories. However, since the semantic space is constrained during fine-tuning, the additional parameters may not model the temporal information properly when facing unknown categories. This concern grows as the test categories are less semantically correlated with the fine-tuning categories. Thus, we propose the modulation module that measures the confidence of the temporal feature by the semantic association of proxy fine-tuning and text categories. This strategy allows us to improve the model generalization while keeping its specialization performance.

We will refine the above discussion and add it to the revised manuscript.

---

6. Improper trade-off metric.

Thanks for your constructive question! We agree that the reviewers' concern is thoughtful and reasonable. We change the trade-off metric to the harmonic mean of the zero-shot performance A and close-set performance B, calculated as $\frac{2AB}{A+B}$. This metric is not sensitive to extreme values and provides a more accurate indication of the overall performance.

---

7. The relationship between experts and data bias views, and how the MoE approach leverages multiple data bias views to improve model performance.

Thanks for your suggestion, and sorry for the confusion. ''Data bias views'' indicates the diverse knowledge learned from various optimization trajectories. We set different optimization trajectories for each expert and construct a more generalized model using distinct knowledge learned across experts. We will add the above description to the revised manuscript.

---

We sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.

We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.

---

1. Discussion on FROSTER.

Thanks for your kind reminder! Our motivation does resemble FROSTER's in some way but differs in the following aspects.

Motivation:

• FROSTER aims to mitigate the catastrophic forgetting caused by fine-tuning the model, while our method focuses on eliminating the catastrophic forgetting caused by the integration of the additional trainable parameters. The two motivations stem from different observations and perspectives.

• FROSTER aims to strike a balance between generalized knowledge and video-specific learning specifically for zero-shot video recognition, while our work pursues this objective in more general and various task settings (close-set, zero-shot, and few-shot recognitions). Since the close-set setting requires much more video-specialized knowledge than the zero-shot setting, our objective is more challenging and practical.

Contribution:

• Our method and FROSTER address the objective with different methodologies. FROSTER focuses on knowledge distillation from the frozen CLIP, while our method concentrates on the model structure design and loss landscape regularization. We offer distinct contributions to the community.

We will add the above discussion and performance comparison in the revised manuscript.

---

2. Whether the model can be effectively integrated with alternative network architectures, such as adapter-based networks?

Thanks for your constructive suggestion! To further demonstrate the generality and scalability of MoTE, we apply our method to ST-Adapter [1]. For simplicity, we remove the depth-wise Conv3D operation in our adapter. In this case, our baseline is slightly lower than ST-Adapter, but it doesn't affect justifying the effectiveness of our method. The results are shown in the table below.

Our method delivers a notable generalization performance improvement while maintaining the close-set result, indicating the effectiveness of MoTE applied to alternative networks.

[1] ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. NeurIPS 2022.

---

3. Additional computational and training time costs associated with implementing the method in conjunction with the baseline model.

Thanks for your comment! We show the training time costs in Table 8 of the supplementary material. Below, we include the table for your convenience and additionally add GFLOPs for reference. GPU days are calculated by the number of GPUs multiplied by the training time in days.

As shown in the table, applying our method on the baseline introduces slight computational and training time costs, illustrating the efficiency of our method.

---

4. More detailed analysis of the specific action types that each expert excels at recognizing.

Thanks for your constructive suggestion! We perform a category-wise performance analysis of the merged experts and each individual expert. We visualize the Top-1 classification accuracy for each video category, which is sampled from the UCF-101 dataset. The experiment is conducted on UCF-101 under the zero-shot setting. The visualization is provided in the attached global response PDF.

We find that different experts exhibit distinct performance for each video category. This is because diverse generalization knowledge is learned across experts, as we demonstrate in the paper. The merged experts benefit from the aggregation of knowledge across experts, especially for some video categories where temporal information is particularly needed, for example, 'Body Weight Squats', 'Handstand Pushups', and 'Wall Pushups'. This suggests that each expert can learn various temporal patterns for the same category. For more visualizations, please refer to Figure 5 and Figure 6 in the supplementary material, which shows more intuitive visualizations.

---

We sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.

Thank you to the authors for their comprehensive rebuttal, which has addressed many of my concerns.

However, I remain unconvinced by the explanation provided for the first point of contention.

FROSTER aims to mitigate the catastrophic forgetting caused by fine-tuning the model, while our method focuses on eliminating the catastrophic forgetting caused by the integration of the additional trainable parameters. The two motivations stem from different observations and perspectives.

Nevertheless, upon examining the FROSTER paper, it becomes evident that the term fine-tuning the model encompasses approaches that incorporate additional trainable parameters within CLIP. Notably, in Figure 1 and Table 3 of the FROSTER paper, experiments with adapter-based methods such as AIM and ST-adapter are presented, which clearly involve the addition of extra parameters.

Given this context, the authors’ response does not sufficiently address the overlap between the two methodologies. I would appreciate further clarification on this matter.

Besides, I would like to know how the authors initialize the parameters of MOTE.

Thanks!

We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.

---

1. The experimental setting may hide the weakness of the proposed method. It would be great to also fine-tune the model on the small-scale UCF-101 and evaluate it on K400.

Thanks for your thoughtful and reasonable suggestion! Our experimental setting follows previous works (Methods listed in Table 3) in fine-tuning on large-scale K400 and then evaluating zero-shot performance on relatively small downstream datasets. Following your suggestion, we train the ViT-L/14 network on UCF-101 and evaluate it on K400, results are shown in the table below.

MoTE yields a 0.5 improvement over the Baseline on UCF-101, while zero-shot performance on K400 significantly outperforms both Raw CLIP and Baseline. This demonstrates the applicability of MoTE to small-scale datasets and its ability to learn generalized knowledge from limited data.

---

2. Comparison with the baseline variant. Is it really necessary to train a unified model with such much cost?

We adopt the recommended baseline and present the results in the table below. Our method still shows notable advantages in zero-shot performance.

We would like to claim that the most significant value of our work is its ability to simultaneously introduce new specialized knowledge while keeping original generalized knowledge, rather than solely achieve the best performance on a particular dataset. Although the recommended baseline performs fairly on close-set and zero-shot tasks, it does not really achieve our goal. This can be evidenced when facing a middle-ground task between close-set and zero-shot tasks, for example, the few-shot task. It requires both specialization and generalization capability to rapidly adapt using limited samples. We present the results of the few-shot task in the table below. Our method significantly outperforms the recommended baseline.

From the perspective of model structure, the recommended baseline can only be applied when the additional parameters are separated from the CLIP encoder. Our proposed method can be applied to various forms of additional parameters (e.g. adapter) that are integrated into the CLIP encoder, demonstrating the versatility of our method. The details of this part please refer to the response of the second question to the reviewer s5Zs. Moreover, in real-world applications, the data distribution shift is often hard to identify. The recommended baseline faces the choice of whether to use the temporal feature when facing a moderate or unknown distribution shift, which limits its practicality. This further demonstrates the necessity of training a unified model.

---

3. For different text-query, the influence of noisy data is diffrerent, and one fixed K may not be optimal. Is there any solution for this issue. Is the selection of K have large influence on the performance?

Thanks for your interesting question! We agree with your opinion and conduct an experiment. In this experiment, we employ the K-means algorithm to cluster the fine-tuning and test category text features and compute the semantic association by retrieving the most similar cluster-centered feature. In this case, the retrieved cluster-centered feature may represent different numbers of data points. We hope this strategy can better represent the semantic associations between fine-tuning and test categories. We also test the effect of the number of K. The results are shown in the table below (TFM indicates the Temporal Feature Modulation).

We find that applying the clustering algorithm does not lead to performance improvement. The reason lies in that using the semantic association to measure the confidence of the temporal feature is actually a rough estimation process. Since this process is inherently somewhat noisy, better-selected data may not largely affect it. This also explains why the selection of the K-value does not have a significant influence on the performance. However, we still observe performance degradation when K is too large, indicating that the noise introduced from the selection of the K dose has a negative influence on the performance.

---

We sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.