ZeroI2V: Zero-Cost Adaptation of Pre-Trained Transformers from Image to Video

Q1: Referring to it as zero-cost adaptation may be an overstatement.

We sincerely apologize for any confusion. We have repeatedly emphasized in the abstract and the main paper that our meaning of “zero-cost adaptation” is that there is no additional computational and parameter cost during inference. We would change the name "zero-cost adaption" to "adaption with zero extra inference cost" or "zero-extra cost adaption" in the final version for clarity.

Q2: the full adaptation diagram still requires a significant amount of computational resources and memory for backpropagation. Is there a training wall clock time comparison with prior works?

We have supplemented the comparison of ZeroI2V and the previous PETL method regarding training time and training GPU memory usage in Table 14 of Appendix B. It can be seen that although our method’s training efficiency is not as prominent as the inference efficiency compared to the previous PETL method, it is also not much more cumbersome. Compared to AIM that impose an additional computational burden for temporal modeling, our STDHA method, which does not introduce extra learnable parameters, ensures that ZeroI2V maintains superior training efficiency. And we believe that it is worthwhile and acceptable to exchange some training cost for a reduction in inference cost.

Q3: What is the total training parameter of linear adapters used during training in Table 2/3?

I presume you might be referring to Table 3/4? As in Table 2, we have already listed the number of tunable parameters. As for the total training parameters of ZeroI2V in Table 3/4, we provide the final setting that we used in Table 6 in Appendix A.1. For most cases, we use 4 adapters per ViT block (14M for ViT-B and 50M for ViT-L).

Q4: I noticed that the best figure in Table 2 for SSv2 Top-1 is 66.3. However, in Table 4, the corresponding number is 67.7. Which setting accounts for this improvement?

The result of 66.3 was obtained using a single view test (center crop), while the result of 67.7 was obtained using a multi-view test (three crops). This evaluation method is consistent with previous work.

Q5: How do you select a specific head for different Δt? Has there been any ablation study conducted?

Due to the existence of serial learnable adapters, we believe that it is unnecessary to deliberately select heads for different $\Delta t$. This is because different heads actually correspond to different feature dimensions, which in turn correspond to $W_{qkvo}$ in the attention layer. We believe that the learnable linear adapter can adaptively adjust the weights of $W_{qkvo}$ corresponding to each head during low-rank adaptation.

Thank you for your valuable suggestion. Here we provide further explanation for your existing concerns:

1. Comparsion with DualPath: We have also noticed this work. DualPath achieves high computational efficiency by using fewer sampling frames in the spatial adaptation branch. In the temporal adaptation branch, it uses more sampling frames, and after reducing the resolution of each frame, it stitches them into a grid-like frameset to control the computation.

   • In terms of technical contributions, our method and DualPath’s method are complementary. We simply followed the settings of previous works like ST-Adapter and AIM. In fact, we can also replace DualPath’s temporal adaptation branch with our temporal modeling, or enhance the temporal modeling capability of the DualPath’s spatial adaptation branch without increasing additional computation. Our linear adaptation strategy can also reduce the parameter and computational load of DualPath during inference.
   • In terms of performance, we believe that the temporal modeling capability of our STDHA actually surpasses the “Grid attention” used by DualPath. This can be seen from the comparison with SIFAR [1] in Table 13 of Appendix B (SIFAR uses the same temporal modeling method as DualPath).
   • We actually tried to reproduce the performance of their method using the code provided by the DualPath authors, but we failed. The authors did not provide any further replies or explanations, nor did they provide model checkpoints. Our own implementation also failed to reproduce the performance, we also notice similar reproduction issue is also encountered by others in the GitHub issues.

   Therefore, we did not compare with DualPath in the origin paper. We can supplement this in later versions.

2. The contribution of our method: We agree with your statement that the application of shifting operations and reparameterization techniques is common in our community. However, our work is not a simple stack of these techniques:

   • The shift operation is not the core contribution of our STDHA. We just use it to provide an efficient implementation of STDHA (for details, please refer to our answers to Q3 of Reviewer PNMk and Reviewer Z2GV);
   • The reparameterization technique is also just a means for us to implement linear adaption. Our contributions lie in our full adaption strategy and our comprehensive experiments.

[1] Fan, Quanfu, Chun-Fu Chen, and Rameswar Panda. "Can an Image Classifier Suffice For Action Recognition?." International Conference on Learning Representations. 2021.

Thanks for your constructive suggestions again. We value your comments and have made efforts to revise the paper. Please kindly let us know if our response addressed your concerns. We are willing to respond to any further questions before the rebuttal ends (within 6 hours) :-)

Q1: The results are convincing but needs a bit more illustration

Thank you for your valuable suggestions. Due to space constraints, we did not elaborate in detail on the comparison of results. We will supplement it here and update it in our paper after adjusting the layout:

• Why MViTv2-L works better on SSv2？ Firstly, from the perspective of model design, MViTv2 introduces more efficient downsampling designs compared to the original ViT, thereby resulting in lower FLOPs. This allows it to use a higher spatial resolution (312) and sampling frame number (40). Secondly, our method is based on image pre-training, while MViTv2 is based on video weights pre-trained on K400. As SSv2 places more emphasis on temporal modeling capabilities, MViTv2-L can achieve a better performance trade-off.
• Why AIM better on the K400？ We understand that the K400 dataset places more emphasis on spatial modeling capabilities. Our STDHA, by making some heads that originally performed spatial modeling complete temporal modeling, has to some extent reduced the model’s spatial modeling capabilities. On the other hand, AIM, through the stack of temporal attention and spatial attention, fully retains the model’s original spatial modeling capabilities. Therefore, it achieves slightly stronger performance on K400, but at the cost of nearly 30% additional computational overhead.

Q2: The advantage and disadvantages of the proposed STDHA comparing against the commonly used action recognition models

• advantage：The first obvious advantage is that STDHA is highly efficient, requiring no additional learnable parameters or computational overhead. In addition, compared to other efficient temporal modeling methods, our STDHA is specifically designed for the characteristics of the attention mechanism and has stronger spatio-temporal modeling capabilities (we have discussed this in Section 3.2 of the main paper). Finally, due to the popularity of the attention mechanism, our method has a very wide range of applications and is plug-and-play (we have shown in Table 13 of Appendix B that our method can also achieve good performance when using Swin Transformer as the backbone).
• disadvantage：The main drawback of STDHA is that it makes some heads, which originally performed spatial modeling, complete temporal modeling. This to some extent impairs the model’s spatial modeling capabilities. This may result in slightly poorer performance in scenarios that require high spatial information modeling (for example, its performance on K400 is slightly inferior to AIM).

Q3: Why pick the inter frame attention as part of the proposed STDHA:

• Motivation of the choice of inter-frame attention: Our method is designed for image-to-video adaptation, so we start from the perspective of trying to reduce the difficulty of image model transfer. A direct idea is to tile video frames into images for spatio-temporal modeling, but this is not ideal in terms of both computational efficiency and performance. Then we considered inter-frame attention. Since only the key and value of the current frame are replaced with those of other frames, it can be assumed that the data distribution of the key and value has not changed much when calculating attention. From this perspective, we replaced the temporal attention in AIM with inter-frame attention. We found that it has stronger temporal modeling capabilities in the image-to-video scenario:

Moreover, the implementation of inter-frame attention is very simple and has the same computational cost as the original spatial attention, we can implement STDHA in a highly efficient manner.

• The inter-frame attention is not originally from this paper： We believe that the key contribution of our proposed STDHA is not just in our use of inter-frame attention for temporal modeling. Rather, it lies in demonstrating the effectiveness and efficiency of using different heads for simultaneous spatial and temporal modeling, and in providing an efficient implementation based on inter-frame attention.

Q4: CLIP VIT activation map there for comparison.

Thank you for your valuable suggestions. We have added the CLIP VIT activation map for a more intuitive comparison in Figure 3 of Appendix C in our revised paper. It can be seen that STDHA pays more attention to the interaction of objects in the video (such as the touch between hands and cups or Rubik’s cubes), while CLIP only focuses on individual objects and does not capture the spatio-temporal dynamics in the video well.

Thanks for your constructive suggestions again. We value your comments and have made efforts to revise the paper. Please kindly let us know if our response addressed your concerns. We are willing to respond to any further questions before the rebuttal ends (within 6 hours) :-)

Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

What is the benchmark mentioned on page 2 in “establish a benchmark” on page 2?

We agree with your statement that a benchmark typically introduces a new dataset. In this case, we use the existing Something Something v2 dataset as our dataset for evaluating the temporal modeling capabilities of our model after transfer. We compared common image-to-video adaptation methods under fair comparison conditions, so we refer to it as “establishing a benchmark”. Similar expressions are also used in the introduction of ST-Adapter.

How is “offering powerful capabilities” a "key perspective" of temporal modeling?

What we wanted to express here is that the “Capability of capturing the temporal dynamics” is a “key perspective” of temporal modeling. We have corrected this in the revised paper.

Saying ZeroI2V is “reducing the difficulty” of image-to-video adaptation is also vague

Here, we are not just aiming to reduce the inference cost, but we are also concerned with reducing the training cost in image-to-video adaptation. STDHA, by not introducing additional learnable parameters and retaining the original spatial head for image modeling, also reduces the difficulty of training to a certain extent.

Q1: Trainability of ZeroI2V

We have supplemented the comparison of ZeroI2V and the previous PETL method regarding training time and training GPU memory usage in Table 14 of Appendix B. It can be seen that although our method’s training efficiency is not as prominent as the inference efficiency compared to the previous PETL method, it is also not much more cumbersome. Compared to AIM that impose an additional computational burden for temporal modeling, our STDHA method, which does not introduce extra learnable parameters, ensures that ZeroI2V maintains superior training efficiency. We believe that it is worthwhile and acceptable to exchange some training cost for a reduction in inference cost.

Q2: There need to be experiments on another video task

Thank you for your valuable suggestions. To understand the capability of our method in fine-grained spatial understandingWe evaluate our method on action detection dataset AVA. Following the setting of VideoMAE. As shown in Table 9 of Appendix B , compared to using the original image CLIP features, our ZeroI2V achieved a significant performance improvement (26.4 vs 18.3) with the same number of parameters and computation. It’s noteworthy that our method was not pre-trained on action recognition datasets such as Kinetics. Instead, we directly applied image-to-video transfer on the AVA dataset. Remarkably, our method still managed to achieve performance on par with full-finetuning methods and self-supervised methods that underwent pre-training using the Kinetics dataset, even when using only 8 frames as input. In summary, our ZeroI2V demonstrates outstanding potential in video tasks beyond recognition.

Q3: Can you clarify the novelty of ZeroI2V compared to the Patch Shift Transformer introduced?

The TPS operation in the Patch Shift Transformer is placed before the self-attention layer. It achieves efficient modeling by mixing information from frames at different times, which is more like an extension of the channel shift proposed in TSM, rather than a profound improvement on the original MHSA operation. Our STDHA, on the other hand, starts from the redundancy of heads in MHSA. By assigning some heads that originally performed spatial modeling to complete temporal modeling tasks, we believe it maintains the purity of information within the same head (in contrast, in TPS, each head contains a portion of channels from other frames when calculating attention, i.e., all heads are simultaneously performing spatio-temporal modeling). This allows us to ultimately achieve better spatio-temporal modeling capabilities. This point is also demonstrated in Table 13 of Appendix B, where under the same settings, Swin-B w/ STDHA achieved significantly better performance than PST-B (70.0 vs 67.4).

Additionally, we understand that the reason you may find this confusing is because we use inter-frame attention for temporal modeling in STDHA, which also requires us to use a shift operation in our implementation. However, we believe that the key contribution of our proposed STDHA is not just in our use of inter-frame attention for temporal modeling. Rather, it lies in demonstrating the effectiveness and efficiency of using different heads for simultaneous spatial and temporal modeling, and in providing an efficient implementation based on inter-frame attention. In the future, we will explore more temporal modeling operations for the temporal head.

Q4: What is the difference between head relocation QKV and head relocation KV (STDHA) in Table 1a?

In the temporal head, when calculating attention, “head relocation QKV” means that we use the QKV corresponding to non-current moment frames for attention calculation. On the other hand, “head relocation KV (STDHA)” means that we use the Q of the current moment frame and the KV corresponding to non-current moment frames for attention calculation, thereby implementing inter-frame attention.

Q5: Which configuration/hyperparameter setting is chosen for ZeroI2V

In Appendix A.1, Table 6, we provide the final setting that we used in Section 4.3.

Q6: Is the channel change ratio R_c supposed to be the ratio k:ℎ−k?

$R_c$ represents the ratio of the number of channels involved in temporal modeling to all channels. Specifically, $R_c$ denotes $k:h$ in STDHA.

Q7: In Table 5, why is ST-Adapter missing?

We have already shown the latency when using two ST-Adapters in Table 2, which is significantly higher than ZeroI2V (38.8 vs 28.9). ST-Adapter uses a different number of adapters for K400 and SSv2, so we need two lines to present it. Due to space constraints, we did not include it in the comparison in Table 5. We will update it in our paper after adjusting the layout.

Thanks for your constructive suggestions again. We value your comments and have made efforts to revise the paper. Please kindly let us know if our response addressed your concerns. We are willing to respond to any further questions before the rebuttal ends (within 6 hours) :-)

Thank you for your comprehensive response to my questions. It has improved my understanding of of ZeroI2V's implementation. Additionally, the efficacy of ZeroI2V on the action detection task shows that it may be valuable for more dense prediction tasks, although action detection does not require as much pixel-wise spatial understanding in videos compared to video object detection or object segmentation. My remaining concerns are the following:

1&5. Regarding the trainability of the method, it is concerning that different hyperparameters are chosen for different datasets. This suggests that a careful choice of hyperparameters is necessary to achieve good performance in practice, which might make the method cumbersome to use. When it comes to the presentation, it is misleading to put the term "zero-cost" in the paper when the training time is increased as it is compared to the state of the art.

3. It is still a bit difficult to understand how large of a difference the implementation of ZeroI2V is compared to previous methods. A figure showing how tokens are used in the attention operations in ZeroI2V vs previous methods like TPS would probably be helpful in clarifying the novelty.

4. Unless I'm mistaken ST-Adapter is missing from Table 5 in the most recent version, perhaps this can be updated in the final version?

The concern around trainability and ease of use in practice is the main point that leads me to keep my rating.

Q1: how is the STDHA implemented?

Our choice of inter-frame attention for temporal modeling in STDHA allows us to directly apply the shift operation to the key and value tensors associated with the temporal head. Shift operation is very lightweight and can be almost ignored compared to other operations that are counted into FLOPs (such as matrix multiplication). Specifically, we directly modified the nn.MultiHeadAttention in PyTorch. It’s worth mentioning that our STDHA does not conflict with some commonly used efficient attention implementations such as FlashAttention, and we can combine them to achieve higher computational efficiency.

Q2: In Table 1 (b), what is the meaning of 1/2 head?

$\Delta t=1/2$ means that half of the channels in this temporal head have the time offset of 1 and the other half have -1. This experiment actually violates our original intention of keeping the purity of the head, so we just treat it as an extreme case of ablation experiments.

Q3: It would be better to show the full comparison with ST-Adapter

Thank you for your valuable suggestion. Due to space constraints, we choose one method for comparison at different model sizes. We will add a more comprehensive comparison after adjusting the layout in the future.

Q4: It could better show the effectiveness of the proposed method to show some results on other datasets such as Diving48, Epic-kitchens

We have already demonstrated on the SSv2 dataset that ZeroI2V can still achieve good performance even when there is a large distribution difference with the k400 dataset. Additionally, following your suggestion, we have supplemented the experimental results on Diving48. We achieved a top-1 accuracy of 91.4%, significantly surpassing AIM (90.6%). This further showcases the robust performance of our method on small datasets. Detailed results and comparisons can be found in Table 11 of Appendix B in our revised paper.