Extending Video Masked Autoencoders to 128 frames

We appreciate the reviewer’s valuable feedback on our work, and for noticing the effectiveness of our method in improving both pre-training efficiency and downstream accuracy. Below, we address some of the concerns and questions raised by the reviewer.

Q1. Gradually increasing the number of frames:

Our motivation is to study whether we can effectively pre-train MAE on long-videos and whether such pre-training helps downstream tasks. For this, we selected 128 frames as our target (as it is 8x the typical MAE pre-training frames already), and designed our method around achieving this target. Post-facto, we agree that it’s interesting to study the effect of varying number of frames. To satisfy reviewer’s curiosity, we present some related findings:

In the table below, we ablate the effect of increasing the number of frames while keeping the mask ratio fixed at 15%. During inference, we ensure that the model sees the same number of frames and report Epic-Kitchens-100 action Top1 Accuracy. We clearly observe a trend of the accuracy improvement with the number of frames similar to what we presented in Table 2 of the main paper.

Below, we repeat the same experiment without decoder masking and observe OOM for 64 and 128 frames, whereas our proposed decoder masking method unlocks additional gains in the previous table. This further demonstrates the effectiveness of our method.

Q2. "Why does this work convert MAGVIT to FSQ-MAGVIT for reconstruction targets and adaptive token selection?"

We thank the reviewer for raising this question. We have now added a justification in the common response on our choice of tokenizer and the quantizer, and we request the reviewer to consult the same. Our studies showed that our choice of FSQ achieved the highest PSNR and the highest downstream EK100 accuracy, however the differences in downstream EK100 accuracies are not much with different choice of quantizers.

Q3. Only 2 downstream tasks are used for the main experiments.

During the rebuttal period, we additionally tested our method on Something-Something-v2 dataset. We presented the corresponding results in the common response and observe the same trend as seen in Table 2 of the main paper.

Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you

We appreciate the reviewers comments on our work being well-written, its importance in the field of video understanding, orthogonality to existing works, and that the effectiveness of our approach is verified through ablation experiments. Below, we address the additional questions raised by the reviewer.

Q1. Limited dataset sizes and small model sizes

We tested our approach on conventionally used model sizes that include Base (B) and Large (L) transformer variants on two of the widely used datasets, namely EPIC-Kitchens-100 and Diving48. Although there exists literature [1, 2, 3, 4] on large-scale pre-training and large model sizes for video understanding, we focussed our study on answering the question of how we can effectively pre-train MAEs on 128 frames and whether such pre-training helps. Extending our approach and adopting it in large-scale models is definitely an exciting prospect that we would like to explore, however it is beyond the reach of the current work.

Q2. Limited and narrow range of tasks

During the rebuttal period, we additionally tested our method on Something-Something-v2 dataset. We presented the corresponding results in the common response and observe the same trend as seen in Table 2 of the main paper. Due to time/resource constraints of the rebuttal period we were not able to run similar ablations for models with a VIT-L backbone and for different task types including temporal action localization, but we hope to include these experiments as well in a future version.

Q3. “10%+5% masking even exceeds the case without decoder masking”

We appreciate that the reviewer noticed this. Videos contain redundant information, and we hypothesize that when we reconstruct a lower number of high importance (or higher rank order) tokens during pre-training, the gradients can potentially be stronger than when we give equal importance to all tokens, and this would explain the stronger pre-trained encoder. However, we didn’t notice this behavior consistently across datasets, namely on SSv2. We will include this discussion in the paper.

[1] Wang, Yi, et al. "Internvideo: General video foundation models via generative and discriminative learning." arXiv preprint arXiv:2212.03191 (2022).
[2] Li, Kunchang, et al. "Unmasked teacher: Towards training-efficient video foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[3] Zhao, Long, et al. "Videoprism: A foundational visual encoder for video understanding." arXiv preprint arXiv:2402.13217 (2024).
[4] Wang, Yi, et al. "Internvideo2: Scaling video foundation models for multimodal video understanding." arXiv preprint arXiv:2403.15377 (2024).

Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you

Differentiating EK-Noun SoTA with EK-Verb SoTA

We appreciate the reviewer’s feedback and agree that our EK-100 noun accuracy is below SoTA. However, we would like to humbly point out that SoTA methods for EK-Noun rely on large-scale pretraining, while ours does not. The EK-100 noun categories such as hands, gloves, spoon, knife etc. routinely appear as annotations in large-scale pre-training datasets such as ImageNet21k [1], Ego4D [2] etc. (282/300 nouns from EK-100 also appear in ImageNet21k). As noted by Verbs-In-Action paper [3], verbs are relatively scarce in existing datasets. This explains why methods [4, 5] that use large-scale datasets excel at noun classification. On the other hand, our approach doesn’t use large-scale pre-training datasets and learns long-range spatio-temporal dynamics to achieve absolute SOTA on verb classification. Hence, we argue that our contribution is orthogonal to existing works per Reviewer UhsZ, and we make progress on a different aspect of the problem than data-scaling, namely long-context spatiotemporal actions.

Q1. “Why does the approach only make sota for the verbs?”

Please see above discussion, and also find related discussion in lines 297-304 in the main paper. Performing well on nouns requires understanding object semantics which are benefited by large-scale pre-training data, while performing well on verbs requires spatiotemporal understanding. The focus of our paper is on the second question.

Q2. “... The approach needs to be tested on longer sequence videos”

We extend the temporal context of MAE encoders from typical 16 frames to 128 frames. We found that reconstructing <15% of tokens can lead to performance drop, and with 15% we are already at the limit of the high-bandwidth memory (HBM) of the TPU accelerators at 128 frames. Orthogonal works use sliding window inference [6, 7], non-differentiable external memory [8, 9], frame filtering etc [10] to scale 16 frame video features to longer sequence tasks. In this paper, we demonstrate the utility of pretraining MAEs with longer context and provide novel techniques for doing so efficiently. It is exciting future work to combine what we have discovered with these other orthogonal approaches to further scale up to even longer videos.

Q3. “There is limited innovation on the actual network used”

On the encoder part, it is our intended choice to use a well-known model architecture as vision transformers are the model of choice for video foundation models [11, 12, 13], allowing potential adoption of our method. While MAE architecture has minimal modification, our dual innovation is in (1) designing an adaptive tokenizer and (2) designing a methodology of using importance score from adaptive tokenizer for latent reconstruction to achieve the required decoder efficiency while maintaining / improving performance.

We refer the reviewer to the qualitative results in Figure 3 and Figure 4 of our Appendix that shows that our method of selecting tokens is indeed superior to random selection or flow based methods, and this is corroborated by our empirical results from Table 1 of the main paper.

Q4. “What is the limit on the length of tokens”

We limit the number of decoder tokens to be in the same ballpark as the number of encoder tokens in our MAE pre-training setup, so that the memory, FLOPs and training time are not disproportionately affected by the decoder. Typically, MAE encoders use only 10% of the video tokens and are trained for 1600 to sometimes 3200 epochs [14]. However, as shown in Figure 1 (right) of our main paper, decoder memory and FLOPs disproportionately dominate at 128 frames making such training inefficient. Hence, it is crucial to limit the number of tokens to be approximately same as the number of encoder tokens during pre-training.

Q5. “... dataset performance doesn't seem to make it worth it.”

In Tables 3 (a) and 3 (b), we show SOTA performance on EK-100 verbs and Diving48. We further improved these numbers leading up to the rebuttal as shared in the common response. Moreover, we pre-train in an unsupervised manner without relying on large-scale datasets compared to related approaches.

[1] Ridnik, Tal, et al. "Imagenet-21k pretraining for the masses." arXiv preprint arXiv:2104.10972 (2021).
[2] Grauman, Kristen, et al. "Ego4d: Around the world in 3,000 hours of egocentric video." In CVPR. 2022.
[3] Momeni et al., “Verbs in Action: Improving verb understanding in video-language models”. In ICCV 2023.
[4] Zhao, Yue, and Philipp Krähenbühl. "Training a large video model on a single machine in a day." arXiv preprint arXiv:2309.16669 (2023).
[5] Xiong, Xuehan, et al. "M&m mix: A multimodal multiview transformer ensemble." arXiv preprint arXiv:2206.09852 (2022).
[6] Wang, Xiang, et al. "Proposal relation network for temporal action detection." arXiv preprint arXiv:2106.11812 (2021).
[7] Chen, Guo, et al. "Video mamba suite: State space model as a versatile alternative for video understanding." arXiv preprint arXiv:2403.09626 (2024).
[8] Wu et al., “Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition”. In CVPR 2022.
[9] Balazevic et al., “Memory Consolidation Enables Long-Context Video Understanding”. arXiv 2402.05861.
[10] Tan et al., “Koala: Key frame-conditioned long video-llm”. In CVPR 2024.
[11] Wang, Yi, et al. "Internvideo: General video foundation models via generative and discriminative learning." arXiv preprint arXiv:2212.03191 (2022).
[12] Li, Kunchang, et al. "Unmasked teacher: Towards training-efficient video foundation models." In ICCV 2023.
[13] Zhao, Long, et al. "Videoprism: A foundational visual encoder for video understanding." arXiv preprint arXiv:2402.13217 (2024).
[14] Tong, Zhan, et al. "Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training." Advances in neural information processing systems 35 (2022): 10078-10093.

Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you

We thank the reviewer for the detailed review and for recognizing the novelty, effectiveness and impact of our proposed approach. While we designed our experiments to study the impact of long-video MAE pre-training as such has not been attempted before, we agree with the reviewer that the proposed model can work well across model sizes and video lengths and we also agree that our justification on the choice of tokenizer is lacking. We hope that our below response addresses your concerns:

Q1: Short-video datasets

We favored datasets containing longer action sequences as they can benefit the most from the increased context-length. In addition, we present a study on Something-Something-V2 dataset in the common response where we show the same trend as observed in Table 2 of the main paper.

Q2: Scale model size vs scale number of frames, given our decoder efficiency

This is an interesting direction that we haven’t considered in our study as we focused on how to scale the number of frames. Below, we present some preliminary findings where we study the effect of varying model sizes vs varying the number of frames. For this experiment, we first fix the memory budget and masking ratio to the reference base model trained with 128 frames at 15% decoder masking, and try to maximize the number of frames that we can fit in this memory budget. We report final accuracy and FLOPs relative to the base model.

We find that with larger model size and fixed memory budget the accuracy improves despite lower frames but at the cost of significantly increased compute. We will include this discussion in our paper. Finally, we’d like to reiterate that all the three settings above are only made possible due to the memory savings from our proposed strategy.

Q3: Different number of frames and mask ratios

Our motivation is to study whether we can effectively pre-train MAE on long-videos and whether such pre-training helps. For this, we selected 128 frames as our target (as it is 8x the typical MAE pre-training frames already), and designed our method around achieving this target. As shown in Figure 1 of the main paper, at 15% masking ratio, the decoder memory becomes less than encoder memory and hence we chose it as our masking ratio. Post-facto, we agree that it’s interesting to study the effect of varying number of frames and mask ratios. To satisfy reviewer’s curiosity, we present some related findings:

Varying mask ratio

We ablate the effect of the decoder mask ratio on accuracy while keeping the encoder mask ratio fixed at 10%. For this experiment we use our adaptive masking strategy, a 128 frame context for pre-training and fine-tuning and use adaptive MAGVIT tokens as pre-training targets. A 15% mask ratio yields the highest accuracy, significantly outperforming even the much more expensive setting with 50% mask ratio that requires >4x more memory than the 15% setting.

Varying number of frames

In the table below, we ablate the effect of increasing the number of frames while keeping the mask ratio fixed at 15%. During inference, we ensure that the model sees the same number of frames and report Epic-Kitchens-100 Top1 Noun-Verb Accuracy. We clearly observe a trend of the accuracy improvement with the number of frames similar to what we presented in Table 2 of the main paper. Note that both 64 and 128 frame MAE models go OOM without our decoder masking strategy.

Q4: Tokenizer clarifications

Why does utilizing such a tokenizer significantly reduce the number of tokens?

We believe that the tokenizer can be used to reduce tokens due to two reasons: (1) When we compare with pixels as targets, tokens can encode more surrounding context, and thereby they can have more redundancy. This choice allows us to afford higher masking ratios than used by VideoMAE-V2 [1] with pixels as targets (2) Our adaptive tokenizer training process forces the tokenizer to learn a good importance weighting scheme that gets reflected in the adaptive mask. This lets us have a high masking ratio. In addition, since our mask is learnt independently of MAE, unlike EVEREST [2], our masking scheme outperforms all the other schemes as shown in Table 1 of the main paper.

Why choose MAGVIT instead of other options? Please refer to the common response where we discussed the tokenizer and quantizer.

The 3D tokenizer compresses the temporal dimension several times; could this lead to inferior performance? Temporal downsampling is an inherent result of using the 3D-CNN MagViTv2 architecture (which reduces resolution but provides spatiotemporal features). We have not ablated the effect of temporal compression in the tokens on the downstream performance. However, we believe compression doesn’t necessarily lead to performance drop, as prior works [3] have shown that reconstructing high-level targets in MAE is better than reconstructing low-level targets and this is also corroborated by Table 1 of our main paper.

[1] Wang, Limin, et al. "Videomae v2: Scaling video masked autoencoders with dual masking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[2] Hwang, Sunil, et al. "EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens."
[3] Yu et al. “Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation”. In ICLR 2024.

Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a common response. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you