ViDROP: Video Dense Representation through Spatio-Temporal Sparsity

1. In Table 1(d) and lines 358-359, the experiments should be conducted on the same batch size while altering the drop rate, to mitigate the impact of the batch size on training efficiency (and perhaps the final performance). Additionally, the authors found that the model trained on full tokens (0% drop rate) achieves significantly lower performance than the base model. In this case, the authors are encouraged to study more drop ratios, e.g. which ratio leads to the best performance.
2. It is not very clear how K-means works in Section 3.4, what is the actual input to the model after clustering?
3. The authors claimed that K-means accelerates the data loading process. The comparison of data loading time as well as the training time should be reported. As a fair comparison, the efficiency and performance of training on randomly selected image patches (vs K-means) are not reported.
4. Regarding action recognition, the authors are encouraged to additionally report the full fine-tuning results.

The main drawback for me is the way the rest of the paper (apart from abstract and introduction) is written. It is mostly confusing, and sometimes misleading and missing a lot of details that should have been included to answer the questions of the readers. I will give plentiful examples for this in the rest of the review.

• It seems the technical contribution of this paper is limited. The proposed dropping and masking strategies are both existing techniques. The combination of them is also very straightforward. Using the dropping strategy to speed up the pre-training process of contrastive learning is not new, and has been studied in the following paper.

• The whole pre-pretraining pipeline and loss functions are very similar to DINO v2 and the description in Section 3.2 is very short, indicating the techinical novelty is limited.

• The technical description is very short and some remains unclear: For example, the illustration in Figure 2 is not clear and can not tell how to calculate the training loss. Meanwhile, it is unclear how to model the connection between the large crops and small crops. It is suggested to provide a clear and detailed pipeline figure to illustrate the whole pre-training process. Second, the motivation and description on lossy data loading is unclear. The very short description in Section 3.4 is really hard to find the implementation details.

• About evaluation, although linear probing evaluation with frozen backbone is very efficient, the full fine-tuning is more effective and can test the up-bound performance of the pre-trained model. Meanwhile, the linear probing evaluation is more friendly to consistency-based method than reconstruction methods. The contrastive loss is a kind of discriminative objective and encourges the learned representation to be more sepearable for classification. But, the reconstruction loss is only for pixel prediction and is less effective for classification if the pre-trained is frozen. So, I think we should test and compare the effectivness of learned representation in both linear probe setting (for generation power) and full fine-tuning setting ( for its performance limit).

• The performance comparison in Table 6 is confused. The performance drop against VJEPA is very large on the SSv2 dataset. Is there any reason behind this result? Meanwhile, from ViT-Large to ViT-Huge, the performance improvement is very small on the Kinetics dataset.

• The proposed method is not independent SSL method and its good performance is still dependent on the VideoMAE pre-trained or VJEPA pre-trained models. It is really important to improve the performance without initialization from previous checkpoints.

1. The writing of the paper needs substantial improvement, many technical details are missing or not clear, and some experiment results are not explained.

1.1. What are the global-local consistency loss and global-global consistency loss in L212 -- L213? If the losses are used in DINOv2, maybe it's more clear to specify the equation number or the corresponding name in DINOv2.

1.2. In L303, why the ViT-L and ViT-H are trained without compression at all? Is it because video compression provides less speedup when larger models is used (data loading is not a bottleneck anymore). I'd suggest the authors to describe the experimental setup and details more clearly.

1.3. From L311 and L372, I don't really understand the difference of the masking strategies between the two halves, and the reason (or ablation) for using different strategies is not explained.

1.4. Table 1f shows that better results can be obtained by using (min=0.5, max=0.7), and the improvement is even more significant than those shown in other ablation. Why not using this as the default setting?

1.5. It's not clear how the K-Means based video compression method provides speedup for data loading. Is it due to the lower resolution when compressing each 10x10 patch? If so, it'll be also interesting to compare the compression quality (PSNR) with some simple strategies such as taking the mean or median for each 10x10 patch.

1.6. In Table 6, significant performance drop is observed for K400-Attn. and SSv2-Linear settings. The paper doesn't provide any explanation or analysis to these two important results.

2. It is not necessarily a key concern but overall the novelty of this paper is limited because most of the training strategies are proposed in prior work such as MAE and DINOv2.

Apples-to-apples comparison in main results (Sec 4.3 and Table 6)

• The paper specifies the use of VideoMAE checkpoints at 1600 epochs as initialization (lines 464-466), and use them as baselines to report performance improvements in Table 6
• However, since the proposed approach performs additional training on top of these checkpoints, it is not clear whether these comparisons are fair
• Couple of ways the comparison can be made apple-to-apples is
  • to compare VideoMAE(1600ep) with VideoMAE(800ep) + ViDROP(800ep).
  • or training VideoMAE(1600ep) for the additional number of epochs that the paper uses to train ViDROP
• The authors listed a strategy to scale model size from scratch in lines 305-309, and provide results for ViT-Huge variant in Table 6. Do the authors have the results for ViT-Large trained from scratch?

Is linear probing the best way to evaluate VideoMAE features?

• The authors consistently use linear probing to draw comparisons with VideoMAE, but only fine-tuning is used in their paper [1]. Did the authors try fine-tuning?

Naive baseline

• In Figure 4, the authors made a comparison with a variant of the model with no token dropping, which is close to DINOv2 approach (Fig 2c), and shows that it converges slower
• On the other hand, the approach also address the limitation of VideoMAE-style encoder-decoder approach (Fig 2a). One baseline that could be included is to use the same drop+mask strategy but reconstruct mask tokens using a decoder. This would help draw direct comparisons with VideoMAE architecture and give readers some insights.
• Moreover this would highlight whether the improvement is brought by the use of additional losses or model architecture

Lack of notations and uniform use of terminology

• Table 1 caption (line 272) specifies a “masking ratio is 85%” but lines 319-320 mentions “mask uniformly between 10% to 40% of the tokens”. This creates a confusion between mask and drop tokens, and readers are left to parse the details.

  • Notations for drop and mask ratios could be introduced in the text to as done in previous works such as VideoMAEv2 [1] ($\rho$)

• Further, tokens that are masked are denoted as “MSK” in Figure 2 and lines 203-204, but later denoted as “MASK” in Table 1 and line 323

• In Table 1, although 85% is desirable w.r.t. accuracy and time trade-off, the accuracy in bold should be the first row (for 80%)

Additional details (such as supplementary material) would have been useful

• A model card containing complete hyperparameter setting for the experiments and details on model architecture. For example, VideoMAEv2 [1] shared details of model architecture along with pre-training, post pre-training, and fine-tuning settings
• Details regarding k-means clustering, and how it is done on 10x10 pixel patches (lines 255-257) would be useful

Justification

The paper shows strong performance of the approach on various benchmarks, but it is not clear whether we should favour the proposed approach over VideoMAE or DINO line of research. The results seems to lack head-to-head comparisons, use of naive baselines, and additional details to verify the claims and draw conclusions.

Please note

• I have not checked the accuracy of the claim that linear probing results are state-of-the-art
• And I did not fully understand the k-means clustering details mentioned in lines 255-257

References

[1] Wang, Limin, et al. "Videomae v2: Scaling video masked autoencoders with dual masking." In CVPR 2023.