Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis

We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.

Q1. Were all methods pretrained on the same union of 7 datasets as M$^2$CRL?

Yes, for a fair comparison, all methods were pretrained on the same union of 7 datasets as our M$^2$CRL. We apologize for this and we will make careful revisions in subsequent versions.

Q2. Was Endo-FM used as is for downstream fine-tuning or retrained on this pretraining dataset?

Endo-FM [1] was originally pre-trained on this pretraining dataset, and its backbone was subsequently used for fine-tuning in downstream tasks. Endo-FM serves as our baseline, and the datasets we use are also provided and processed by it.

Q3. How were hyperparameters selected and were they the same for each model and task? Section 4.2 could clarify these details.

For the other methods, we used the hyperparameters as documented in their original papers or provided in their code. Similarly, for the hyperparameters of downstream tasks, we followed the guidelines outlined by Endo-FM [1], which are comprehensively explained in their paper and open-source code. We pretrained all methods on the same union of 7 datasets and then loaded them into the backbone of downstream tasks for fine-tuning. We apologize for this unclear presentation and we will make careful revisions in subsequent versions.

Q4. Appendix A should additionally include fine-tuning hyperparameters.

Thank you for your valuable suggestions. We will update Appendix A to include the fine-tuning hyperparameters as requested.

Q5. The passage from L225-233 is confusing.

We propose a Multi-view Masked Contrastive Representation Learning framework. In the manuscript, L225-233 primarily introduce the contrastive representation learning component of our framework.

L225-229 describe self-distillation and explain the relationship between contrastive learning and self-distillation. It is clarified that the contrastive representation learning component in our framework is achieved through self-distillation.

L229-233 provide a description of the specific workflow of our framework, including data generation, model composition, and data flow within the model. Our model employs a teacher-student architecture, consisting of a student network and a teacher network, which consists of a student network and a teacher network sharing the same structure. It should be noted that the two student networks depicted in Fig. 1(a) actually represent a single student network. We illustrated two student networks in the figure to more clearly convey the data flow.

Q6. Explain why only local views are fed to the student network.

We create two spatiotemporal views (global and local views) of the input endoscopic videos with varying spatial sizes and frame rates. The global views are fed into both the teacher and student networks, while the local views are only fed into the student network. Both the teacher and student networks process these views and predict one view from another in the latent feature space. This approach enables the model to learn spatiotemporal invariant features that can be transferred across different endoscopic domains and disease types.

• Cross-View Matching: The global views are fed into the teacher network primarily because it provides stable and reliable outputs as references for the student network. These global views contain comprehensive information about the video, which assists the teacher network in generating accurate feature representations. The student network processes both the global and local views. By aligning the global features of the student network with those of the teacher network, the student network can learn rich global information. Furthermore, by aligning the local views of the student network with the global features of the teacher network, it can acquire more fine-grained feature representations. This cross-view matching approach involving global and local views facilitates a more comprehensive learning process for a rich feature representation within this model.

• Increased Learning Challenge: Not feeding the local views to the teacher network introduces an information imbalance between the teacher and student networks. This design introduces a learning challenge for the model, as it requires the student network to exert more effort in extracting and integrating information from the local views to compensate for the lack of global information. Consequently, this approach enhances the contrastive learning process by necessitating that the student network maximize agreement between representations of different views despite their partial and incomplete nature. Ultimately, this design encourages the model to develop robust and discriminative features.

Q7. L267 Is this saying these were the baselines considered in the Endo-FM paper?

Endo-FM [1] compares several methods, including TimeSformer, CROP, FAME, ProViCo, VCL, and ST-Adapter. These methods are used for comparison rather than serving as baselines. The authors of Endo-FM pre-trained these methods on the same union of 7 datasets and then fine-tuned them on downstream tasks. Since we followed the same experimental procedure, we have directly utilized the recorded results from Endo-FM instead of re-running the experiments.

Q8. Minor comments:

Thank you for your comments. We will replace the term “mask strategy” with “masking strategy” throughout the entire manuscript in subsequent versions. We will also provide a detailed textual explanation of what Equation (5) represents. Furthermore, we will remove “significant” from Line 275. We sincerely appreciate once again your valuable suggestions, which have greatly contributed to the improvement of our manuscript.

[1] Wang Z, et al. Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train. MICCAI 2023.

We sincerely appreciate your positive feedback on our manuscript. We're glad our efforts to address your questions were satisfactory. Thank you for your support.

We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.

W1 and Q1. Downstream tasks are evaluated on a single endoscopic modality.

In our study, we used multiple publicly available endoscopic video datasets. These datasets cover 3 types of endoscopic procedures (colonoscopy, gastroscopy, and laparoscopy) and 10+ different diseases. We believe that this comprehensive and large-scale dataset is valuable for endoscopic research (as detailed in Table 1). In downstream tasks, classification is for gastroscopy, while segmentation and detection are for colonoscopy.

Based on your suggestion, we have added the Cholec80 dataset in downstream tasks. This dataset contains 80 complete laparoscopic cholecystectomy videos and is specifically designed for laparoscopic surgical recognition. Followed by previous works [1, 2], we used 40, 8, and 32 videos from Cholec80 as our training, validation, and test sets, respectively. We compared our approach with the classical endoscopic foundation model Endo-FM [3] and the masked video modeling method VideoMAE [4] using F1-score in Table 2. The experimental results demonstrate that our approach achieves superior performance. This experiment further validates the robustness and effectiveness of our approach across diverse endoscopic video tasks.

Table 1. Details of pre-train and downstream datasets

Table 2. Results of surgical phase recognition

W2. A cross-validation like approach.

We conducted five-fold cross-validation on the 3 downstream tasks, and the experimental results are presented in Table 3. We observe that the results of cross-validation are lower than those recorded in our original manuscript. We think that this difference is caused by the different ways of data division. For a fair comparison, we strictly followed the public partitioning of the datasets for conducting experiments in our study. Considering that there is no existing work to cross-validate this dataset, we also cross-validated the classical endoscopy foundation model Endo-FM [3] for comparison. The results show that our method outperforms Endo-FM in all three tasks.

Table 3. Results of cross validation for 3 downstream tasks

Q2. Why the segmentation tasks might not take significant benefits compared with the other tasks?

Due to varying requirements in different downstream tasks, the features acquired through different pre-training methods may exhibit distinct performances on specific downstream tasks. Previous studies, such as VideoMAE, DropMAE, and VideoMAE V2, have shown impressive performance in segmentation tasks due to their reliance on masked modeling. The strength of masked modeling lies in capturing rich pixel-level information, making these single-task pretraining approaches particularly effective for dense pixel-level tasks like segmentation. However, their performance tends to underperform in structural tasks that focus on global features and the holistic understanding of objects in images or videos.

Our method integrates masked modeling with contrastive learning, which not only encourages the model to capture fine-grained pixel-level features but also compels it to learn comprehensive discriminative representations. During the pre-training process, the model needs to carefully consider feature selection tradeoffs across multiple pre-tasks. This consideration may result in some downstream tasks exhibiting less prominent performance compared to single pre-task training. Although the improvement of our method in segmentation tasks is not as significant as that using mask modeling techniques alone, our method still achieves a significant improvement in segmentation tasks compared to the method using only contrast learning. Overall, our method achieves robust performance across both structural and pixel-level tasks, highlighting its versatility and effectiveness in learning both detailed and global features.

[1] Ramesh S, et al. Dissecting self-supervised learning methods for surgical computer vision. Medical Image Analysis, 2023.

[2] Czempiel T, et al. Surgical phase recognition with multi-stage temporal convolutional networks. MICCAI 2020.

[3] Wang Z, et al. Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train. MICCAI 2023.

[4] Tong Z, et al. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022.

Thank you for recognizing our efforts in the rebuttal. We sincerely appreciate your consideration in raising our paper's score to 7: Accept. Your feedback has been invaluable to our work. Thank you for your support.

We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.

W1 and Q1. Detail information about downstream tasks.

Our downstream tasks include classification, segmentation, and detection, each addressing different aspects of endoscopic video analysis.

• Classification Task: The classification task is a binary classification problem aimed at diagnosing polyps. We use the PolypDiag [1] dataset, which follows the gastroscope examination protocol and includes 253 videos and 485,561 frames. Each video is annotated to indicate the presence or absence of a lesion. The dataset is divided into 71 normal videos without polyps and 102 abnormal videos with polyps for training, and 20 normal videos and 60 abnormal videos for testing.

• Segmentation Task: The segmentation task aims to perform polyp segmentation. We use the CVC-12k [2] dataset, which follows the colonoscopy examination protocol and serves as the official training dataset for the Automatic Polyp Detection sub-challenge of the MICCAI 2015 competition. This dataset includes 29 videos and 612 frames, with 20 videos allocated for training and 9 videos for testing. Each frame in the videos is annotated with ground truth masks (with a single class) to identify the regions covered by polyps.

• Detection Task: The detection task aims to perform polyp detection. We use the KUMC [3] dataset, which follows the colonoscopy examination protocol and is sourced from the Kansas University Medical Center. This dataset includes 53 videos and 19,832 frames. Each frame in each video is annotated with bounding boxes and polyp categories, with 36 videos allocated for training and 17 videos for testing, including hyperplastic polyps and adenomatous polyps.

For a fair comparison, all datasets are sourced from Endo-FM, and the data partitioning for all datasets follows Endo-FM guidelines. For more details on the experimental reproduction of downstream tasks, we will provide more detailed relevant hyperparameters in the appendix.

W2. Source code is not (yet) released.

We will release the code and pre-trained checkpoints upon acceptance. For reproducibility we build on the open source implementation of Endo-FM and provide all the relevant hyperparameters in the appendix.

W3. Minor Remarks.

Thank you for your valuable suggestions. We will remove the particular results from the Abstract and make careful writing revisions in subsequent revisions.

Q2. Are any modifications needed to employ this method to other spatio-temporal data?

Although our work is based on an endoscopic video dataset, it is important to note that the video format of endoscopic videos is the same as that of natural videos (Kinetics, SSv2). The basic techniques for processing and analyzing video data, such as frame extraction and video decoding, are essentially the same for both types. Therefore, applying our method to other spatio-temporal data does not necessitate specific modifications; rather, it only requires adherence to standard techniques for processing video data.

Limitations. The potential negative societal impact is not sufficiently discussed.

Our study involves self-supervised learning, which requires extensive pre-training leading to significant energy consumption. Additionally, large-scale model training requires high-performance computing hardware (GPUs), leading not only to increased hardware costs but also resource overconsumption and electronic waste. These negative impacts underscore the necessity of considering environmental protection and resource conservation. In future work, we will adopt more efficient training methods and optimization strategies to address these issues.

[1] Tian Y, et al. Contrastive transformer-based multiple instance learning for weakly supervised polyp frame detection. MICCAI 2022.

[2] Bernal J, et al. WM-DOVA maps for accurate polyp highlighting in colonoscopy. Computerized medical imaging and graphics 2015.

[3] Li K, et al. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations. Plos one 2021.

Thank you for recognizing our efforts in rebuttal. Your comments have greatly improved the quality and clarity of our paper. We appreciate your support!

We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.

W1. The novelty.

Existing self-supervised pre-training methods for endoscopic videos predominantly rely on contrastive learning. However, using contrast learning alone is not sufficient to capture the fine-grained feature representations required for endoscopic videos.

• To address the limitations of current methods, we have integrated contrastive learning with masked modeling to effectively acquire endoscopic video representations that possess both comprehensive discriminative capability and fine-grained perceptive ability.

• Unlike the general approach of combining self-distillation and masking modeling, our method specifically designed a novel multi-view masking strategy for endoscopic video features, which significantly improved the model performance. This strategy involves utilizing the frame-aggregated attention guided tube mask to capture global-level spatiotemporal contextual relationships from the global views, while also employing the random tube mask to focus on local variations from the local views.

• We conducted extensive experiments on 10 endoscopic video datasets to evaluate the performance of M$^2$CRL in comparison to other methods. Experimental results demonstrate the superiority of our method.

This work holds inherent value in clinical practice. The development of robust pretrained models for endoscopic video analysis through self-supervised learning can effectively support various downstream tasks, ultimately enhancing clinical workflow efficiency.

W2. Innovation in FAGTM.

Unlike [1] and [2], our work focuses on video rather than static images. Videos have a specific temporal dimension that should be considered in the mask design process to fully leverage spatiotemporal information. Furthermore, endoscopic videos present unique challenges due to their domain-specific characteristics. In this study, we propose a multi-view masking strategy tailored to endoscopic video features, which enhances the model’s robustness and effectiveness.

Specifically, to address the instability of inter-frame variations in endoscopic videos, we designed the FAGTM to capture global spatiotemporal representations. Traditional image-based attention mechanisms only focus on the spatial information of the current frame, neglecting dependencies across the entire video sequence. In contrast, our FAGTM aggregates the attention of frames to the mean value to obtain a simplified and holistic attention distribution. This attention distribution is capable of obtaining an approximation of the critical region of the video that are being attended to from both the temporal and spatial dimensions, while reducing the impact of excessive variations in individual frames or regions. Thus, our FAGTM provides more comprehensive and context-aware guidance for the masking process.

W9. Comparison with more methods.

As you suggested, we have included comparisons with MME, AdaMAE, and UTM methods, as shown in Table 5. The results indicate that our method achieves the best performance in classification and detection tasks and performs moderately well in the segmentation task. This is primarily due to the fact that these mask modeling methods excel in capturing rich pixel-level information, making them particularly effective for dense pixel-level tasks such as segmentation. However, their performance tends to underperform in structural tasks that focus on global features and the holistic understanding of objects in images or videos. Our method integrates masked modeling with contrastive learning, which not only encourages the model to capture fine-grained pixel-level features but also compels it to learn comprehensive discriminative representations. This is friendly for both pixel-level and structured tasks, but it also increases the complexity of model pre-training. During the pre-training process, the model needs to carefully consider feature selection tradeoffs across multiple pre-tasks. This consideration may result in some downstream tasks exhibiting less prominent performance compared to single pre-task training. Overall, our method achieves robust performance across both structural and pixel-level tasks, highlighting its versatility and effectiveness in learning both detailed and global features.

Table 5. Comparison with more methods

W16. The performance on other datasets like Cholec80.

Based on your suggestion, we have added the Cholec80 dataset in our downstream tasks. We compared our approach with the classical endoscopic foundation model Endo-FM [4] and the masked video modeling method VideoMAE [5] using F1-score in Table 9. The experimental results demonstrate that our approach achieves superior performance.

Table 9. Results of surgical phase recognition

Additional comment: We would have liked to include the rest of answers to the questions mentioned by Reviewer wGJR (marked as W3, W4, etc..). Unfortunately, we did not have enough space in this rebuttal box. As soon as the discussion phase will begin, we will include the mentioned answers in an additional comment for the reviewer.

As we mentioned in the main rebuttal, we include the answers to the additional questions of Reviewer wGJR. We hope that this helps to address all remaining concerns and we thank again for taking the time to review our work.

W3. Why self-distillation is a form of contrastive learning？

The pretraining tasks for self-supervised learning are summarized into two categories: contrastive and generative. Contrastive methods focus on maximizing the similarity between different views of the same image after augmentation, while also potentially minimizing the similarity between views of different images after augmentation. Self-distillation is optimized by comparing the feature representations of the teacher and student models to extract a closer representation from the same image. In our work, the self-distillation component also serves to align features across different views, which follows the paradigm of contrastive learning. Therefore, self-distillation can be considered a form of contrastive learning. Furthermore, several other studies [1-3] have also classified self-distillation in self-supervised learning as a type of contrastive learning.

W4. Ablation on teacher’s block used for FAGTM.

In our study, we used the last layer block of the teacher ViT-B for FAGTM. The higher layer block incorporates lower-level features and object-level semantic information, offering comprehensive and abstract features that are essential for the model. Thus, it effectively guides the student network in masking. Table 1 shows that it is most beneficial for FAGTM to utilize the last layer block of the teacher network.

Table 1. Ablations on blocks

W5. Whether to initialize ViT-B?

We pre-trained ViT-B with weight initialization to accelerate convergence and enhance training stability. This way also ensures consistency with the baseline (Endo-FM), which similarly employed weight initialization for pre-training ViT-B.

W6 and W10. Ablation on pre-training epochs. Evaluation or metric to stop pre-training.

For a fair comparison, we used the same number of epochs as Endo-FM. As you suggested, we performed ablation studies on different pre-training epochs, as shown in Table 2. While increasing the number of epochs generally enhances performance, excessive training may result in diminishing returns or overfitting, particularly when dealing with a smaller endoscopic dataset compared to Kinetics. This overfitting can impair the model’s generalization.

During the pre-training process, we stopped pre-training based on monitoring the loss on the training set. If there is no significant decrease in training loss over multiple epochs or if the loss curve becomes flat, it suggests that the model has likely acquired most of the features. At this point, we consider stopping pre-training.

Table 2. Ablations on epochs

W7. Ablation on different architecture.

For a fair comparison, we used the weights for initialization as Endo-FM did. However, since this work did not provide weights for ViT variants, we were unable to conduct ablation experiments on different architectures with weight initialization. Consequently, we conducted a set of ablation experiments without weight initialization for the backbone, as shown in Table 3. It was observed that the performance improvement is more pronounced with larger models due to their increased parameters and more complex structures, enabling them to capture more intricate features. In contrast, smaller models have limited feature extraction capabilities and cannot fully extract visual features. Although larger models exhibit stronger learning abilities, they are more prone to overfitting during training. Additionally, larger models require higher computational resources and longer training times. In conclusion, choosing ViT-B as the pre-trained backbone is a suitable compromise.

Table 3. Ablations on architecture

W8. Pre-train VideoMAE and other baselines using your dataset? Some results using kinetics pre-trained SSL weights.

Yes, to ensure the fairness of experiments, all compared methods were pretrained on the same union of 7 datasets as our M$^2$CRL. As you suggested, Table 4 presented results using SSL weights pretrained on Kinetics for 3 downstream tasks. We observed that the performance of these methods pretrained on Kinetics is comparable to those pretrained on the endoscopic datasets. This can be attributed to the significantly larger size of the Kinetics dataset compared to the endoscopic datasets. Pretraining models on such a large dataset allows them to learn more generalized features and patterns. Consequently, when these models are transferred to endoscopic downstream tasks, their robust feature extraction capabilities allow them to effectively fine-tune on the endoscopic datasets, thereby adapting efficiently to the new tasks. Moreover, despite the substantial differences in content between endoscopic datasets and Kinetics, both consist of color images, which means they share some similar fundamental visual features such as edges and colors. Therefore, models pretrained on Kinetics can capture these common features to some extent, enabling them to perform well when transferred to endoscopic downstream tasks.

Table 4. Results of using Kinetics pre-trained SSL weights on 3 downstream tasks

W11. Can FAGTM be used on local views?

FAGTM cannot be used on local views. In our method, the global views are fed into both the teacher and student networks, allowing the teacher to generate attention corresponding to the global views to guide the student model in masking. However, the local views are only fed into the student networks, thus the teacher does not have attention for the local view to guide masking. This approach is inspired by self-supervised visual transformers [6, 7], where a teacher-student framework is used for contrastive pre-training. In this framework, different views of the video are processed by the teacher and student networks, and predictions are made between views in the latent feature space, enabling the model to learn spatiotemporally invariant features.

W12. The approach looks very sensitive to $\gamma$.

The parameter $\gamma$ serves as the threshold for FAGTM, which is designed to allow the model to sample visible tokens from its attention regions and mask the rest in a reasonable manner. This ensures that our method can effectively perform reconstruction tasks even at a high masking rate. A lower value of $\gamma$ implies selecting visible patches from a smaller high-attention region, potentially leading to an overemphasis on non-critical areas during reconstruction, thus contradicting the objectives of the self-supervised pretraining task. Conversely, a higher value of $\gamma$ means selecting visible patches from a broader region, which may dilute the focus on high-attention areas and adversely affect the model’s learning efficiency.

W14. Ablation of VideoMAE using linear layer decoder.

As you suggested, we evaluated VideoMAE on downstream tasks after pre-training with a linear layer decoder. As shown in Table 7, the impact of the decoder on the experimental results is minimal and almost negligible. The prediction head can be of arbitrary form and capacity, as long as its input conforms with the encoder output and its output accomplishes the prediction target. This has already been validated in SimMIM [8].

Table 7. Ablations on decoder

W15. Why use L1 loss for reconstruction?

The mask modeling component of our model follows SimMIM [8], which employs the L1 loss function. The study demonstrates that different loss functions have minimal impact. To maintain consistency, we used the same loss function as SimMIM. Furthermore, we conducted ablation studies to demonstrate that different loss functions have a negligible effect on our results.

Table 8. Ablations on loss

W17. Some typos.

We apologize for any typos. We will proofread the whole manuscript carefully and make revisions in subsequent versions.

W13. Ablation on different masking ratio for each of the masking strategy.

We conducted ablation experiments on different masking rates for FAGTM (global views) and RTM (local views). As our work is video-related, we aligned our masking rates with those used in other video masking modeling studies. As shown in Table 6, both excessively high and low masking rates are unfavorable for endoscopic representation learning. A high masking rate increases the difficulty of pre-training and hinders the ability of the model to learn effective representations, while a low masking rate reduces the challenge for the model and fails to fully utilize the advantages of masked learning to extract potential features from the video.

Table 6. Masking ratio

[1] Dong X, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining. CVPR 2023.

[2] Gupta A, et al. Siamese masked autoencoders. NeurIPS 2023.

[3] Chen X, et al. Context autoencoder for self-supervised representation learning. IJCV 2024.

[4] Wang Z, et al. Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train. MICCAI 2023.

[5] Tong Z, et al. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022.

[6] Caron M, et al. Emerging properties in self-supervised vision transformers. ICCV 2021.

[7] Ranasinghe K, et al. Self-supervised video transformer. CVPR 2022.

[8] Xie Z, et al. SimMIM: A simple framework for masked image modeling. CVPR 2022.

[9] Wu Q, et al. Dropmae: Masked autoencoders with spatial-attention dropout for tracking tasks. CVPR 2023.

[10] Sun X, et al. Masked motion encoding for self-supervised video representation learning. CVPR 2023.

[11] Bandara, et al. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. CVPR 2023.

[12] Li K, et al. Unmasked teacher: Towards training-efficient video foundation models. ICCV 2023.

I sincerely thanks authors for clarifications and experiments. However, I still have some questions regarding novelty and experiments. I would also suggest the authors to add all new experiments in the revision.

$**Novelty**$

I agree that for surgical videos, most of the papers have been using contrastive learning for pre-training, but recently, there are a few papers that have studied self-distillation and contrastive leanring. EndoFM uses global and local views and apply self-distillation. [1] comprehensively studies the self-distillation and contrastive learning approaches for endoscopic videos. The only component that differentiates this approach with other methods is applying masked modeling on both global and local views. FAGTM module is mere extension of attention guided masking for images.

[1] Dissecting Self-Supervised Learning Methods for Surgical Computer Vision. Medical Image Analysis 2023

$**Comparison with more methods.**$

Thanks for adding more experiments. Did you pre-train MME and AdaMAE from the scratch or did you fine-tune these models for endoscopic downstream task?

$**Cholec80**$

I see some missing baselines like TeCNO[2], LoViT[3]

[2] Tecno: Surgical phase recognition with multi-stage temporal convolutional networks

[3] LoViT: Long Video Transformer for Surgical Phase Recognition

In the above paper [1], table 6 shows F1 score on Cholec-80 dataset. SimCLR, when fine-tuned with 10 labeled videos, can achieve 85.0 and can go upto 93.6 when fine-tuned will all labeled videos. Same goes for other self-supervised approaches like DINO, MoCo v2 and SwaV with temporal model TCN.

$**initialize ViT-B?**$

What would be the performance gap if ViT-B is not initialized with weights? Can we pre-train M$^{2}$CRL from the scratch?

$**Pre-training epochs?**$

The pre-training epochs is too low considering the we are pre-training a transformer using some proxy tasks. I think merely looking at the loss doesn't really justify to stop the pre-training.

$**Ablation on different architecture?**$

I think it would be great to pre-train a larger model using M$^{2}$CRL and compare it random initialization or imagenet/kinetics weight. The current set of experiments don't really tell the whole story. The only thing I can infer is the performance of M$^{2}$CRL can be achieved without weight initialization or pre-training.

$**VideoMAE and other baselines?**$

Did you pre-train them from initialized SSL weights? From the Table 4, these SSL methods achieve the same performance without pre-training on pixel-level tasks like segmentation and detection. M$^{2}$CRL only achieves significant performance on classification task.

$**Approach sensitive to$\gamma$**$

There's a drastic difference in performance when using 0.5 and 0.6 values. I understand the performance variability is using smaller or higher value, but performance gap in 0.5 and 0.6 don't correlate with the authors justifications.