One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation

Thank you very much for your sincere review, especially for summarizing the strengths of our work, including 1) interesting and well-written work, 2) clear presentation of the claims and confirmed by the results, 3) novel training method of occlusion that is easy to follow and 4) good experiments with common benchmarks, additional promising ablations and generalization validations. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly.

1. (Comment & Question) occlusion time. Thanks so much for your sincere and constructive suggestions. Different from the standard augmentations used in traditional contrastive pre-training, we use the torch module with CUDA acceleration to implement the occlusion operations for our method. Quantitatively, our whole model has a floating point of operations (FLOPs) of 12.03 G, while the operation of occlusion only has 0.12 G FLOPs, accounting for about 1%. It is confirmed that occlusion operations take up very few computation resources and runtime but bring great efficiency improvements.

   Besides, to better demonstrate the efficiency of our model, we conduct the table following the MixMAE to illustrate the comparison with previous methods on ImageNet-1K classification with ViT-B model. All methods are evaluated by Aug., Epoch (Ep.), FLOPs/G, Parameters (Param.)/M, linear probing (LIN) and fine-tuning (FT). The resolution of images is fixed to 224×224. Aug. indicates the utilization of handcrafted view data augmentation during pre-training. FLOPs/G is utilized to show the runtime and computational resources of the pre-training. Param./M is calculated for the encoders of the pre-training model, following the MixMAE. $\dagger$ denotes the results are copied from the MixMAE. Top-1 accuracy (Acc) is used as the metric.

   	Aug.	Ep.	FLOPs (G)	Param. (M)	LIN	FT
   Masked Image Modeling
   BEiT $\dagger$	w/o	800	17.6	87	-	83.2
   MAE $\dagger$	w/o	1,600	17.5	86	68.0	83.6
   CAE $\dagger$	w/o	1,600	17.5	86	70.4	83.9
   I-JEPA	w/o	600	17.5	86	72.9	-
   Contrastive Learning
   DINO	w/	1,600	74.7	171	78.2	82.8
   MoCo v3	w/	600	74.7	171	76.7	83.2
   Masked Image Modeling with Contrastive Learning
   SiameseIM	w/	1,600	16.3	88	78.0	84.1
   ccMIM	w/o	800	39.1	86	68.9	84.2
   ConMIM	w/	800	17.5	86	-	85.3
   MixMAE $\dagger$	w/o	600	15.6	88	61.2	84.6
   iBOT $\dagger$	w/	1,600	17.5	86	79.5	-
   OCL	w/o	800	12.0	86	74.2	83.4

   From the table, our method achieves 12.0 G FLOPs, surpassing the second-best method (MixMAE at 15.6 G) and reducing computational costs by approximately 23%. It is verified that our occlusion operation achieves considerable enhancement of efficiency. We have revised our paper and added more detailed descriptions about occlusion time. Thanks for your sincere suggestion.

Thank you once more for generously dedicating your time to provide a thoughtful review. Your feedback is tremendously valuable, and we are open to hearing from you at any time. If you find our response satisfactory, we would greatly appreciate your assistance in improving our rating score.

Thank you very much for your review, especially for summarizing the strengths of our work, including 1) well-motivated method 2) well-written paper and easy to follow and 3) comprehensive ablation studies. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly.

1. (Weakness 1) computational costs. Thanks very much for your detailed review. Concerning traditional contrastive paradigms such as MoCo v3 and DINO, they use student-teacher dual networks as two branches to process distinct views, leading to almost double the computation cost for one image. However, we did not use two networks as two branches, while we utilized two different parts of an image as two branches. Thus, two branches of contrastive learning in our method will not bring additional computation costs. Cooperating with the masking strategy, we can further reduce the amount of computation for an image. Moreover, our model does not depend on an additional transformer decoder to reconstruct the image, leading to less computation. In summary, we use the above contributions to significantly reduce the computational costs of the pre-training paradigm and improve the efficiency.

   • To better demonstrate the efficiency of our model, we conduct the table following the MixMAE.

     https://s2.loli.net/2025/03/31/RfKjqYMtAxoVQHC.png (Same Table as in Reviewer 2yVR, sorry for limited chars.)

     From the table, our method achieves 12.0 G FLOPs, surpassing the second-best method (MixMAE at 15.6 G) and reducing computational costs by approximately 23%.

   • Moreover, we have presented Figure 3 in our manuscript to illustrate the efficiency and scaling ability of our model, and discussed the computational costs in line 248. OCL is highly scalable compared to previous methods, requiring less computational resources while achieving comparable and competitive results and without relying on handcrafted data augmentations. https://s2.loli.net/2025/03/31/VqHwXUrMgpcGTt1.png

   • Furthermore, we have also conducted ablation experiments on mask ratio and discussed it in section 3.2.1. MASKED RATIO of our manuscript. The results demonstrate that a lower masking ratio (0.4) optimally balances computational efficiency and visual representations. https://s2.loli.net/2025/03/31/VfcqPNk7SXuoIjO.jpg

   • Besides, We also provide the runtime with FLOPs and parameters for Table 4 to better illustrate the ablation of the MLP head from the efficiency perspective. | MLP Head | FLOPs(G) | Param. (M) | Pre-training Hours | Eff. Bsz. | LIN | FT | |--|--|--|--|--|--|--| | w/o | 42.3 | 303 | 559 | 1,800 | 77.4 | 85.7 | | 2-layer | 43.6 | 305 | 600 | 1,800 | 76.7 | 85.6 | | 3-layer | 43.7 | 306 | 611 | 1,800 | 76.6 | 85.6 |

   We have added the discussion about the computational costs of two branches in our contrastive paradigm, and added the Table of pre-training details of the ViT-B model to show the runtime and effective pre-training epochs, and revised our Table 4 to add FLOPs and Parameters. Thanks for your constructive suggestions.

2. (Weakness 2) fine-tuning and linear-probing results. Many thanks for your sincere review. We are sorry for the misunderstanding of our theme in this paper. Our goal is to provide a new pre-training paradigm for efficient visual representation with affordable training time and reasonable computation cost. Actually, our method reduces computational costs while maintaining efficiency. From the FLOPs comparison in the table above, we can see that our FLOPs of 12.0 G significantly exceed the second-best MixMAE with 15.6 G, improving the efficiency by about 23%. With such significant improvement in efficiency, our method achieves comparable and competitive results of fine-tuning and linear-probing to previous SOTA contrastive learning and MIM methods. We have added a more detailed discussion about the fine-tuning and linear-probing results to clarify the misunderstanding. Thanks for your sincere review.

3. (Weakness 3) detection and segmentation results. Thanks for your comments. We found it is quite similar to the weakness 2. We provide a variety of downstream tasks to show that our pre-trained model has the generalization ability competitive with previous SOTA methods. However, our core contribution lies in proposing a simple pre-training paradigm that balances efficiency and performance, no bells and whistles. We hope it helps us train larger vision models and replicate the success of LLM. We have also added a discussion about the relationship between downstream tasks and our theme. Thanks for your sincere review.

We appreciate your time and thorough review. Your feedback is highly valuable to us, and we welcome further communication from you. If you are satisfied with our response, we would be grateful for your support in enhancing our rating score.

We sincerely appreciate your thoughtful and detailed review. Your positive feedback serves as a great source of encouragement for our team, especially recognizing the strengths: 1) efficient and simple framework, 2) not relying on specific masking strategies and 3) higher level of semantic abstraction than standard MIM methods. Furthermore, we have diligently responded to each of your questions and incorporated your feedback into our manuscript revisions:

1. (Weakness 1) efficiency about epochs and views generations Thanks very much for detailed and thoughtful review. We sincerely apologize for the typo in Table 6, that we actually used 800 epochs for pre-training, which is consistent with our Table 7, Table 8 and the description in line 380 of the paper. We have corrected this typo and double-checked the entire manuscript to make sure there are no further typos.

   Besides, we are sorry for the misunderstanding of the efficiency of views generation. We do take into account the time it takes to generate two views. As illustrated in Algorithm 1, we conduct generations of two views in torch module MiCLAutoencoderViT (code in models_mae.py) with CUDA acceleration, instead of image pre-processing using CPU like DINO and MoCo v3. Thus, our method has considered the efficiency of views generation. And we have clarified this misunderstanding in our revised manuscript.

2. (Weakness 2) runtime and training epochs. Many thanks for your kind suggestions. To better demonstrate the efficiency of our model, we conduct the table following the MixMAE.

   https://s2.loli.net/2025/03/31/RfKjqYMtAxoVQHC.png (Same Table as in Reviewer 2yVR, sorry for limited chars.)

   From the table, our FLOPs of 12.0 G significantly exceed those of the second-best method, MixMAE (15.6 G), improving efficiency by about 23%. Concerning DINO and MoCo v3, they leverage student-teacher dual networks to pre-train, leading to higher FLOPs and Parameters of the pre-training encoder. We have revised the paper and added this table to illustrate the efficiency of our method more clearly.

3. (Weakness 3 & Question 2) the sources of the speedup. We sincerely appreciate your kind and insightful suggestions. The acceleration of our method primarily stems from two key factors:

   • First, the masking strategy significantly reduces the number of tokens processed by the ViT for a single image. With a ratio of 0.4, only a part of tokens participate in pre-training, dramatically decreasing the computational load required by the ViT.
   • Second, the contrastive framework eliminates the need for additional modules to reconstruct the image. Traditional MIM methods and most MIM-CL hybrid approaches employ a transformer decoder for image reconstruction, which incurs substantial computational overhead. In contrast, our OCL requires neither a decoder nor an MLP head, thereby reducing pre-training computations and enhancing efficiency.

   As shown in Table of Answer 2, our method requires significantly less training time per epoch compared to MAE and I-JEPA. Specifically, OCL achieves 12.0G FLOPs, surpassing MAE of 17.5G by approximately 31% through elimination of decoder dependency. Moreover, I-JEPA’s higher FLOPs (29.8 G) result from its dual encoders (context and target) to generate different views. We have revised the paper to provide a more detailed explanation of these acceleration mechanisms.

4. (Weakness 4) frozen patch layer. Sincerely thanks for your thoughtful and helpful suggestions. We did experience training failures. While we were more concerned about the efficiency of the model's pre-training, so we simply changed the random seed and retrained. We will continue to explore solutions to model failures in future research.

5. (Question 1) global masking strategy. Thank you for valuable suggestions. The global masking strategy is designed to enhance representation learning by addressing semantic redundancy in images. As highlighted by the MAE and discussed in line 32 of our paper, images inherently carry redundant semantics. Global masking prunes unnecessary patches, forcing the model to abstract high-level semantic patterns. We have revised the paper to provide a detailed explanation of the global masking strategy.

6. (Question 3) the relationship between MLP head and loss. Thanks for valuable suggestions. Though distinct in mechanism, both T-SP loss and MLP head enhance visual representations. We have added additional ablation experiments and related discussion to validate the relationship between MLP head and T-SP loss.

   T-SP Loss	MLP Head	LIN	FT
   √	√	77.0	85.5
   x	√	72.4	85.1
   √	×	77.9	85.8
   ×	×	61.3	82.6

Thanks again for your valuable time and careful review. Your feedback is immensely valuable to us. If you find our response satisfactory, could you please consider helping us improve our rating score?