Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning

Overall

Thank you for your thoughtful review and for highlighting the applicability and clarity of our framework. We have carefully addressed your concerns below.

W1

Thank you for your insightful comment. While we agree with the reviewer that dual-stream and frequency-domain masking approaches are common, our motivation for FOLK is to address two specific limitations of Masked Frequency Modeling (MFM).

Firstly, MFM lacks a mechanism to expose the model to natural views of images during pre-training. This limitation especially hampers performance in few-shot learning scenarios (see “MFM” in Table 2). To specifically address this limitation, we incorporated self-distillation into FOLK, as it perfectly serves to familiarize the model with natural images during pre-training. In our approach, the teacher model processes the original image (information-rich), while the student model processes the frequency-masked image (information-poor) and receives guidance from the teacher’s output. Though this is analogous to DINO’s [1] teacher global views (information-rich) and student local views (information-poor), FOLK’s specific self-distillation design has not been explicitly explored by previous studies, and it does not require as many student frequency-masked views as DINO’s local views (i.e. 6). Such an approach allows the student to benefit from both masked and natural image representations, enhancing training stability and improving downstream task performance. This has been elaborated in lines 107-114, 312-318, 388-401, and 1345-1352 in the paper.

Secondly, though MFM offers unique advantages compared to Masked Image Modeling (lines 52-80), the constant frequency filters used in the original MFM result in training inefficiency (lines 248-253, and “MFM*” compared to “MFM + R/Com*” in Tables 1 and 2). To address this limitation, we propose the Com/Rcom filters to adaptively mask (in)significant frequencies based on each image’s unique frequency components, hence presenting a challenging pretext task for effective pre-training (lines 255-264, Tables 1 and 2, and Appendix Section B.5.5).

[1] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

W2

Please see our comments on Q1 and Q2 below.

Q1

Thank you for raising the question regarding the selection of views u and v in the FOLK framework. We agree that the data augmentation techniques used to create different views are critical factors for model training. Previous studies have extensively explored this topic, including iBOT [1], DINO [2], AttMask [3], and FixMatch [4]. In our work, we have followed the same augmentations used in DINO to facilitate a fair comparison with other methods.

We apologize for not being clear on how we chose these augmentation steps, and we have now clarified it in the updated manuscript. Please refer to lines 321-322.

[1] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).

[2] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[3] Kakogeorgiou, Ioannis, et al. "What to hide from your students: Attention-guided masked image modeling." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." Advances in neural information processing systems 33 (2020): 596-608.

Q2

Please see our comment on Weakness 1 (W1) from reviewer tJxA.

Dear Reviewer nBwS,

We appreciate your valuable comments on our paper. We have prepared a rebuttal with an updated manuscript and tried our best to address most, if not all, of your concerns. We notice that the author-reviewer discussion period is coming to an end, and we are willing to answer any unresolved or further questions that you may have regarding our rebuttal if time is allowed.

If our rebuttal has addressed your concerns, we would appreciate it if you would be willing to consider raising your original rating. Thank you for your time and consideration.

Best,

Authors

Thank you for answering my concerns.

Overall

Thank you for your positive feedback. We are pleased to hear that you found our work "novel" and our presentation "clear." We appreciate your recognition of our contributions to the field of self-supervised learning.

We have carefully addressed your concerns below. Regarding one of your points, we kindly refer you to our response to another reviewer, as this concern overlaps with theirs. Answers:

W1

Thank you for highlighting this important consideration regarding the training cost of our proposed method. We acknowledge that our dual-branch framework, which incorporates knowledge distillation, introduces additional computational overhead compared to the original MFM. This increased cost stems from processing both the teacher and student models during pre-training.

However, we designed this framework specifically to address key limitations of MFM, particularly its inability to effectively handle few-shot learning scenarios. By enabling the model to learn from both masked and unmasked views of the data through self-distillation, we achieve significant improvements across various downstream tasks, especially in few-shot learning scenarios. We believe that these improvements justify the additional computational resources required during pre-training.

Moreover, it's important to note that pre-training is a one-time investment. Once the model is pre-trained, it can be fine-tuned for various tasks without incurring the same level of computational cost again. In many practical applications, the enhanced performance and versatility of the pre-trained model can outweigh the initial training overhead. The following table provides a detailed comparison between FOLK and MFM in terms of training time (number of seconds per epoch on the ImageNet-1K dataset), GPU usage, and few-shot learning accuracy (from Table 2 in the paper).

W2

Thank you for highlighting the need for more explicit formulations of the Com and RCom masking filters. While Section 3.2.1 (“Informed Filters”) in the main paper discusses the conceptual design of these filters, we agree that providing exact equations or pseudocode would greatly enhance the clarity and facilitate a better understanding of our method for the readers.

We have now added pseudocode to the appendix (Section A.4) to better illustrate the implementation details of the proposed Com and Rcom filters.

W3

Please see our comment on Question 1 (Q1) of reviewer nBwS.

Dear Reviewer tJxA,

We appreciate your valuable comments on our paper. We have prepared a rebuttal with an updated manuscript and tried our best to address most, if not all, of your concerns. We notice that the author-reviewer discussion period is coming to an end, and we are willing to answer any unresolved or further questions that you may have regarding our rebuttal if time is allowed.

If our rebuttal has addressed your concerns, we would appreciate it if you would be willing to consider raising your original rating. Thank you for your time and consideration.

Best,

Authors

Overall

Thank you very much for your positive and encouraging feedback on our paper. We are glad to hear that you found our method “well-motivated”, the paper “well-written”, and that our proposed solutions effectively address the key limitations of MFM. Your support means a lot to us, and we hope our work provides valuable insights to the community.

We have carefully addressed your concerns below. For one of your points, we would like to kindly direct you to our response to another reviewer, as this concern is shared between both reviews.

W1

Thank you for your insightful observation regarding the reliance on pre-defined thresholds in our adaptive filters. We agree that hyperparameter tuning can be a concern when applying the method to different datasets. To address this, we have provided guidance in the paper (line 269 in the main text, and Appendix Section B.5.1) on selecting threshold values without extensive tuning.

In practice, we found that selecting thresholds that retain or mask approximately 0.2% to 7% of the highest-magnitude frequencies yields good performance across various datasets. This range ensures that the frequency-masked images are visually distinct, enhancing the diversity of the training data. By observing the images produced with these thresholds, practitioners can balance retaining important image structures with introducing sufficient masking to challenge the model during pre-training. Please also see the images in Tables 13, 14, and 15 in the appendix that illustrate the effect of frequency masking with different thresholds.

Additionally, our empirical results demonstrate that the model's performance remains robust as long as the thresholds are spread out to produce varying levels of frequency masking (Appendix B.5.1 and Table 8). This approach minimizes the need for fine-grained tuning of threshold values, simplifying the practical application of our method across different datasets.

W2

Thank you for pointing out your concern regarding the effectiveness of FOLK with CNN architectures in the full fine-tuning setting. We acknowledge that, as shown in Table 4 of Section B.2, FOLK does not exhibit significant gains over MFM for CNN-based models when fully fine-tuned. However, we'd like to emphasize that one of FOLK's key strengths lies in few-shot learning scenarios, which are particularly critical in the self-supervised learning domain due to the common scarcity of labeled data.

In lines 961 to 999 in the appendix, and as presented in Table 5, we demonstrate that FOLK significantly outperforms MFM in few-shot settings, even with CNN backbones. Specifically, FOLK achieves substantially higher accuracy with only 1% and 10% of labeled data compared to MFM. This highlights FOLK's robustness and effectiveness in scenarios with limited labeled data, making it a superior choice for practical applications where acquiring large amounts of labeled data is challenging.

W3

Please see our comment on Weakness 1 (W1) from reviewer tJxA.

Dear Reviewer 6EN5,

We appreciate your valuable comments on our paper. We have prepared a rebuttal with an updated manuscript and tried our best to address most, if not all, of your concerns. We notice that the author-reviewer discussion period is coming to an end, and we are willing to answer any unresolved or further questions that you may have regarding our rebuttal if time is allowed.

Thank you once again for your time and your encouraging feedback on our paper.

Best,

Authors

Overall

Thank you for your thoughtful review and for acknowledging the effectiveness of our proposed method. We have carefully addressed your concerns below.

W1

Thank you for pointing out that our experiments did not directly address the two limitations proposed in the introduction. We apologize for this oversight and appreciate the opportunity to clarify how our work addresses these issues.

Addressing the First Limitation:

To demonstrate our filters are effective, we show that they present a harder pretext task for the model during pretraining compared to the constant filters originally used in MFM, in terms of the loss progression. The results are as follows:

The higher loss values for MFM + Com/RCom indicate that our informed filters present a more challenging task for the model compared to the constant filters used in MFM. A higher loss suggests that the model finds it more difficult to predict the masked frequencies, which implies that our filters are more effective in encouraging the model to learn richer representations. This directly addresses the first limitation by providing a more complex and informative masking strategy.

We have now included this analysis in Appendix Section B.5.5 in the updated manuscript.

Addressing the Second Limitation:

Regarding the second limitation, we would like to highlight the significant improvements achieved by our FOLK model in few-shot learning scenarios compared to MFM. While FOLK offers only marginal performance improvements over MFM in the full fine-tuning scenario, these gains are still meaningful. More importantly, in few-shot learning scenarios, FOLK demonstrates significantly greater enhancements, which more accurately reflect the limitations of MFM. As MFM lacks a mechanism to expose the model to natural images during pretraining, its performance in few-shot learning is greatly hampered—a limitation that may remain unnoticed in full fine-tuning due to the abundance of training data. In contrast, FOLK incorporates a self-distillation design that enables the student model to be exposed to natural, unmasked images by learning from the teacher model during pre-training. This approach mitigates MFM’s limitation and enhances the model’s adaptability and performance when labeled data is scarce.

We appreciate your insightful feedback and hope that this clarification demonstrates how our experiments support the claims made regarding the proposed limitations.

Q1

In our paper, we focused on the Com and RCom filters because they are specifically designed to highlight the main semantics of an image by preserving significant frequencies (Com) and reconstructing finer details (RCom). We discuss the rationale for choosing these filters in Section 3.2.1 (lines 255–264).

To address your concern, we would like to kindly refer you to our experiments with additional filtering methods, including Gabor filters and other techniques. The results of these comparisons are detailed in Appendix B.5.4. Our ablation studies demonstrate that the Com/RCom filters outperform other filters in terms of model accuracy. For instance, as shown in Table 11, the Com/RCom filters achieved higher accuracy compared to Gabor filters, token masks, and circle masks when using a ViT-S/16 model. We believe this demonstrates the robustness and generalizability of our approach.

We appreciate your recommendation and will consider exploring additional filtering methods, such as wavelet transforms, in future work to further validate and enhance the FOLK framework.

Q2

Thank you for highlighting the importance of evaluating our proposed method on low-level vision tasks for a comprehensive assessment. We fully agree with your suggestion. In our paper, we have conducted experiments on semantic segmentation, which is a representative low-level vision task and the only one evaluated by MFM. Specifically, we evaluated FOLK on the ADE20K dataset, following the methodology used by MFM. The results of these experiments are detailed in Appendix B.1. Our findings show that FOLK outperforms MFM and other baseline methods on semantic segmentation, demonstrating that it not only excels in high-level vision tasks but also provides improved performance on low-level vision tasks.

Dear Reviewer DqRd,

We appreciate your valuable comments on our paper. We have prepared a rebuttal with an updated manuscript and tried our best to address most, if not all, of your concerns. We notice that the author-reviewer discussion period is coming to an end, and we are willing to answer any unresolved or further questions that you may have regarding our rebuttal if time is allowed.

If our rebuttal has addressed your concerns, we would appreciate it if you would be willing to consider raising your original rating. Thank you for your time and consideration.

Best,

Authors