Self-Guided Masked Autoencoder

We appreciate the reviewers' positive comments and constructive feedback. We have made efforts to address each of the concerns raised.

[W1] Quality of informed masks

We acknowledge the concern that initial clusters may not be well-formed for some images. Although we have shown that MAE properly clusters patches through the observations in Sec. 3, it is true that the clustering quality may sometimes be suboptimal. To address this issue, we proposed a 'relevance score' that measures the relationship of the entire token set to the bi-partitioned cluster and then generates masks based on this score (Sec. 4 L222). As a result, our approach yields robust informed mask even in the case when the clustering and bi-partitioning is imperfect, illustrated in Fig. II in Appendix A.

[W2] Training cost

Our method only requires only one more step to generate masks, which empirically increases the pre-training time about 0.25% (for 400 epochs pre-training). We will make this clearer in camera-ready.

[W3] Additional baselines

We thank this consturctive feedback. As AMT is the only model soley relying on MAE archietcture itself, we chose it as baseline. We exclude CL-MAE [1] and ADIOS [2] from our baselines since both of them utilize additional module such as curriculum masking module or occlusion module which are parameterized and need to be trained in addition to MAE. The reported figures in their original papers are not comparable, since CL-MAE experimented only on a subset (20%, randomly selected 200 classes) of ImageNet-1K (IN1K), which does not align with our main purpose of self-supervision on large datasets. Also, as it takes about 6x training time for CL-MAE empirically, we will add CL-MAE in camera-ready due to the short rebuttal period. ADIOS was also experimented in its original paper with a downsized version (224->96) of a subset (10%, ImageNet-100) of IN1K and is not applied to the MAE architecture. Admitting their relevance, we will introduce CL-MAE and ADIOS in the related work section.

Reflecting the reviewer's point, we compare with two other models (SemMAE [3] and HPM [4]) using additional module or external model in Table B of the rebuttal PDF. Since these baselines incorporate additional parameters (external models or extra modules attached to MAE), they slightly outperform ours. However, these models incur significantly larger training cost for the additional parameters, while our method requires only a constant additional time (about 0.25% of the training cost of MAE) for mask generation relying solely on MAE itself without introduction of any additional resources.

[Q1] Ablation on masking ratio

We provide an ablation study on masking ratio in Table C in the rebuttal PDF. A masking ratio of 0.6 shows slightly better performance (+0.1%), while a masking ratio of 0.9 shows degraded performance, which shows similar trend to the results in [5]. We had to conduct this experiment with 200 pre-training epochs due to the short rebuttal period, but will add a similar table with regular 400 epochs of pre-training in camera-ready.

[Q2] Hinting strategy

Both ideas are very interesting, and we explored them in an ablation study. Consideration of using $S_i$-based sampling is shown in the ablation studies (Ln 273-277, Table 3) in main paper, where it exhibited slightly degraded performance. Sampling hint tokens from different clusters is presented in Table I in the supplementary material. We sampled hint tokens from two major clusters alternately throughout the training epochs, which also resulted in slightly lower performance compared to our default setting.

[1] Madan, Neelu, et al. "CL-MAE: Curriculum-Learned Masked Autoencoders." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[2] Shi, Yuge, et al. "Adversarial masking for self-supervised learning." International Conference on Machine Learning. PMLR, 2022.

[3] Li, Gang, et al. "Semmae: Semantic-guided masking for learning masked autoencoders." Advances in Neural Information Processing Systems 35 (2022): 14290-14302.

[4] Wang, Haochen, et al. "Hard patches mining for masked image modeling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Thank you for your valuable feedback. We have tried to alleviate all the concerns raised.

[W1] Additional baselines

We agree that it would be valuable to compare our method with others that rely on external pre-trained models or additional modules, as long as the comparison is fair. We included a table comparing our method to these models, including HPM [1] in rebuttal PDF. (We will update the table with unified 400 pre-training epochs in camera-ready.)

Upon thorough understanding of AttnMask [2], the model suggested by the reviewer, we conclude that this approach DOES supervise the model with labels, making it no longer comparable with our self-supervised method. For this reason, we confirm to exclude this for fair comparison. We highlight that our main contribution is achieving complete independence from any extra resources while successfully expediting the learning process of MAE, as demonstrated in our analysis.

[W2] Scalability

We thank the reviewer for the feedback on scalability. Scalability on pre-training epochs can be found in the supplementary materials (~1600 epochs). Although we would like to include what the reviewer suggested now, it is challenging to conduct experiments on larger model sizes and datasets during this short rebuttal period. We will add the experiments on ViT-L/16 with 400 pre-training epochs compared to MAE in camera-ready.

[Q1] Applicability across various domains and modalities

Since our method accelerates the training process of MAE without altering MAE's inherent learning (i.e., patch clustering), we believe it is applicable to any domain or modality where MAE itself is suitable.

[1] Wang, Haochen, et al. "Hard patches mining for masked image modeling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Kakogeorgiou, Ioannis, et al. "What to hide from your students: Attention-guided masked image modeling." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Thanks for your rebuttal for addressing my questions! I have raised the rating to 6.

I would like to note that clarifications about the comparisons with other works should be included in the final manuscript to improve the clarity of the work.

Thank you for your thorough review and constructive feedback; we have carefully considered your comments and made efforts to address each of the concerns raised. Due to length limitations, we have had to use references from the main paper. We apologize for any inconvenience this may cause.

[W1] Does Fig. 2 indicate that MAE clusters patches based on visual patterns?

We understand that Fig. 2 may not be sufficient to claim visual pattern-based patch clustering. To support this, we refer to Figure III in Appendix, which qualitatively shows token clustering based on their visual pattern. Following your suggestion, we will refer to [40], which demonstrates that 1) MIM models can distinguish tokens and 2) MIM models focus on high-frequency information.

[W2] Attention map with CLS token

Correct. Since the CLS token is not updated during training, it does not contain particularly meaningful information. The main point of Fig. 2c is that if patch vectors are well clustered, they can be distinguished by any random vector. We will clarify Fig. 2c in camera-ready.

[W3] Unclear statement (L126)

We agree that MAE mainly aims to obtain a well-trained encoder. However, we discuss both encoder and decoder to show that the entire architecture of MAE aims to learn patch clustering, as our analysis targets the whole MAE. Also, we apologize for the misuse of the term 'pattern' in L126, which was confusing with 'visual pattern'. It would be better to be replaced with 'trend'.

[W4] Drawbacks of random masking (L195)

Thanks for this clarifying question. In Sec. 3, we discover that MAE is already well-trained to separate the image into major feature clusters in early epochs. After the initial $T$ epochs, MAE becomes sufficiently trained to distinguish tokens into major clusters. Random masking, however, keeps producing easily separable training examples, without taking advantage of its learned knowledge so far. We propose a method to exploit this to expedite the learning process in Sec. 4. In this context, we describe it as 'delaying the learning of patch clustering' and identified this as a drawback of random masking. We will clarify this in the paper.

[W5-1] Difference from [40] (L33)

To clarify, the demonstration in [40] about MIM models is confined to 'token distinguishment', distinct from our observation of 'pattern-based patch clustering'. We claim that merely distinguishing tokens is not sufficient to establish our method, as it does not indicate that MAE can generate meaningful patch clusters for informed masking.

[W5-2] $T=40$ epochs is not early

It may be subjective whether 40 epochs are extremely early or not, but our contribution lies in the discovery that MAE trained for less than $T$ epochs is insufficiently trained and would be inadequate for downstream tasks. As this is the empirical minimum, any pre-training method with $T$ or longer epochs will benefit from our method, and in this context we argue $T$ epochs to be an 'early' stage.

[W6] Difference from prior works (L37)

While our masking strategy shares some common aspects with them, the fundamental difference is that ours relies solely on MAE itself, established through our original analysis of MAE's internal operations. AttnMask [28] relies on supervised training with external labels, unlike our complete self-supervision. We highlight that our main contribution is achieving complete independence from any extra resources while successfully expediting the learning process of MAE.

[W7,9,10,11] Suggestions for writing quality

We thank for the suggestions to improve our manuscript. We will reflect them (overview diagram, notation summary, and summary of experimental implications) in camera-ready.

[W8] Why do we need exploitation rate?

Employing NMI or attention distance might look feasible to decide if the MAE is sufficiently trained to provide informed masks, but they pose the challenge of setting a proper threshold, since they are unbounded. Our exploitation rate quantifies the shared information within the mask tokens compared to the visible tokens, providing a bounded score and thereby easing the thresholding.

[W12] Effectiveness of bi-partitioning in diverse image contexts

We acknowledge the concern regarding images with large objects or multiple objects.

For an image with a large object, only their sub-parts will be masked because the object occupies more than the masking ratio. This scenario still benefits from our method, as intensive masking on the main object still leads to better representation (see Fig. II in Appendix).

As you pointed out, images with many objects might present a challenge for applying informed masking effectively. As there will be many small clusters, masking specific clusters with informed masking may yield similar masks to random masking. However, we emphasize that this limitation is not unique to our method but is a common issue across various informed masking strategies. We will explicitly note this limitation in the paper to provide a clearer understanding of the scope and applicability of our method.

[W13] Using the last encoder layer for informed masking (L238)

We agree with your insight. We simply follow the analysis results and employ the second-last layer, but as pointed out, using the last layer has a minor impact on performance. We will note this in camera-ready.

[W14] Additional baselines

We provide Table B in rebuttal PDF to compare with other methods that use external pre-trained models or additional parameterized modules. Thanks to the external pre-trained models or additional parameterized modules, they slightly outperform our method. However, we emphasize that our method is entirely independent of any extra resources, almost maintaining the original cost of MAE (+0.25% of additional cost), in contrast to other models which require extra cost for training external models and incorporating additional parameters into MAE.

We appreciate the reviewer for positive comments and constructive feedbacks. We have tried to address each of the concerns raised.

[W1] Reason for the analyzing MAE with 400 pre-training epochs

To use a 1600-epoch pre-trained MAE for analysis, we would need to train MAE, MoCo, and ViT for 1600 epochs for a fair comparison. Due to the excessive time and resources required for this setting, we set the training time to 400 epochs following the literature, MAE [1], MoCo [2], and ViT [3]. These seminal works have analyzed with 400 or less epochs, reflecting the fact that 400 epochs is sufficient to obtain a well-trained embedding space for comparison. Nevertheless, we have shown the results with 1600 epochs to demonstrate scalability and effectiveness of our method. Also, all the anlayses have been conducted with models trained on the whole IN1K training set.

[W2] Using fewer tokens for downstream tasks

We think this is an interesting idea. Because MAE distinguishes features solely based on their visual patterns, if we can extract specific clusters of tokens tailored to a given downstream task, it would be feasible to use only these subsets of tokens for those tasks. Although this is beyond the scope of this paper, it would be a great future research topic.

[W3] Training process of our method

The model is trained in a single stage; at epoch $T$, we begin generating informed masks and continue the training process without interruption. We will state this point more clearly in the revised paper.

[W4] Using decoder features for downstream tasks

It is an interesting suggestion to use the decoder layers for downstream tasks, if the MAE is sufficiently trained. Using decoder features might have slight benefit for the tasks requiring pixel-level details, considering the fact that decoder features are closer to the pixel-level space. However, using decoder layers would be less efficient since it requires additional inference time. This would be an interesting future direction for this line of research.

[W5] Training cost

Our method only requires only one more step to generate masks, which empirically increases the pre-training time about 0.25% (for 400 epochs pre-training). We will make this clearer in camera-ready.

[W6] Ablation on masking ratio

We provide an ablation study on masking ratio in Table C in the rebuttal PDF. A masking ratio of 0.6 shows slightly better performance (+0.1%), while a masking ratio of 0.9 shows degraded performance, which shows similar trend to the results in [1]. We had to conduct this experiment with 200 pre-training epochs due to the short rebuttal period, but will add a similar table with regular 400 epochs of pre-training in camera-ready.

For hint tokens, we start by providing 10% of the masked tokens and gradually decrease the ratio to 5% by the end. We will include this information as well.

[W7] Reconstruction loss with and without hints

As shown in the ablation studies in Table 3 in main paper, the learning process can become too challenging with informed masking, making the reconstruction task difficult and ultimately resulting in degraded performance. The reconstruction losses (MSE) after 400 epochs of training for vanialla MAE, ours (with hint) and ours (without hint) are 0.41, 0.56, and 0.64 respectively as shown in Table D in rebuttal pdf.

Although the model can fail to learn generalized feature with a too difficult task, we successfully addressed this issue by providing hint tokens and consistent performance improvements in various downstream tasks verify our claim.

[W8] Difference from SemMAE [4]

Both masking strategies aim to mask out important regions of the image. However, the fundamental difference between SemMAE and ours is that SemMAE is 'supervised' by an external pre-trained model which requires extra training cost for pre-training this model, while ours is completely 'self-supervised'. As SemMAE is supervised by external model, it is not guaranteed that SemMAE still learns what MAE actually learns, i.e., pattern-based patch clustering. The training process of SemMAE can be interpreted as indirect feature distillation via reconstruction task, since it heavily depends on the quality of the features extracted from the external model, e.g., iBoT [5] features containing semantic segmentation information. On the other hand, our work identifies the property of MAE (i.e., patch clustering) emerging during the training process and utilizes this observation to accelerate MAE's learning of this property. In this context, we refer to our method as providing 'acceleration' rather than 'performance improvement' because it speeds up MAE's ability to learn its inherent features, rather than enhancing its feature space with external resources. Since our method relies solely on MAE itself, it is completely free from relying on the attributes of feature representation from external models, and this is the key difference between our method and SemMAE.

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[3] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[4] Li, Gang, et al. "Semmae: Semantic-guided masking for learning masked autoencoders." Advances in Neural Information Processing Systems 35 (2022): 14290-14302.

[5] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).

We appreciate the detailed feedback and valuable insights provided by the reviewer, and we have tried to address all concerns and suggestions.

[W1-2] Connection between the explotation rate analysis in Sec. 3 and our method

We thank the reviewer for this constructive feedback. This will be explicitly stated at the beginning of Sec. 4 in the main paper as detailed below.

In Sec. 3, we show that the MAE encoder learns patch clustering from an early stage through bi-partitioning and KL divergence analyses. As bi-partitioning is sufficient to distinguish tokens into two major token clusters, we can bi-partition the image and mask out one of them. This result indicates that we can generate informed masks with MAE itself early in the pre-training phase and use these informed masks for the remainder of the training. Then, the next question is when exactly the MAE can properly cluster the patches. When the encoder is sufficiently trained to cluster patches, the encoder outputs reflect this information. Then, they are utilized to constitute mask tokens in the decoder. This means the mask tokens possess this patch clustering information and start to be highly exploited for reconstructing masked-out patches. By reversing this order, it can be inferred that a high exploitation rate of mask tokens in the decoder indicates that mask tokens have proper patch clustering information conveyed from the encoder, implying that the encoder can cluster the patches. We verified this via exploitation rate analysis, showing that mask tokens start to be exploited as much as visible tokens from an early epoch ($T$ epoch). This finding allows us to confidently generate informed masks at epoch $T$, ultimately leading to the design of our method.

[W1-1] Analyzing tool in Sec. 3

Following your suggestion, we verify our method using the analysis tools in Sec. 3 in Table A in the rebuttal PDF. The results indicate that our method shows more diversified feature space via higher feature variance and similarity variance, aligned well with the analysis in Sec. 5.

[W2] Difference between SemMAE [1] and our method

Both masking strategies aim to mask out important regions of the image. However, the fundamental difference between SemMAE and ours is that SemMAE is 'supervised' by an external pre-trained model which requires extra training cost for pre-training this model, while ours is completely 'self-supervised'. As SemMAE is supervised by external model, it is not guaranteed that SemMAE still learns what MAE actually learns, i.e., pattern-based patch clustering. The training process of SemMAE can be interpreted as indirect feature distillation via reconstruction task, since it heavily depends on the quality of the features extracted from the external model, e.g., iBoT [2] features containing semantic segmentation information. On the other hand, our work identifies the property of MAE (i.e., patch clustering) emerging during the training process and utilizes this observation to accelerate MAE's learning of this property. In this context, we refer to our method as providing 'acceleration' rather than 'performance improvement' because it speeds up MAE's ability to learn its inherent features, rather than enhancing its feature space with external resources. Since our method relies solely on MAE itself, it is completely free from relying on the attributes of feature representation from external models, and this is the key difference between our method and SemMAE.

[W3, W4] Lack of explanation and inconsistent notation

Thanks for your thorough review. As suggested, we will define the term 'visual pattern' earlier in the paper and explain the hint strategy in the section pertaining to Table 3. Also, the notation $*M*$ in Sec. 4 will be replaced with an alternative notation.

[W5] Performance improvements

We admit that the degree of improvement varies across various tasks, as the reviewer pointed out. We emphasize, however, that our method consistently improves performance across various tasks with different nature, e.g., requiring global understanding (classification) and pixel-level details (segmentation).

Additional comment on limitations

Our method may show less significant improvement when training with excessively fragmented images, especially for the segmentation task. In detail, since there would be numerous clusters within each image, masking specific clusters with informed masking may yield similar masks to random masking. We will note this as a limitation in the paper.

[1] Li, Gang, et al. "Semmae: Semantic-guided masking for learning masked autoencoders." Advances in Neural Information Processing Systems 35 (2022): 14290-14302.

[2] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).

Thank you for your positive feedback. We are pleased that our responses have addressed most of your concerns. We would also like to address your last concern regarding the effectiveness of our method compared to SemMAE.

As shown in Table B, SemMAE requires about 3.15x the training time due to the additional cost of the external model, whereas our approach requires approximately 1.0025x the training time. In this context, we respectfully suggest that for a fair comparison, SemMAE should be evaluated after 133 epochs, while our method would be evaluated after 400 epochs. Even if we optimistically assume that SemMAE maintains the 4.9% performance gap (as reported at 800 epochs) compared to the original MAE at 200 epochs, it may achieve at most around 58.8% accuracy in linear probing at 200 epochs. This indicates that even under this optimistic assumption, where SemMAE is trained for 200 epochs instead of 133, our approach (which achieves 62.9%) would still outperform SemMAE when comparing under the same training cost.

As suggested, to clearly convey the effectiveness of our method especially compared to SemMAE, we will update the performance comparison with SemMAE in Table B to reflect a unified training time in camera-ready.

Regarding the overall presentation, we will clearly explain the connection between the analysis in Section 3 and our method, addressing the initial concerns raised. Additionally, we will include the evaluation of our method using the analysis tools discussed in Section 3 in Section 5.4 for further verification of our method. We will also ensure that the comparison of our contributions relative to SemMAE is explicitly stated in the revised manuscript.

Again, we sincerely appreciate the reviewer’s thoughtful feedback and the effort invested in improving our paper.

I appreciate the author's clarification. The author's response partially addressed my concerns (W3, W4, and partial aspects of other questions). I hope the authors resolve the lack of explanation, inconsistent notation, and limitations of their approach during the revision.

[W1-1] Analyzing tool in Sec. 3

The additional experiments demonstrated the proposed method's improvements in both feature variance and similarity. If possible, during the revision, I hope the authors also show the relation between "the feature variance and similarity" and some quantitative performances, so that enhancing the metrics leads to better performance in downstream tasks

[W2] Difference between SemMAE and our method

I understand the design difference between SemMAE and the proposed method. However, the comparison between them raises questions about whether the proposed method outperforms the masks generated by SemMAE in enhancing MAE's ability to learn its inherent features. It would be valuable to explain how the masks generated by each method differently impact MAE pre-training and to determine which approach is more effective.

[W1-2] Connection between the exploitation rate analysis in Sec. 3 and our method

Thanks to the detailed explanation, which has clarified the background of the proposed method. However, it seems that all the analyses in Section 3 only explain 'what and how MAE exactly learns' and 'when the clustering is sufficiently done'. The justification for "masking the cluster containing the main object" should be provided to bridge the gap between the background in Section 3 and the design choice of the proposed method. Could the authors provide more insight into this?