Empirical Study on Enhancing Efficiency in Masked Image Modeling Pre-training

W1: The problem of low efficiency in MIM has been pointed out by previous works as the paper stated. The idea of using low-resolution images to expedite training is too simple and has been implemented in previous works. Additionally, the superiority of reconstructing HOG features has been demonstrated by its original paper.

A1: Thank you for your valuable feedback. We respectfully highlight the following key aspects to address these concerns and clarify the contributions of our paper:

1. Positioning of our paper as a Practical Paradigm

   • First, weFastMIM is positioned as an empirical study, inspired by works [1,2], that thoroughly examines prior MIM methods and identifies a critical research gap: self-supervised pre-training demands enormous computational resources, making it impractical for single deep learning machines. Our philosophy is to break the barrier of costly pre-training by providing a solution that is both efficient and deployable, enabling broader accessibility to MIM research.

   [1] Xinlei Chen, Saining Xie, et al. "An empirical study of training self-supervised vision transformers." ICCV 2021.

   [2] Zi-Yi Dou, Yichong Xu, et al. "An empirical study of training end-to-end vision-and-language transformers." CVPR 2022.

2. Key Contributions
   While both small input sizes and HOG targets have been explored independently in prior works, our contributions are two-fold:

   • First, we demonstrate for the first time that combining these two factors leads to a qualitative transformation in efficiency and practicality for MIM. Unlike previous works that consider these aspects in isolation, our study reveals the synergistic effect of small input sizes and HOG targets in addressing computational bottlenecks, offering a clear and actionable pathway for efficient pre-training.

   • Second, our motivations are fundamentally different from related works such as SimMIM. For instance:

     • SimMIM only slightly reduces input size (from $224^2$ to $192^2$), aiming to mitigate memory overhead. This minor reduction, however, is insufficient to drastically improve training efficiency.

     • In contrast, FastMIM adopts a significantly smaller input size, reducing the resolution by a factor of four. This results in a substantial acceleration in training and memory savings, as shown in Table below:

       Case	GPU	Memory	# GPUs	PT Days
       EVA	A100	~27GB	128	~14.5
       SwinV2	A100	<40GB	8	~20
       CAE	V100	<32GB	8	~12
       SimMIM	V100	~28GB	8	~7
       FastMIM	V100	~14GB	8	~1.6

       FastMIM reduces memory consumption to 14GB on a V100 GPU (compared to $\sim$28GB for SimMIM) and achieves pre-training in 1.6 days, compared to $\sim$7 days for SimMIM, $\sim$12 days for CAE, and $\sim$14.5 days for EVA. These results clearly demonstrate the practicality and scalability of our approach.

3. To substantiate our claims, we conducted extensive experiments:

   • Robustness to Input Resolution: Our findings (Table 1a in the main paper) indicate that a wide range of input resolutions yield comparable performance. This highlights that the efficacy of MIM is preserved even with smaller inputs, a result not emphasized in prior works.

   • Significance of Input Size in Pre-training: We explicitly analyze the impact of input resolution during pre-training and fine-tuning (Table 8 in the main paper), providing new insights into why reducing resolution during pre-training improves both efficiency and downstream performance.

W1: The proposed method appear less novel.

A1: Thank you for your valuable feedback. We respectfully highlight the following key aspects to address these concerns and clarify the contributions of our paper. Our paper is positioned as an empirical study, inspired by works [1,2], that thoroughly examines prior MIM methods and identifies a critical research gap: self-supervised pre-training demands enormous computational resources, making it impractical for single deep learning machines. Our philosophy is to break the barrier of costly pre-training by providing a solution that is both efficient and deployable, enabling broader accessibility to MIM research.

[1] Xinlei Chen, Saining Xie, et al. "An empirical study of training self-supervised vision transformers." ICCV 2021.

[2] Zi-Yi Dou, Yichong Xu, et al. "An empirical study of training end-to-end vision-and-language transformers." CVPR 2022.

The two key factors behind our method, (i) smaller input size and (ii) HOG target, have been proposed in earlier works. However, our contributions lie in two critical differences:

• Combination of Factors: We are the first to highlight that the combination of these two factors enables a significantly more efficient and practical MIM framework, a direction not yet explored in the literature.
• Different Motivations: The motivations of our FastMIM differ fundamentally from those of SimMIM and MaskFeat. While SimMIM slightly reduces input size to mitigate memory overhead, our FastMIM deliberately adopts a much smaller input resolution to expedite training while maintaining comparable performance. Additionally, our experiments reveal that a wide range of input resolutions achieve similar fine-tuning results, underscoring the flexibility of our approach.

Comparison with SimMIM

• Motivation: SimMIM focuses on improving representation learning performance by analyzing various components. In contrast, FastMIM is designed to provide a faster training paradigm with reduced memory consumption.
• Input Resolution: SimMIM pre-trains ViT-B with a 224$^2$ input size and Swin-B/L/H with 192$^2$ input size to address computational overhead but does not explore the effects of further reducing input resolution. By directly lowering the input resolution to $128^2$, FastMIM significantly reduces the number of patches, leading to faster training with minimal performance loss.
• Target Resolution: Section 4.1.4 of SimMIM ablates the prediction (output) resolutions rather than input resolution, which offers negligible computational savings. The input resolution in SimMIM remains fixed at 192$^2$, limiting its potential for accelerating training.

Comparison with MaskFeat

Our primary goal is to accelerate the pre-training process. Using low-resolution inputs directly, however, often causes significant performance drops when employing pixel targets commonly used in MIM.

Inspired by MaskFeat, which demonstrates the robustness of HOG targets against color ambiguities, we further elaborate on HOG's invariance to resolution changes. Our experiments and visualizations reveal that HOG retains better fine-tuning accuracy and achieves lower pre-training loss under low-resolution inputs compared to pixel targets. Thus, we adopt HOG to mitigate the loss of critical information (e.g., textures and edges) that typically occurs with smaller input resolutions, achieving both efficiency and robustness in the pre-training process.

W2: these aspects are common knowledge in MIM research, thus providing limited contribution to the community.

A2: Our primary focus is distinct: we aim to demonstrate that significantly reducing the input resolution during pre-training has minimal impact on final fine-tuning performance.

This insight is both practical because it enables the development of faster and more memory-efficient baselines without compromising accuracy. By systematically evaluating the interplay of input resolution and prediction targets, we provide a streamlined approach for pre-training, which can serve as a solid reference for future work.

While our contributions may not introduce highly novel methods, we believe the value of this work lies in its practical guidance and the ability to deliver competitive baselines that are both efficient and effective.

W3: When erifying the shallow layers during pre-training, the linear probing accuracy is missing.

A3: Since we used fewer layers during pretraining, we introduced new layers (random initilized) during fine-tuning. This demonstrates that the layers closer to the end do not necessarily require self-supervised pretraining, rather than suggesting that our network inherently does not need the final layers. Our experiments have validated this hypothesis. However, because the final layers were not pretrained, it is not feasible to perform corresponding experiments using linear probing.

W1 : From Figure 4, the accuracy does not demonstrate a clear saturation trend. Providing the fine-tuning accuracy after 1600 pre-training epochs would make the results more convincing.

A1: Thanks for your advice. We have updated the results based on ViT-B and will provide the Swin-B results once the experiments are completed.

W2: Missing citation.

A2: Thank you for the reminder. We have updated the revised paper accordingly.

W3: While the reviewer acknowledges the efficiency of the proposed method, the performance gains of FastMIM on ImageNet-1K and ADE20K are marginal.

A3: Our main contribution is to expedite and alleviate the computational overhead for practical MIM-based pretraining. As shown in Figure 1 and Figure 2, FastMIM reduces memory consumption by approximately 3x and accelerates the pretraining procedure by around 5x, while achieving comparable or better downstream performance compared to previous frameworks.

W4 & Q1: The effectiveness of FastMIM for large-scale models such as ViT-L and Swin-L

A4: Thank you for your valuable suggestion. In the revised version, we have included experimental results for ViT-L in Table-3. Consistent with the trend observed from Swin-B to Swin-L, we find that reducing the input resolution results in slightly more pronounced accuracy degradation as the model scales up from ViT-B to ViT-L. This further validates that larger models indeed tend to be more data-hungry, and a smaller number of input tokens may lead to insufficient pretraining. However, by leveraging the progressively enlarged input resolution strategy, we effectively mitigate this issue and achieve significant improvements in accuracy for larger models. The updated discussion in the revised manuscript further elaborates on these findings.

W1: However, since model pre-training does not occur frequently in practical applications, the time cost of pre-training is acceptable. The main purpose of pre-training is to obtain better foundational model representations.

A1: Thanks for your valuable comments. I agree with your point that, on a macro level, the time cost of pretraining is acceptable. However, we would like to emphasize that the motivation of this paper is to provide a baseline method that makes current pretraining more efficient and deployable. Our goal is to enable relevant research to be conducted on smaller, commercially available GPUs, rather than requiring high-end GPUs like the A100, while also significantly saving time.

FastMIM is positioned as an empirical study, inspired by works [1,2], that thoroughly examines prior MIM methods and identifies a critical research gap: self-supervised pre-training demands enormous computational resources, making it impractical for single deep learning machines. Our philosophy is to break the barrier of costly pre-training by providing a solution that is both efficient and deployable, enabling broader accessibility to MIM research.

[1] Xinlei Chen, Saining Xie, et al. "An empirical study of training self-supervised vision transformers." ICCV 2021.

[2] Zi-Yi Dou, Yichong Xu, et al. "An empirical study of training end-to-end vision-and-language transformers." CVPR 2022.

Our contributions are two-fold:

• We demonstrate for the first time that combining these two factors leads to a qualitative transformation in efficiency and practicality for MIM. Unlike previous works that consider these aspects in isolation, our study reveals the synergistic effect of small input sizes and HOG targets in addressing computational bottlenecks, offering a clear and actionable pathway for efficient pre-training.

• Our motivations are fundamentally different from related works such as SimMIM. For instance:

  • SimMIM only slightly reduces input size (from $224^2$ to $192^2$), aiming to mitigate memory overhead. This minor reduction, however, is insufficient to drastically improve training efficiency.

  • In contrast, FastMIM adopts a significantly smaller input size, reducing the resolution by a factor of four. This results in a substantial acceleration in training and memory savings, as shown in Table below:

    Case	GPU	Memory	# GPUs	PT Days
    EVA	A100	~27GB	128	~14.5
    SwinV2	A100	<40GB	8	~20
    CAE	V100	<32GB	8	~12
    SimMIM	V100	~28GB	8	~7
    FastMIM	V100	~14GB	8	~1.6

    FastMIM reduces memory consumption to 14GB on a V100 GPU (compared to $\sim$28GB for SimMIM) and achieves pre-training in 1.6 days, compared to $\sim$7 days for SimMIM, $\sim$12 days for CAE, and $\sim$14.5 days for EVA. These results clearly demonstrate the practicality and scalability of our approach.

Q1: The caption is incorrect in Figure 4, the text in the top left corner of the figure should annotate the line segment as HOG instead of pixel.

A1: Thanks for raising this point, we have corrected the corresponding errors in our revision.

Q2: How does the method perform compared to other methods on larger pretraining datasets such as IM-21K or with longer pretraining epochs?

A2: Thanks for your suggestion. In Table 8, we compare pretraining on ImageNet-21K and ImageNet-1K, and we observe that larger pretraining datasets can significantly improve performance. But using SimMIM on the 22K dataset with 192² input resolution requires around 30 days to complete pretraining, which demands substantial computational resources, and we were unable to obtain results within this time frame, we will update this result once the experiments are completed. Additionally, we have included results on ViT-L in Table 3, further demonstrating the effectiveness of our method on larger models.

Q1: Risk of Information Loss

A1: Thank you for raising this point. We acknowledge that reducing resolution and using HOG features may lead to the loss of fine-grained details, which could impact tasks requiring precise texture or color information. However, our primary goal is to improve the efficiency and practicality of pretraining for vision classification tasks, where such fine details often contribute less to downstream performance compared to the overall structure and layout of the image. Our experiments demonstrate that, despite this potential information loss, the performance on tasks such as ImageNet classification remains on par with or better than previous methods.

Q2: Task-Specific Adaptability & Method Generality

A2: We appreciate the concern about task-specific adaptability. Our work focuses on a specific class of general vision tasks, particularly classification, segmentation, and detection, where HOG features perform robustly. We agree that tasks such as image generation or super-resolution reconstruction might demand finer details like texture and color information. While our current approach may not be directly applicable to such tasks, we believe it provides a solid foundation for efficient pretraining in its target domain. Given that task-specific adaptability is not the primary focus of our current research, we have incorporated a corresponding discussion in Line 542-552 based on your comment.