Morphing Tokens Draw Strong Masked Image Models

We deeply appreciate the positive reviews and valuable and insightful comments that encourage our work. We have carefully read the comment and addressed the concern through this response.

Q1. The applicability to hierarchical ViTs

A1. Thank you for the comment. We did not compare with methods based on hierarchical architectures because our approach focuses on masked image modeling (MIM) pre-training, specifically handling all tokens simultaneously for Vision Transformers. In contrast, hierarchical architectures, which usually share a non-isotropic design, typically require tailored MIM techniques.

However, following Reviewer L29Y's suggestion, we give some experimental results using our method DTM built upon a hierarchical architecture in a pre-training phase. Due to time constraints and the higher training costs of hierarchical ViTs (which are believed to be optimized further), we employed the ConvMAE-Small architecture to showcase that DTM could improve performance even with fewer training epochs on ImageNet-100, pre-training and fine-tuning for 25 epochs and 100 epochs, respectively. The results in the table below demonstrate that our method when applied during pre-training can enhance the performance of a hierarchical ViT. We will provide full training results on ImageNet-1K in the revised paper.

• Applicability of DTM on ConvMAE-Small
  • Pre-trained for 25 epochs and fine-tuned for 100 epochs | Method | Fine-tuning Accuracy (%) | |------------------------------|:-----------------------:| | ConvMAE | 82.4 | | ConvMAE + DTM | 83.0 |

Q2. How do the authors address spatial inconsistencies when existing methods like EVA, EVA02, and Fast-iTPN have already achieved strong results?

A2. The suggested methods such as EVA01/02 or Fast-iTPN have shown outstanding performance. However, we hypothesize that spatial inconsistencies still exist in the frameworks due to their reliance on supervisory signals from a pre-trained teacher like CLIP or EVA-CLIP during pre-training.

To investigate this, we examine the spatial inconsistency of visual token predictions generated by other supervisory models following the same approach used for CLIP in Sec 3.1, by evaluating token-wise zero-shot classification results. As suggested by Reviewer L29Y, we employ a strong and large-scale model EVA-02 and analyze its teacher model EVA-01-CLIP-g/14. We believe this analysis highlights the spatial inconsistency beyond CLIP, which may incur inaccurate supervision when used as a supervisory signal.

We first provide an illustrative figure (please refer to Fig A in Appendix) which follows the same illustrative approach as Fig. 1 in the main paper. We observe that token-wise zero-shot predictions without token aggregation exhibit spatially inconsistent results compared with the zero-shot prediction results with token aggregation (it follows a trend similar to CLIP). Moreover, we observe that zero-shot accuracies computed using token-wise ensembles achieve 53.1% with token aggregation compared to 51.2% without aggregation. This highlights spatial inconsistency still exists in even a stronger pre-trained model, which can still benefit from token aggregation when used as a supervisory signal.

Another option suggested by Reviewer L29Y, EVA-01 benefits from significantly larger training datasets (including ImageNet-21K and additional datasets) while utilizing the CLIP supervision and a BEIT framework very similar to ours. Therefore, we believe our analysis with CLIP (in the main paper), which revealed spatial inconsistency, would impact further improving EVA-01 in that it employs the same supervisory signal (CLIP).

We plan to improve EVA01/02 using DTM during pre-training by replicating the setups detailed in their respective publications in our revised paper.

Q3. Missing COCO experiments

A3. Thank you for suggesting the experiment. We actually reported ADE20K semantic segmentation results in Table 3 to demonstrate that our method performs effectively on a dense prediction task.

Following reviewer L29Y’s suggestion, we have run some COCO experiments and report the performance improvement achieved by employing a DTM-enhanced ImageNet-pretrained backbone, which is further fine-tuned on the COCO dataset. We compare the pre-trained model using DTM with the baseline model that does not incorporate DTM. We employ Mask R-CNN with ViT-B/16 and initialize its encoder with the baseline and our DTM models, respectively. Due to time constraints and the high computational cost associated with high image resolutions on COCO, we adopted a lighter experimental setup than the standard training protocols used in previous self-supervised learning methods. We resized the larger side of input images to 1024 and fine-tuned the models for 10 epochs; we further reduced the image resolution to 512 and fine-tuned the models for 18 epochs as another experimental setup. We observe that DTM enhances the baseline, delivering consistent improvements across all the metrics in both experimental setups.

We again appreciate your understanding that we employed lighter experimental setups due to time and resource constraints, training with fewer epochs (upper table) and smaller resolution (lower table). We observe that the backbones pre-trained with our method achieve significant gains across both experimental setups. We will include experiments with the standard experimental setups using larger models in the revised paper.

• Mask RCNN fine-tuning (res: 1024, trained for 10 epochs) | Method | bbox/AP | bbox/AP50 | bbox/AP75 | bbox/APs | bbox/APm | bbox/API | segm/AP | segm/AP50 | segm/AP75 | segm/APs | segm/APm | segm/API | |------------------------------|---------|-----------|-----------|----------|----------|----------|---------|-----------|-----------|----------|----------|----------| | Baseline | 46.7 | 69.4 | 51.1 | 29.6 | 50.5 | 61.6 | 41.9 | 65.6 | 45.0 | 22.2 | 44.7 | 61.3 | | DTM | 47.4 | 69.8 | 51.9 | 30.6 | 51.3 | 62.3 | 42.5 | 66.6 | 45.6 | 22.6 | 45.7 | 61.3 | | Gain | 0.7 | 0.4 | 0.8 | 1.0 | 0.8 | 0.7 | 0.6 | 1.0 | 0.6 | 0.4 | 1.0 | 0.0 |

• Mask RCNN fine-tuning (res:512, trained for 18 epochs) | Method | bbox/AP | bbox/AP50 | bbox/AP75 | bbox/APs | bbox/APm | bbox/API | segm/AP | segm/AP50 | segm/AP75 | segm/APs | segm/APm | segm/API | |------------------------------|---------|-----------|-----------|----------|----------|----------|---------|-----------|-----------|----------|----------|----------| | Baseline | 41.8 | 63.9 | 45.3 | 21.1 | 46.1 | 61.4 | 37.5 | 60.1 | 39.7 | 15.0 | 40.2 | 61.0 | | DTM | 42.9 | 65.0 | 46.3 | 22.2 | 47.5 | 63.0 | 38.2 | 61.0 | 40.6 | 16.1 | 41.1 | 61.8 | | Gain | 1.1 | 1.1 | 1.0 | 1.1 | 1.4 | 1.6 | 0.7 | 0.9 | 0.9 | 1.1 | 0.9 | 0.8 |

Dear Reviewer L29Y

We sincerely appreciate your thoughtful positive review and constructive comments, which greatly encouraged our work. As the discussion period nears its end, we would like to leave a gentle reminder to consider our responses. We have made every effort to address all your concerns by including requested experimental results and detailed explanations (along with paper revisions). Please feel free to let us know if you have any additional questions or require further clarification.

Thank you once again for your valuable reviews.

Best regards,

Authors

Dear Reviewer L29Y

We sincerely appreciate your response and consideration of our paper as a weak acceptance.

In the current era, extensively utilizes CLIP-supervisory signals, we believe our method would offer broader applicability by effectively enhancing the signals in that the effectiveness has already been demonstrated in various ways. In our paper, we showcased the general applicability of DTM across several architectural variations (i.e., MAE+CLIP, BEiT-v2, and BYOL) and a variation of CLIP (i.e., SLIP). During the rebuttal period, we further investigated the applicability of DTM to EVA CLIP, validating its general effectiveness on variations of CLIP.

Besides the general applicability of DTM, recent Masked Image Modeling (MIM) studies have mainly focused on innovations in framework design and improved masking strategies. This remains significant potential for refining their MIM supervision using CLIP or other signals. From this perspective, we believe our work can become a baseline to improve MIM supervision through an intuitive plug-and-play manner, with the potential to inspire future research in this direction.

Finally, we sincerely thank you again for your positive feedback and greatly welcome any additional concerns or feedback required for more favorable consideration.

Best regards,

Authors

We deeply appreciate the detailed reviews and valuable and insightful comments that encourage our work. We have carefully read the comment and addressed the concern through this response.

Q1. Enhancing clarity and details in Section 4.2

A1. We apologize for the lack of detailed descriptions addressing the Reviewer gb6h's concerns. Following the advice, we have improved the manuscript and included an algorithm flow table to explain the morphing matrix generation process in greater detail (please refer to Section 4.2 and Section B in Appendix). The revised texts have been marked in blue for clarity. The key updates are summarized as follows: We have added a detailed description of the process of performing k iterations with bipartite matching; We revised to clarify how the aggregated results are derived from the token morphing matrix M; An algorithm flowchart for the token morphing function, which generates M, has been added.

Q2. More refined writings

A2. We first apologize for the confusion. Following the Reviewer gb6h's suggestion, we have provided additional clarifications. As Reviewer gb6h commented, in line 264, $**v**_ i=f_\xi(x^M_i)$ should be $**v**_ i=f_\xi(x_i)$. The line connections appear inconsistent but should be corrected to have the same shape. We apologize for the confusion. Specifically, the lines from the morphing matrix M to each side now have arrows, which denote the morphed tokens ($**v** → \hat{**v**}$) and ($**u** → \hat{**u**}$) generated by the same morphing operation. In the revised paper, we have corrected the lines in Fig.4 to clarify that the morphing matrix is applied identically to all tokens both $\\{ \mathbf{u}_ {i} \\}^N_{i=1}$ and $\\{ \mathbf{v}_ {i} \\}^N_{i=1}$. We hope our revisions provide greater clarity and address any previously misleading details.

Q3. On faster training and overall training time

A3. Faster training refers to a faster convergence of the training metrics during fine-tuning. As shown in Fig. 5, our approach reaches a given accuracy faster than the baseline and achieves lower loss values earlier, which demonstrates a faster convergence from our perspective. Using the same number of epochs is to remain consistent with the baseline and for a fair comparison. Since our method reaches the same accuracy more quickly, extending the training to larger epochs provides additional performance benefits, as also observed in Fig. 6.

Additionally, regarding training time, Table 6 compares DTM's training speed with the baseline. While DTM introduces minor additional computational costs due to token morphing and alignment loss, its impact on throughput is minimal (<1%) and yields significant performance improvements. As a result, DTM maintains a training speed very similar to the baseline, though the overall training time may not be reduced.

Q4. Explanation for not using superpixel-based clustering

A4. The main reason for not using superpixel-based clustering was its inefficiency, despite a slightly improved distillation performance shown in Table 2. While the results in the table suggest the potential of context-aware aggregation methods, we argue their effectiveness should be validated through a more practical MIM-focused study that aligns more closely with our goal. Specifically, as noted in lines 237-238, our pilot experiment found that learning with superpixel-based clustering achieved a baseline improvement of 87.1%; however, employing alternative - Bipartite Matching (used in DTM) - yielded a higher performance of 87.9% (over the baseline 79.5%). We attribute the suboptimal result of the superpixel-based clustering to two factors: 1) its reliance on a fixed number of clustering centers and 2) its limitation of generating a single set of morphed tokens.

Q5. On the lower performance without the dynamic mechanism in Table 7

A5. Without a dynamic approach, Table 7 exhibits that token aggregation uses a fixed number of tokens in a single stage, which can lead to performance degradation. This occurs because determining the optimal number of tokens is challenging—morphing too many tokens erases detailed information while morphing too few experiences a limited impact. Additionally, performing morphing in a single step may reduce token diversity and introduce noise due to the inherent randomness of the matching algorithm (e.g., random splits in Bipartite Matching). This can result in misaligned tokens which would restrict the model's capability to learn more localizable representations. The observed performance drop highlights how undesirable, non-dynamic morphing affects representation quality; we believe this result emphasizes the need for the dynamic mechanism in DTM.

Dear Reviewer gb6h

We sincerely appreciate your thoughtful review and constructive comments, which have greatly encouraged our work. As the discussion period nears its end, we would like to leave a gentle reminder to consider our responses. We have made every effort to address all your concerns by providing detailed explanations and corresponding paper revisions. Please feel free to let us know if you have any additional questions or require further clarification.

Thank you once again for your valuable reviews.

Best regards,

Authors

We sincerely appreciate the positive reviews and the valuable, insightful comments encouraging our work. We have carefully considered the feedback and addressed the concerns in this response.

Q1.More introductions and references to the concept for clarity

A1. Thank you for the advice. We have complemented the explanation of the introduced concepts and revised some explanations. In “4.1 Preliminary", we have revised the manuscript according to the Reviewer xEMo's suggestion as follows:

• We have added references [A, B] to support the block-wise masking approach;
• We have clarified some missing descriptions of the concepts we introduced such as the inputs and outputs of the online and target encoders, as well as the definitions of the terms themselves.
  • The terms "online encoder" and "target encoder" are derived from existing literature [C, D, E]. We have updated the manuscript to include the description.

We hope these revisions improve clarity and address the concerns.

[A] BEiT: BERT Pre-Training of Image Transformers, ICLR 2022

[B] Beit v2: Masked image modeling with vector-quantized visual tokenizers, arxiv 2022

[C] Bootstrap your own latent: A new approach to self-supervised Learning, NeurIPS 2020

[D] iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022

[E] data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language, ICML 2022

Q2.Request for the illustrative figures of the morphing matrix M

A2.

• We have added an illustrative figure in the methods section (please refer to Fig.5 in Section. 4) to illustrate the overall process of obtaining the morphing matrix M, in response to the comment.

• Additionally, we have included an algorithm flowchart for the Token Morphing Function (in lines 864-881), which details the iterative process of generating the morphing matrix M.

Dear Reviewer xEMo

We sincerely appreciate your thoughtful and positive review, as well as constructive comments, which have significantly encouraged our work. As the discussion period nears its end, we would like to leave a gentle reminder to consider our responses. We have made every effort to address all your concerns by incorporating the requested paper revisions and providing detailed explanations. Please feel free to let us know if you have any additional questions or require further clarification.

Thank you once again for your valuable reviews.

Best regards,

Authors

Dear Reviewer xEMo

We greatly appreciate your positive feedback and are glad to hear that our responses have addressed your concerns. We welcome any additional feedback or suggestions for further improvement of our paper.

Best regards,

Authors