Thanks for your valuable comments and your appreciation for our technical contributions. We will fix the typos. Below, we discuss in detail the points you have raised.

Q1: Is the bottleneck of scalable visual representation learning not about the methodology but more about data?

In our paper, we conducted experiments with the same dataset (IN-1K) and the same setting (multi-stage distillation). We specifically focused on masked knowledge distillation, which utilizes masked image modeling as the pretext task for self-supervised training. We observed that the target representation is not crucial for mask knowledge distillation when distillation is performed in stages. However, we believe that new sophisticated pretext tasks can further enhance the performance. The bottleneck of scalable visual representation learning arises from both methodology and data.

Q2: Does the invariance of target representation suggest that the masking operation is the key in MKD? Does the implementation of masking also affect the scalability of pre-training?

The masking operation is indeed the key in masked knowledge distillation for self-supervised learning. Through the same pre-training task, the target representation is not important in multi-stage distillation, and dBOT can achieve a similar weight distribution. The implementation of masking also affects the scalability of pre-training. Like MAE, even changing the mask ratio can affect the model performance, as shown in our ablation study.

Q3: More backbone architecture.

In our experiment, we used the same architecture for both the student and teacher networks, as the teacher's weights are loaded from the pre-stage student's weights in multi-stage distillation. Therefore, the teacher and student networks should be the same. Previous works such as DINO and DALL-E were pre-trained on vanilla VIT, which facilitated our study on masked knowledge distillation with vanilla VIT as the backbone. As you mentioned ConvNext, a possible experiment would involve performing DINO and DALL-E on ConvNext first, and then applying masked knowledge distillation. However, whether DINO and DALL-E are suitable for ConvNext is a question that goes beyond the scope of our paper. We would like to leave this as an open-ended question.

We appreciate your valuable feedback and will continue to explore and expand the scope of our work.

Thanks for your valuable comments and your appreciation for our technical contributions. We will improve our draft following your suggestions. Below, we discuss in detail the points you have raised.

Q1: Does performance saturate and not drop with more stages and training?

We perform multi-stage training using pixels as the target. The fine-tuning performance on IN-1K at the 0th, 1st, 2nd, and 3rd stages is 83.6, 84.3, 84.4, and 84.3, respectively. The performance saturates after two-stage distillation and fluctuates around the saturation performance.

Q2: Different models for different tasks.

As we explained in response to Q1, the performance will fluctuate around the saturation performance. Downstream users can load any model after two-stage distillation with a slight performance difference.

Q3: Do the trends hold across random seeds?

During our experiments, we found that the performance of classification and detection is quite stable, except for the segmentation tasks. The segmentation performance reported in our paper is averaged over 5 random seeds, which we will clearly state in our revised version.

Q4: Is the total number of pre-training iterations controlled across different methods in Tables 2, 3, 4, and 5?

In Table 5, we conducted ablation studies to choose the best settings for dBOT. Then, the best setting was used in Tables 2, 3, and 4. The model performance saturates after multi-stage distillation.

Q5: Why is MAE faster than dBOT when both use an asymmetric encoder-decoder architecture?

MAE uses the original pixel as the target, which does not involve a teacher network, while dBOT uses an extra teacher to extract the reconstructed token.

Q6: It seems like MAE's attention distance plot is similar to the distilled models'.

They are similar but there are differences between them. Taking the last layer as an example, the maximum average attention distance for MAE is 85, while it is 70 for dBOT. The minimum average attention distance for MAE is 40, while it is 30 for dBOT. dBOT achieves more local attention than previous works.

Q7: Shouldn't a lesser correlation in the model's output result in lower redundancy?

We apologize for this mistake and will correct it in the revised version.

Q8: Why may the "existence of decoder" hurt?

The encoder and decoder are both used in the pre-training stage, but the decoder is discarded in the fine-tuning stage. When the target is low-level, the decoder is needed to decode the high-level representations encoded by the encoder. However, when the target contains high-level semantics, the existence of the decoder may even restrain the encoder from fully grasping knowledge from the teacher, leading to degraded performance.

We appreciate your valuable feedback and will continue to explore and expand the scope of our work.

Thanks for your valuable comments and your appreciation for our technical contributions. We will clearly state the technical details in the revised version. Below, we discuss in detail the points you have raised.

Q1: Technical details.

In our pre-training and fine-tuning, we use IN-1K as the dataset and masked autoencoder as the student models. This is described in section 3 (common setup). We will clearly state the technical details in our revised version.

Q2: The hyper-parameter setting.

The setting for distilling MAE in the two-stage process is selected based on the ablation in section 5.4. In Table 1, all the results are set the same for a fair comparison. Distilling the mask autoencoder for two stages with any teacher achieves similar performance in classification, object detection, and segmentation.

Q3: Linear probing results and the details in section 6.

The linear probing results of dBOT using DINO as the teacher are 77.35%, while dBOT using other teachers achieves similar results (69.1%). We attribute this to the multi-crop technique and contrastive learning used in DINO. In section 6, all the models are pre-trained models before fine-tuning.

Q4: The meaning of recipe ⚗

The token ⚗ represents the distillation of dBOT with CLIP as the teacher, as described in appendix C.1. We will clearly state this in the revised version.

We are investigating dBOT's effectiveness in other potential distillation methods except for l2 distillation. We appreciate your valuable feedback and will continue to explore and expand the scope of our work.

Thanks for your valuable comments. Below, we discuss in detail the points you have raised.

Q1: The relation between dBOT and the observation about the choice of teacher model.

Indeed, the performance of dBOT comes from the multi-stage training. The major finding in our paper is that the choice of the teacher model does not have to be carefully chosen if the distillation is done in stages. Specifically, we use a randomly initialized model as a teacher to formulate dBOT and compare the results with previous state-of-the-art (SOTA) models. dBOT still performs well when switching teacher networks since the choice of the teacher model is not crucial.

Q2: The observation is somewhat aligned with the observation of MAE.

In the early stages of Masked Image Modeling (MIM), researchers used different targets for performing MIM, such as pixels, Histogram of Oriented Gradients (HOG), and tokens from a pre-trained teacher. Researchers have different opinions on target selection and different results are achieved with different targets. MAE simplifies MIM by using pixels as the target and achieves better results than Beit and maskfeat, which demonstrates that using pixels is better than HOG and tokens from a pre-trained teacher. However, our paper finds that the choice of the teacher network is not crucial if the distillation is done in stages. In other words, similar results can be achieved by using any target (as shown in Table 1), even a randomly initialized teacher. This is the main difference between dBOT and MAE.

Q3: Reconstructing raw pixels.

MAE achieves good results by simply using pixels as a target. However, later works, like [1], achieve better results by using tokens from pre-trained teachers as targets. Different results are achieved with different targets. The motivation of our paper is that there is no work that provides a system-level study on the importance of how to choose an adequate target representation or teacher network to guide the learning of Masked Knowledge Distillation (MKD). Our work reveals that the choice of teacher is not crucial if the distillation is done in stages. In particular, we find that using a randomly initialized model as a teacher can achieve similar results through multi-stage distillation, which we formulate as dBOT. This alleviates the painstaking selection of target representations since we achieve similar performances with different target representations in multi-stage distillation.

Q4: The results of randomly initialized teacher.

The results of the 0th stage are the initialized teacher, and the results of the 1st and 2nd stages are the distilled student, which is described in Section 3.1. Therefore, 77.1 is the result of fine-tuning a randomly initialized model on IN-1K.

[1] Wei, Yixuan, et al. "Contrastive learning rivals masked image modeling in fine-tuning via feature distillation." arXiv preprint arXiv:2205.14141 (2022).

Here we would like to highlight the main contributions:

a) Pioneering System-Level Study in MIM with dBOT: Our work, dBOT, stands as the first to conduct a system-level study on the role of the 'teacher' in Masked Image Modeling (MIM). While previous studies (such as MAE, MaskFeat, and MVP) suggest that the choice of target may not be pivotal for MIM's success, this hypothesis remained untested. dBOT is the first to empirically demonstrate this, marking the first contribution in the field.

b) Highlighting the Importance of Multi-Stage Masked Distillation: We reveal that the key to effectively training high-capacity models lies in our novel multi-stage masked distillation pipeline, rather than in the meticulous selection of a training target. This approach contrasts with the typical one-stage training methodology prevalent in prior research, marking another contribution of this work.

We are eager to engage in further discussions and questions!

Thanks for your prompt reply! I appreciate the potential effort it might take.

However, after reading the authors' response, and opinions from other reviewers, I still feel this work has the following weakness:

(1) As agreed by the authors, the superior performance of dBOT comes from its multi-stage training design, not from its analysis about the role of pre-trained teacher model. I feel not so comfortable that this paper tries to pitch its analysis "target representation" as the main driving force of its performance, as they are somewhat unrelated.

(2) I agree that dBOT performs extensive analysis on the role of pre-trained teacher model in MIM, which is different from the raw pixel construction in MAE. However, I doubt if those analysis is really valuable, after MAE has proved that there is no need for such a pre-trained teacher. Indeed, as pointed out by Reviewer tvFW, dBOT is even slower than MAE due to its adoption of a teacher model. Plus, as I said earlier, there is no painstaking selection of the target representations if there is no teacher.

(3) Following (2), one could argue that the fearetus extracted by a randomly initialized teacher is somewhat similar to raw pixels in the MIM framework, which is why those seemingly random features could work in the first place.

(4) I checked [1] as the authors pointed out it achieves better results by using tokens from pre-trained teachers as targets. However, if my understanding is correct, [1] focuses on improving contrastive learning to the level of MAE by adopting stronger CLIP pre-trained teacher models. It is not a work of MIM. Hence, I doubt if that work could justify the motivation of this work, especially after there is already MAE. In other words, I cannot see any sign why we should adopt any teacher for targer representation in MIM. I feel confident that breaking MAE training into multiple stages would lead to just as good performance.

My original review continues to apply.

[1] Wei, Yixuan, et al. "Contrastive learning rivals masked image modeling in fine-tuning via feature distillation." arXiv preprint arXiv:2205.14141 (2022).