Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners

Q1: [The training time and memory usage.]:

Please refer to Q1 for the general response, we claim that introducing additional teacher models inevitably brings more overhead, but the increase is tolerable. Besides, Hybrid Distill enjoys shorter one-epoch training time and lower memory usage compared to dBOT[1]. Hence, we believe that the computational cost of Hybrid Distill is acceptable compared to other distillation methods.

Q2: [Linear probing results on ImageNet-1K.]:

The linear probing performances on ImageNet-1k are shown in the table below. The subpar performance of dBOT further validates our perspective discussed in Section 2.3, which suggests that its increase in diversity comes at the expense of sacrificing discrimination. In contrast, Hybrid Distill consistently maintains outstanding linear probing performance, thus confirming the effectiveness of our approach in simultaneously promoting diversity and discrimination.

Results for linear probing. $\star$: using MAE+DeiT teachers. $\dagger$: using MAE+CLIP teachers.

[1] Exploring target representations for masked autoencoders, 2022.

Q1: [Sec 2 does not discuss performance.]:

i) We have demonstrated the inherent difficulty of achieving both diversity and discrimination through direct feature distillation in Section 2. Therefore, discussing the performance in Section 2 is essentially equivalent to comparing the superiority of these two properties with respect to different downstream tasks since the student can only guarantee one of these properties, and this comparison has already been discussed in the second paragraph of Section 1 and in previous related works [1-2]. Specifically, models with better discrimination exhibit advantages in recognition tasks, particularly in linear probing, while models with better diversity excel in dense-level tasks.

ii) In Q2 ii) of the general response, we have verified that models with better discrimination indeed exhibit superior linear probing performance. However, according to previous works [2-3] and our experience, longer pre-training or distillation periods (e.g., 1600 epochs in [2-3]) are required to achieve sufficient diversity and reap dense-level downstream benefits. In Section 2, we only conduct 100 epoch distillation to compare the distillation trends, which is insufficient to fully reflect the downstream advantages of diversity. Therefore, performance comparisons at this stage have limited value. Nonetheless, our Hybrid Distill design is based on the analysis presented in Section 2, and we consider its effectiveness on dense-level tasks as reliable evidence supporting this analysis.

iii) Regarding the two papers you mentioned, they primarily discuss the impact of projectors on CL and Supervised models, whereas our focus is on their influence on the trend of distillation. As a result, they have different motivations with our studies.

Q2: [The method makes training harder and requires more memory.]:

i) Please refer to Q1 of the general response, Hybrid Distill enjoys lower memory usage compared to MAE and dBOT. Hence, we believe that the computational cost of Hybrid Distill is acceptable compared to other distillation methods.

ii) Regarding the training difficulty, we argue that Hybrid Distill actually mitigates this issue since the required properties can be directly obtained from teacher models. This claim is supported by the results presented in Table 10, as well as the discussion provided in Section E of the appendix. Specifically, Hybrid Distill can achieve better performance with much fewer training epochs compared to dBOT.

Q3: [Hard to compare the relevant rows in Tables 1 and 2.]:

Thanks for your suggestion and we have reorganized Table 1 and Table 2 in the revised version.

Q4: [The distillation schedule when using ViT-L.]:

For ViT-L, we offer two distinct models that are derived through different distillation processes. The first model is obtained through a 100-epoch distillation process using ImageNet-1k, while the second model is derived from a 40-epoch distillation process with ImageNet-21k. We have made revisions to the description in order to improve its clarity.

Q5: [The hyperparameter α and β.]:

Please refer to Table 13 and Table 14 and the discussion in Section A.5 of the appendix, we ablate the setting of α and β and conclude that: i) Combining two teachers for hybrid distillation can lead to performance gains towards only using one teacher, regardless of the value of hyperparameters α and β. ii) Ensuring the weights of the more powerful model among the two teacher models count for better performance. iii) Setting both α and β to 1 for simplicity can already yield satisfactory results.

[1] What Do Self-Supervised Vision Transformers Learn? 2023.

[2] Revealing the Dark Secrets of Masked Image Modeling, 2022.

[3] Masked Autoencoders Are Scalable Vision Learners, 2021.

Q1: [Unfair comparison in Table 1.]:

i) Since the main purpose of our work is to reveal the incompleteness of previous works and to verify the benefits of promoting diversity and discrimination simultaneously, we conduct most of our experiments in a basic setting. As shown in Table 1 and Table 9 of our appendix, we conduct 300 epochs of distillation utilizing the ImageNet-1K dataset and ViT-B/L as backbones. We only report the results with ImageNet-21K in the last row of Table 1 to illustrate the potential benefits of larger-scale distillation, while all other results are reported within the basic setting for fair comparisons.

ii) Besides the CLIP+MAE targets you mentioned, Hybrid Distill also provides results of using DeiT+MAE targets and both results showcase consistent benefits towards teacher models and distillation baselines on dense-level detection and segmentation tasks, illustrating the effectiveness of our Hybrid Distill. We have reorganized Table 1 to make the relevant comparisons easy to understand.

Q2: [Average attention distance and NMI are existing metrics.]:

We agree that average attention distance and NMI do not directly indicate good representation, as they are merely tools used to assess model behavior. However, we argue that these two metrics have been inadequately interpreted in previous works. In fact, our work is dedicated to solving the limitations of the distillation works you mentioned, including dBOT [1]. We present evidence that the diversity brought by dBOT is derived from its asymmetric decoder design (Section 2.2), while the asymmetric decoder design de facto hinders discrimination (Section 2.3). This loss in discrimination also explains the phenomenon you mentioned, where models with different teachers (DeiT, DINO, MAE, etc.) exhibit similar behavior in dBOT after multi-stage distillation. In summary, in Section 2, we have confirmed that existing works are still far from achieving good representation from analyzing the diversity and discrimination perspectives.

Q3: [Apply to other architectures such as Swin or MViT or CNNs.]:

The core of our Hybrid Distill is to utilize MAE and Supervised/CL teachers to ensure both diversity and discrimination. Since these two kinds of teachers have been confirmed to demonstrate similar behavior in hierarchical transformer structures (e.g., Swin) [2] when compared to the plain ViT structure, we believe that our designs can be effectively adapted to these hierarchical structures as well. However, our analysis relies on the available pretrained models for efficiency, and the challenge lies in that previous methods have not provided pre-training weights (e.g., CLIP) or baseline results (e.g., [1,3-5]) for these structures. Therefore, further comprehensive validation will be conducted in the future, in parallel with the research progress of the community.

Thanks for your detailed comment.

Q1: [Using in-house data from a different domain.]:

i) Compared with directly distilling on custom data, we think that distilling on a large-scale, general dataset and then fine-tuning on custom datasets (typically with small-scale supervised data or weakly supervised data) may be more applicable for its effectiveness in acquiring high-quality representation and preventing potential overfitting. Additionally, models obtained through large-scale distillation have the capability to quickly adapt to different custom data, as downstream tasks only require further fine-tuning on relatively small-scale datasets. While for large-scale distillation itself, as mentioned in our response to Q5 and Q6, Hybrid Distill has efficiency advantages towards dBOT.

ii) Even in cases where distillation needs to be performed on custom data, the MAE model obtained through large-scale pre-training can still serve as a good teacher. As demonstrated in Section 2 of our updated supplementary materials, the unsupervised training paradigm of the MAE teacher allows it to maintain considerable diversity even when applied to data from different domains. Therefore, we believe that utilizing this model to provide diversity for Hybrid Distill is still effective and the efficiency advantages of our Hybrid Distill can still be maintained.

Q2: [Using a general visual representation on ImageNet.]:

i) In our response to Q6, we have emphasized that Hybrid Distill is more efficient than dBOT in utilizing the off-the-shelf teacher models (e.g., DeiT, CLIP, and MAE) for distillation. While the dBOT models, obtained through distillation, do not belong to these base teacher models and they should be compared with our Hybrid Distill models (also obtained through distillation) for fair comparisons. We have further improved our revised version's description about 'fewer epochs' to improve clarity.

ii) Note that  dBOT also requires custom training, specifically through the process of distillation. The models provided by dBOT to the community have already undergone 1600 epochs of custom training under the guidance of a pre-trained teacher, but this does not mean that custom training is unnecessary for dBOT. The advantages of our Hybrid Distill lie in its distillation strategy, which results in a significantly lower custom training cost compared to dBOT. Therefore, we can also provide the community with performance-enhanced models that require less custom training cost (e.g., 300 epochs, or 500 epochs for further benefits) by leveraging our Hybrid Distill paradigm, and releasing such models is in our plan.

Q3: [MILAN is not cited and compared in the submission.]:

i) MILAN does not directly distill features. Instead, it introduces a prompting decoder design that involves freezing the unmasked tokens. These tokens are then used as keys and values to interact with learnable mask tokens in the decoder. In our updated supplementary materials, we have verified that this design can alleviate the discrimination loss caused by the asymmetric decoder, resulting in improved performance. Therefore, the reason for its performance gain compared to direct distillation can be explained by the metrics in our paper, i.e., it successfully improves diversity while still maintaining a certain level of discrimination. We have cited this paper in the revised version.

ii) MILAN follows the downstream setting of MAE and also does not disclose its details. Therefore, we re-evaluate the performance of MILAN under our 1x downstream setting for a fair comparison, and our method outperforms MILAN by 0.9 $AP_{box}$ and 0.7 $AP_{mask}$, as demonstrated in the table below. Evidence for this improvement can be found in the supplementary materials, i.e., MILAN does not exhibit the same level of diversity gain and discrimination preservation capabilities as our Hybrid Distill.

Results compared with MILAN

[1] Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation, 2022.

[2] Exploring target representations for masked autoencoders, 2022.

[3] Context Autoencoder for Self-Supervised Representation Learning, 2022.

[4] Sdae: Self-distillated Masked Autoencoder, 2022.

[5] Progressively Compressed Auto-Encoder for Self-Supervised Representation Learning, 2023.

Q1: [Questions about diversity and discrimination.]:

We actually share a similar viewpoint as you, i.e., we acknowledge the importance of preserving long-term information (i.e., discrimination) in the last layers and do not believe that diversity is the only property to pursue. The fundamental principle of our Hybrid Distill is thus to retain this long-term information while simultaneously promoting diversity. In the following, we talk about your concerns about diversity and discrimination point by point.

[The definition of diversity.]

The term 'evenly distributed representations' is derived from [1], indicating that the representations are acquired through more evenly distributed attention across different attention distances. Your interpretation is also accurate, as diversity does represent the existence of 'different, or diverse, per-head attention distances'. In the revised version, we have adjusted the description for clarity.

[Is diversity itself clearly worth to pursue?]

i) As mentioned at the beginning, we also recognize the importance of preserving long-term information in the last layers as well as the limitation of judging the model with only diversity like previous works [1-2]. We emphasize that increasing diversity should not be at the expense of sacrificing discrimination (i.e., long-term/global information in the last layer). This notion forms the core of our discussion in Section 2 and serves as the foundation for the development of Hybrid Distill in Section 3.

ii) We acknowledge the significance of long-term information, but the problem lies in the fact that the model uses too many transformer layers to represent homogeneous long-term information. We introduce diversity to address this homogeneity, not to replace this information entirely. It is worth noting that although this problem is more apparent in the later layers, the trend of homogeneity can already be observed in earlier layers of CL/Supervised teachers. Therefore, relying solely on multi-layer features is not an essential solution to this problem. In fact, we have already incorporated multi-layer features in our downstream tests for detection and segmentation (r.f., Section C of the appendix), and the clear advantages of our Hybrid Distill over direct distillation (r.f., Table 1 of our paper) in these downstream provide support for our claim.

[The quantitively evaluation of discrimination.]

Please refer to Q2 of the general response, we utilize average attention distance and NMI to judge both diversity and discrimination. Besides, we have included additional linear probing results in the revised version, and the results align with the analysis of discrimination presented in Section 2.

Q2: [MAE results. ]:

Sorry for the confusion, the difference between reported performance and original values in the MAE paper is attributed to variations in downstream settings. We follow the settings of recent works [3-5] (which are also widely adopted currently) for employing pretrained models in downstream tasks of COCO detection and ADE20K segmentation. For COCO detection:

i) The MAE paper finetunes the pretrained model for 100 epochs, which is time-consuming and not a common downstream setting. Besides, MAE does not release the official codes for COCO detection and some fine-tuning details are missing. Therefore, we follow the common practice in [3-5] that employs a 1x schedule (12 epochs) for fine-tuning, and our reported MAE results align with these papers (48.4 $AP_{box}$ and 42.6 $AP_{mask}$).

ii) In Table 10 of our appendix, we also provide another set of COCO detection results using Cascaded Mask-RCNN for comprehensiveness. According to [2], the reported MAE result under this setting is 50.6 $AP_{box}$, while our Hybrid Distill achieves 53.0 $AP_{box}$, which is 2.4 points higher than MAE. This clearly demonstrates the advantages of our method compared to MAE across different settings.