Accessing Vision Foundation Models via ImageNet-1K

We appreciate the Reviewer's approval and valuable comments. We respond to the Reviewer's concerns as below.

Weakness 1: Same-size teacher-student pair.

Thanks for the great suggestion. We conduct the experiment by utilizing DINOv2-S/14 as the teacher and Proteus-S/14 as the student to examine the effect and we choose smaller size networks considering the short time window. We validate the generalization ability of the distilled network by conducting linear probing on 12 fine-grained classification datasets.

We surprisingly find that we achieve almost the same performance with the oracle method DINOv2-S on the ViT-S architecture where our method exhibits the largest gap with DINOv2-S previously. This suggests that part of the performance gap is caused by the model capacity gap and our knowledge transfer is very efficient and effective. We will add this part in our final version.

Weakness 2: Varying data scales.

Thanks for the comment. We have provided the analysis of proxy dataset in Sec 3.5 of our paper and we examine the effect from two perspectives.

(1) Dataset Diversity: we introduce ImageNet-Merge, which concatenates all the training sets that we utilized for validation, and ImageNet-Single, a 1.2M training set where the images are generated from a single large image by performing extensive data augmentations. We find ImageNet-Merge brings almost 1% average improvement on fine-grained classification with merely 0.2M additional data and ImageNet-Single surprisingly delivers very decent performance (68.7%), considering that all the information comes from a single picture. This demonstrates that Proteus is robust even under the extreme scenario.

(2) Scaling Behavior: we conduct validation on subsets of ImageNet-1K by either sub-sampling a portion of data at each class or sub-sampling a portion of classes from the total 1000 classes. The results demonstrate the robustness of Proteus when subjected to reduced data availability, suggesting that it is feasible to access foundation models with even smaller data scales.

Weakness 3: Validation on other domains.

We thank the Reviewer for the valuable suggestion. Indeed, extending Proteus to LLMs and diffusion models is part of our original plan but we are struggling with computation resources as both of the experiments demand much larger training costs. We will definitely consider it as our future plan and we hope our work could incentivize the exploration of this idea in other modalities to facilitate the research.

Considering the upcoming deadline of the discussion phase, we would appreciate it if the reviewer could provide any feedback based on our reply so that we could further improve our work. Thank you.

We sincerely appreciate the Reviewer’s detailed comments and constructive suggestions for us to improve our work. We make the response as below.

Weakness 1: Difference with prior arts.

We thank the Reviewer for pointing out these two methods, while we emphasize that we focus on the problem of data-free model compression in a task-agnostic context which is intrinsically different from them as we explained in the reply to all reviewers. Moreover, the mentioned approaches are more like pruning methods as they made hand-crafted designs on the architecture level while we did not touch the model structure.

As there is no public implementation or pre-trained models of DC-ViT, we leverage the pre-trained model of MiniViT-B and examine its generalization ability by conducting linear probing on 12 fine-grained classification datasets. The results suggest that it performs on par with conventional knowledge distillation baseline DeiT-B and significantly lags behind our method Proteus-S. We will add the discussion with these methods in our final version.

Weakness 2: Figure 3.

Thanks for the great suggestion. We follow the same visualization protocol in DINOv2 where we compute a PCA between the patches of the images from the same set and colorize by their first 3 components. Figure 3 demonstrates the generalization ability and dense prediction ability of our method because we can accurately detect the foreground / background boundary as DINOv2, even though our training set is much smaller. We will add the description in the final version.

Weakness 3: Implementation details for downstream evaluation.

Thanks for the comment. The criterion of downstream evaluation is that we leverage the same setting for all methods, including training epochs, hyperparameters, etc. For fine-grained classification, we follow the setup in SynCLR where the training objective is minimized using L-BFGS with L2-regularization and we set the maximum iterations to 500. All methods are trained for 60K and 25K iterations for semantic segmentation and depth estimation, respectively.

In terms of computational costs, we adopt the same architecture with DeiT-S/16 and iBOT-S/16 with the only difference in patch sizes as we have to keep the same patch size with our teacher network DINOv2-B/14 and our computational costs would be larger than DeiT-S/16 and iBOT-S/16. However, it is worth noting that Proteus-S/14 even outperforms DeiT-B/16 on fine-grained classification and dense prediction which suggests that our superior performance is not caused by the smaller patch size.

Weakness 4: Figure 7.

Thanks for reading our paper carefully. The reasons are two folds: (1) our method is very robust and sub-sampling data per class does not affect the performance significantly; (2) as we explained in the paper, we hypothesize the increase is due to the variance in linear probing as those models share very similar performance.

Weakness 5: Integrating logits distillation and cross-entropy loss.

We thank the Reviewer for the valuable suggestion. We conduct experiments by further integrating soft-logits distillation (it performs better than hard-logits distillation according to the empirical results) and cross-entropy loss to our framework (denoted as Proteus-S*) and we observe obvious performance decline in the generalization ability as these two designs introduce dataset bias again. We will add the results in our final version.

Considering the upcoming deadline of the discussion phase, we would appreciate it if the reviewer could provide any feedback based on our reply so that we could further improve our work. Thank you.

Thank you for your detailed and thoughtful response.

1. The authors clarify that distilling task-agnostic contextual information from the teacher model through data-free compression is a significant distinction from other knowledge distillation methods. If this is indeed the case, would similar phenomena be observed on datasets of comparable scale to ImageNet-1K? Furthermore, would the teacher model's generalization ability be more significantly affected on smaller or larger datasets? The experiments in the paper, which rely solely on variants of ImageNet-1K, do not sufficiently demonstrate the proposed method's dataset independence. The authors compare the downstream task performance of other transformer-based distillation methods, providing evidence of the effectiveness of the proposed approach in extracting generalization capabilities.
2. I understand the authors’ intended message conveyed by Figure 3. Including more similar results in the supplementary material would make the findings more convincing.
3. Comparisons on downstream tasks are fair when using identical training hyperparameters and iterations.
4. The authors' analysis does not seem to adequately explain the performance degradation observed when increasing the data scale by nearly half, as this counterintuitive result suggests an anomaly. The authors hypothesize that this is due to variance in linear probing. To substantiate this claim, results with finer-grained sampling rates—e.g., ratios of sub-sampled data at 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0—would be more persuasive.
5. The authors provide additional experiments indicating that knowledge distillation using only intermediate results improves the student network's performance on downstream classification tasks. Including experiments on segmentation and depth estimation would further enhance the persuasiveness of the paper.

Overall, leveraging smaller computational costs to extract the generalization capabilities of large-scale models for downstream tasks is a highly meaningful research direction. A smaller distillation dataset may cause the student model to quickly overfit to the distilled data, failing to extract generalization capabilities. Conversely, larger datasets may make learning challenging for the student model, requiring substantial computational cost to extract generalization. Balancing the scale of the distillation dataset and computational cost could be an intriguing research topic. The authors’ additional experiments validate the effectiveness of this approach in extracting transferability, and more experiments is necessary.

We appreciate the Reviewer's feedback. We provide further explanations to clarify the Reviewer's concerns based on several key points as below.

Weakness 1: Comparison with prior arts.

We thank the Reviewer for pointing out these methods, while we emphasize that we focus on the problem of data-free model compression in a task-agnostic context which is intrinsically different from them as we explained in the reply to all reviewers.

The first two methods (ViT-KD and DeadKD) are conventional knowledge distillation methods which leverage projection head and Cross-Entropy loss. These designs will result in dataset bias and hurt the generalization ability of student network based on our results in Table 1.

MiniLM and MiniLMv2 also focus on the problem of task-agnostic model compression but they leverage the same training set as the teacher network, which aligns with the traditional setting in knowledge distillation and is fundamentally different from ours. To examine their effect in our setting, we implement MiniLMv2, which shows better performance compared to MiniLM, based on the public codebase and we validate the generalization ability of the student network on 12 fine-grained classification datasets. MiniLMv2 exhibits obviously inferior performance compared to our method, demonstrating that the problem we target to resolve is indeed challenging. We will add the discussion with these methods in our final version.

Weakness 2: Theoretical analysis.

Thanks for the great suggestion. While we agree that this is an important and intriguing phenomenon, providing a strict theoretical analysis is challenging due to the dynamic nature of the training process in knowledge distillation. However, we have attempted to provide an intuitive theoretical explanation based on the interaction between the CE loss and MSE loss: the CE loss forces the student model to fit the ImageNet-1K labels, aligning its feature space to the distribution of ImageNet-1K $P(T_{train})$, while the MSE loss aligns the student's features to the teacher's features, which are learned from a much larger and diverse dataset $P(T_{teacher})$.

When CE loss is included, the student model's feature space is pulled toward the ImageNet-1K distribution, which introduces dataset bias and may limit its ability to generalize to other datasets $P(T_{test})$, as $P(T_{test})$ may share very different distributions with $P(T_{train})$, e.g., the 12 fine-grained classification datasets (conventional distillation methods mainly focus on the experiments on ImageNet-1K, so $P(T_{train})$ and $P(T_{test})$ are the training and validation sets of ImageNet-1K which are well aligned). While we have the prior knowledge that foundation models have great generalization ability, i.e., $P(T_{teacher})$ aligns better with $P(T_{test})$ than $P(T_{train})$, solely focusing on minimizing the discrepancy between the teacher and student features will allow the student model to better inherit the teacher’s generalization ability.

Question 1: Definition of 'feature'.

We thank the Reviewer for the valuable suggestion. The 'feature' represents the output of the last transformer layer in ViTs and we will add this detail in our final version.

Considering the upcoming deadline of the discussion phase, we would appreciate it if the reviewer could provide any feedback based on our reply so that we could further improve our work. Thank you.

I sincerely thanks the authors providing detailed responses, which solves my concerns in the weaknesses and questions .

Moreover, the "Reply to all Reviewers" section clearly solves my major concern (i.e., the novelty) for this paper. From my personal perspective, the fundamental research question of this paper is actually Knowledge Distillation with limited data. This is an intriguing yet underexplored question, since more and more foundation models are developed with large-scale while unaccessible datasets. Previous works have focused solely on relatively simple tasks [1][2][3][4]. In contrast, this paper firstly explored this interesting research question on vision foundation model level and demonstrate the distilled model with limited data (comparing the scale of ImageNet to LAION, etc) can still maintain the original generalization ability. Considering that the setting itself is sufficiently novel, I believe the proposed token-, feature-, and patch-level distillation serves as a solid baseline in this direction. On the other hand, I would kindly suggest the authors avoid using "data-free", as it typically refers to scenarios involving learning with synthetic data [5][6].

Therefore, I am revising my rating to positive in support of this paper. But this paper should do the following things in the final version:

1. emphasize more on the novel setting itself;
2. discuss more related works in terms of data-free KD, few-shot KD, etc;
3. clearly define the "features"

[1] Li, Tianhong, et al. "Few sample knowledge distillation for efficient network compression." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Sauer, Anna, Shima Asaadi, and Fabian Küch. "Knowledge distillation meets few-shot learning: An approach for few-shot intent classification within and across domains." Proceedings of the 4th Workshop on NLP for Conversational AI. 2022.

[3] Kang, Myeonginn, and Seokho Kang. "Knowledge distillation with insufficient training data for regression." Engineering Applications of Artificial Intelligence 132 (2024): 108001.

[4] Cui, Kaiwen, et al. "KD-DLGAN: Data limited image generation via knowledge distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Chen, Hanting, et al. "Data-free learning of student networks." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[6] Yin, Hongxu, et al. "Dreaming to distill: Data-free knowledge transfer via deepinversion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

We appreciate the Reviewer's feedback. We provide further explanations to clarify the Reviewer's concerns based on several key points as below.

Strengths 2: Experiments.

'Experimental results show that models distilled only on ImageNet using the proposed approach match/outperform same size models trained on significantly larger datasets without distillation.'

We thank the Reviewer for noticing this phenomenon but we stress that the oracle models of DINOv2-based experiments (teacher networks) are obtained by distillation from DINOv2-g on their private large-scale datasets.

Weakness 1: Problem setting.

As pointed out in the paper mentioned by the reviewer ("Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models"), pre-training on large-scale datasets requires significantly more computing which may not be affordable to most researchers, especially the ones in academia. ImageNet-1K has long been the cornerstone for advancements in computer vision and can be accessed by most researchers due to its relatively ‘small’ scale. Since our method has shown the ability to match the performance of oracle models in different tasks and datasets, we believe leveraging a smaller dataset to compress foundation models should be praised instead of criticized as our solution can be accessed by more audiences.

If the paper had compared different datasets in terms of their KD effectiveness or proposed methods to create better general purpose KD datasets, that would have been an interesting and useful contribution to the community.

We have provided the analysis of proxy dataset in Sec 3.5 of our paper and we examine the effect from two perspectives.

(1) Dataset Diversity: we introduce ImageNet-Merge, which concatenates all the training sets that we utilized for validation, and ImageNet-Single, a 1.2M training set where the images are generated from a single large image by performing extensive data augmentations. We find ImageNet-Merge brings almost 1% average improvement on fine-grained classification with merely 0.2M additional data and ImageNet-Single surprisingly delivers very decent performance (68.7%), considering that all the information comes from a single picture. This demonstrates that Proteus is robust even under the extreme scenario.

(2) Scaling Behavior: we conduct validation on subsets of ImageNet-1K by either sub-sampling a portion of data at each class or sub-sampling a portion of classes from the total 1000 classes. The results demonstrate the robustness of Proteus when subjected to reduced data availability, suggesting that it is feasible to access foundation models with even smaller data scales.

Weakness 2: Technical contribution.

While there are different kinds of feature distillation methods, we emphasize that we focus on the problem of data-free model compression in a task-agnostic context which is intrinsically different from them as we explained in the reply to all reviewers. Moreover, empirical results shown in that table demonstrate that directly adapting these methods to our problem does not work well and the problem we target to resolve is indeed challenging.

Weakness 3: Presentation.

We thank the Reviewer for the valuable suggestion and we will revise the terminology in our final version.

Table 1 - What does "Hint" in Table.1 refer to?

Hint refers to feature-based distillation according to the definition in FITNETS[1].

Considering the upcoming deadline of the discussion phase, we would appreciate it if the reviewer could provide any feedback based on our reply so that we could further improve our work. Thank you.

[1] Romero A, Ballas N, Kahou S E, et al. Fitnets: Hints for thin deep nets[J]. arXiv preprint arXiv:1412.6550, 2014.

I thank the authors for their detailed response.

First, I would like to emphasize that while the training datasets used to train some proprietary foundation models are not available to public, there are various alternative large-scale datasets that are available for everyone (including academia) to use. For example, DataComp, LAION, etc. Hence, one need not restrict themselves to ImageNet.

I understand that training on web-scale datasets may not be easy in academia. The reason I find the setting unconvincing is not about the data scale itself, but more about restricting only to ImageNet dataset. ImageNet has about 1.3M training images and the paper uses 300 epochs of training. That is equivalent to seeing around 400M examples during training. Since the goal is to get a small foundation model that works on a wide variety of tasks, it seems restrictive to focus only on ImageNet dataset. One can sample diverse subsets of data from existing web-scale datasets and use them for distillation within the same computational budget of "400M seen training samples". I think a thorough comparison of various alternative distillation datasets derived from existing large-scale computer vision datasets would make the paper more useful to the community.

We appreciate the reviewer's support and the valuable insights that helped us to improve the work. Thank you for being with us so far!

I understand that the proposed strategy is task-agnostic since task-specific data is not used for distillation. But, I do not understand why it is data-free. Distillation is performed using ImageNet dataset, i.e., data is used for distillation. If the authors were using some weight cloning/selection strategies to get the compressed model, the term "data-free" would have been meaningful. But, calling it data-free when performing standard data-based distillation is not appropriate.

We noticed that some reviewers have remaining concerns about introducing additional datasets and we would like to resolve the concerns by adding several key experiments before the end of the discussion phase:

While we have successfully replicated the performance of vision foundation models using ImageNet-1K as the training set, exploring alternative datasets should be viewed as a complementary direction as the results are influenced by various factors, including the number of samples, image quality, resolution, data augmentations, etc. Conducting a comprehensive study of this problem demands significant effort and resources. Considering the short time window, we introduce an additional dataset Conceptual Captions Dataset (CC3M) and download the images following the protocol[1]. To minimize resolution disparity with ImageNet data, we preserved the original aspect ratio of the images and set the shortest side to 384 pixels. In the end, we managed to collect 2.2M images as different errors occurred during downloading (expired links, network issues, etc.) and we followed the default data augmentation operations used for ImageNet.

We conduct two experiments using DINOv2-B as the teacher and Proteus-S as the student: (1) randomly sampling 1.2M data from 2.2M images and keeping the original training epoch of 300; (2) leveraging 2.2M images as the training set and reducing the training epoch to 165 to match the same training time budget. The results in the table suggest that CC1.2M yields a marginal performance decline compared to ImageNet-1K, potentially due to differences in data quality and diversity. Interestingly, CC2.2M exhibits slightly inferior performance compared to CC1.2M, suggesting that scaling up data volume within a fixed training budget does not always translate to better results. Overall, our methods show competitive performance using web-sourced data which demonstrates the robustness of our approach and we hope these findings provide valuable insights for the research community.

We sincerely thank the reviewer for the detailed suggestions to improve our work and we will add these discussions in our final version. Please let us know if you have any remaining questions or concerns.

[1] https://github.com/rom1504/img2dataset/blob/main/dataset_examples/cc3m.md