Data-scarce distillation for large-scale vision language models

We thank the reviewer for their thoughtful comments and willingness to reconsider their score.

Limited Experiments (W1)

The experiments are limited in scale, with no evaluations on large datasets, such as ImageNet.

Our focus here is on a particular task-specific setting. ImageNet includes a broad range of images but our work focuses on transferring a narrow set of knowledge from the teacher to the student. We consider settings where the user only needs their model to perform a limited, specific task (such as car type classification, not broad image categorization), and wants to leverage the power of foundation models to do so efficiently.

Additional Backbones (W2, Q2)

The range of compared models is narrow, including only EfficientNet-B0 (EB0).

Does the proposed method perform similarly across all backbones? Specifically, how does it perform with larger models, such as ResNet-18 mentioned in the paper?

We are unable to pretrain CLIP student models with a variety of backbones including ResNet but we include results for EfficientNet B1 and B2 models (with a more limited pretraining setup). We have added two additional backbones, EfficientNet B1 and EfficientNet B2. These CLIP-initialized models were pretrained with less DataComp data due to time constraints, so results are not directly comparable to EfficientNet B0 results, but we demonstrate that relative trends still hold. We include experiments on only the Stanford Cars dataset for the 1, 2, 4, and 8 shot settings. We note that SIDCLIP provides notable boosts relative to baselines for each model backbone.

8 shot

4 shot

2 shot

1 shot

Quality of Synthetic Data (Q3)

How is the quality of generated data ensured?

We would imagine an intuitive relationship between the quality of generated images and downstream results. We selected a performant existing diffusion model as the focus of our work is not on generative models. We did some limited hyperparameter tuning and manual evaluation of the generated images to ensure that fidelity was sufficient for the task at hand. However, we would like to emphasize that the added benefit of synthetic images is obtained without significant tuning effort needed for the generative model. Please note the images in Figure 4, which include representative samples from each dataset used in our paper. Our minimal hyperparameter tuning included selecting the weighting between images and text, and selecting to perform an interpolation between two images and one text prompt.

We would like to thank the reviewer for their thorough feedback.

Limited Novelty (W1)

Besides combining synthetic data, initialization and distillation for the data-scarcity specific setting, the proposed approach is limited in novelty. The main elements (distillation [1] and data generation [2]) are straightforwardly utilized. While their combination is interesting, the results are not very surprising due to the expected complementary knowledge coming from different sources of information: the generation capacity of latent diffusion informed with the specific dataset at hand, the pretrained ViT-L/14 knowledge and the DataComp data for pretraining as initialization.

We agree that our work is primarily a practical guide to knowledge distillation that leverages foundation models and that the combination of our methods may not be surprising. However, we believe that despite the simplicity of our approach, it still carries value. Synthetic data does not always improve performance, and has been shown to cause model collapse in some settings. Distillation can also be challenging across large size gaps between the teacher and student, particularly across different architectures. Although our result may not be surprising, we believe it may be valuable to the community to demonstrate the combined benefit of our approaches, whether it provides a baseline for future research into foundation model distillation, or whether it can be used as a guide for users who simply want a small but performant task-specific model.

Additional Distillation Methods (W2)

While the classical logit-based knowledge distillation approach seems to be effective, more advanced techniques were introduced in the literature ([A],[B],[C]). Since the approach is straightforward, it would be interesting to explore some other distillation methods as a positioning among other design choices. For example, some distillation methods were shown to improve classical distillation by decoupling target class and non-target class terms [B]. Other methods show that distillation is also possible for "smaller" and "weaker" teacher [A]. Investigating such choices might help answering the question whether ViT-L/14 is needed as a teacher, or if the same or close results can be achieved with a smaller version of CLIP as teacher.

We agree that fruitful future work could examine the effect of using a smaller or weaker teacher model. However, in our paper, we choose to limit ourselves to the setting where we have access to a strong teacher model. Of the methods listed here, we we able to include additional results on decoupled knowledge distillation [B]. We include a limited set of experiments here where we use the CLIP-initialized EfficientNet B0 and the StanfordCars 8 shot dataset. SIDCLIP achieves the highest performance when using DKD. We use alpha=1 and beta=8. We would imagine that other distillation methods would hold a similar pattern, and we will include these results in our final paper.

Limited Datasets (W3)

The method is tested on only 3 datasets. Corroboration on more datasets helps assessing the value of the proposed framework and its practical use, especially that the target is a data-specialized model not a general purpose one. For example, the authors could add more experiments on common datasets used for few-shot learning with CLIP: ImageNet, DTD, EuroSAT, FGVCAircraft, SUN397, Caltech101... Please refer to [D] for more examples of such datasets. Some of these dataset (like DTD, EuroSAT, FGVCAircraft) differ from the used ones in being more specialized and fine-grained. How would each of the components help the total performance for such datasets?

We agree that more datasets would help paint a more comprehensive picture of our results, but unfortunately we are limited by computational constraints. We chose our three datasets for their specialized task-specificity. Each of the datasets we chose is limited to a narrow domain (cars, flowers, food) and the classes within each dataset highlight fine-grained differences between otherwise somewhat similar images.

Thank you to the reviewer for their generally positive comments and rating.

Ablations (W1)

The new proposal is extending several existing methods and putting things together to reach some better performance. While this is innovative, it will be more convincing to analyze the role of each component for the overall performance improvement.

We agree that comprehensive ablations paint a better picture of the strength of our model. Here we include a comprehensive set of ablations of each set of components for a subset of our experimental settings, namely 2, 4, and 8 shot versions of StanfordCars using the EfficientNet B0 model. We will add these to the final paper. We note that synthetic data can sometimes detract from overall performance, but in conjunction with distillation and initialization, it consistently outperforms baselines.

Additional Models (Q1)

Related to the weakness, for example, will the synthetic data help other models also?

We have added two additional backbones, EfficientNet B1 and EfficientNet B2. These CLIP-initialized models were pretrained with less DataComp data due to time constraints, so results are not directly comparable to EfficientNet B0 results, but we demonstrate that relative trends still hold. We include experiments on only the Stanford Cars dataset for the 1, 2, 4, and 8 shot settings. We note that SIDCLIP provides notable boosts relative to baselines for each model backbone.

8 shot

4 shot

2 shot

1 shot

Typos (W2) Thank you for pointing this out - we will fix this typo and double check our paper for copy editing errors.

Thanks for the additional experiments, which make the work more solid for my previous 6 rating. Yet, I do not plan to increase it further.

We would like to thank the reviewer for their thoughtful comments.

Computational requirements (W1, W4, Q1)

The use of text-to-image generation to create synthetic data can introduce computational overhead, potentially making the framework slower or more resource-intensive.

The combination of synthetic data generation, initialization, and distillation steps may limit SIDCLIP’s practicality in scenarios requiring minimal preprocessing. The entire pipeline involves multiple stages that could increase setup and training time. It would be beneficial to provide the computation requirements compared to existing methods.'

Could the authors provide more quantitative details on SIDCLIP’s training efficiency compared to other distillation methods?

The data generation process can be expensive. This work focuses on a setting in which a user has limited real data and wants to have low inference latency. Therefore, we assume that some upfront cost is acceptable. This upfront, one-time cost can be viewed as being amortized over an improved inference time. The user spends some cost to generate synthetic data and perform distillation, but at the end of this process, the resulting small model has high performance and low inference latency and can be used with less overhead than other alternatives, such as using the teacher at inference time. We provide some of the times required for these processes. Using one Nvidia A6000 GPU, the generation of 100 images takes 840.6 seconds. The difference in time for one epoch when using just real data vs real + synthetic data can primarily be ascribed to the increased amount of data being processed. For the 8 real shot setting (8 images per class), our CLIP-initialized EfficientNet B0 model takes around 24 seconds for one epoch of distillation, and the 8 real + 300 syn shot setting (308 total images per class) takes around 1021 seconds for one epoch of distillation.

Additional Backbones (W2)

The paper primarily experiments with EfficientNet B0 as the small model architecture. How does SIDCLIP perform with different or larger backbones? Testing with a broader range of architectures would help demonstrating the generalizability of SIDCLIP.

We have added two additional backbones, EfficientNet B1 and EfficientNet B2. These CLIP-initialized models were pretrained with less DataComp data due to time constraints, so results are not directly comparable to EfficientNet B0 results, but we demonstrate that relative trends still hold. We include experiments on only the Stanford Cars dataset for the 1, 2, 4, and 8 shot settings. We note that SIDCLIP provides notable boosts relative to baselines for each model backbone.

8 shot

4 shot

2 shot

1 shot