Don't Pre-train, Teach Your Small Model

1. A fully finetuned student would be an interesting data point. Intuitively, we believe that it would achieve better performance, but would be the most expensive method on our cost plots. In addition, this would really limit the student to being good on only one task due to the backbone distortion mentioned in section 3.1. However, if you believe that this data would significantly strengthen the main intention of our paper, we can provide it.
2. From a practical standpoint, we believe it’s useful for those who want to use a small model that isn’t available in any of these hubs. Granted, we used small models that were already trained on ImageNet in our illustrative examples, but that was mainly for our convenience and research efficiency. However, the future of model hubs and licenses is unknown, but our method enables practitioners to lower their reliance on hubs for all of their model needs.
3. Sometimes data generation is a key step. Importantly, we find that the synthetic data component is the most significant in the data scarce scenarios. As we can see (Table 3), the datasets with plenty of samples (CIFAR-10, CIFAR-100, CUB-2011) are able to surpass their pre-trained counterparts without adding any synthetic data (DPT (0x)). Overall, synthetic data just boosts performance at test time, which is an observation known to the community for a while now. If it is the case that the synthetic data is not as easily generated, then our method would take a small hit, but by no means would it be failing as a result. Echoing what we said in the Novelty section above, the synthetic data should be viewed as a free benefit of work in pre-trained deep generative models. If it is there and works; great. If not, one still has the teacher and distillation algorithm that can give significant gains regardless.

4. (1, 2, 4) addressed in Novelty above
5. (3) With regards to the distillation algorithm itself, as we mentioned in the paper, our choice of the alignment/uniformity metric was purely illustrative. We appreciate the work of CRD in bridging contrastive learning with KD, but their work was done pre- the modern boom in contrastive learning brought on by SimCLR, MoCo, CLIP, etc. Therefore, we formulate KD as a contrastive objective with the modern notation and considerations, massaging it into an InfoNCE-type form that SimCLR and MoCo build off of. From here, any recent development (most of which took place after the publication of CRD), could be used, and which we assume would also be successful. The A/U metric was chosen since one of the main themes of the paper was efficiency; if we’re going to minimize the training time for the small model, let’s try a method that doesn’t include all the bells and whistles such as large batch sizes or momentum-updated models.
6. [5] We welcome any suggestions for other models you believe would make our case stronger. We believed that choosing a popular convo-based one and transformer-based one was sufficient to demonstrate that our theory was justified in practice. Most other vision models are just derivatives of these 2 model philosophies. In addition, the synthetic data is never seen by the teacher during training (thought this is a good idea!), and we added clarifying language for this. We do not believe this is cause for concern, because for 3 datasets (CIFAR-10, CIFAR-100, and CUB-2011), we are able to demonstrate that without any synthetic samples our paradigm works to beat the baseline.
7. (6) Our reasons for linear probing as opposed to full finetuning come from theoretical and practical considerations. While full finetuning is certainly more robust to distribution shifts, it has been found to lead to a distortion of the backbone which will probably harm future finetuning to other tasks. We wish to leave our teacher “unharmed” in the case where we wish to use the foundation for multiple student tasks. From a practical perspective, most foundations that are finetuned for a task in the modern era are linearly probed for this exact reason. In addition, full finetuning incurs a much higher cost on the practitioners end as they now have to train the teacher end-to-end on each task. With the increasing size of foundations, this would be a significant cost for the intended users of our method. This is fundamentally even closer to the “vanilla KD” regime that you bring up in your first point; one minor improvement we have is that by taking an off-the-shelf pre-trained one, we only need to train one linear layer and our teacher is ready to provide big improvements.
8. (7) This sounds like a grammatical issue, which we don’t agree is contradictory to the message of our paper. We believe it is reasonable for readers to infer that the “Pre-train” in the title refers to the object of the phrase, “small model”, as opposed to being a declaration of not pre-training anything ever. This should also be clear after reading the abstract and/or introduction as well. For how it fits in the broader context of other methods, see above “Comparison to NAS, Pruning, Quantization”.
9. For the datapoints requested: (1) the situation where the teacher is exposed to the synthetic data and (2) where it is fully finetuned, we believe that these would just lead to better performance altogether, but would not significantly alter our conclusions. However, if you disagree, we can attempt to provide them but may not be able to gather all the data by the deadline.

1. We added references to these works, thank you for pointing them out. However, we do not believe a comparison to them is appropriate. KDEP is fundamentally a pre-training optimization, whereas we want to completely avoid pre-training of any kind. KCD takes an E-M approach to the choice of the transfer dataset each iteration, whereas we propose to augment it at the beginning. Both of these are great ideas, and there is no reason why one cannot apply them to our formulation as well (i.e. running KCD on a synthetically-augmented transfer set, or doing KDEP to warm up the student, although this loses the spirit of our approach). We believe both these approaches tackle different aspects of the training procedure that do not intersect with our work. TrustAL is an interesting approach, though their work concerns the language domain. More importantly, we do not see how the problem that they tackle, example forgetting, is related to our work that justifies a comparison. Sure, they use a teacher model to help minimize forgotten knowledge, but that is fundamentally a different use of a teacher model compared to ours. Again, there is an opportunity here to utilize their method in tandem with ours in an active learning setup.
2. Addressed above in Novelty section
3. KDEP is a great work that proposes a well-justified and effective optimization to pre-training. We particularly enjoyed their observation that entire foundation datasets may not even be necessary to pre-train! However, we maintain the position that a direct comparison to their work is unnecessary. In addition, the “16 hours” claimed is not a comparable data point without accounting for differences in hardware used. Their paper states that their timing numbers were done on 4 V100 GPU’s, whereas we used 1 P100, an older and slower architecture. Even if we ignore the supremacy of the Volta (V) architecture over Pascal (P), this translates to KDEP taking around 64 hours to achieve that 82%, which is 3x longer than our method; however, they do achieve a higher accuracy at this high cost. We believe that the desired “comparable scenario” is addressed in Section 4.3, where we fix the same teacher-student, use the same transfer dataset, but compare to popular and effective distillation algorithms: vanilla KD, CRD, and SRRL. There, we see that our method is competitive, but no method comes out to be the outstanding winner. Since our main contribution is the overall recipe (see Novelty response above), we are not looking to show purely the supremacy of our contrastive-based distillation algorithm, but that we can formulate KD as a contrastive objective, and any algorithm (like the Align/Uniform we choose) can be competitive to SoTA. Also, our experiments do indeed combine the real and synthetic datasets. DPT (1x) combines the real dataset with an equally sized synthetic one, while DPT (2x) combines the real with a synthetic one twice the size. If there is anywhere in the paper that we can clarify this better, please let us know.

1. Addressed in Novelty section above
2. We welcome suggestions as to other datasets or pairings you believe would strengthen our case. Granted, we chose 1 modality, but in this modality we attempted to use datasets that reflect different realities (data is abundant vs data is scarce). For the models, we carefully chose pairings to exemplify how flexible our method is. One conv-based and one transformer-based teacher have no problems working with small models that have different design approaches.
3. The conclusion to be taken from the tables is that the overall flow is highly effective and multiple distillation methods can be used to get around similar levels of performance. Vanilla KD can’t always be used (e.g. self-supervised regime, Table 4 in Appendix A.1), and many new KD algorithms tend to over-engineer, in our opinion, the knowledge transfer. Instead, our generic contrastive formulation is straightforward, makes no assumptions about the underlying model architectures, and performs competitively to previous SoTA works. We find that most modern works in KD that are accepted to similar venues propose some new heuristic distillation algorithm and test it on certain model pairings that show their superiority. However, when subjecting just 2 of these, CRD and SRRL, to reproduction in our paper, we find that while they perform well, none are overall the “best”. We wish for attention to focus on the bridge between contrastive learning and distillation; practitioners can then decide which contrastive algorithm they wish to try and empirically find to work best. In addition, S-LP is defined in section 4.1, where we claim that “S” refers to the student, and “LP” refers to pre-trained on ImageNet and linearly probed for the task.
4. We only did 1 experiment for each. More experiments are always better, but given our time and resources, we do not believe we can run every experiment multiple times by the deadline. However, a future version of this paper would definitely have them.
5. If by “pre-training cost”, you are referring to the cost of pre-training the teacher and generative models, we chose to omit those since those costs are not absorbed by the intended practitioners of this method. They can assume that these models already exist and are available for the using immediately. We added a clarification about this in our updated version.
6. Addressed in Comparison section above
7. The baselines were not exposed to the synthetic samples. The only scenarios that were were DPT (1x) and DPT (2x). We updated the wording in our results section to clarify that only the transfer dataset is augmented. We can certainly provide those if desired, but does the reviewer believe that these would lead to a different or stronger conclusion than is currently provided?

I thank the authors for their reply, and will respond to the more general statements here first.

On the novelty
(This is a copy of my individual reply). I do not believe that the shared response answers this convincingly. Is it not true that the primary proposal in this paper is to leverage distillation from a teacher that wasn't trained from scratch to some distillation datasets, but rather pretrained? If so, I do not see any novelty in the conclusions that are drawn here - it is still distillation from a stronger teacher, with the minor difference that it was pretrained. It seems that the authors agree on this aspect as per the shared response. But given that, it remains unclear to me what the actual contribution in this paper is. Because it is not the insight that you can distill from a stronger teacher, and it can't be the fact that training on synthetic data benefits model performance or allows useful representation to be learned (see e.g. [1,2,3,4]), particularly when leveraging generative models that were trained on much more expansive datasets. And using both together in my eyes is not sufficient contribution.

[1] Beery at al. 2019, "Synthetic Examples Improve Generalization for Rare Classes"
[2] Lehner et al. 2023, "3D Adversarial Augmentation for Robust Out-of-Domain Predictions"
[3] Tian et al. 2023, "StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners"
[4] Azizi et al. 2023, "Synthetic Data from Diffusion Models Improves ImageNet Classification"

On the comparison to NAS / Pruning / Quantization

... all these methods incur tremendous additional costs. One of the largest benefits of our method is how lightweight it is.

I don't agree with this statement - generating hundreds of thousands synthetic examples, with comparable number of forward passes through the foundation model to distill from is anything but lightweight. Indeed, from a cost perspective, I'd wager pruning to be notably cheaper - though I'm not 100% sure, and this is where the issue of lacking comparisons to these existing approaches comes into play. A direct comparison would give the reader much more intuition into when and why the proposed approach should be applied. In my eyes, it isn't even a performance competition, different approaches may operate orthogonally. But without any comparisons, no clear statements can be made in that regard.