Can pre-trained models assist in dataset distillation?

1. Lack of analysis of the training time/storage cost of leveraging PTMs in DD, especially the domain-specific case. Pretraining models with different initialization and model architecture over the target dataset seems time-consuming (even with early-stopping).
2. The analysis of cross-architecture generalization. The reported number in table 4 seems inconsistent with that in table 3 of the original DM paper.

1. The model diversification and whether or not the models need to be well-trained have been studied in CVPR 2023 paper, "Accelerating Dataset Distillation via Model Augmentation". This paper pointed out these two points in their paper.
2. The improvement with CCLoM (Table 2) is very marginal, the average gain only up to 1.35%.

1. A primary concern is the seemingly ambiguous use of the concept of Pre-trained Models (PTMs). There are very limited details on the data that are used for PTMs during the pretraining stage. The introduction and related work sections give the impression that pretraining follows the convention of using the ImageNet dataset, similar to methods they cited. However, I realize until section 5.2 when the discussion on pretraining across various epochs (1, 4, 6, 10...) is provided, that this is probably not pretrained on the imagenet dataset. However, if the model is pretrained on CIFAR10 (that further used for distillation), one would expect it to naturally have a better performance during distillation. How is it different from the buffer training stage in previous works e.g. [1,2]?
2. The overall structure and writing of this paper lack clarity and cohesiveness. For example, the decision to space out section 3.2 (experiment setup) and section 5, and to introduce losses in sections 4 and 5.3 separately, results in disjointed reading and necessitates excessive cross-referencing.
3. Two of the highlighted key findings do not appear particularly new. The model diversification is a pretty established technique that are widely used in a lot of DD works, and domain knowledge part also seems to be a widely known concept. The point is, I do appreciate the authors provided very detailed experiments but I don't think the authors communicate their key findings very effectively in the introduction part.
4. The paper has a lot of technique details missing e.g. dataset resolution, full dataset training performance, and specifics about datasets used for model pretraining.
5. For the distilled dataset evaluation using data augmentation techniques. What's the size of the final distilled-then-augmented dataset? Similarly, as mentioned in point 4, there's an absence of full dataset training performance metrics, especially when data augmentation techniques are applied, which makes it hard to evaluate the results.
6. In Figure 1, on which dataset was this experiment conducted? It is not identified or described within the paper.

[1] Cazenavette, George, et al. "Generalizing Dataset Distillation via Deep Generative Prior." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Zhou, Yongchao, Ehsan Nezhadarya, and Jimmy Ba. "Dataset distillation using neural feature regression." Advances in Neural Information Processing Systems 35 (2022): 9813-9827.

The paper primarily focuses on empirical studies and experimental results. It would be valuable to provide more theoretical analysis and insights into why the proposed approach works, such as discussing the underlying principles or mathematical foundations.

This paper investigates the application of pre-trained models (PTMs) in dataset distillation (DD) and their potential to enhance the performance of synthetic datasets. The authors introduce two novel loss terms, CLoM and CCLoM, to leverage PTMs as guidance for optimizing synthetic datasets. The paper systematically explores various aspects of PTMs, including initialization parameters, model architecture, training epoch, and domain knowledge, offering valuable insights. The comprehensive analysis of PTM options and their impact on synthetic datasets provides a thorough understanding of the individual factors' influence. The presentation of findings and analysis is clear and well-organized, making it accessible for readers to follow and grasp the experimental results. Furthermore, the paper's focus on dataset distillation and the exploration of factors influencing synthetic datasets has practical implications in the machine learning and data preprocessing domain, offering insights into effectively leveraging PTMs for dataset enhancement. However, the paper could benefit from a more in-depth theoretical analysis. While the empirical studies and experimental results are valuable, providing a deeper understanding of why the proposed approach is effective would enhance the paper's quality. A discussion of the underlying principles or mathematical foundations would further strengthen the theoretical foundation of the research. Furthermore, the manuscript exhibits some minor imperfections, with an identified inaccuracy in Equation (1).