DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation

Thank you for the valuable and constructive comments. We will incorporate all suggestions in our revision. Below, we provide further clarifications to the reviewer's questions.

Q1: Comparison to previous methods.

Thanks for your comments. We are confident that no prior work has specifically focused on EarlyLate training in the context of dataset distillation. While our method is technically straightforward, it achieves state-of-the-art accuracy across multiple benchmarks, from small-scale to large-scale datasets. Therefore, we believe our contribution is significant and not merely incremental for dataset distilaltion task.

Q2: The motivation and the advantages of the "Selection Criteria" in the initialization approach are not clear. And I am confused about how to rank, which is presented in Fig 5, could the authors explain it here?

Thanks for highlighting this point. We select the N images with scores around the median from the teacher model: the score being the probability of the true class. The motivation is that such images have medium difficulty level to the teacher, so they have more room for information improvement via distillation gradients. We further empirically validate this strategy by comparing different approaches in Table 3b.

For instance, in Figure 5, ranking is based on the probability of the true class, and the figure shows selecting (IPC=3) images around the median score, to get the medium difficulty images.

Q3: There are a lot of hyperparameters involved. How should these hyperparameters be tuned? Are there any principled approaches?

The original hyperparameters follow baseline methods of CDA and SRe$^2$L, our method have a hyper-parameter for the Round Iterations (RI) that specifies the number of rounds = Total Iterations / RI. As highlighted in Table 3.c, less round iterations generates more rounds and thus contributes to more diverse synthesized images and better performance.

Q4: I want to know the impact of the initialization method. In the ablation study, only CDA+init is shown. More advanced methods with init and whether EarlyLate uses init are not presented.

Thanks for pointing this out, we hightlight that DELT, the EarlyLate, indeed uses the mentioned initialization. As for the other methods, we focused on gradient-based methods capable of scaling to large datasets. Therefore, we compared using the advanced CDA or SRe$^2$L, and the CDA was better. We provide the performance comparison below for IPC 50 on ImageNet-1K:

As we can see, our EarlyLate strategy enhances the performance of around +1% over the initialization. Without initialization, our method improves even more, with 2.4% as follows:

In brief, initialization alone enhances the performance over the basic CDA/SRe$^2$L. The proposed EarlyLate strategy further enhances the performance by +1% over the initialization.

[1] Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. Advances in Neural Information Processing Systems, 2023. [2] Dataset Distillation in Large Data Era. arXiv preprint arXiv:2311.18838.

Q5: The performance of other sota methods on MobileNet-v2 is not presented in Table 1. Is the proposed method still better than other sota methods on MobileNet-v2?

We appreciate your suggestion. We provide the comparison of our method against RDED when using MobileNet-v2 as below:

Q6: Training tricks like random crop play a significant role in methods such as SRe2L. I would like to know to what extent the method proposed in this paper relies on such tricks.

Thank you for highlighting this point. We have included a table comparing different initial crop ranges in random crop augmentation for our DELT method. As shown in the results, the 0.08-1.0 range yields the best performance, which is why it is the default setting in our framework.

We appreciate the reviewer's detailed comments and constructive suggestions, and would like to address some key points from our submission that may have been overlooked, which might have led to confusion and concerns to the reviewer.

W1: Initialization with real samples is common. Suggest adding diversity analysis and comparison of the distilled data.

Thank you for your suggestion. Our initialization method differs from RDED as we use median scores from the teacher model. We do not select the easiest images as RDED because we want to pick the ones with room for information enhancement via gradient updates. We also have credited RDED appropriately and will add further acknowledgment regarding RDED's initialization idea for the DD task in our revision. However, the core of our method is the EarlyLate training approach, not the initialization. Even without this initialization, our method significantly outperforms SRe$^2$L and CDA, with improvements of 9.1% and 2.4%. We also have provided a diversity analysis, specifically the intra-class semantic cosine similarity of synthetic images in Figure 2 left of our submission. Our method achieves the lowest similarity, indicating the highest diversity.

W2: It seems that only the real initialization contributes to the diversity.

Figure 7 (last column, top to bottom) in our submission clearly demonstrates how our EarlyLate strategy enhances diversity in batch-to-global optimization methods. Please refer to it for details.

We also highlight that our method produces more diverse images than RDED, as demonstrated in Figure 2 left, even though RDED merges 4 patches together initialized from the original dataset. These statistics demonstrate that simple initialization is not as diverse enough as our EarlyLate strategy.

Q1: Equation 7: A comparison between random order and ascending/descending order by patch probability could rationalize model design.

Thanks for the suggestion. In our DELT, we select the N patches with scores around the median from the teacher model, where the score represents the probability of the true class. To order them, we start with the median, and we go back and force expanding the window around the median until we cover the number of IPCs, refer to Figure 5 for details. The rationale is that these patches present a medium difficulty level for the teacher, allowing more potential for information enhancement through distillation gradients while having a good starting point of information. We empirically validate this approach by comparing different strategies in Table 3 (b) of our paper. We provide these analyses here in the below table including the random order.

We present the impact of using different ordering on ImageNet-100 when having the same initialized images, those around the median, as below:

Furthermore, we also include a comparison of the performance of different initialization strategies based on the order. Unlike the previous table, the initialized images are different:

Q2: Table 3 (c).

Thank you for the insightful suggestion. Table 3 (c) is used to identify the optimal hyperparameter for the EarlyLate interval and the overall training budget. We have included additional results below and will polish this section in our revised paper accordingly.

Q2-1: Comparison of 1K/1K is non-trivial (baseline).

Thanks for your suggestion. We provide the results of 1K/1K as below:

We highlight that the 1K/1K configuration yields the lowest score because it cannot apply the EarlyLate strategy. With only one round (1K/1K = 1), all IPC images are updated for 1K iterations, which fails to enhance diversity as seen in other experiments in the table.

Q2-2: It is interesting that 4K/2K is worse than 2K/1K.

Thank you for highlighting this point. More round iterations (RI) do not necessarily lead to better performance with the same number of rounds due to initialization factors. However, increasing total iterations while also increasing the number of rounds is beneficial, as it enhances diversity and overall performance, as shown in Table 3 (c).

Q3: In Table 5: 4K-iteration is not fair for comparison since original SRe2L only trains for 1K iterations.

We appreciate the reviewer's comments. To clarify, the default setting for SRe$^2$L also uses 4K iterations for their final reported models, as described in their paper (Sec 3.2, "Recover Budget"). In our Table 5, as reiterated in the caption, SRe$^2$L and all other methods are trained with 4K iterations using the official code to ensure a strictly fair comparison with our approach.

Additionally, if the base algorithm converges within 500 iterations, DELT would become 500/100 or 500/250, still significantly faster than the base algorithm. The mechanism and design of our DELT ensure it will always be much faster than the base method, regardless of the number of iterations needed for convergence.

Minors:

Line 66, footnote links to Fig.2

Thank you for pointing this out. We have double-checked the footnote to ensure it references the correct position.

A figure with training time as x-axis and training sample number as y-axis.

Thank you for the insightful comment. Following the suggestion, we have included a figure (Figure 3) in the PDF attachment to directly illustrate the reduction in computation time.

We appreciate the reviewer's valuable and constructive commetns. We will accommodate all of the suggestions in our revision. In the following, we make further clarifications to reviewer's concerns.

Q1: The motivation for the research is unclear, lacking an explicit articulation of the unifying challenges faced by current state-of-the-art works.

Thanks for the suggestion. Our proposed DELT method is motivated to address the widely-recognized less-diverse data generation issue in batch-to-global methods, which is our primary goal of this work. As we have briefly introduced in the Abstract section, prior state-of-the-art dataset distillation methods like SRe$^2$L (NeurIPS'23) employ batch-to-global optimization, where target images are generated by independently optimizing samples using shared global supervision signals across different synthetic images, which suffers limited supervision and synthesis. G-VBSM (CVPR'24) improves matching precision by utilizing a diverse set of signals from multiple backbones and statistical metrics, but the increased model diversity also adds to the overall complexity of the framework, reducing its conciseness. RDED (CVPR'24) uses a train-free image stitching method that crops original images into patches ranked by realism scores from an observer model, but it does not enhance or optimize the visual content within the distilled dataset. Consequently, the diversity and richness of information depend heavily on the original dataset's distribution. That is why RDED does not improve the diversity much (refer to Figure 2 left). Our EarlyLate optimization-based solution enjoys the flexibility to fine-tune images with efficient training, offering the advantages of both strategies.

Q2: The resolution of the figures is inadequate, impeding clear interpretation of the results.

We have included a higher-resolution version of Figure 1 in the rebuttal PDF, please check it out. Datasets like CIFAR and Tiny-ImageNet have low original resolutions (32 $\times$ 32 in Figure 11 and 64 $\times$ 64 in Figure 9), so images may appear blurry when enlarged. We will aim to provide figures with the highest possible resolution in our revised paper.

Q3: There is inconsistency in the styling of table borders and captions, with captions for Table 1 and 2 placed in different positions compared to subsequent tables, some above and some below the table.

Thanks for pointing this out. We have made the caption location consistent in our manuscript and will update into our revised submission.

Q4: The experimental settings are not uniformly aligned, and efforts should be made to cover all datasets and settings consistently across all experiments to ensure comparability and rigor.

Thanks for raising this concern. We ensure that all experimental settings across our tables are fully consistent across different methods. This alignment has been thoroughly double-checked during the rebuttal. We are confident that the superior performance of our method is entirely based on the fair comparisons and our proposed approach. If the reviewer has further concerns about the specific table, kindly point them out and we are happy to clarify.

Thank you for pointing this out! The mistake in the caption was unintentional, likely due to following other works. We will remove it in the revision to ensure it aligns with our table.

We thank the reviewer for the constructive comments. We will accommodate all the suggestions in our revision. In the following, we make further clarifications to reviewer's concerns.

Q1: Introduction describes previous methods in too much detail.

We appreciate the constructive suggestion. We will streamline the introduction, refine the related work section to merge overlaps in the introduction and focus on providing a concise overview of previous works in the revision.

Q2: dedicate more space to explaining Concatenation Training and Training Procedure.

Thanks for the suggestion. We explain the detailed process as follows, which will be included in our revision. The representative batch-to-global DD methods like SRe$^2$L, CDA, G-VBSM, and our method contain three stages: 1) Pretrain/Squeeze model, the objective of this stage is to extract crucial information from the original dataset. 2) Data synthesis, this phase involves reconstructing the retained information back into the image space utilizing class labels, regularization terms, and BN trajectory alignment. 3) Post-training on synthetic data and evaluation using soft labels. We elaborate on each stage in detail using formulas as follows:

1. Pretrain/Squeeze Model:

The learning process can be simplified to regular model training on the original dataset using an appropriate training recipe:

$

\boldsymbol{\theta}_{\mathcal{T}}=\underset{\boldsymbol{\theta}}{\arg \min } \mathcal{L}_{\mathcal{T}}(\boldsymbol{\theta})

$

where ${\mathcal{T}}$ is an original large labeled dataset and we train the model with parameter $\boldsymbol{\theta}$ on it for data synthesis/recovery. $\mathcal{L}_{\mathcal{T}}(\boldsymbol{\theta})$ typically uses cross-entropy loss as:

$

\mathcal{L}_{\mathcal{T}}(\boldsymbol{\theta})=\mathbb{E}_{(\boldsymbol{x}, \boldsymbol{y}) \in \mathcal{T}}[\boldsymbol{y} \log (\boldsymbol{p}(\boldsymbol{x}))]

$

2. Data synthesis:

Our EarlyLate Optimization:

$

\mathrm{Round \ 1}: \underset{\mathcal{C}_{\mathrm{IPC}_{0:k-1}},|\mathcal{C}|}{\arg \min } \ell\left(\phi_{\boldsymbol{\theta}_{\mathcal{T}}}\left(\widetilde{\boldsymbol{x}}_{\mathrm{IPC}_{0:k-1}}\right), \boldsymbol{y}\right)+\mathcal{R}_{\text {reg }}

$

...

$

\mathrm{Round \ M-1}: \underset{\mathcal{C}_{\mathrm{IPC}_{0:Mk-1}},|\mathcal{C}|}{\arg \min } \ell\left(\phi_{\boldsymbol{\theta}_{\mathcal{T}}}\left(\widetilde{\boldsymbol{x}}_{\mathrm{IPC}_{0:Mk-1}}\right), \boldsymbol{y}\right)+\mathcal{R}_{\text {reg }}

$

where $\mathcal{C}$ is the target small distilled dataset with $\widetilde{\boldsymbol{x}}$. $\mathrm{M}$ is the number of batches and $\mathrm{M}>1$ (If $\mathrm{M}=1$, training will degenerate into a way without EarlyLate). This process is referred to in Figure 4 of our submission. $\mathcal{R}_{\text {reg }}$ is the regularization term, we also utilize the BatchNorm distribution regularization term to improve the quality of generated images:

$

\mathcal{R}_{\mathrm{reg}}(\widetilde{\boldsymbol{x}}) = \sum_l\left|\mu_l(\widetilde{\boldsymbol{x}})-\mathbf{B N}_l^{\mathrm{RM}}\right|_2+\sum_l\left|\sigma_l^2(\widetilde{\boldsymbol{x}})-\mathbf{B N}_l^{\mathrm{RV}}\right|_2

$

where $l$ is the index of BN layer, $\mu_l(\widetilde{\boldsymbol{x}})$ and $\sigma_l^2(\widetilde{\boldsymbol{x}})$ are mean and variance. $\mathbf{B N}_l^{\mathrm{RM}}$ and $\mathbf{B} \mathbf{N}_l^{\mathrm{RV}}$ are running mean and running variance in pre-trained model at $l$-th layer, which are globally counted.

3. Post-training on synthetic data and evaluation:

$

\widetilde{\boldsymbol{y}}_i=\phi_{\boldsymbol{\theta}_{\mathcal{T}}}\left(\widetilde{\boldsymbol{x}}_{\mathbf{R}_i}\right)

$

where $\widetilde{\boldsymbol{x}}\_{\mathbf{R}\_i}$ is the $i$-th crop in the synthetic image and $\widetilde{\boldsymbol{y}}\_i$ is the corresponding soft label. Finally, we can train the model using the following objective:

$

\mathcal{L}_{\text {syn }}=-\sum_i \widetilde{\boldsymbol{y}}_i \log \phi_{\boldsymbol{\theta}_{\mathcal{C}_{\text {syn }}}}\left(\widetilde{\boldsymbol{x}}_{\mathbf{R}_i}\right)

$

Regarding concatenation training, we elaborate:

• Our EarlyLate tries to enhance the diversity of the synthetic data by varying the number of iterations for different IPCs during data synthesis pahse.
• This means the first IPC can be recovered for 4K iterations while the last IPC will only be recovered using 500 iterations.
• To make this process efficient, we share the recovery time (on the GPU) across the different IPCs via concatenation to minimize the time as much as possible.
• Therefore the first image IPC will start recovery for a couple of iterations, and when it completes iteration 3,500 the last IPC will join it in the recovery phase to get its 500 iterations.

Q3: A more sound explanation.

Thank you for the suggestion. We have included an illustration in the attached PDF to explain our method from an optimization perspective. Following [1], we visualize the optimization trajectory of the network loss landscape using the same supervision and training data but with different training budgets of 50, 100, 150, and 200 steps (similar to our EarlyLate training). This demonstrates that each sub-batch is optimized in different directions, even though the supervision signal appears to be the same global signal used in other batch-to-global methods.

[1] Li, et al. "Visualizing the loss landscape of neural nets".

Q4: When training the later sub-batches in DELT, are previous parts frozen, or are they trained together?

Thank you for raising this suggestion. In DELT, later sub-batches join the previous sub-batches in recovery/training, rather than freezing the earlier sub-batches. We will clarify this in our revision.

Thank you for your kind words and for raising the score. We have no intention of complaining but are thankful and sincerely appreciate the time and effort you invested in reviewing our work, and further helping us improve it. We assure that we will polish our paper carefully according to the suggested comments. Wishing you a great day!