Prioritize Alignment in Dataset Distillation

We sincerely thank the reviewer UeUP for valuable feedback. We make responses as follows.

W1: PAD introduces two new sets of parameters, which add to the complexity of tuning

Thanks for raising this concern. We would like to make the following clarifications:

• They don't need to be changed according to the target datasets. Through experiments, we find that setting IR=75% and AEE=40 generalizes well across various datasets. In all experiments reported in the paper, we use these settings by default except for changing AEE to 20 on CIFAR-10.
• Generally, 75% ~ 80% for IR and 20 ~ 40 for AEE are good settings as the performance doesn't change too much within these ranges. We show ablation results on the other two benchmarks in the tables below:

CIFAR-100, IPC10 (notion: initial ratio = IR, addition end epoch = AEE)

Tiny ImageNet, IPC1

W2: PAD's improvement may not justify the added complexity

Thanks for the comment. We would like to clarify that PAD doesn't add complexity to DATM from the following three perspectives.

• Expert training: With our difficulty measurer and data scheduler, PAD filters out misaligned information for all IPCs in one set of expert trajectories. This means we only need to train expert trajectory once, instead of training experts according to IPCs like in DATM. We list the comparison of expert training costs in the table below:

• Trajectory matching: The matching loss is computed on fewer model parameters since we filter out part of model parameters that introduce misaligned information. In this case, the cost of distillation also doesn't increase. The comparison of running time of PAD with that of DATM is shown below. The distillation costs of these two methods are very close.

  IPC	DATM	PAD
  50	8.3 hrs.	8.1 hrs.
  1000	11.1 hrs.	11.2 hrs.

• Hyper-parameter tuning: PAD introduces new hyper-parameters such as AEE and IR. However, these parameters are not costly to tune as we explained above.

Q1: How is data selection applied in Figure 2 (b)? Are you using untrained agent models but applying data selection score (EL2N) from a differently pre-trained model? Are you replicating the score from the paper and apply it in all your methods or are you recomputing it based on different distillation architectures?

Thanks for the questions. We would like to elaborate the process as follows:

1. Before the distillation begins, we use an untrained ResNet-18 to compute EL2N scores (this is the default setting of EL2N) to evaluate the difficulty of the data sample.
2. Then, after we filter out misaligned data points according to their scores, only reserved difficulty-matched data are used for the distribution matching.

In other words,

• We didn't modify the workflow of DM except now DM can only select samples from the pruned dataset during the distillation.
• For the EL2N score, we replicate it from the paper and use it for all our methods instead of recomputing it on different architectures.
• The implementation of computing EL2N is based on DeepCore.

Q2: Can the same analysis be applied to more recent SOTA methods such as $SRe^2L$

Thanks for the question. $SRe^2L$[8] is an excellent work on DD that proposes a "squeeze, recover, label" process to decouple previous bi-level optimization. It achieves great success on large-resolution benchmarks such as ImageNet-1k. To show PAD's compatibility, we apply PAD in the squeeze stage of SRe2L. Please see the experiment results and analysis below:

• Setting: We carry out data selection to filter information extraction in the squeeze stage. For simplicity, experiments are done on CIFAR-100. All parameters are the same as the original paper. The model used is ResNet-18.

• Results: Please see the table below

  IPC	SRe2L	SRe2L + PAD(FIEX)
  1	25.4	26.7 ($\uparrow$ 1.3)
  10	28.2	29.3 ($\uparrow$ 1.1)
  100	57.2	57.9 ($\uparrow$ 0.7)

• Analysis: After applying PAD to filter out misaligned information extracted in the squeeze stage, the performance of $SRe^2L$ improves on both small and large IPC settings. This further validates our hypothesis that filtering misaligned information is effective.

• Conclusion: Filtering misaligned information can improve $SRe^2L$. It further supports that PAD is beneficial to matching-based methods for performance improvement.

We will add the discussion of $SRe^2L$ in the revision as follows:

• section 6, paragraph 5, "$SRe^2L$ introduces a "squeeze, recover, relabel" procedure that decouples previous bi-level optimization and achieves success on high-resolution settings with lower computational costs." (short version)

We will add the comparison with $SRe^2L$ in the revision as follows:

• section 5.5, a new section to discuss the compatibility of PAD on other DC/DM-based methods, add "We implement the Filtering Information Extraction (FIEX) on $SRe^2L$ which involves a component similar to DM. As shown in Table 5, compared with the baseline, filtering misaligned information brings remarkable improvements on both small and large IPC settings." (short version)

Dear Reviewer UeUP,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarized our responses here.

Reviewer UeUP:

$**New hyper-parameters introduced**$ :

Response:

• We explain that these two hyper-parameters don't need to be changed according to different datasets, so they are not hard to tune.
• We provide experimental results on other benchmarks to show that AEE=20~40 and IR=75%~80% are good settings and can be generalized well.

$**PAD adds more complexity**$ :

Response:

• We explain why PAD doesn't add any complexity from trajectory training, distillation, and hyper-parameter tuning perspectives.
• We provide overhead comparisons with DATM in trajectory training and distillation to further demonstrate that PAD is efficient.

$**Data selection in Figure 2(b)**$ :

Response:

• We provide detailed steps on how to apply data selection in DM.
• The data selection is done before the distillation starts. Misaligned samples are pruned and won't be used later.
• We replicate the EL2N score in the paper and use default settings to compute the score.
• The implementation is based on Deepcore.

$**Discussion of SRe2L**$ :

Response:

• We summarize the work of SRe2L and discuss the differences between PAD and SRe2L.
• We apply PAD on SRe2L and provide experimental results to show that PAD can bring remarkable improvements to SRe2L.
• We promise to include a discussion of SRe2L and the experiment in the revision.

Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best wishes,

Authors

We sincerely thank the reviewer emvM for valuable feedbacks. We make responses as follows.

W1: Experiments to support the two strategies are not sufficient

Thanks for the comment. We provide more results as follows (will be included in the revision)

• Settings:

  Dataset: CIFAR-10

  Arch: ConvNetD3.

  IPC 1/10: remove hard samples.

  IPC 500: remove easy samples.

  Other parameters are the same.

  Init: real images.

• Results: Pls ref to Author Rebuttal Table 1~6.

• Analyses:

  • FIEX:

    • Small IPCs:

      DC/DM: the performance consistently exceeds the baseline within 50% of removal.

      MTT: removing up to 30% hard samples helps improve the performance.

    • Large IPCs: removing up to 20% easy samples improves the distillation performance in large IPC settings.

    • Percentage Range: The average percentage of data removed that exceeds baseline is nearly 40%, indicating misaligned information widely exists during the information extraction stage.

  • FIEM:

    • Both small and large IPCs benefit from discarding shallow-layer parameters.
    • Discarding smaller ratios of shallow-layer parameters is more effective on small IPCs.
    • Discarding larger ratios of shallow-layer parameters is more effective on large IPCs.

• Conclusion: Two strategies are effective in improving performances.

CIFAR-100 is still running. Will report when finished.

W2: The comparison of discarding deeper-layer parameters is missing.

Thanks for the comment. Here are results of discarding deep-layer parameters (will be included in the revision):

• Settings:

  Dataset: CIFAR-10

  Arch: ConvNetD3

  Discard ratios: 25%, 50%, 75%.

• Results: Please refer to Author Rebuttal Table 8

• Analysis: Discarding deep-layer parameters significantly reduces the distillation performance. This indicates that useful information is mainly distilled by deep-layer parameters.

• Conclusion: Deep-layer parameters are more important than shallow-layer parameters.

W3: Discussion of other difficulty-aware DD

Thanks for the advice. We provide thorough analysis as follows (will be put into the revision, pls ref to Author Rebuttal Revision for more details).

Prune-then-Distill[5] propose to prune the target dataset with data selection methods before the distillation. For each IPC, only data with high EL2N score are then used for distillation since they are recognized being important.

• Differences:
  • We use EL2N score to evaluate the difficulty of training examples, to filter out misaligned information. In practice, we propose to reserve samples with high EL2N scores in large IPC settings but reserving samples with low EL2N scores in small IPC settings, which turned out to be effective to all matching-based distillation methods.

BLiP[4] proposes part of data samples in the dataset are redundant. It designs a data utility indicator to evaluate if samples are useful for the distillation given an IPC setting, then samples with low utility are pruned.

• Differences:
  • We find a few samples of target dataset are actually harmful for the distillation since they will provide over-hard or over-easy information. So we filter samples according to their difficulty measured by the EL2N score.
  • The data scheduler allows our method to only buffer expert trajectories once for a dataset, which can be utilized by arbitrary IPC settings.

RDED[6] proposes to extract image patches directly from the real dataset and rearrange key ones according to their scores, instead of synthesizing new images.

• Differences:
  • Our method focus on addressing the information misalignment problem for matching-based distillation methods, which will generate new synthetic samples during the distillation.

DATM[7] is the first to find that the difficulty of information embedded into the synthetic data should be aligned with the IPC setting.

• Differences:
  • DATM only aligns the information by controlling the matching range of training trajectories, which is only effective for methods based on matching trajectory.
  • We find out misaligned information exists in the Information Extraction stage and Information Embedding stage. By filtering out misaligned information, our method outperforms DATM on every benchmark.

Q1: Experiments to support Line 125-127

Thanks for the comment. We report experimental verification as follows(will be included in the revision):

• Setting:

  Dataset: CIFAR-10

  IPC: 500

• Results: Please refer to Author Rebuttal Table 9

• Analysis: Remove easy samples in one operation performs better. This supports our conclusion that after being trained on the full dataset for some epochs, letting the model focus on learning hard information is more effective for the distillation in large IPC cases.

Q2: Can combine PAD with DATM?

Yes, it is. As DATM is the SOTA trajectory matching method, to verify the effectiveness of PAD, our implementation is based on DATM. As shown in Table 1, PAD outperforms DATM in every setting.

Moreover, PAD can also be combined with other matching-based algorithms such as DC and DM (please refer to Author Rebuttal Table 1~6).

Q3: More ablation studies in Table 3(b)

Thanks for your advice. We add ablation on CIFAR-100 in the revision. Here are the results:

• Settings:

  • Dataset: CIFAR-100
  • IPC: 50

• Results: Pls ref to Author Rebuttal Table 10

• Analyses: Two filtering modules both bring improvements. FIEM brings more improvements.

• Conclusion: Both modules are effective.

Tiny-ImageNet is still running. Will report when finishes.

Dear Reviewer emvM,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarized our responses here.

$**More explanations and experiments on Figure 2 and 3**$ :

Response:

• We offer results on more IPCs and test ratios as required in Table 1~6.
• New results further support our conclusion that misaligned information exists in matching-based DD, and our filtering strategy is effective.

$**Discussion of listed works**$ :

Response:

• We discuss all mentioned works in details, including Prune-then-Distill, BLiP, DATM, and RDED.
• We provide thorough analyses of the differences between PAD and these works.
• We promise to add discussions of these works in the revision.

$**Experimental results for Line 125-127**$ :

Response:

• We provide comparisons in Table 9 as required.
• Through experimental comparisons, our conclusion that directly removing easy samples during trajectory training is more effective has been further supported.

$**Combine PAD with DATM**$ :

Response:

• We answer the question that PAD and DATM can be combined and we built PAD upon DATM.
• In Table 1 of the submission, PAD consistently improves DATM in all settings, further supporting our conclusion.

$**More ablation studies**$ :

Response:

• We provide more ablation experiments on CIFAR-100 in Table 10 as required.
• The results match our conclusion: both filtering information extraction and filtering information embedding improve the performance.

Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards,

Authors

We thank the reviewer xUdN for the feedback. We make responses as follows.

W1: Performance gains are limited and may have other explanations

Thanks for the question. In Table 1, we achieve 11 SOTAs out of 12 settings. Moreover, PAD can be generalized to methods based on matching gradients (DC[1]) and distributions (DM[2]), further showing the effectiveness of filtering out misaligned information.

To clarify our differences between "PDD" and "BLiP", we make analyses and experimental verification as follows (will be put into the revision, please refer to Author Rebuttal Revision for more details):

• PDD[3] progressively synthesizes multiple sets of images, to capture the training dynamics of different stages. It achieves success in enhancing existing DD methods. Differences:
  • Data Perspective: PDD uses all samples of the target dataset. PAD filters out samples that contain misaligned information according to the IPC setting, and they won't be used during the entire distillation process.
  • Trajectory Perspective: PAD aligns the difficulty of information distilled by different model parameters with each IPC capacity and improves the distillation performance.
• BLiP[4] proposes that it is not necessary to use all samples of the target dataset to perform the distillation since they are redundant. It greatly improves the distillation efficiency by pruning a large percentage of redundant data. Differences:
  • Data Perspective: BLiP proposes a data utility indicator to evaluate if samples are 'useful' given an IPC setting, then samples with low utility are pruned. We find that a few samples of the target dataset are actually harmful for the distillation since they will provide over-hard or over-easy information. So, PAD filters data samples according to difficulty measured by EL2N scores.
  • Trajectory Preparation Perspective: For trajectory-matching-based methods, BLiP needs to train expert trajectories case by case due to different pruning ratios. Our data scheduler allows our method to only buffer expert trajectories once, which can be utilized by arbitrary IPC settings.

Experimental Comparison with BLiP:

• Settings: For a fair comparison, we only compare the improvement brought by our data selection module (FIEX) with BLiP. Experiments are done on CIFAR-10 and we use MTT[5] as the distillation method. The data pruning ratios and IPCs tested are the same as BLiP.

• Results: Pls ref to Author Rebuttal Table 7

• Analyses:

  • PAD brings better performance improvements on IPC1/10/50.
  • Under the given data-dropping ratios, PAD's improvements over BLiP get larger as the IPC increases.

• Conclusion: Difficulty misalignment between IPC and real data used is more harmful. PAD's data selection module is more effective in removing such misaligned information.

W2: Figure 2 needs to be explained

Thanks for the comment. In Figure 2, we show a group of ablation results to demonstrate that removing misaligned information from the information extraction step helps improve the performance. We make additional clarifications as follows:

• The percentage of easy(hard) samples to remove is not always the higher the better for large(small) IPCs. When the removing ratio crosses a certain value, we may lose too much information to achieve a good performance. For example, the performance of trajectory matching on IPC500 is below baseline when removing more than 20% of easy samples. This explains why some points are below the baseline.
• We find the percentage of easy(hard) samples that are not aligned with large(small) IPCs resides in a large range (around 40%, please refer to Tables 1,2,3 in the Author Rebuttal). This means misaligned information widely exists in previous DD and our proposed solution is effective.

W3: Ablation under more IPCs and reduction ratios in Figure 2.

Thanks for the advice. More ablation studies are reported as follows:

• Settings:

  Dataset: CIFAR-10

  Architecture: ConvNetD3. IPC 1/10: remove various ratios of hard samples. IPC 500: remove various ratios of easy samples.

  Initialization: real images.

  Other parameters: same as the default.

• Results: Please refer to Author Rebuttal Table 1~3.

• Analyses:

  • Small IPCs: Removing part of hard samples helps improve the distillation performance.

  • Large IPCs: Removing part of easy samples improves the distillation performance.

  • Percentage Range: The average percentage of data removed that gives better performance than the baseline is nearly 40%, indicating that misaligned information is a common issue for existing DD, and our filtering strategy is effective.

• Conclusion: Misaligned information indeed exists and filtering it out can alleviate the negative effect.

Q1: Results of DC and DM in Figure 2, 3 don't match Table 1

We are sorry for the confusion. We elaborate on the reasons as follows:

• In the DC and DM, there are two ways of synthetic image initialization, random gaussian noise and images from the real dataset. In the current version, we initialize the synthetic data with random noise.
• We used smaller inner loop and outer loop steps in previous experiments for convenience.

Thanks for the reminder. To avoid confusion, we adjust our experiments with real image initialization and official hyper-parameter settings. New results are reported as follows:

• Settings:

  Init: real images

  Other parameters: same as the default.

• Results: Please refer to Author Rebuttal Table 1~6.

• Analysis: As can be observed, the trend still matches our previous experiments. Removing difficulty misaligned information distilled by shallow-layer model parameters according to IPCs achieves better performances.

• Conclusion: Misaligned information exists in the matching-based methods. Filtering it out brings improvements.

Dear Reviewer xUdN,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here.

$**Other explanations to performance gains**$ :

Response:

• PAD achieves SOTAs on 11 out of 12 settings.
• We provide a detailed discussion of the two mentioned works, BLiP and PDD.
• We also provide experimental comparisons between PAD and BLiP in Table 7: PAD is more effective than BLiP on MTT.

$**More explanations and experiments on Figure 2**$ :

Response:

• We offer more clarifications on the trend of Figure 2 and explain why some points could be below the baseline.
• According to the requirement, we add more IPCs and test ratios and listed all results in Author Rebuttal Table 1~3.
• Results still support our conclusion that misaligned information exists in existing matching-based DD, and removing it improves the performance.

$**Figure 2,3 results and Table 1 don't match**$ :

Response:

• We explain reasons why these results don't match: we used noise for synthetic image initialization and we reduced the inner and outer loop steps.
• We rerun all experiments with the default hyper-parameter setting and fix the results in Figure 2,3. Updated results are shown in Author Rebuttal Table 1~6. The trend still matches our previous results.
• We will update all results in the revision.

Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards,

Authors