Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation

Thank you for recognizing the significant performance gains of AT‑BPTT. Below we will address specific questions.

W1/Q1: Clarity and concerns about the link

Thank you for pointing out the need to clarify the definitions and the link between [1] and our method.

We appreciate that your interpretation is consistent with our formal definitions of stages and phases. For clarity, we will provide the descriptions below to explicitly define three stages and two phases, and these will be provided in the main text:

• Outer-loop Training Stages: We use Early, Middle, and Late to describe the global stages of the entire dataset distillation process. These stages are counted by epochs.
• Inner-loop Learning Phases: We use 'Preliminary' and 'Post' to describe sub-phases within the unrolled learning trajectory. These phases are counted by timesteps.

Additionally, we are sure that there is a profound inherent link between the paper [1] and our method. The essence of our three-stage division lies in focusing on the learning priorities of the model at different stages. This aligns exactly with the conclusion in [1]: early stages prioritize simple patterns and later stages refine complex features. Based on Reviewer aCEq's Q1, we can confidently state that our method not only has a strong formal link with paper [1], but also extends its theoretical framework to dataset distillation.

W2/Q2: Hyperparameter fine-tuning

Thank you and Review zMmc for your suggestion regarding hyperparameter fine-tuning. Based on your comments and Reviewer zMmc's W2/W3, we offer the following comprehensive reply.

We agree that clarifying the hyperparameter sensitivity, computational overhead, and practical strategies for setting these parameters is crucial for reproducibility and real-world deployment. Therefore, we provide clear guidelines to minimize tuning effort. Our analysis divides these hyperparameters into three categories based on dataset sensitivity, and we discuss empirical basis, practical strategy and computational cost.

Sensitivity Category

• Low sensitivity: Fixed defaults suffice across datasets due to universal gradient dynamics
• Medium sensitivity: Adjusted via simple scaling rules, minimizing need for per-dataset tuning
• High sensitivity: Only M and N require dataset-specific tuning

Empirical Basis and Practical Strategy

Computational Cost of Hyperparameter Tuning

The computational cost for tuning AT-BPTT hyperparameters is low, as most parameters are low-sensitivity or medium-sensitivity. As shown in table above, the costs of low-sensitivity or medium-sensitivity hyperparameters are negligible. The hyperparameter tuning is thus focused on M and N. And this process for other datasets is similar to that of ablation study in Figure 5. The computational costs on Nvidia A800 are shown in table below:

If the paper is accepted, we will provide the analysis above in the camera-ready version.

W3/Q4: Ablation study of PSP (or LRHA?)

Thank you for your comments on the Patch-wise Semantic Preservation (PSP) module. Our intention is not to design PSP as an inseparable component of AT-BPTT, but rather as a plug-and-play paradigm solving the challenges when applied to high-resolution data. To clearly differentiate the individual impact of three components, we have already conducted a detailed ablation study (lines 283-294), with the results presented in Table 2. In fact, the results of Tiny-ImageNet and ImageNet-1K are included in Table 2. Additionally, the results of RaT-BPTT with and without PSP are also included in the second and fifth lines in Table 2, since RaT-BPTT is our backbone.

However, based on the questions you've raised, particularly "the authors should report the accuracies with and without patch-wise semantic preservation", we infer that there might be a typo in your comments. Could you please clarify whether you are referring to the Low-rank Hessian approximation (LRHA) instead of PSP in W3/Q4?

To fully address your concerns, we have further conduct relevant experiments for LRHA that might meet your requirements. Since we have conducted its ablation study for low-resolution datasets in Appendix C.2, the results shown below are the extention of Table 5.

Table above demonstrates that LRHA can significantly reduce memory and training time while only slightly reducing accuracy. If the paper is accepted, we will provide the ablation experiments above in the camera-ready version.

Q3: Relationship and comparison with Teddy

Thank you for your suggestion to add details about Teddy [2] method. Teddy is an interesting DD method, and both it and our method have their own strengths. We will elaborate on the relationship and comparison between Teddy and our work.

Relationship:

Both methods have made improvements to reduce computational resource consumption in the meta-learning optimization objective. For the second-order derivative in the meta-learning objective, Teddy converts it to a first-order derivative using a Taylor approximation, while AT-BPTT approximates the Hessian matrix generated in the second-order derivative using the LRHA module. The methodologies of the two are complementary. Teddy uses Taylor approximation and model pooling to optimize global distillation, making it suitable for distillation with large-scale data; while AT-BPTT uses dynamic truncation to focus on fine-grained control of training dynamics, making it suitable for various inner-loop optimization scenarios.

Comparison:

Since Teddy is not open-sourced, we use the results of our reproduction for Teddy. We compare Teddy with our AT-BPTT in three aspects: test accuracy, cross-architecture, and high-IPC settings.

• Test accuracy: As shown in Table 1, although our method is lower than Teddy on high-resolution datasets, it significantly outperforms Teddy on various low-resolution datasets.

• Cross-architecture: Under similar settings in Appendix C.4, we conduct cross-architecture evaluations for Teddy and ours. Table below shows that AT-BPTT consistently outperforms Teddy across four tested architectures.

• High-IPC settings: Under similar settings in Appendix C.5, we conduct evaluations in high-IPC settings for Teddy and ours. Table below shows that AT-BPTT consistently outperforms Teddy across various high-IPC cases.

W4: Formal issues

Thank you for your suggestion regarding the format. Hypotheses (1) and (2) will be more appropriately expressed as research questions; we will fix "optimal" to "better" in the main text.

References:

[1] A closer look at memorization in deep networks. ICML 2017.

[2] Teddy: Efficient large-scale dataset distillation via taylor-approximated matching. ECCV 2024.

Thank you for highlighting our solid technical contributions, integrated innovations, extensive validation, and scalable efficiency across diverse datasets. Below we will address specific questions.

W1: Justification and ablation of three stages

Thank you for your suggestion regarding spliting the learning process into three stages. This is a question that interests Reviewers aCEq, zMmc and B86A. We address it with a Justification and Ablation Study:

Justification:

Based on the theory in paper [1] that is cited as [1] in the main text (early stages prioritize simple patterns and later stages refine complex features), we initially divided the stages into two parts (early and late). However, such improvements in results are limited, as shown in the the table below. This inspires us to explore a more refined staged strategy. Additionally, according to paper [2] that is cited as [9] in the main text, overfocusing the truncted window on a certain stage is not conducive to global learning. Therefore, it is crucial to retain a period of random windowing to cover the entire trajectory. Together, a three-stage strategy is needed. According to the analysis of Figures 1 and 3 in the main text, this extra stage is defined as the middle stage, and finally boosts performance.

In addition, why not more stages, such as four stages? First, each extra stage adds two new hyperparameters in the stage transition, which complicates tuning and deployment; Moreover, according to the analysis in Figure 1 in the main text, middle-stage performance demonstrates negligible sensitivity to truncation position (lines 46). A finer partitioning thus produces negligible change. Therefore, we ultimately chose to use a three-stage strategy.

Ablation Study:

We conduct ablation experiments on three variants to verify the effectiveness of three-stage stragety. Using CIFAR-10 with IPC=10 (600 epochs) as an example, the setups are as follows:

• Two-stage variant: The training is split into two stages of 300 epochs each. In the first stage, truncations position follow $P_{trunc}(t)$ for early stage in Equation 6. In the second stage, truncation positions follow $P_{trunc}(t)$ for late stage in Equation 6.
• Three-stage variant: The training is split into three stages of 200 epochs each. The first and third stages are identical to the first and second stages of the two-stage variant, respectively. The second stage employs random truncation.
• Four-stage variant: The training is split into four stages of 150 epochs each. The strategies for the first and third stages align with the first and second stages of the two-stage variant, while the second and fourth stages employ random truncation.

The experimental results are shown in the table below:

Table above demonstrates that the three-stage variant consistently outperforms both other two variants.

In conclusion, both Justification and Ablation Study support our choice of a three-stage design as an effective and balanced truncation strategy.

W2/W3: Hyperparameter fine-tuning

Thank you and Review B86A for your suggestions regarding hyperparameter fine-tuning. Based on your comments and Reviewer B86A's W2/Q2, we provide following comprehensive reply.

We agree that clarifying the hyperparameter sensitivity, computational overhead, and practical strategies for setting these parameters is crucial for reproducibility and real-world deployment. Therefore, we provide clear guidelines to minimize tuning effort. Our analysis divides these hyperparameters into three categories based on dataset sensitivity, and we discuss empirical basis, practical strategy and computational cost.

Sensitivity Category

• Low sensitivity: Fixed defaults suffice across datasets due to universal gradient dynamics
• Medium sensitivity: Adjusted via simple scaling rules, minimizing need for per-dataset tuning
• High sensitivity: Only M and N require dataset-specific tuning

Empirical Basis and Practical Strategy

Computational Cost of Hyperparameter Tuning

The computational cost for tuning AT-BPTT hyperparameters is low, as most parameters are low-sensitivity or medium-sensitivity. As shown in table above, the costs of low-sensitivity or medium-sensitivity hyperparameters are negligible. The hyperparameter tuning is thus focused on M and N. And this process for other datasets is similar to that of ablation study in Figure 5. The computational costs on Nvidia A800 are shown in table below:

If the paper is accepted, we will provide the analysis above in the camera-ready version.

W4: Comparison with generative DD methods

Thank you for your suggestion regarding the comparison with SOTA generative DD methods. We provide the experimental results in the table below:

Through comprehensive comparison, we get several conclusions:

• The results on ImageNet-1K of Minimax and IGD are both 44.3 and 46.2, which are higher than our results.
• These two generative methods can not generalize across datasets with various resolutions like CIFAR-10, CIFAR-100, and Tiny-ImageNet, while our AT-BPTT achieves comparable results across these datasets, especially in low-resolution datasets.
• These two generative methods necessitate the incorporation of diffusion models for distillation, thereby incurring substantial GPU memory consumption and additional computational overhead. In contrast, our method with a LRHA module achieves a $3.9 \times$ training speedup and 63% memory savings compared to the baseline.
• As shown in Table 8 in Appendix C.5, our AT-BPTT extends the scale of synthetic dataset to high-IPC cases and achieves expressive performance.

If the paper is accepted, we will add the new comparison table to the camera-ready version.

W5: Application of downstream tasks

Thank you for highlighting the importance of downstream evaluations. As stated in Appendix F Discussion, we outline several downstream tasks as our future work, like neural architecture search and federated learning. To initially verify the effectiveness of our method for downstream tasks, we have extended language tasks in Appendix C.8 and pretrained model fine-tuning in Appendix C.9. For instance, the results shown in Figure 7 demonstrate that pre-training on data distilled by AT-BPTT significantly outperforms random selection. These experiments demonstrate that our method preserves critical representations that are transferable to complex downstream tasks beyond simple classification. Additionally, we further discuss AT‑BPTT’s theoretical benefits for diverse applications in Appendix F Discussion. Therefore, we will continue to explore AT-BPTT to other downstream tasks in future work.

References:

[1] A closer look at memorization in deep networks. ICML 2017.

[2] Embarrassingly simple dataset distillation. ICLR 2024.

[3] Efficient Dataset Distillation via Minimax Diffusion. CVPR 2024.

[4] Influence-Guided Diffusion for Dataset Distillation. ICLR 2025.

Thank you for appreciating our clear motivation, detailed presentation, and the significant accuracy and efficiency gains of our method. Below we will address specific questions.

W1: Lack of justification to the algorithm design

Thank you for recognizing our strong motivation and gains. Based on Q1, we infer that your concerns come from the three-stage strategy, which we will address in the next section.

Q1: Justification and ablation of three stages

Thank you for your suggestion regarding spliting the learning process into three stages. This is a question that interests Reviewers aCEq, zMmc and B86A. We address it with a Justification and Ablation Study:

Justification:

Based on the theory in paper [1] that is cited as [1] in the main text (early stages prioritize simple patterns and later stages refine complex features), we initially divided the stages into two parts (early and late). However, such improvements in results are limited, as shown in the the table below. This inspires us to explore a more refined staged strategy. Additionally, according to paper [2] that is cited as [9] in the main text, overfocusing the truncted window on a certain stage is not conducive to global learning. Therefore, it is crucial to retain a period of random windowing to cover the entire trajectory. Together, a three-stage strategy is needed. According to the analysis of Figures 1 and 3 in the main text, this extra stage is defined as the middle stage, and finally boosts performance.

In addition, why not more stages, such as four stages? First, each extra stage adds two new hyperparameters in the stage transition, which complicates tuning and deployment; Moreover, according to the analysis in Figure 1 in the main text, middle-stage performance demonstrates negligible sensitivity to truncation position (lines 46). A finer partitioning thus produces negligible change. Therefore, we ultimately chose to use a three-stage strategy.

Ablation Study:

We conduct ablation experiments on three variants to verify the effectiveness of three-stage stragety. Using CIFAR-10 with IPC=10 (600 epochs) as an example, the setups are as follows:

• Two-stage variant: The training is split into two stages of 300 epochs each. In the first stage, truncations position follow $P_{trunc}(t)$ for early stage in Equation 6. In the second stage, truncation positions follow $P_{trunc}(t)$ for late stage in Equation 6.
• Three-stage variant: The training is split into three stages of 200 epochs each. The first and third stages are identical to the first and second stages of the two-stage variant, respectively. The second stage employs random truncation.
• Four-stage variant: The training is split into four stages of 150 epochs each. The strategies for the first and third stages align with the first and second stages of the two-stage variant, while the second and fourth stages employ random truncation.

The experimental results are shown in the table below:

Table above demonstrates that the three-stage variant consistently outperforms both other two variants.

In conclusion, both Justification and Ablation Study support our choice of a three-stage design as an effective and balanced truncation strategy.

Q2: Lengths of the three stages

Table below reports the average number of epochs in each stage for IPC = 10 across our four benchmarks.

As detailed in Threshold-guided Stage Transition mechanism (lines 195-207), the transitions between stages are triggered automatically by the model's performance dynamics during training. Therefore, each stage’s duration naturally adapts to the dataset’s learning dynamics under a given IPC.

Q3: Impact of LRHA

We have evaluated the impact of our LRHA in Appendix C.2 (lines 538-544). As shown in Table 5, we test the experimental results before and after the introduction of LRHA, using CIFAR-10 IPC=10 as an example. After the introduction of LRHA, the accuracy is reduced from 72.7% to 72.4%, with a negligible drop of just 0.3 %. In contrast, LRHA cuts peak memory usage by 63.3 % and speeds up training by $3.9 \times$. These results demonstrate that LRHA introduces minimal approximation error while delivering substantial computational savings. Additionally, the experimental results reported in the main text are the results after applying LRHA.

Q4: Performance of PSP

We appreciate your interest in the behavior shown in Table 2. In fact, PSP performs better on high-resolution datasets, while PSP has only negligible improvements on low-resolution datasets. This discrepancy stems from low-resolution patches containing too little semantic content for PSP to exploit.

For low-resolution images (e.g., CIFAR-10, $32 \times 32$), PSP results in each patch being only $8 \times 8$ pixels in size. An $8 \times 8$ pixel block contains limited semantic information, making it difficult for the model to learn meaningful representations. Therefore, for datasets like CIFAR-10 or CIFAR-100, distilling the complete $32 \times 32$ image directly is a much more effective strategy. For high-resolution images (ImageNet-1K, $224 \times 224$), following the same $4 \times 4$ division rule, a $224 \times 224$ image is segmented into 4 patches, each of size $56 \times 56$ pixels. A $56 \times 56$ patch is large enough to preserve rich local structure critical for distillation. Therefore, PSP is a specialized component designed to tackle the challenges of distilling high-resolution images.

Paper Formatting Concerns:

Thank you for your suggestion on paper formatting. We will fix the $P_{trunc}$ in Equation 6 to $P*_{trunc}$.

References:

[1] A closer look at memorization in deep networks. ICML 2017.

[2] Embarrassingly simple dataset distillation. ICLR 2024.

Thank you for acknowledging our novel dynamic adjustment, resource‑saving Hessian and semantic preservation contributions, SOTA results across benchmarks, and comprehensive evaluations with available code. Below we will address specific questions.

W1/W2/W3: Presentation of algorithm, missing citation and typo

Thank you for your suggestions regarding the presentation of algorithm, missing citation and typo. If the paper is accepted, we will adjust the layout to place the algorithm in the main text; We will clearly indicate the positions of the inner and outer loops using comments; We will add the BPTT reference; We will correct the typo of "Segmennt" to "Segment".

Q1: Data initialization

We have reported the data initialization procedure in Appendix B (lines 507-509). Specifically, We draw synthetic images from a standard Gaussian distribution and assign labels so that each class has an equal number of samples. Following this random generation, we apply a norm-normalization step to unit length. This initialization creates balanced, model‑agnostic starting data that is comparable across experiments.

Q2: Performance of high-IPC setting

We appreciate your interest in the high‑IPC settings on ImageNet-1K. We have completed this experiment during the rebuttal period. With IPC=50, AT-BPTT achieves a test accuracy of 46.5% on ImageNet-1K. This accuracy is substantially higher than the 30.6% at the IPC=10 setting. It provides strong evidence that our method's performance scales robustly with an increasing IPC, rather than being effective only at low-IPC settings. Additionally, we have reported the results on other three datasets (CIFAR-10/100, Tiny-ImageNet) with high-IPC settings in Appendix C.5 and Table 8.

Q3: Data visualization

While the rebuttal system does not support displaying figures, we will include a detailed ImageNet-1K data visualization in the final version, including:

• We will present a side‑by‑side grid of $5 \times 2$ distilled samples alongside Teddy [1] and other high‑resolution baselines.
• Compared to low‑resolution distilled datasets, our ImageNet‑1K distillates exhibit sharper, more recognizable object outlines.
• The backgrounds become washed‑out or texture‑like, since the model integrates more class features.

Q4: Performance on ResNets

We have conducted experiments using the ResNet-18 architecture in Appendix C.4 and C.5.

• In Appendix C.4 (lines 557-564), we have evaluated the distilled CIFAR-10 dataset with IPC=10 and 50 across four distinct architectures, including ResNet-18. As shown in Table 7, the performance of our method on ResNet-18 is lower than on ConvNet, which is the same as the judgement you mentioned. In this case, our AT-BPTT still outperforms all baselines, achieving the current SOTA accuracy of 75.6% under the IPC=50 setting. This provides strong evidence that AT-BPTT has excellent cross-architecture generalization.

• In Appendix C.5 (lines 565-578), we have further conducted experiments with high-IPC settings, consistently using ResNet-18. The results in Table 8 demonstrate that AT-BPTT comprehensively surpasses all baselines, across CIFAR-10/100 and Tiny-ImageNet.

In conclusion, these findings confirm AT‑BPTT’s robust cross‑architecture generalization, even on ResNets.

References:

[1] Teddy: Efficient large-scale dataset distillation via taylor-approximated matching. ECCV 2024.