Towards Mitigating Architecture Overfitting in Dataset Distillation

Dear reviewer w1VW,

Thanks for your constructive suggestions and insightful comments. In response to your questions, we offer the following point-to-point answers.

For the problems mentioned in Weaknesses:

1. >>> The results are impressive, however, the method contributions seem relatively marginal. Though the adapted method absorbed from previous works is non-trivial, the discussion lacks enough insight but is empirical.

   Reply: Although our design is motivated by some existing techniques such as DropPath and knowledge distillation, we adapt them in the context of dataset distillation. For DropPath, we make it applicable to single-branch networks and propose improved shortcut connection and three-phase keep rate to boost the performance. For knowledge distillation, we pioneer the use of small networks as teacher models, which is different from current common practices. In the experiments, our method significantly mitigates the architecture overfitting in dataset distillation and improves the generalization of large networks on small datasets. Section 3 comprehensively demonstrates the motivation of our proposed methods. More fundamental analysis of DropPath and knowledge distillation were well analyzed in many works [1 - 3].

2. >>> The three-phase Keep Rate looks quite complex for tuning. How is the tuning complexity and robustness if we use the proposed method for many different models?

   Reply: In the ablation study, we evaluate the performance under different hyperparameters (HPs). Moreover, additional ablation studies on HPs of three-phase keep rate were added. The result is reported below and in Figure 4 (b) and (c). Compared to the significant performance improvement brought by our method, if we choose different HPs in an appropriate but wide range, the performance fluctuation caused by different HPs is negligible. For example, when the final keep rate ranges from 0.6 to 1.0, KD weight ranges from 0.3 to 0.9 and KD temperature ranges from 1 to 10, the performance fluctuation is with 1 percentage point. In addition, although lower minimum keep rates and longer periods of decay induce better performance, they also mean a longer training phase. We set them to 0.5 and 500 to balance the efficiency and performance, respectively. Despite that, the performance fluctuation caused by these two HPs is still insignificant. Therefore, we can demonstrate that our training method is robust to different HPs. In addition, the selected HPs also perform well in different settings, including different distilled data, different IPCs and different test networks.

3. >>> Though the bag of methods works well, the whole paper gives the readers a feeling of separation.

   Reply: The main contribution of our work is proposing a bag of methods to mitigate architecture overfitting in dataset distillation. These methods perform like regularization from different perspectives, such as network architecture (i.e. DropPath with three-phase keep rate), and training scheme (i.e. knowledge distillation, data augmentation and optimizer). Instead of directly using these methods, we adapt them to our scenario to further improve the performance. In the experiments, we demonstrate that our method not only mitigates architecture overfitting in dataset distillation but also improves the generalization of large models on small datasets. We believe our work provides practitioners with a toolbox and some intuition to solve related problems.

3. >>> Table 2 provides lower results for MTT baselines. From the original paper, MTT reaches 65.3/71.6 accuracy on CIFAR-10, while in Table 2 the authors reported 63.6 and 70.2. I am curious why there is a performance gap since it may create invalid performance advantages for the proposed techniques.

   Reply: We carefully checked the hyperparameters of MTT and found the only difference is that the learning rate they adopted in the evaluation is a learnable parameter, which is learned during the distillation process. To ensure a fair comparison with other methods, we disabled this mechanism and used the initial learning rate they defined before distillation, i.e., 0.01. In addition, as pointed out in the original MTT paper [1], the test accuracies fluctuate a lot with different random seeds. Moreover, the accuracies obtained with our hyper-parameters in Table 2 of the manuscript are in the fluctuation range reported in the MTT paper at some runs, e.g., 64.8 / 71.5. Finally, we need to point out that if we compare MTT with the baselines as listed in [1], the fluctuation of the test accuracies does not challenge the performance superiority of MTT.

4. >>> Section 4.2 is interesting, but I think it is off-topic. The paper is trying to solve the so-called "architectural overfitting" but it will not happen when there is no actual "fitting" process. Therefore, I think Contribution 3 does not count and the contribution bullets should be adjusted.

   Reply: As pointed out in the answer to Question 1, architecture overfitting accounts for the additional generalization gap when we train models of different architectures on the distilled dataset, compared with training on small datasets. In section 4.2, we demonstrate that our methods contribute more performance improvement in the case of training on distilled dataset than in the case of training on sub-sampled dataset, so it is useful to mitigate architecture overfitting. In addition, section 4.2 validates the effectiveness of our methods in sub-sampled dataset, which indicates the versatility of our method.

Table 1: Comparison between our method and baselines. Note that p in DropPath and Dropout denotes the keep rate and alpha in MixUp and CutMix is the parameters $\alpha$ and $\beta$ in beta distribution, where $\alpha$ and $\beta$ are the same.

[1] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750–4759, 2022.

Dear reviewer wW36,

Thanks for your constructive suggestions and insightful comments. In response to your questions, we offer the following point-to-point answers:

1. >>> I am not sure whether it is really the "overfitting". Why this is an overfitting? Is there any evidence to show that the performance already saturates on one backbone, but decreases on the other backbones? It could be the case that the distillation process is even "underfit" on a given architecture - and the performance drop during "transferring" (i.e., distill with one backbone and then evaluate on the others) could be just some normal "generalization error". I would suggest to use the terminology more carefully to avoid such a confusion, or provide evidence to support the terminology.

   Reply: We are sorry about the confusion. From the perspective of evaluation, we argue that the architecture overfitting is consistent with the classical definition of overfitting: a model suffers from overfitting if the gap between its performance on the training set and the test set is big. In our cases, the training accuracy on the distilled dataset is always 100% for all network architectures, but the test accuracy on the real dataset is lower, so overfitting definitely occurs as we see generalization gaps. Moreover, the generalization gap is bigger when we train larger networks on the distilled dataset than when we use a small network with the same architecture as the training network. We call such difference architecture overfitting, meaning the additional overfitting arising from the architecture difference between the training network and the test network in dataset distillation. From the dataset perspective, we agree with the reviewer that the distilled dataset is "unfit" to the original large dataset given a network architecture. We believe that the distilled dataset loses some useful information from the original large dataset, leading to noticeable performance degradation when training models on the distilled dataset. Nevertheless, it is important to note that this paper primarily focuses on examining the network architecture perspective. In this context, we utilize the term architecture overfitting to describe the phenomenon under investigation.

2. >>> From what I am understanding, this work actually alters the evaluation process (the process of training a model on the distilled image set). Augmenting the training process is not novel at the high level as many previous works have already done so. A lot of augmentation to architectures (like DropOut and DropPath etc.) and data (MixUp, CutMix, AutoAugment, etc. ) could be applied here and I think that can serve as potential baselines.

   Reply: We have added more experiments in the revised manuscript in Table 5 of Appendix B.1, where we adopt DropOut, DropPath, MixUp and CutMix as baselines and evaluate their performance with the same setting in the ablation study (ResNet18, FRePO, CIFAR10, IPC=10). Note that AutoAugment only supports uint8 data, the pixel value can only be in {0, 1, ..., 255}. However, distilled data are float32 and their pixel values are neither bounded by [0, 1] nor [0, 255]. As a result, we do not include AutoAugment in the comparison. Similar to AutoAugment, we already adopted 2-fold data augmentation in our method, which also randomly samples 2 augmentations from a pool of augmentations. In addition, from the results shown in Table 2 (w/o DP & KD v.s. Full), where w/o DP & KD already uses better data augmentation and better optimizer, we can expect that pure data augmentation will not outperform our method. The results of the comparison with different baselines are reported in the table below. We can observe that simple DropPath and DropOut with a constant keep rate can even deteriorate the performance. In addition, MixUp and CutMix only lead to marginal performance improvement, which further demonstrates the effectiveness of our method.

For the first question I am partly agree with the authors. The overfitting does happen in the retraining (evaluation) phase. But I do not agree with how the authors divide the scope. When I first read this paper I had the first impression that the authors are trying to address the overfitting in the distillation phase. This leads me to confusion why architecture overfitting even happens. So either make it more clear or change slightly the writing. This is also the main reason hinders me from providing a higher score.

For the MTT part, could the authors provide the results with learnable learning rates? I think that is the standard "MTT" people are trying to compare with. Or mark it explicitly and explain why you adopt a fixed learning rate in the paper.

Scores have been adjusted. New experiments are helpful. Thank you.

Dear reviewer MbH7,

Thanks for your insightful and constructive comments. In response to your questions, we offer the following point-to-point answers:

1. >>> I do not think the cross-architecture generalization problem should be addressed this way. The authors do not modify the process of obtaining synthetic datasets during dataset distillation. Instead, they modify the training strategies given synthetic datasets. In dataset distillation, we should definitely focus on the former and should not make any assumptions on how users should use provided synthetic datasets for downstream training. What I want to see is actually a thorough algorithm for the process of dataset distillation that can improve the cross-architecture performance following the original evaluation protocols. The current form does not really enhance dataset distillation. The improvement is from the strategies of using distilled datasets.

   Reply: We agree with the reviewer that proposing novel dataset distillation methods may help mitigate architecture overfitting and is definitely a direction for exploration. However, we argue that it is also meaningful to improve the training strategies using the distilled dataset. Our argument is supported by the following points:

   • The overfitting issue studied in this work is not solely caused by dataset distillation because a similar overfitting phenomenon is also observed when we train models on tiny real datasets. For example, the results in Section 4.2 indicate overfitting happens when we use a tiny sub-sampled training set to train deep neural networks. Our proposed methods can also improve the performance in this context.

   • Following the first point, from the results in Section 4.1 and 4.2, we observe that training on distilled datasets has better performance than training on sub-sampled datasets. However, the performance gaps between the small and big networks are also larger when training on the distilled datasets. In this context, our proposed methods can induce more performance boosts, so it is more meaningful to apply our methods for distalled datasets, which is also the focus of this paper.

   • In addition, our methods are orthogonal to dataset distillation since our method focuses on how to use the distilled dataset, and dataset distillation focuses on how to obtain the distilled dataset. These two categories of methods can help improve each other. For example, by adopting our method, we can more comprehensively evaluate the performance on different distilled datasets and thus compare different dataset distillation methods fairer.

2. >>> If the paper focuses on how to use distilled datasets, the problem can be cast to some more classical problems, like how to avoid overfitting, synthetic-to-real generalization, few-shot learning, etc. The authors fail to make a broader discussion.

   Reply: As mentioned above, we argue that discussing our method in the context of dataset distillation is practical. In addition to the distilled dataset, in section 4.2, we also demonstrate the effectiveness of our method when we train deep neural networks on small but real sub-sampled training sets. Although we see bigger performance improvement in applying our method to distilled datasets, its effectiveness on sub-sampled datasets indicates its broader applications to avoid overfitting. However, we argue that the other two cases pointed out by the reviewer are different from our problem settings.

   1. Synthetic-to-real generalization: It measures the performance of the models that are trained on the synthetic data but tested on the real data. The goal of this task is to narrow down the gap between the performance on synthetic and real data and thus to make the models more robust to domain shifts [1]. That is to say, the key challenge in generalization lies in addressing the domain shifts rather than insufficient training data or architecture differences as in our work.

   2. Few-shot learning: it aims to adapt pre-trained models for new tasks from very few instances. The key to few-shot learning is how to construct a pre-train model on several related tasks with enough training data in the meta-training phase so that it can generalize well to unseen (but related) tasks with just a few examples during the meta-testing phase [2]. However, we focus on training a model from scratch on a small dataset, which is different from few-shot learning settings.

   In our revised manuscript, we add a brief discussion about these related tasks. Customizing our method for them is out of the scope of this paper. We leave them as future works.

Dear reviewer Mz7n,

Thank you for your constructive comments and suggestions, especially mentioning our work is unique in the context of dataset distillation and is significant in practice. For your questions, we offer our point-to-point responses as follows:

1. >>> I'm not sure I followed the reasoning behind the scaling of the DropPath output maps. It would be nice to have a derivation in the Appendix for how this scaling matches the expectations from training to test.

   Reply: Eq. (1) on page 4 shows that $\mathtt{DropPath}(\mathbf{x}) = \frac{m}{p}\cdot \mathbf{x},\quad m=\mathtt{Bernoulli}(p)$, where $p \in [0, 1]$ denotes the keep rate, $m = \mathtt{Bernoulli}(p) \in \{0,1\}$ outputs 1 with probability $p$ and 0 with probability $1-p$. We consider the module output $\mathbf{y} = \mathtt{DropPath}(\mathbf{x})$ in the training phase, then the expectation of $\mathbf{y}$ given $\\mathbf{x}$ is $\mathbb{E}(\mathbf{y}) = p \cdot \frac{1}{p} \cdot \mathbf{x} + (1 - p) \cdot \frac{0}{p} \cdot \mathbf{x} = \mathbf{x}$. In the test phase, DropPath is disabled, so the module output is simply $\mathbf{x}$ and consistent with the expectation in the training phase. If there is no scaling factor $1 / p$ in Eq. (1) and $p < 1$, the expectation of the module outputs in the training and test phases will be different, which leads to performance degradation.

2. >>> There is a statement that architecture overfitting occurs due to depth and not width. Has this been recognized in existing work?

   Reply: First, to clarify, we do not claim that the width has no impact on architecture overfitting. We use the architectures evaluated in existing works, including [1, 2], and observe that the architecture overfitting occurs and becomes more severe when the models get deeper (from 3-layer CNN to 50-layer ResNet). We do not discuss the impact of width in this context.

3. >>> In the Three-Phase Keep Rate section on page 4, it is said that the variance increases as p increases: isn't the variance maximal at p = 0.5?

   Reply: We are sorry about the confusion, and we agree with your insightful comment. For a Bernoulli distribution, the variance is indeed the largest when $p=0.5$. We have updated the corresponding statement (please check the text highlighted in blue in our revised manuscript). The key point of three-phase keep rate is to control the effective depth of the model by the value of $p$. In the early phase of training, the model is underfitting, stochastic architecture brings optimization challenges for training, so we turn off DropPath by setting the keep rate $p = 1$ in the first few epochs to ensure that the network learns meaningful representations. We then gradually decrease $p$ to decrease the effective depth of the model and thus to narrow down the performance gap between the deep sub-network and the shallow training network until the keep rate reaches the predefined minimum value after several epochs. In the final phase of training, we decrease the architecture stochasticity by increasing the value of $p$ to a higher value to ensure training convergence.

4. >>> There seem to be some hyper-parameters involved, such as the values shaping the shape of the three-phase cycle and the temperature in knowledge-distillation. How are these hyper-parameters tuned? The experiments suggest direct evaluation on test set performances.

   Reply: In Section 4.3, we conduct ablation study to evaluate the performance under different hyper-parameters (HPs) when using FRePo-generated distilled dataset, IPC = 10 and ResNet18 as the test network. More ablation studies on HPs of three-phase keep rate have been added in the revised manuscript. The results are reported below and in Figure 4 (b) and (c). We can conclude that the performance fluctuation by different HPs in an appropriate but wide range is negligible, compared with the significant performance improvement brought by our method. For example, when the final keep rate ranges from 0.6 to 1.0, KD weight ranges from 0.3 to 0.9 and KD temperature ranges from 1 to 10, the performance fluctuation is with 1 percentage point. In addition, although lower minimum keep rates and longer periods of decay induce better performance, they also mean a longer training phase. We set them to $0.5$ and $500$ to balance the efficiency and performance, respectively. Despite that, the performance fluctuation caused by these two HPs is still insignificant. In addition, our results in Section 4.1 demonstrate that the selected HPs by ablation studies also perform quite well under other different settings, including different distilled datasets, IPs and test networks. In summary, many experimental observations indicate that our training method is robust to different HPs.