Efficiency for Free: Ideal Data Are Transportable Representations

[W1] The way the paper is written it almost appears that authors are approaching the concepts of dataset distillation considerations (eg in Table 1) as if such matters are unexplored in literature, which I would sort of find surprising. I think the paper would have been better structured as leading with the ReLA-D model convention for data set distillation as a more prominent part of the writeup, including more detail as to how that relates to prior work.

[R1] Thank you for your valuable feedback. We will revise the manuscript to explicitly situate the ReLA-D model within the context of existing literature on dataset distillation. Additionally, we will enhance the discussion to clarify how our work builds upon and differentiates from prior research, as outlined in our [R3] below.

[Q1] I am partly fascinated by scope not addressed by the paper as to where these forms of distilled data sets might be of added benefit due to their less noisy representations.

[R2] Thank you for your feedback. We have conducted additional experiments and further explorations, as detailed in attached PDF and our response to Reviewer zbwg.

In the attached PDF, we have included additional experiments covering: (a) the higher-resolution CelebA-HQ dataset (1024 × 1024); (b) a continual learning task; (c) comparisons with more state-of-the-art baselines; (d) reports on computation time and peak GPU memory usage; (e) a segmentation task on the VOC 2012 dataset; and (f) the larger ImageNet-21K dataset.

[Q2] More of a hypothetical question, no need to answer: Are there any known applications where data augmentation is known to interfere with implementations? Perhaps that might be a starting point of where valuable sources of impact of this form of distillation could be considered.

[R3] To the best of our knowledge, no such applications currently exist.

[Q3] Can you talk further about the extent of novelty and related work for the ReLA-D tool?

[R4] Additional discussions include the following:

1. As detailed in Appendix K, the SOTA dataset distillation methods generally demand a higher computational budget compared to training on the full dataset. For instance, to distill ImageNet-1K, the SOTA efficient methods [a,b,c] require a model that is well-trained on ImageNet-1K. In contrast, our ReLA can distill a dataset using less than the budget required for training a single epoch (refer to our analysis in Appendix K). The technical solution is detailed in Section 4.
2. Our ReLA is capable of distilling large datasets for both self-supervised and supervised learning tasks, whereas conventional methods are typically limited to distilling small datasets or are exclusively applicable to supervised learning.

[a] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." CVPR 2024.

[b] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurlPS 2023 Spotlight.

[c] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." CVPR 2024 Highlight.

[W1] […] I recommend introducing the technical solution first and then discussing the underlying motivations.

[R1] Thank you for your feedback. We will revise the paper to present the technical solution earlier, enhancing clarity and focus.

[W2] Technically, my major concern lies in the motivation of the proposed method. In Conjecture 1, […] This conjecture is based on an over simplified example that conducts classification for a Gaussian Mixture Model with 2 Gaussian distributions. […]

[R2] We would like to clarify the following points:

1. Prior research efforts [n,o], including the Outstanding Paper at NeurIPS 2022, have utilized simplified mixtures of Gaussian distributions to study data efficiency and other properties, acknowledging the complexity of analyzing real-world data distributions. These studies typically extend theoretical insights to more complex datasets through empirical validation.
2. Our work provides theoretical insights into data properties relevant to efficient training. These insights do not suggest a reduction in the variance of real-world data but rather support our Conjecture 1, which forms the basis of our proposed technical solution. The validity of our solution is demonstrated in Section 5, where we present extensive experimental results on real-world datasets.

[W3] In fact, this conjecture, as well as the final solution that minimizes the cosine loss with sample predictions and targets generated by a well-trained teacher, is contradictory to many recent studies. For example, in RDED [a], hard patches are selected to form the synthetic samples. After all, hard patches should have larger classification loss, corresponding to samples closer to the decision boundary in the toy case. Recent studies like [b] and [c] also demonstrate that weak trajectory or teacher models may benefit learning more. […]

[R3] We would like to clarify the following points:

1. In fact, RDED [a] promotes the selection of easy patches (those with smaller classification loss, as indicated in their Equation (8)) and utilizes well-trained teacher models for target generation, as specified in their Algorithm 1. This methodology is consistent with our theoretical and empirical analyses and aligns with the recommendations found in several recent studies [e,f,g,j,k,l].
2. The papers reviewer referenced [b, c] utilize weak teachers for trajectory matching or patch selection in image generation. However, none of these studies advocate for the use of weak teachers in target generation.

[W4] Based on above analysis, I would like to believe the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved. […]

[R4] We would like to clarify the following points:

1. Unlike prior studies [a,b,e,f,g,j,k,l] that directly utilize a well-trained model specific to the dataset, our approach, ReLA, leverages any publicly available model to generate supervision signals. For instance, ReLA can use a model pre-trained on CIFAR-10 to generate supervision signals that accelerate training for BYOL on a 10% subset of ImageNet-1K, resulting in a 7.7% increase in accuracy compared to the original BYOL (see Sections 4 and 5).
2. Our aim is to propose a simple, effective, plug-and-play, and theoretically supported method to accelerate representation learning. In addition to empirical validation in our paper and similar evidence in prior studies [d,e,f,g,j,k,l], we also provide theoretical insights into how these generated supervision signals enhance the convergence rate of learning (see Section 3.2).
3. We have not transformed a self-supervised learning problem into an almost fully supervised one. The additional supervision loss is integrated with the original loss term and modulated by a dynamic coefficient (see Section 4.2). Furthermore, our ablation study in Section 5.2 illustrates that using only the supervision loss can result in catastrophic performance.
4. Moreover, our method can also be used to accelerate supervised learning, as demonstrated in our experiments and analysis in Appendix L.

[W5] The authors analyze the optimal properties of distilled data. […], different methods would result in different distill data. Not all of them satisfy the proposed properties.

[R5] We would like to clarify the following points:

1. Our analysis try to identify the ideal properties of efficient data to offer insights that support our proposed framework. Most existing dataset distillation methods fail to fully satisfy these properties. The relationship between our analyzed properties and previous studies is discussed in Section 3.2 and Appendix H.
2. However, our theoretical results are consistent with the empirical results in many previous studies [e,f,g,j,k,l,m], which indicate that: (i) selecting/synthesizing samples that are easy or close to the class data mean, and (ii) using well-trained models to relabel samples, can achieve superior performance with fewer training steps.

[W6] I do not think the introduction of dynamic dataset distillation is necessary. […], the results may not appear surprising since the method transforms a self-supervised learning problem to an almost fully supervised one.

[R6] We would like to clarify the following points:

1. Our framework ReLA is defined under the concept of dynamic dataset distillation because ReLA involves dynamically editing samples and labels during training.
2. Our approach does not rely solely on the introduction of a pre-trained teacher, as detailed in our analysis in [R4]. Furthermore, our experiments in Section 5 show that introducing weak teachers also contributes positively to our framework.
3. In addition to self-supervised learning tasks, we demonstrate that our ReLA framework outperforms state-of-the-art dataset distillation methods in conventional supervised learning tasks (see Appendix L).

[a] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." CVPR 2024.

[b] Zhou, Daquan, et al. "Dataset quantization." ICCV 2023.

[c] Lee, Yongmin, and Hye Won Chung. "SelMatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching." ICML 2024.

[d] Lee, Dong Bok, et al. "Self-supervised dataset distillation for transfer learning." ICLR 2024.

[e] Noroozi, Mehdi, et al. "Boosting self-supervised learning via knowledge transfer." CVPR 2018.

[f] Xu, Guodong, et al. "Knowledge distillation meets self-supervision.” ECCV 2020.

[g] Fang, Zhiyuan, et al. "Seed: Self-supervised distillation for visual representation.” ICLR 2021.

[h] Wang, Kai, et al. "Dim: Distilling dataset into generative model." Preprint 2023.

[i] Chen, Xuxi, et al. "Data distillation can be like vodka: Distilling more times for better quality." ICLR 2024

[j] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurlPS 2023 Spotlight.

[k] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." CVPR 2024 Highlight.

[l] Shao, Shitong, et al. "Elucidating the Design Space of Dataset Condensation.” Preprint 2024.

[m] Zhao, Bo, and Hakan Bilen. "Dataset condensation with distribution matching." WACV 2023.

[n] Sorscher, Ben, et al. "Beyond neural scaling laws: beating power law scaling via data pruning." NeurlPS 2022 Outstanding Paper.

[o] Loureiro, Bruno, et al. "Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions.” NeurlPS 2021.

[W1] The paper lacks generalization ability to high-resolution datasets, such as 1Kx1K, which are common in practical datasets like clinical and aerial images.

[R1] Thank you for your feedback. We have conducted additional experiments using the CelebA-HQ dataset (1024 $\times$ 1024). The results, presented in Table 1 of the attached PDF, indicate that our ReLA effectively accelerates self-supervised learning on high-resolution datasets.

[W2] The practical applications of dataset distillation are not discussed. Demonstrating applicability in neural architecture search (NAS), continual learning, federated learning, and privacy preservation would be valuable for both the community and real-world scenarios.

[R2] We would like to emphasize that the primary application of our ReLA method is to accelerate representation learning. This extends the scope of conventional dataset distillation, as traditional methods require more computational resources in distillation process than training on the full dataset, making them unsuitable for accelerating learning processes (refer to our detailed analysis in [R4]).

Furthermore, we have applied ReLA to continual learning, with the experimental results presented in Table 2 of the attached PDF. These results demonstrate that ReLA outperforms the baseline in this context.

[W3] Comparisons with some state-of-the-art methods, including DREAM [a], DataDAM [b], and SeqMatch [c], are missing from Table 6. Including these comparisons would strengthen the evaluation of the proposed method.

[R3] We excluded comparisons with methods [a,b,c] because they are inefficient when dealing with large-scale datasets, such as ImageNet-1K, particularly with large backbone architectures like ResNet-18, as demonstrated in previous studies [d,e,f]. Hence, we evaluate these methods [a,b,c] against our proposed ReLA on four datasets with smaller backbone networks. The results of these comparisons are presented in Table 3 of the attached PDF, demonstrating that ReLA consistently outperforms these methods.

We will add these baselines into our manuscript.

[W4] An analysis of the computational costs is absent. It is crucial to examine the training costs, including both memory and training time, for the proposed process and state-of-the-art methods to ensure data-efficient training algorithms.

[R4] As detailed in Appendix K, the SOTA dataset distillation methods generally incur higher computational costs than training on the entire dataset. For example, the SOTA efficient method RDED [d] necessitates a fully trained model on ImageNet-1K for distillation. Additionally, several other SOTA methods [a,b,c] also demand higher computational budgets compared to training on the full dataset, as documented in their respective studies.

In contrast, our proposed ReLA method can distill a dataset using less than the computational budget required for a single epoch of full dataset training (see Appendix K for a detailed analysis).

The computational costs of ReLA, along with comparisons to baseline methods, are presented in Table 4 of the attached PDF. The experimental results indicate that while ReLA slightly increases computational time and peak GPU memory usage when using a partial dataset for training, it significantly enhances accuracy, surpassing the performance achieved by training on the entire dataset.

We will add these comparisons into our manuscript.

[W5] The theoretical analysis does not generalize to complicated data distributions, such as two arbitrary mixtures of Gaussian distributions or a Mixture of Generalized Gaussian distributions (MGG).

[R5] The theoretical analysis presented in this paper can indeed be generalized to arbitrary mixtures of Gaussian distributions. Although Section 3 exemplifies this with a specific mixture of Gaussian distributions, the proofs and theoretical framework detailed in Appendices B and C provide a generalized approach applicable to any mixture of two Gaussian distributions, regardless of their means and variances.

[Q1] Can RELA be extended to ImageNet-21K? If so, please provide some experimental results.

[R6] The ImageNet-1K dataset used in this study is already a substantial and challenging benchmark in the field of dataset distillation [a,b,c,d,e,f]. We extended our experiments to the ImageNet-21K dataset. As shown in Table 6 in the attached PDF, our ReLA approach also accelerates training for BYOL on ImageNet-21K, achieving higher performance with the same data usage as the original BYOL.

[Q2] How does the proposed framework perform on other downstream tasks, such as object detection or segmentation?

[R7] We evaluate the pre-trained models on a downstream segmentation task, as detailed in Table 5 of the attached PDF. The results demonstrate that models trained with our ReLA framework outperform baseline methods in both classification and segmentation tasks.

[a] Liu, Yanqing, et al. "Dream: Efficient dataset distillation by representative matching." ICCV 2023.

[b] Sajedi, Ahmad, et al. "Datadam: Efficient dataset distillation with attention matching." ICCV 2023.

[c] Du, Jiawei, Qin Shi, and Joey Tianyi Zhou. "Sequential subset matching for dataset distillation." NeurIPS 2023.

[d] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." CVPR 2024.

[e] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurlPS 2023 Spotlight.

[f] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." CVPR 2024 Highlight.

Thank you for addressing some of my comments in the rebuttal. After reviewing the appendices again, I still do not see how the theoretical analyses can be generalized to two Mixture of Generalized Gaussian distributions (MGG). This requires further discussion.

However, given that you have extended the work to high-resolution images, various tasks, and complex datasets, I am inclined to increase the contribution rating to 4. If you provide more discussion on the theoretical analysis and address all comments in the final draft, I will consider raising my score to a strong acceptance.

Proof of MGG version of Theorem 1：

Setup

Notation: $\textup{N}(\mu,\alpha,\beta)$ denotes the generalized Gaussian distribution with pdf $\frac{\beta}{2 \alpha \Gamma(1 / \beta)} e^{-(|x-\mu| / \alpha)^\beta}$, $\textup{B}$ for Bernoulli distribution. We focus on the 1-dim situation. Assume that $\mu_1 < \mu_2$. Define the original data distribution ($\mathcal{N}_0 = \textup{N}(\mu_1, \alpha_0,\beta_0)$ and $\mathcal{N}_1 = \textup{N}(\mu_2, \alpha_0,\beta_0)$):

$G := \{(x, y) \mid y \sim 2 \cdot \textup{B}(1,\frac{1}{2}) - 1, x \sim \frac{1-y}{2} \cdot \mathcal{N}_0 + \frac{1+y}{2} \cdot \mathcal{N}_1 \}$

and the modified one ($\mathcal{N}_0' = \textup{N}(\mu_1, \alpha,\beta)$ and $\mathcal{N}_1' = \textup{N}(\mu_2, \alpha,\beta)$):

$G' := \{(x, y) \mid y \sim 2 \cdot \textup{B}(1,\frac{1}{2}) - 1, x \sim \frac{1-y}{2} \cdot \mathcal{N}_0' + \frac{1+y}{2} \cdot \mathcal{N}_1' \}$

Our task is predicting $y$ given $x$. Note that $y \in \{\pm1\}$, which is a bit different from the definition in Section 3.2 . In the 1-dim situation, we just need one parameter for this classification task, so define $f_{\theta}(x) := \textup{sign}(x + \theta)$ to fit the distribution.

We could compute the generalization loss on the original distribution:

$\mathcal{L}(f _{\theta}) = \left( \int _{-\theta}^{+\infty} \, dF _- + \int^{-\theta} _{-\infty} \, dF _{+} \right) / 2 = \left( 1 - \int _{-\frac{\theta + \mu _2}{\alpha_0}}^{-\frac{\theta + \mu_1}{\alpha_0}} \, dF \right) / 2$

Obviously $\theta^\star = -\frac{\mu_1 + \mu_2}{2}$, we have:

$\mathcal{L}(f_{\theta}) - \mathcal{L}(f_{\theta^\star}) = \left( \int_{-\frac{\mu_2 - \mu_1}{2 \alpha_0}}^{\frac{\mu_2 - \mu_1}{2 \alpha_0}} \, dF - \int_{-\frac{\theta + \mu_2}{\alpha_0}}^{-\frac{\theta + \mu_1}{\alpha_0}} \, dF \right) / 2 \leq C_1 \cdot (\theta - \theta^\star)^2 \quad (\text{or } C_1' \, |\theta - \theta^\star|)$

where $C_1$, $C_1'$ are constants, $F_0$, $F_1$, and $F$ denote the CDF of $\mathcal{N} _0$, $\mathcal{N} _1$, and $\textup{N}(0,1,\beta_0)$ respectively.

The inequality above is due to the fact that the function $h(x) = \left( \int^{1}_ {-1} \, dF - \int _{x-1}^{x+1} \, dF \right) / x^2$ has limits at 0 and so is bounded.

Algorithm

For a dataset $\{(x_i, y_i)\}_{i=1}^{n}$,

set the loss function $L(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell \left[ y_i (x_i + \theta) \right]$, $\ell(v) = \tfrac{1}{2} (1-v)^2$.

We apply the stochastic gradient descent algorithm and assume the online setting ($n=1$): at step $t$, draw one sample $(x_t, y_t)$ from $G'$ then use the gradient $\nabla L(\theta_t)$ to update $\theta$ ($\eta \in (0,1), t \in \mathbb{N}$):

$\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$

$\nabla L(\theta_t) = \theta + (x_t - y_t)$

It can be observed that the randomness of $x$ leads to noise on the gradient.

Bounds with Variance

We prove the proposition that lower variance of GG can make convergence faster, i.e., $\mathbb{E}\left[ \mathcal{L}(f_{\theta_t}) - \mathcal{L}(f_{\theta^\star}) \right]$ is bounded by an increasing function of the variance ($t$ fixed).

Proof

From above, we get:

$\theta_t = (1 - \eta)^{t} \theta_0 - \eta \left[ (x_{t-1} - y_{t-1}) + (1 - \eta) (x_{t-2} - y_{t-2}) + \dots + (1 - \eta)^{t-1} (x_0 - y_0) \right]$

and so:

$\mathbb{E}\left[ \mathcal{L}(f_{\theta_t}) - \mathcal{L}(f_{\theta^\star}) \right] \leq C_1 \mathbb{E}\left[ (\theta_t - \theta^\star)^2 \right]$

= C_1 \mathbb{E} \left$ \left[ (1 - \eta)^{t} (\theta_0 - \theta^\star) - \eta \sum_{j=1}^{t} (1 - \eta)^{j-1} (x_{t-j} - y_{t-j} + \theta^\star) \right]^2 \right$

$= C_1 \mathbb{E} \left[ (1 - \eta)^{2t} (\theta_0 - \theta^\star)^2 + \eta^2 \sum_{j=1}^{t} (1 - \eta)^{2(j-1)} (x_{t-j} - y_{t-j} + \theta^\star)^2 \right]$

= C_1 \left$ (1 - \eta)^{2t} (\theta_0 - \theta^\star)^2 + \frac{\eta}{(2 - \eta)} (1-(1 - \eta)^{2t})\left[\frac{\alpha^2 \Gamma(3 / \beta)}{\Gamma(1 / \beta)} + \left( 1 - \frac{\mu_2 - \mu_1}{2} \right)^2\right] \right$ The last two equalities are due to the fact that for $(x, y) \sim G'$:

$\mathbb{E}\left[ x - y + \theta^\star \right] = 0$

$\mathbb{E}\left[ (x - y + \theta^\star)^2 \right] = \frac{\alpha^2 \Gamma(3 / \beta)}{\Gamma(1 / \beta)} + \left( 1 - \frac{\mu_2 - \mu_1}{2} \right)^2$

[W1] The writing style can improve and make the paper approachable. The notations are often terse and lack details about symbols until later. Especially for proofs in appendix.

[R1] Thank you for your feedback. We will revise the paper to improve its clarity and make it more accessible. Specifically, we will provide more detailed explanations of symbols and notations earlier in the text, particularly in the proofs section of the appendix.

[W2] It might be more prudent to add experiments of larger scale.

[R2] Thank you for your feedback. The ImageNet-1K dataset utilized in this study is already a substantial and challenging benchmark in the field of dataset distillation [a,b,c,d,e,f,g]. We have conducted additional large-scale experiments, as detailed in attached PDF and our response to Reviewer zbwg.

Our additional experiments detailed in the attached PDF include the following: (1) Utilization of the higher resolution dataset CelebA-HQ (1024 × 1024). (2) Implementation of a continual learning task. (3) Comparisons with more state-of-the-art (SOTA) baselines. (4) Reporting of computation time and peak GPU memory usage. (5) Segmentation task performance on the VOC 2012 dataset. (6) Evaluation on the larger ImageNet-21K dataset.

[a] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." CVPR 2024.

[b] Lee, Yongmin, and Hye Won Chung. "SelMatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching." ICML 2024.

[c] Lee, Dong Bok, et al. "Self-supervised dataset distillation for transfer learning." ICLR 2024.

[d] Chen, Xuxi, et al. "Data distillation can be like vodka: Distilling more times for better quality." ICLR 2024

[e] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurlPS 2023 Spotlight.

[f] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." CVPR 2024 Highlight.

[g] Shao, Shitong, et al. "Elucidating the Design Space of Dataset Condensation.” Preprint 2024.