Distill Gold from Massive Ores: Efficient Dataset Distillation via Critical Samples Selection

1. I think the loss indicator is also very heavy in computation because according to the description it requires 50~100 epochs of training for 50 trails. The Monte-Carlo indicator also seems to require a long running time. It would be greatly appreciated if the authors can provide running time comparison. While Section 4.3 provides some ways to mitigate this, but I cannot find what would be the final performance using this "economic" version of algorithm.

2. A red flag for me is that, it seems that the proposed methods become less effectiveness when the dataset distillation methods are becoming stronger. More specifically, MTT has higher performance compard to DC and DSA, but it also requires more data to achieve the same level of performance. There could be a chance that this method will become less effective, if not effective at all. I would suggest the authors to benchmark on more advanced methods such as RFAD [r1] to show the proposed method can still be effective. For example, why did the authors not report performance on IDC in Table 7, while providing IDC's performance in somewhere else like Table 6.

3. How to decide the data dropping ratio in principle? Deciding such a threshold could also bring overhead.

4. Table 9 shows a simple twist can help improve the performance but it seems that the gaps will be drastically closed when the number of IPC increases.

5. It would be better if the authors can provide results of IPC=10 / 50 in Table 15 to show that the dataset can be pruned at higher budgets.

[r1] Efficient Dataset Distillation using Random Feature Approximation

W1. Technical rigor

While there are attempts to provide a formal treatment to the problem, the technical rigor of this paper is not satisfactory.
a) The terms “utility” and “capacity” are commonly utilized throughout the paper to describe the value of a particular sample for distillation and how much information a synthetic distilled dataset can hold, including as part of formal definitions. However, the precise definitions of these terms are in fact quite vague and unclear, instead resembling pseudo-information theory. This paper’s usage of utility and capacity in definitions and equations is not appropriate.
b) Concepts of “critical sample size” and “critical sample ratio” are introduced, by these are also not precisely defined, which renders the results in Table 1-4 seemingly arbitrary.
c) There are a number of other issues with notation and some of the tables/figures. See Miscellaneous + Questions below.

W2. Novelty + Significance

a) The proposed method more or less boils down to doing dataset distillation on a subset of the data. The fact that this is more efficient than doing it on the whole dataset isn’t particularly surprising, and that careful selection of data is better than random selection is in line with existing works on data pruning (e.g. [a]).
b) Loss as a measure of utility is more or less the same as techniques commonly used in active learning, or several branches of approaches continual learning (e.g. coreset selection in replay-based approaches, or importance weighting in regularization-based ones [b]). These similarities aren't discussed in the current draft, but should be.
c) The second paragraph of the Introduction is a written a little too harshly. While it’s true that data distillation taking a long time is a bad thing, that doesn’t render it completely unusable as a technique, as there are still applications where the portability of data matters more than having a trained model. Some may argue that the primary utility of dataset distillation is data portability; in many cases, distillation can be done offline, so how long it takes, while still important, is less of a concern. Also, while the “100x longer” statistic is eye-catching, this is also a cherry-picked example that is especially bad, and not necessarily representative of dataset distillation as a whole.

W3. Experiments

a) One of the headliner claims of this paper is that it is able to extend to larger-scale datasets like ImageNet-1K and Kinetics-400. However, the ImageNet accuracy is ~2% for 1 IPC, and ~9% for 50 IPC. While I understand that this is compressed to a much smaller synthetic set and better than the baseline, this is not anywhere close to how modern methods perform on ImageNet, and not what I consider as “working”; same goes for Kinetics-400. Simply running a method on a dataset regardless of results is not the same as saying the method works well on it. As such, the claims here are misleading.
b) A common use case of dataset distillation is continual learning. It could have been nice to see some experiments in this setting.

W4. Writing

The writing could use some improvement. There are a number of grammatical or idiomatic errors, and in a number of instances, the word choice implies the wrong thing, which can be distracting. I list out some examples below under “Miscellaneous”, but this is not an exhaustive list. I recommend the authors give this manuscript another careful round of edits.

The organization of this paper is also somewhat non-standard. Experiments are mixed in through Section 3 (Preliminaries) and 4 (Estimating Data Utility). There is no independent Experiments section. This isn’t necessarily a requirement for writing a paper, but following a more typical separation will help readers more easily find particular content. For example, the current organization is almost chronological in nature, adding new wrinkles or changes to the methodology after pervious benchmarking had already been done. This makes it hard to tell what finally this paper’s contributions are.

Miscellaneous:

• pg 1: “has become critical which enables high performance” <= awkward wording
• pg 1: “reduce the model size” <= extraneous “the”
• pg 1: “maintaining the model performance trained on the synthetics” <= I think this is actually trying to say that the goal is to be able train a model on the small synthetic dataset that has similar performance to training the real large dataset.
• pg 1: “data utilization” doesn’t seem to be the heart of the issue the authors are seeking to address. If I understand correctly, the authors are more concerned about the efficiency of data compression
• pg 1: “ a natural assumption comes up” <= assumptions are made by choice. Also, what follows is not an assumption, but rather a hypothesis/design choice.
• Figure 1 right: Rather than have points off the chart, can this be shown in log scale instead?
• pg 3: “we first argue that data redundancy is extremely severe in distillation” <= This is not a property of distillation, but rather the dataset for the specific learning task.
• pg 3: “to support the general the observation”
• pg 4: Tables 1-4: Methods/architectures should be cited.
• pg 4: “… real dataset and we have more freedom …” <= run-on sentence
• pg 4: “… high computation cost and MTT exceeds …” <= run-on sentence
• Figure 3 is confusing. Why does the subset have the same utility as the full dataset? Why does the model have more utility than the synthetic bottleneck?
• Definition 3: Cardinality is more commonly expressed with vertical bars like $|\mathcal X|$.
• pg 4: “A Larger” <= second capitalized word
• pg 6: Using $M$ for the size of the subset and $m$ for the size of the whole dataset feels backwards.

[a] Sorscher, Ben, et al. "Beyond neural scaling laws: beating power law scaling via data pruning." NeurIPS 2022.
[b] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." PNAS 2017.

1. Incomplete Results: Some of the tables lack IPC-50 results for specific datasets, creating an inconsistency in the presentation of the findings.
2. Clarity on Data Selection Methods: It would be beneficial to clarify how the data selection methods were compared. For instance, it is unclear if all samples were selected at the beginning of the training or if the selection was epoch-based.

• Hope to see a clear theoretical analysis or justification for the proposed data utility indicators and selection methods. It is not clear how they relate to the information content or transferability of data samples.

• The paper does not show significant performance improvement over the state-of-the-art distillation algorithms, please check the sota methods in: https://github.com/Guang000/Awesome-Dataset-Distillation . The performance gains are mostly marginal.