MITIGATING BIAS IN DATASET DISTILLATION

• The experimental benchmarks are limited to the synthetic datasets. In recent research, several real-world datasets have been employed for evaluation, such as CelebA-HQ, IMDB, and BFFHQ.
• The logic from Section 3 to Section 4 seems somewhat disjointed to me. Specifically, it's not evident how the concepts of 'bias amplification' and 'bias suppression' are factored into the design of the reweighting scheme.
• While the paper did assess performance against various distillation methods, I believe it would be beneficial to include other reweighting algorithms as baselines for a more comprehensive comparison (e.g., to justify the choice of KDE and Supervised Contrastive Learning for distance computation).

W1: It appears that the authors are trying to convince that their sample reweighting strategy can be utilized as a plug-and-play scheme for the matching-based dataset distillation method. However, they only test this strategy by combining with mere two earlier arts including distribution and gradient matching methods, providing insufficient evidence to prove the generalization ability and thus limiting the reliability.

W2: The method proposed in this paper seems to excel only on very simple datasets with sufficiently significant biases (e.g., CMNIST and BG FMNIST). On more complex datasets that lack remarkable biases (e.g., Corrupted CIFAR-10), the performance appears to be modest. This raises concerns about the practicality and robustness of the proposed reweighting scheme.

W3: The concerns raised in W2 lead me to question the necessity of addressing biases during the dataset distillation process. I posit that the performance ceiling of such reweighting mechanisms is inherently constrained by the number of unbiased samples in the original dataset. For instance, in simple datasets with strong biases, like CMNIST and BG FMNIST, distilled data can extract ample features from a limited pool of unbiased samples. However, for more intricate datasets where biases are not as pronounced, such as Corrupted CIFAR-10, it is unsurprising that this reweighting scheme falls short. This limitation stems from its inability to effectively harness the features present in biased samples.

Given this analysis, from my perspective, wouldn't a logical progression be to first create an unbiased dataset using methods like DFA or SelectMix, followed by dataset distillation?

The types and importance of bias: In this paper, the author investigates three types of bias (color, background, and corrupted data). I wonder if there exist other types of bias. In real datasets, bias may be mixed and complex. Does your method still work in such settings? The time complexity of your method: Though the method is quite simple, the additional operation like KDE comes with a cost. How much time complexity does your method introduce? The baselines are limited. please refer to https://github.com/Guang000/Awesome-Dataset-Distillation

1. A sample reweighting scheme utilizing kernel density estimation is not novel to me. Overcoming the bias problem with a reweighting strategy is a well-known approach in the traditional ML paradigms.

2. The kernel density estimation reweighting-based approach is not scalable. This work only shows experiments on three toy datasets. The performance on the popular benchmark datasets, i.e., CIFAR100 and ImageNet tiny, are not reported in this work.

3. It would be great if the authors could show the proposed debiasing method and other mitigation strategies can be combined to further improve the debiasing performance.

4. Synthesized biased datasets are applied in the experiments. I assume there are no real-world tiny biased datasets for this task. There might be some, but only large-scale datasets, in which the proposed method does not work well.