Learn to Synthesize Compact Datasets by Matching Effects

1. Although the proposed formulation is novel in the literature on dataset distillation, I cannot get the motivation of the proposed objective against the original formulation of BPTT. Specifically, BPTT wants to minimize the error on real data for models trained by synthetic data, i.e., $\mathcal{L}(P,\theta^*_{A})$, while the matching effect is to minimize $|\mathcal{L}(P,\theta^*_{D})-\mathcal{L}(P,\theta^*_{D-G+A})|=|\mathcal{L}(P,\theta^*_{D})-\mathcal{L}(P,\theta^*_{A})|$, given that we would like to replace all the real data with synthetic data and thus we can assume $D=G$.
   1. I do not get why the authors believe the latter can be better than the former.
   2. In the practical cases of dataset distillation, since the synthetic dataset is small, the error on the real data of models trained by the synthetic dataset is usually larger than that trained by the real dataset, in most cases $|\mathcal{L}(P,\theta^*_{D})-\mathcal{L}(P,\theta^*_{A})|=\mathcal{L}(P,\theta^*_{A})-\mathcal{L}(P,\theta^*_{D})$, which is equivalent to the original formulation because the term $\mathcal{L}(P,\theta^*_{D})$​ is not relevant to optimization. From this point of view, the proposed method can conduct some rectification for the opposite case. But I am not sure if this is how the method works in fact and how we could benefit from this rectification. In summary, a more comparative analysis is necessary.
   3. It seems that the above analysis is also applicable to the proposed approximation, which can be viewed as a variant of the previous gradient matching scheme.
2. From Eq. 6, it seems that the approximated error is quite large because it is dominated by the farthest distance the neural network parameters move away from their initial state during training when any subset is used as the training set. The authors can provide some analysis on whether the bound is tight. If it is indeed tight, I am not sure whether it is useful in practice. The authors are encouraged to provide some toy experiments to illustrate this approximation.
3. Accordingly, in the experiments, I recommend the authors provide more ablation studies to compare the proposed method with the original formulation, i.e., BPTT and gradient matching while maintaining other factors the same. Given the current results in Tab. 1, which are not evidently strong, we cannot state that the proposed method is better. More analysis is encouraged to figure out in what cases the method is superior and in what cases it is not.

1. It seems that this paper is not finished. There is only one incomplete table in the experiment. Although the proposed method performs better than the other methods on digital datasets with IPC=1/10, it performs worse on the other datasets. So, the experiments cannot demonstrate the superiority of the proposed method.
2. An ablation study on the hyper-parameters is required.
3. The summarised contributions are not matched to the method.
4. For Eq.(6), reducing the steps of network optimization $T$ can help to close the gap of approximation error. However, a smaller $T$ usually means a sub-optimal performance of a network.

1. Instead of directly matching training trajectories, this method is proposed to match the 'effect', which is measured by the differences between probability distribution predicted by models trained on synthetic data and real data. This means that naturally, this method will have worse performance than matching training trajectories (MTT) [1]. Because MTT directly minimizes the differences between parameters of models trained on synthetic data and real data, where models' predictions will be the same ideally (which is the optimization goal of the method proposed in this paper). Coinciding with this, TESLA [2] (following work of MTT), which also uses soft labels, always performs better than this method.

2. I notice this method outperforms TESLA in large IPC cases, is it because this method uses the difficulty alignment trick (control matching range) proposed by DATM [3]? The author should report the hyper-parameters to improve clarity.

3. What are the benefits of replacing matching parameters with 'effects'? Being more efficient? Have better generalizability? The paper only reports one comparison, I think more comprehensive comparisons can improve the quality of this paper.

[1]. Dataset Distillation by Matching Training Trajectories, CVPR 2022.

[2]. Scaling up dataset distillation to imagenet-1k with constant memory. ICML 2023.

[3]. Towards lossless dataset distillation via difficulty-aligned trajectory matching. ICLR 2024.

Scope of Experiments The paper only evaluates the method on classification tasks, which might limit its applicability to other machine learning tasks such as regression. Dataset Diversity The experiments are conducted on a limited number of datasets, which raises questions about the method's generalizability to other data distributions. Computational Complexity While more efficient than some alternatives, the methodʼs computational cost could still be a concern for large-scale datasets or real-time applications.

pls add comparisons with more recent methods