Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator

Thank you for your insightful comments! We answer your questions as below.

Q1(Weakness 1 & 2): discussion of related works.

• inter-class and inter-class relations

IID [1] introduces a class centralization constraint aimed at enhancing class discrimination by clustering samples more closely within classes. In contrast, our proposed INFER focuses on enhancing cross-class features to mitigate feature duplication. Both approaches are conceptually sound and cater to distinct use cases effectively. Class-centralized method, IID, excel on simpler datasets and with smaller IPC settings, where preserving the most representative features of each class is sufficient to maintain performance. However, as dataset complexity increases or the IPC grows, these representative features become inadequate to fully capture the dataset distribution. At this point, the advantages of our method, INFER, become evident. This is clearly demonstrated in following experimental results. Besides, IID not only centralizes features but also incorporates a covariance matching constraint to enhance feature diversity. This concept aligns with our approach, as both methods aim to reduce feature duplication and better approximate the real data distribution.

• CIFAR-10:

• CIFAR-100:

• multiple backbone architectures for dataset distillation

In the proposed INFER framework, we utilize multiple backbone architectures, each individually responsible for training the one compensator. The compensators learned by each architecture are illustrated in Appendix A.4, Figure 6, and the ablation studies examine each architecture’s contribution to the final performance are provided in Tables 4 and 6. Although G-VBSM [2] also employs diverse backbone architectures, it adopts a random sampling strategy, resulting in a synthesized dataset that is a mixture of outputs from all architectures. we evaluated the feasibility of this random sampling approach by incorporating it into our INFER framework.

• CIFAR-100

We observe that, given the same number of compensators, the random sampling method performs worse compared to the model-by-model strategy. This is because compensators optimized using randomly sampled architectures tend to converge toward a single optimal solution, thereby diminishing the diverse and complementary information provided by different architectures.

Q2(Weakness 3): speed up distillation process.

Thank you for the comment. Here, we provide a comparison of both computational cost and peak memory usage against existing methods following the defination in SRe2L [2].

• Tiny-ImageNet: Time Cost represents the consumption (ms) when generating one image with one iteration update on synthetic data. Peak value of GPU memory usage is measured or converted with a batch size of 200.

As our distillation process involves multiple architectures, the time cost and GPU memory usage vary accordingly. While the number of optimizable parameters is reduced, capturing inter-class features requires the participation of original images in both forward and backward propagation. Consequently, the generation speed and memory requirements are similar to those of SRe2L.

Thank you for your time and insightful feedback. We have provided detailed responses to all your questions below and hope they effectively resolve your concerns.

Q1(Weakness 1): notion defination.

Thank you for pointing that out. We have made the correction in the revised version (see Figure 2).

Q2(Weakness 2): comparison with SOTAs.

Thank you for the comment. As shown in Tables 1 and 2, INFER and INFER+dyn outperform other methods in most cases, except for the evaluation on CIFAR-10 with ConvNet under both $`ipc`=10$ and $`ipc`=50$. This is because, unlike other comparison methods, our methods do not involve the validation architecture (ConvNet) during the synthesis phase, thereby leading to cross-architecture generalization issues. To address this, we have included results where ConvNet is used during the distillation phase. Additionally, we incorporated a more recent SOTA method, DATM (ICLR 2024) [1], into the comparison.

• CIFAR-10:

The performance of INFER+dyn shows significant improvement, surpassing other SOTA methods under the $`ipc`=50$ setting. However, at lower $`ipc`$ values, both INFER and INFER+dyn fail to provide notable benefits. This can be attributed to the fact that, under high compression ratios, preserving simple and easy-to-learn patterns, rather than cross-class compensatory features, is more effective in maintaining model performance. All the compared methods, being class-specific, naturally hold an advantage in such scenarios. Notably, INFER demonstrates its superiority as the distillation budget increases or when applied to more complex datasets, such as CIFAR-100 and Tiny-ImageNet.

• CIFAR-100:

• Tiny-ImageNet:

* Note that the results reported here involve ConvNet for both distillation and validation in these experiments, instead of ResNet-18 as described in the original submission.

Q3(Weakness 3): cross-arch experiments.

Thank you for the suggestion. We have incorporated additional cross-architecture experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet, utilizing networks such as MobileNetV2, EfficientNetB0, ShuffleNetV1, ResNet-50, and ResNet-101. These results demonstrate the robust cross-architecture generalization capability of our method.

• CIFAR-10 ($`ipc` = 10$)

• CIFAR-10 ($`ipc` = 50$)

• CIFAR-100 ($`ipc` = 10$)

• CIFAR-100 ($`ipc` = 50$)

• Tiny-ImageNet ($`ipc` = 50$)

Thank you for your time and constructive comments. We have responded to all your questions as follows and hope these address your concerns.

Q1(Weakness 1 & Question 1): calculation of distillation budget

A1: Thank you for bringing this to our attention. We must apologize for the incorrect statement regarding $K = `ipc`$ in line 261. As clarified in line 407, ‘Lastly, we also average the number of UFCs into ipc for fair comparison,’ the correct computation for $K$ is $K = \lfloor `ipc` \times \frac{C}{C + M} \rfloor$. Specifically, we use $K = 7, 35$ for CIFAR-10 with $`ipc` = 10, 50$; $K = 9, 48$ for CIFAR-100 with $`ipc` = 10, 50$; and $K = 9, 49$ for Tiny-ImageNet and ImageNet-1k with $`ipc` = 10, 50$.

Q2 (Weakness 2):comparison with SOTA method DATM [1]

A2: Thank you for the suggestion. Several works aim to enhance the efficiency of budget utilization. For instance, DATM (ICLR 2024) [1], SeqMatch (NeurIPS 2023) [2], and PAD (Arxiv 2024) [3] have proposed various strategies to reallocate distillation budgets based on the difficulty of learning patterns. Below, we provide an experimental comparison on the CIFAR and Tiny-ImageNet datasets. To ensure fairness, ConvNet is involved in distillation and used for validation in these experiments, instead of ResNet-18 (as reported in the original submission).

• CIFAR-10:

• CIFAR-100:

• Tiny-ImageNet:

The proposed INFER demonstrates better or comparable performance on these benchmarks. At smaller ipc, these class-specific distillation methods retain an advantage. This can be attributed to the fact that, under high compression ratios, retaining simple and easy-to-learn patterns rather than cross-class compensatory features effectively sustains model performance. Notably, INFER will demonstrate its superiority as the distillation budget increases or when applied to more complex datasets, such as CIFAR-100 and Tiny-ImageNet.

Q3 (Weakness 3): privacy-related question

A3: There are two key reasons for including real instances. First, similar to the SOTA method RDED [4], real instances sampled from the original distribution enhance the realism and diversity of the distilled dataset, mitigating feature duplication and overfitting to specific distillation architecture. Second, while privacy is important, our primary focus is on efficiency. Real samples are essential for achieving an aggressive compression ratio through static labeling, as natural instances inherently follow the linear interpolation of labels. This is further supported by applying static labeling to SRe2L [5], a method that does not utilize real samples. The observed significant performance degradation highlights the critical role of real instances in improving efficiency. Moreover, the inferior performance of SRe2L to INFER also reinforces the importance of realism and diversity provided by real instances, as outlined in the first point.

• CIFAR-100 (ResNet-18)

Q4 (Question 2): local and global inter-class feature

A4: Thank you for the question. We actually considered the so-called "global" UFCs but decided to deprecate them during the examination of our method. Below, we first present the comparison experiments between the "local" and "global" UFCs.

• CIFAR-10 (ipc=10)

We elaborate on the reasons for deprecating the global UFCs as follows:

• The performance drop is the primary reason to adopt "local" UFCs as shown in the above table.
• Optimizing UFCs across the entire set $\mathcal{P}$ would incur extensive memory consumption, requiring $K$ times the memory of "global" UFCs.
• The entire set $\mathcal{P}$ contains many instances belonging to the same class, causing the "global" UFCs to also capture intra-class features.

Dear reviewer hymx,

Thank you for your questions regarding the review of our paper. We understand that your schedule is very busy, but we wanted to kindly remind you that the discussion period is nearing its conclusion. You raised several questions during the first round of review, and we have carefully addressed each of them in our rebuttal.

All other reviewers have responded to our rebuttal, and we sincerely hope you can review the responses, and we could help you address all your concerns about our submission.

Thank you for your time and thoughtful consideration.

Best The authors

I would like to thank the authors for your response and most of my concerns are resolved. However, I still have doubts about the effectiveness of the proposed idea under high compression ratios.

The authors explained that the class-specific distillation method is more effective at preserving simple and easy-to-learn patterns under high compression ratios. If so, does this mean that the cross-class distillation method structurally fails to preserve such simple and easy-to-learn patterns effectively? If the cross-class distillation method is indeed ineffective under high compression ratios, I believe it has limitations as a universally applicable approach in the most challenging tasks (e.g., IPC = 1). I believe there should be a clear explanation and justification for this. Furthermore, I think Figure 1, which appears to suggest its effectiveness even under high compression ratios, needs appropriate revisions.

Additionally, consistent statements regarding high compression ratios are necessary. I believe the authors used the justification of simple and easy-to-learn patterns to defend the significant performance gap with prior studies when considering CIFAR-10 and IPC=10. However, in Table 2, the proposed idea demonstrates higher performance than the baseline while using a lower budget (high compression ratio), which the authors emphasized. These contradictory experimental results make it difficult to clearly understand the effectiveness under high compression ratios.

Nevertheless, I agree with defining the performance degradation of dataset distillation as IPC increases as a feature duplication problem and proposing an idea to mitigate it. Thus I decide to increase my score to 5, but I will not insist on rejection if all other reviewers champion acceptance.

Thank you for your constructive comments! We give point-to-point replies to your questions in the following.

Q1(Weakness 1 & Question 1): comparison with RDED

A1: We compare the performance of our proposed INFER to RDED as you suggested as below:

Our INFER outperforms RDED on CIFAR-100 but falls short on the other three datasets. This difference can be attributed to the use of dynamically generated soft labels in RDED, as highlighted in [5], which significantly enhance performance. While RDED employs dynamic soft labels updated each epoch, INFER relies on static soft labels. To isolate the effect of static labeling, we compare the performance of RDED using static labels under the same conditions as INFER:

• ImageNet-1K

The performance of RDED lags significantly behind INFER when evaluated under the same static labeling protocol.

The non-optimization method proposed by RDED is a type of patch-wise dataset pruning technique. The performance of RDED can be further enhanced by applying our optimization-based UFC to it.

• ImageNet-1K (ipc=10):

As evidenced by the above experimental results, non-optimization method of RDED also overlooks the inter class features and can therefore be improved by our optimization-based UFC.

Q2(Weakness 2 & Weakness 3 & Question 2 & Question 3): explaination of experimental results.

A2: We appreciate you bringing this to our attention and apologize for the imprecise conclusion in our submission. We will revise our statement to: “Static labels suffice for simpler networks, but dynamic labels are observed to offer stronger supervision for some more complex architectures.”

We performed a statistical analysis and demonstrate this tendency as below. In the following tables, a ticked cell indicates better performance, and the architectures are roughly arranged by complexity from simple to complex across the rows.

• CIFAR-100 (ipc=10)

• CIFAR-100 (ipc=50)

For CIFAR-100, regardless of whether ipc=10 or 50, dynamic labeling demonstrates superior performance on larger networks such as ResNet-18 and ResNet-50.

• ImageNet-1K (ipc=10)

• ImageNet-1K (ipc=50)

For ImageNet-1K, the results remain consistent across architectures except for ResNet-101, which appears to be an outlier in these experiments.

Thank you for your insightful response!

**Q1 :**Performance Comparison with prior research [1]:

A1: Our method outperforms LPLD[1] significantly on the ImageNet-1k, as indicated below:

Table 1: Results of Resnet-18 on ImageNet-1k.

Table 2: Results of Resnet-50, ResNet-101 on ImageNet-1k.

All results of LPLD [1] are cited from their paper [1]. The results indicate:

• Our proposed method, INFER (with static labels), achieves significantly better performance than the previous state-of-the-art (SOTA) LPLD [1] (with improvements ranging from 7.6% to 20.9% on ImageNet-1k).
• More importantly, our INFER (with static labels) is equivalent to $300\times$ compressed labels under the same standard as stated in [1]. (This is because SRe2L, CDA [3], and RDED [2] use dynamic soft labels over 300 epochs, whereas INFER uses static labels in just 1 epoch, which is $300\times$ compressed labels.) This demonstrates that our INFER not only outperforms the previous SOTA LPLD [1] but does so with a much lower label budget, highlighting the superior competitiveness of our method.

Additionally, LPLD [1] was released on ArXiv on October 21, 2024, and is motivated by the need to prune the soft labels of SRe2L. In contrast, our INFER, submitted on October 1, 2024, is driven by the goal of leveraging overlooked inter-class features to achieve significantly better performance than LPLD [1]. Therefore, we disagree with the claim that "This raises concerns regarding the novelty and competitiveness of the proposed method."

Q2 : the observation that static labels outperform dynamic labels

A2: We appreciate your keen observation and insightful question! This is indeed a very critical and interesting point that warrants further investigation. To address your query, we use the results from ImageNet-1k (ipc=50) as a case study.

We observe that the "dynamic labels" approach outperforms "static labels" on both ResNet-18 (Dynamic: 55.6% vs. Static: 54.3%) and ResNet-50 (Dynamic: 63.4% vs. Static: 62.5%). However, it performs significantly worse than "static labels" on the largest dataset, ResNet-101 (Dynamic: 60.7% vs. Static: 63.0%). The superior performance of "static labels" on ResNet-101 is attributed to the degradation of "dynamic labels" in this case. This degradation occurs because the "dynamic labels" approach relies on the technique of knowledge distillation.

The synthetic data augmented by is equvalient to the generated data of data-free knowledge distillation, which transfers knowledge through the generated soft labels of the "teacher network" in every epoch to train the "student network". Consequently, the student network is expected to have a simpler architecture and exhibit worse performance compared to the teacher network.

We list the number of parameters for each networks in experiments as below:

The results of "dynamic labels" on ResNet-101 degrade significantly because the teacher network, ResNet-18, is much simpler than the student network, ResNet-101. Such degration is also observed in LPLD[1] (ResNet-50 42.3% vs ResNet-101 42.1%) as indicated in Table 2 of our A1. However, our method, INFER, is not affected by the performance degradation caused by the network complexity gap. Unlike the "dynamic labels" approach, our INFER does not rely on Knowledge Distillation but instead adheres to the natural dataset principle of "one instance, one label." This highlights the great potential of INFER to achieve superior distillation performance.

[1] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? NeurIPS, 2024 [2] On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. CVPR, 2024. [3] Dataset distillation in large data era.

Thank you for the detailed feedback. However, I still have some concerns:

(1) The results presented about [Weakness 1 & Question 1] indicate that the proposed method underperforms RDED across most scenarios. The shortfall is attributed to static labels that are designed for label storage efficiency. However, prior research [1] has demonstrated that under substantial label space compression (e.g., 40× reduction on ImageNet-1K), their approach achieves superior performance compared to state-of-the-art methods, including RDED [2] and CDA [3]. This raises concerns regarding the novelty and competitiveness of the proposed method.

(2) Furthermore, the observation that static labels outperform dynamic labels in some scenarios needs a deeper analysis. Could you provide a comprehensive explanation of the underlying factors driving this phenomenon?

[1] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? NeurIPS, 2024.

[2] On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. CVPR, 2024.

[3] Dataset distillation in large data era.

Thank you for your detailed response. I acknowledge a misunderstanding of certain technical aspects regarding CDA.

However, a primary concern of mine is the fairness of the performance comparisons reported in Tables 1, 2, and 4, as the batch sizes employed for the comparison methods lack consistency. For instance, in Table 1, the primary baseline, SRe2L [1], utilizes a batch size of 128 on CIFAR-100 (as stated in Table 10 on page 17 in [1]), whereas this study employs a batch size of 64 (Table 5 on page 15 in this paper). Similarly, in Table 2, on ImageNet-1K, SRe2L [1] uses a batch size of 1024 (referenced in Table 7 on page 15 in [1]), while this study adopts a batch size of 32 (Table 5 on page 15 in this paper).

Previous research [2, 3] and results presented by the authors indicate that batch size significantly affects performance. For example, using a smaller batch size in SRe2L, RDED, and G-VBSM can yield notable performance improvements. Therefore, the performance comparisons presented in this paper may be unfair.

I welcome further discussion on this matter and ensure a fair evaluation framework.

[1] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS, 2023

[2] Dataset distillation in large data era.

[3] Elucidating the Design Space of Dataset Condensation. NeurIPS, 2024.

I'm grateful for the detailed response and experiments. It is widely recognized in the deep learning community that smaller batch sizes do not consistently yield better performance. If this were true, batch sizes of 1 would be universally adopted.

Furthermore, I have concerns regarding the following points:

1. Papers such as SRe2L and G-VBSM do not claim to have conducted comprehensive experiments to determine optimal batch sizes, contrary to the authors' assertions.

2. The impact of small batch sizes is extensively analyzed in EDC both empirically and theoretically. The paper illustrates that smaller batch sizes effectively increase the number of iterations, aiding in the prevention of model under-convergence. The authors' paper lacks a detailed empirical or theoretical analysis to substantiate this claim.

3. The authors' own experiments indicate that batch size adjustments significantly affect performance. For instance, increasing the batch size from 32 to 128 results in a performance drop from 37% to 34.85% for INFER.

In conclusion, I believe the authors have selectively chosen SRe2L as the primary baseline, overlooking the potential for performance improvements in the baselines through batch size adjustments, resulting in an unfair comparison.

Thank you for your responses. However, we would like to gently remind you of the conclusion from the batch size experiments—requested by you but subsequently overlooked—that our method, INFER, demonstrates outstanding performance when evaluated using the same batch sizes of 32 and 128. Notably, this performance is achieved using only 1/300 (or 3.33%) of the soft labels compared to the other methods.

We acknowledged the performance degradation in our third response when using a batch size of 128, as per your request. However, the conclusion of these experiments, which you also appear to have overlooked, demonstrates that we achieved a 14.85% performance improvement while using significantly fewer soft labels (300× compressed labels for INFER vs. 40× for LPLD). We believe a rigorous evaluation should prioritize acknowledging the experimental results over dismissing them based on intuition.

Additionally, your claim:

G-VBSM do not claim to have conducted comprehensive experiments

This is incorrect, as the authors conducted batch size ablation study in Figure 6.

Your claim:

the authors have selectively chosen SRe2L as the primary baseline

This is biased, as SRe2L is the first work to use a model inversion approach for synthesizing distilled datasets on ImageNet-1k. Consequently, SRe2L should be considered a foundational reference for subsequent dataset distillation research.

Throughout the six rounds of discussion, you have referenced SRe2L, CDA, RDED, LPLD, and G-VBSM solely to critique the experimental results. However, you have not offered any opinions or feedback on the core insights, theoretical contributions, or foundational aspects of these related works or our submission.

After we demonstrated the superiority of our proposed INFER, you repeatedly overlooked the positive conclusions supported by our experimental results. Instead, you consistently referenced additional papers to request further numerical comparisons, repeatedly shifting focus away from the demonstrated strengths of our method. However, your continuous misinterpretations of the referenced papers, including CDA and G-VBSM, suggest that you have not thoroughly read these works. Instead, you appear to cite their experimental results solely because their values differ/are higher from ours, without considering the broader context or methodology underlying those results. However, we demonstrated that our proposed INFER outperforms the papers you referenced.