DRUPI: Dataset Reduction Using Privileged Information

Overall, the biggest drawback of this work is what I mentioned in Summary, the generalization of this paradigm, does it violate the concept of dataset condensation? Is the additional storage overhead introduced acceptable? Besides that, some expressions in the paper need to be improved.

Presentation:

1. The concept of dataset distillation is misunderstood. The authors classify dataset reduction into coreset selection and dataset distillation (line 125). However, this raises the question: how should algorithms that synthesize data without training, such as RDED [1], be categorized? The authors argue that all dataset distillation methods adhere to the distillation paradigm and synthesize only unseen data from the training set. In contrast, coreset selection is restricted to algorithms that select a subset of the original dataset.

2. The authors later make a definitional mistake by assuming that all dataset condensation that requires training is bi-level optimization paradigm, which is false. None of the researches [1,2,3,4,5,6] require bi-level optimization.

3. On line 253, Figure 3(a)(c) is a typo.

4. The theoretical analysis provided by the authors was unclear to me, and I am interested in seeing how other reviewers perceive it. Overall, it did not provide any significant insights and felt as though it was included primarily to meet the expectation of a theoretical component.

[1] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm, CVPR, 2024

[2] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective, NeurIPS, 2023

[3] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching, CVPR, 2024

[4] Information Compensation: A Fix for Any-scale Dataset Distillation, CVPR Workshop, 2024

[5] Elucidating the Design Space of Dataset Condensation, NeurIPS, 2024

[6] Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment, NeurIPS, 2024

Method

1. The full paper, including the appendices, does not address the additional storage overhead introduced by the use of privileged information. This omission is significant, as the storage burden imposed by the model's intermediate layer outputs is much greater than that of soft labels. For example, on the full ImageNet-1k dataset, the soft labels for IPC 50 alone amount to tens of gigabytes.

2. The generalizability of DRUPI is a significant concern. The results presented in Table 4 are based solely on experiments using convolutional model. I believe this paradigm may not be applicable when using a Vision Transformer as the teacher model and convolutional model as the student model. Furthermore, I think the paper should detail how the authors solved the gap in the feature map due to dimensional inconsistency.

3. The authors likewise need to clarify whether this paradigm violates the classical paradigm of dataset condensation.

Experiment

1. The authors tested DRUPI with only a few algorithms, specifically MTT and DC, which is insufficient. To date, dataset condensation algorithms that require training have consistently outperformed dataset selection algorithms in terms of performance. The authors should consider evaluating DRUPI with additional algorithms, such as gradient matching and distribution matching.

2. In addition, decoupled distillation (as referenced in [1-6]) represents the current state-of-the-art (SOTA) algorithm for large-scale datasets, including the full 224x224 ImageNet-1k and the full 224x224 ImageNet-21k. The authors should build on these algorithms to apply DRUPI. Restricting the experiments to small-scale datasets, such as the subset of ImageNet-1k used in the paper, does not significantly advance the field of dataset condensation. This limitation also hinders the potential for dataset condensation to play a meaningful role in the era of foundation models.

DRUPI uses the output of a pre-trained model as privileged information and further applies an MSE loss to enforce consistency between the model’s predictions and the privileged information. This approach shares similarities with knowledge distillation techniques. While the method is straightforward and yields competitive results, it inherently limits the upper bound of DRUPI's performance to the quality of the pre-trained model. Consequently, if the pre-trained model is suboptimal, the privileged information may not fully capture the necessary features, potentially constraining the overall effectiveness of DRUPI.

Given the conceptual overlap between DRUPI and knowledge distillation, it would have been beneficial for the paper to include a direct experimental comparison with established knowledge distillation techniques. Such comparisons would provide more insight into how DRUPI differentiates itself from or improves upon these methods, as both share the goal of leveraging additional information to enhance model performance.

The core innovation of DRUPI lies in the introduction of the output from a pre-trained model and the use of consistency constraints to guide model training. However, this concept has been extensively explored in the domain of knowledge distillation. While DRUPI's application of this idea within the context of dataset reduction is novel, the paper does not sufficiently acknowledge the wealth of prior work in knowledge distillation, where similar strategies have been widely adopted.

1. Some details of the method are not clarified, especially on the pruning part. The authors described the algorithm in the context of DC but have yet to explain how pruning can incorporate learnable feature labels.
2. Analysis in section 5 is trivial and doesn't provide enough insights. More discussion can be included.
3. The scalability on large IPCs may not be good enough according to the reported results in Table 2 and Appendix.

For the weaknesses mentioned above, I'll raise the corresponding questions below.

[W1] The paper lacks a thorough analysis of computational costs, which is critical for assessing the method’s efficiency. I suggest reporting specific metrics such as training time, memory usage, and storage overhead when generating and storing privileged information. It would be helpful to compare these costs with baseline methods to provide a clear understanding of DRUPI’s performance in terms of resource consumption. This analysis will clarify the computational trade-offs involved in using privileged information versus traditional dataset distillation methods.

[W2] Concerns about generalizability remain, particularly regarding the method's applicability to high-resolution datasets and different model architectures. I recommend testing DRUPI on 256x256 ImageNet-1k and on practical datasets such as clinical or aerial images to address this. Additionally, evaluating its performance on ViT models for cross-architecture generalization would provide further insights into the method's broader applicability across deep architectures.

[W3] While the practical applications of dataset distillation are not explored in depth, I understand this may be beyond the scope of a single paper. However, it would strengthen the paper to include a discussion on DRUPI’s potential applications, particularly in areas like neural architecture search (NAS) or continual learning. If feasible, adding an experiment or case study in one of these domains could demonstrate the method’s utility in real-world scenarios.

[W4] The paper does not include comparisons with key state-of-the-art methods like DREAM, DataDAM, and SeqMatch in Table 2. Including these methods would provide a more comprehensive evaluation of DRUPI’s effectiveness. If there are reasons why these methods were not compared, it would be helpful for the authors to explain this and clarify how DRUPI relates to these recent approaches.

References:

• [a] Liu, Yanqing, et al. "DREAM: Efficient dataset distillation by representative matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
• [b] Sajedi, Ahmad, et al. "DataDAM: Efficient dataset distillation with attention matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
• [c] Yin, Zeyuan, et al. "Squeeze, recover and relabel: Dataset condensation at ImageNet scale from a new perspective." Advances in Neural Information Processing Systems 36 (2024).

1. The paper does not sufficiently acknowledge prior work, for instance, it should compare and discuss the proposed method with previous dataset distillation methods that use data-soft label structures, which are similar in approach.
2. The novelty of the privileged information (feature and attention labels derived from feature labels) is limited and similar to approaches such as Re-labeling ImageNet and FKD.
3. The theoretical analysis using VC theory is general and not specific to the proposed method, making it insufficient to substantiate the approach.
4. The baselines in Table 1 are weak. In Table 2 of many cases, only marginal improvements are shown over prior methods. Additionally, the lack of results on full ImageNet weakens the validity of the experiments and the effectiveness.
5. Baseline approaches like DC and MTT are somewhat outdated, newer stronger methods such as SRe2L and RDED are not included in the comparison.