Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios

W1. What is the exact meaning of 'common pattern'? More detailed explanation is needed.

W2. The statement "low-loss signals typically correspond to common patterns" lacks specific reasoning to explain why dropout should be applied to address this. Can the authors elaborate on the rationale for using dropout?

W3. Observations were made in the introduction; however, they were based on CIFAR10 and IN1K-CIFAR10. Do similar results generally appear on other datasets as well, without cherry-picking?

W4. In a complex scene, object sizes are generally small, which could potentially be a reason for the smaller regions.

W5. Can the proposed method be applied to other methods besides Grad-CAM?

W6. How can the authors be certain that this is a discriminative area? This part is difficult to understand, and an explanation and justification are needed.

W7. What happens in a general environment (like the one conducted in DATM) rather than a complex DD benchmark?

• Some details of the proposed method are missing, as noted in the questions section below.
• This work introduces many hyperparameters, i.e. target supervision dropout ratio ($\alpha$), activation map update frequencies ($K$), and enhancement factor ($\beta$). A more in-depth analysis of the hyperparameter space and its influence on performance would improve the method's usability and adaptability.

• Inaccuracy of Grad-CAM Definition in Section 2.2: According to the original Grad-CAM paper, $M^c$ should be a gradient-weighted sum of all feature maps in the last convolutional layer. Therefore, the symbol $l$ in Equation (3) should denote the $l$-th feature map instead of the $l$-th convolutional layer.

Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 618-626). doi:10.1109/ICCV.2017.74

• Dependency on Hyperparameters: EDF’s performance depends heavily on various hyperparameters (dropout ratio $\alpha$, Grad-CAM update frequency $K$, enhancement factor $\beta$), requiring fine-tuning for a particular dataset, which could limit usability in practical applications without extensive experimentation.

• Generalizability to Non-Complex Scenarios: EDF shows significant improvements on complex datasets; however, its benefits on simpler datasets (like CIFAR) are less clear. It’s valuable to show EDF’s generalizability across datasets of various complexity.

• Limited Complexity of Comp-DD: ImageNet is a curated dataset, with most object-related images in a portrait style. Consequently, the subsets chosen from ImageNet in Comp-DD are likely also portrait-oriented with larger background regions. This setup may limit the true complexity of the benchmark. To further increase complexity, incorporating zoomed-out, cropped objects with relevant labels from uncurated datasets such as COCO, Objects365, or SA-1B could provide a more challenging and diverse set of scenarios, better suited to the intended purpose.

• Model Bias in Generating Activation Maps: It is well-known that neural networks can capture biased or non-generalizable features, as discussed in the Grad-CAM paper. If the activation maps focus on such less-generalizable regions, this could undermine the effectiveness of the Discriminative Area Enhancement (DAE) module. However, it is unclear how the proposed method addresses this issue of potential bias in the activation maps. Clarification on how EDF mitigates or adapts to these biased features would strengthen the approach.

• Clarification on Low-Loss Regions and Discriminative Features: The paper assumes that low-loss regions correspond to common, less-discriminative features. Although the paper demonstrates activation area shifts with different loss levels during distillation (Fig. 2a), additional qualitative results—such as visual analysis with superposed images and highlighted activation regions—would help support this assumption. It is important to note that a smaller activation region does not necessarily indicate less-discriminative features.

Recommendation: Overall, my recommendation for this paper is borderline reject. However, the score could be improved if the concerns outlined above are properly addressed.

1. Using the original image for initialization seems to be critical. As indicated by Figure 5, discriminative features are highly contingent on the initialized image. If random initialization is employed, the calculation of Grad-CAM may lose its significance and potentially lead to convergence issues, whereas the baseline method MTT [1] is viable under random noise initialization. The authors should include experiments involving random noise initialization.
2. This method seems to be a combination of DATM and Grad-CAM. While the authors demonstrate the advantages of Grad-CAM over DATM [2], the versatility of this method requires further validation. Can EDF be integrated with additional baseline methods such as DC [3], DM [4], CAFE [5], and DREAM [6]?
3. The authors should explicitly provide the proportion of wall clock time allocated for calculating Grad-CAM during training. Furthermore, can EDF surpass that of DATM when both methods are trained within the same wall clock time?
4. The network utilized for Grad-CAM calculation must first be trained on the same real dataset. Thus, its inherent bias may act as a bottleneck for gradient updates. The authors can conduct an ablation study to evaluate the impact of the network's capabilities on EDF.
5. It is worth discussing how the authors compare EDF’s approach, which focuses on highly activated regions, with previous methods aimed at reducing spatial redundancy [7][8] and learning a highly representative basis [9].
6. As shown in Table 11, each IPC configuration corresponds to distinct hyperparameters, which may complicate practical applications, especially given that MTT [1] and DATM [2] already involve numerous hyperparameters that require tuning.
7. The authors claim to be the first to achieve lossless results on ImageNet subsets; however, based on the trends depicted in Table 1 and the characteristics of DATM [2], it can be inferred that DATM may also attain lossless results at IPC 200/300. It becomes necessary to supplement experimental validation and accuracy comparisons.
8. I'm curious whether cross-architecture evaluations can achieve lossless results, such as with ResNet50 and other advanced networks. If lossless results are attained only on less robust networks (e.g., ConvNetD5), the practical significance may be limited due to lower performance ceilings. Recent studies have reported higher test accuracy on larger networks [10][11][12][13].

[1] "Dataset distillation by matching training trajectories." CVPR 2022.

[2] "Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching." ICLR 2024.

[3] "Dataset condensation with gradient matching." ICLR 2021.

[4] "Dataset Condensation with Distribution Matching." arXiv preprint arXiv:2110.04181 (2021).

[5] "Cafe: Learning to condense dataset by aligning features." CVPR 2022.

[6] "Dream: Efficient dataset distillation by representative matching." ICCV 2023.

[7] "Dataset condensation via efficient synthetic-data parameterization." ICML 2022.

[8] "Sparse parameterization for epitomic dataset distillation." NeurIPS 2023.

[9] "Dataset distillation via factorization." NeurIPS 2022.

[10] "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurIPS 2023.

[11] "Dataset Distillation in Large Data Era." arXiv preprint arXiv:2311.18838 (2023).

[12] "Curriculum Dataset Distillation." arXiv preprint arXiv:2405.09150 (2024).

[13] "Large Scale Dataset Distillation with Domain Shift." ICML 2024.