Rethinking Dataset Quantization: Efficient Core Set Selection via Semantically-Aware Data Augmentation

We sincerely appreciate the reviewer's thoughtful analysis. Below we address each concern in detail:

1 Regarding Large-scale Dataset Evaluation: We acknowledge the limitation of not evaluating on ImageNet-1k. This was primarily due to computational constraints, as our research was conducted using a single RTX 4090 GPU. While this limitation affects the direct comparability with some previous works, our results on multiple smaller-scale datasets demonstrate the effectiveness of our approach. The consistent performance improvements across diverse datasets suggest potential scalability to larger datasets.

2 Regarding Recent Baseline Comparisons: We sincerely acknowledge that not including recent state-of-the-art coreset selection and dataset quantization methods is a limitation of our work. Our initial comparison focused primarily on DQ as we considered it the state-of-the-art approach at that time and mistakenly assumed it had encompassed comparisons with most existing methods. While our method demonstrates advantages in computational efficiency and improved performance compared to DQ, we recognize that a more comprehensive evaluation including recent methods would have provided a more complete positioning of our approach within the current landscape of coreset selection techniques.

3 Regarding Training Costs and Dataset Size: We appreciate this concern about potential training cost increases. However, we would like to clarify that while our method employs data augmentation, it does not increase the final training data volume. Similar to the original DQ's use of MAE for data enhancement, we employ CNN-based augmentation but maintain the same final dataset size. Specifically, we create a mixed dataset by combining 50% original images and 50% augmented images, ensuring the total size remains consistent with the original DQ method's selection ratio. Therefore, the training costs are comparable to other coreset selection methods as we maintain the same proportion of training data. This approach enhances data diversity while maintaining computational efficiency.

Our method aims to provide a more efficient alternative to DQ by eliminating the need for large pre-trained models while preserving its core advantages. The identified limitations in our evaluation framework provide valuable direction for future work, where we can more comprehensively assess our method against the full spectrum of current approaches in coreset selection.

We sincerely appreciate the reviewer's thoughtful analysis. Below we address each concern in detail:

1 Regarding Fair Comparison with Coreset Selection Methods: We believe our comparison is fair for two key reasons. First, while we employ data augmentation, we maintain dataset size parity during the bin creation and selection process by mixing original and augmented images in equal proportions. This ensures that the total number of images participating in coreset selection remains identical to the original dataset size. Second, regarding data regularization, it's worth noting that the original DQ method also employs implicit regularization through its MAE model, which was pretrained on ImageNet-1K. In contrast, our method uses a randomly initialized model, avoiding the introduction of any external knowledge. This actually makes our approach more "pure" in terms of only learning from the target dataset.

2 Regarding Literature Review and Baselines: We sincerely acknowledge that not including recent important works (Tan et al., 2024; Xia et al., 2022; Yang et al., 2022; Maharana et al., 2023; Yang et al., 2023) is a limitation of our work. Our initial comparison focused primarily on DQ as we considered it the state-of-the-art approach at that time and mistakenly assumed it had encompassed comparisons with most existing methods. While our method demonstrates advantages in computational efficiency and improved performance compared to DQ, we recognize that a more comprehensive evaluation including these recent methods would have provided a more complete positioning of our approach within the current landscape of coreset selection techniques.

3 Regarding Semantic Ambiguity in Background Replacement: We appreciate this concern about potential semantic ambiguity. However, our method is specifically designed to avoid introducing unintended semantic information. As demonstrated in Figure 5a, the replacement backgrounds consist of randomly shuffled patches that contain only texture information, not semantic content. These randomized, small-scale patches, when combined, are highly unlikely to form recognizable objects or introduce semantic ambiguity. The background replacement process preserves the semantic integrity of the main object while only introducing variation in non-semantic texture patterns.

4 Regarding Training Cost Analysis: We acknowledge the limitation in our quantitative analysis of computational costs. While our method inherently requires less computation by eliminating the need for a large pretrained model (replacing ViT-Large with ResNet-18), we agree that explicit measurements of training costs and coreset selection computational overhead would have strengthened our claims. This would indeed provide a more comprehensive assessment of our method's practical efficiency.

5 Regarding Experimental Validation: As mentioned in our response to the literature review concern, we acknowledge that our experimental comparisons could have been more comprehensive by including recent state-of-the-art methods. While our focus on comparing with DQ was motivated by its significance as a recent breakthrough in coreset selection, we recognize that including comparisons with the mentioned recent works would provide a more complete evaluation of our method's practical significance.

These limitations in our evaluation framework provide valuable direction for future work, where we can more comprehensively assess our method against the full spectrum of current approaches in coreset selection.

We sincerely appreciate the reviewer's thoughtful analysis. Below we address each concern in detail:

1 Regarding Motivation and Core Contributions: We respectfully disagree with the characterization that storage reduction is DQ's primary contribution. The main value of DQ lies in its computational efficiency while maintaining high performance in coreset selection. While our method does require more storage (as shown in Table 1), it significantly reduces computational complexity by eliminating the need for the ViT-Large architecture used in MAE, which is the most computationally intensive component of DQ. In today's computing landscape, where storage is relatively inexpensive and even large-scale datasets like ImageNet-1K only require around 100GB of storage space, the computational cost far outweighs storage concerns. Our method preserves DQ's advantages while further reducing computational overhead, making coreset selection more accessible and efficient.

2 Regarding Method Logic and Design Choices: The sequence of applying Tobias data augmentation before dataset selection is a deliberate design choice. This approach allows the coreset selection algorithm to consider both augmented and original images simultaneously when identifying the most representative subset. If we were to perform coreset selection first or apply Tobias augmentation after DQ, the selection process would not account for the augmented samples, potentially leading to less representative core sets. Our approach ensures that the selection algorithm can optimize across both original and augmented images to identify the most effective subset.

3 Regarding Dataset Size Analysis and MAE Performance: We acknowledge that our discussion around line 210 could have been more precise. Our intention was to highlight that MAE's effectiveness varies with image resolution rather than dataset size per se. For larger resolution images (224×224), MAE can leverage more information during reconstruction, potentially providing more effective data augmentation and implicit regularization. Conversely, for smaller resolution datasets like CIFAR, MAE's reconstruction capabilities may be limited by insufficient information. This observation is supported by MAE's strong performance on Food-101, which also features high-resolution images. We acknowledge the reviewer's astute observation that MAE's pretraining on ImageNet-1K (He et al., 2022) might contribute to its effectiveness on ImageNette. However, we note that the strong performance on Food-101 suggests the benefits extend beyond just ImageNet similarity.

Regarding the specific negative effects of MAE: There are several important considerations. First, the computational overhead of MAE, which uses a ViT-Large architecture, is substantial and arguably unnecessary. This is particularly problematic since coreset selection methods are typically employed in resource-constrained scenarios, where running such large models may be impractical or inefficient. Second, and perhaps more fundamentally, the use of an ImageNet-1K pretrained MAE model introduces implicit prior knowledge into the process. This pretraining effectively incorporates external information and regularization priors from ImageNet-1K into the coreset selection process. Consequently, when evaluating DQ against other coreset selection methods, the comparison isn't entirely fair - it's unclear whether DQ's performance stems from the training data alone or is significantly influenced by the implicit ImageNet-1K knowledge encoded in the MAE model. We believe this might be one of the key reasons behind DQ's remarkable performance on ImageNet-1K. While DQ's authors presented MAE primarily as a means for storage reduction, our analysis suggests that the MAE model might be serving another crucial role - providing implicit regularization through its pretrained knowledge. This hypothesis is supported by our experimental results comparing performance with and without MAE across different datasets. Our method achieves comparable or better performance while avoiding these potential concerns, offering a more transparent and computationally efficient approach that relies solely on the target dataset.

[Reference] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).

We sincerely appreciate the reviewer's thorough evaluation and constructive feedback. Below, we address each concern raised:

1. Regarding Computational Efficiency Analysis: We acknowledge the limitation in our quantitative analysis of computational gains. While our method utilizes ResNet-18 instead of ViT-large (as used in MAE), which inherently suggests significant computational savings due to the substantial architectural differences (ResNet-18: ~11M parameters vs. ViT-large: ~307M parameters), we agree that explicit measurements would have strengthened our claims. This architectural efficiency stems from eliminating the need for a large-scale pre-trained model while maintaining comparable performance.

2. Regarding Missing Baselines: We acknowledge that not including recent important baselines (D2 Pruning, CCS, and Moderate) is a limitation of our work. Our initial comparison focused primarily on DQ as we considered it the state-of-the-art approach at that time and mistakenly assumed it had encompassed comparisons with most existing methods. This was an oversight on our part. While our method demonstrates advantages in computational efficiency and improved performance compared to DQ, we recognize that a more comprehensive evaluation including these recent methods would have provided a more complete positioning of our approach within the current landscape of coreset selection techniques. This comparison would be valuable for the community to better understand the relative strengths and trade-offs of different approaches.

3. Regarding ImageNet-1k Evaluation: We acknowledge the limitation of not evaluating on ImageNet-1k. This was primarily due to computational constraints, as our research was conducted using a single RTX 4090 GPU. While this limitation affects the direct comparability with some previous works, our results on multiple smaller-scale datasets demonstrate the effectiveness of our approach. We believe the consistent performance improvements across diverse datasets suggest potential scalability to larger datasets.

4. Regarding Data Augmentation Strategy: We appreciate the opportunity to clarify our data augmentation approach. We would like to emphasize that our method does not actually increase the dataset size. Similar to the original DQ's use of MAE for data enhancement, we employ CNN-based augmentation but maintain the same final dataset size. Specifically, we create a mixed dataset by combining 50% original images and 50% augmented images, ensuring the total size remains consistent with the original DQ method's selection ratio. This approach enhances data diversity while maintaining computational efficiency.

5. Regarding Ablation Studies: We acknowledge that a more comprehensive ablation study would strengthen our work. While we did conduct experiments without augmentation that showed significant performance degradation compared to the original DQ, we agree that examining various augmentation configurations would provide valuable insights. This observation led us to understand that MAE in the original DQ effectively functions as a data augmentation method, introducing beneficial prior knowledge and implicit regularization into the training process.

We appreciate the reviewer's careful evaluation and believe addressing these points would strengthen our work. While the submission period has concluded, these insights will be valuable for future research directions in this area.