When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration

Dear reviewer 3S6N,

Thank you for your careful review and constructive suggestions on our work. We appreciate your recognition of our work's strengths, e.g, novelty, well-structured experiments.

We provide responses to address the comments as follows:

• Q1: More results in Table 1.

• A1: Thank you for your suggestions. We have extended our evaluation to include more results in Tab. 1 (as Tab. 4 shows results on Tiny-ImageNet).

  As shown in Tab. D-1, while both baselines benefit from augmentation, our method consistently achieves higher accuracy. This indicates that the performance gains are not solely due to augmentation but stem from our method's ability to identify samples suitable for augmentation. Thus, by unifying dynamic data selection and augmentation, our framework achieves both training acceleration and enhanced performance.

Table D-1: Additional results for Tab. 1.

• Q2: Clarification on the fine-tuning process.

• A2: We would like to clarify that the fine-tuning process does NOT use the selection&augmentation methods. The CLIP backbone is kept frozen; we only train the lightweight adapter from scratch.

• Q3: Performance of the fine-tuned model.

• A3: Good question. First, in Tab. D-2, we show the accuracy of CLIP before and after fine-tuning. It shows that the fine-tuned CLIP achieves notable improvements, indicating enhanced dataset-specific alignment ability. Second, we employ the pretrained CLIP without fine-tuning in our method. In Tab. D-3, while the vanilla CLIP achieves high performance, the fine-tuned model further improves the performance.

Table D-2: Performance of CLIP in classification accuracy.

Table D-3: Performance of CLIP with Tiny-ImageNet.

• Q4: Clarification on the cost in Tab. 2.

• A4: Yes, the cost of fine-tuning is included in the total costs reported in Tab. 2 to ensure a fair comparison with other methods. We freeze the CLIP backbone and only fine-tune a lightweight adapter (a simple linear layer constituting only 0.04% of CLIP ViT-B/32’s parameters) with minimal training iterations. Thus, the fine-tuning process can be completed efficiently.

• Q5: More results for Tab.3.

• A5: Thank you for the suggestion. As shown in Tab. D-4, our method maintains superior performance across both settings and selection ratios. Notably, Infobatch performs worse than the random baseline in some cases in noisy conditions. This is because it prioritizes high-loss samples, which are often noisy or corrupted under noisy scenarios. Applying augmentation to noisy samples further exacerbates their negative impact as more misleading training signals are introduced, ultimately degrading performance.

  This highlights that our performance gain is not solely due to augmentation, but rather due to selecting the semantic-representative samples for training.

Table D-4: Additional results for Table 3.

• Q6: Clarification on the selection ratio.
• A6: We would like to clarify that, following common practice in data augmentation and selection research, our method trains models only on the augmented data. In each epoch, we generate exactly one augmented sample per selected data point, and only these augmented samples are used in both forward and backward passes. Thus, the selection ratio is exactly equal to the proportion of the data used for training.
• Q7: Clarification of multimodal focus in the Introduction.
• A7: Thank you for the suggestion. We will revise the intro part, clarifying the multimodal characteristic and focus of our framework.
• Q8: Typo.
• A8: We have corrected the mentioned typo and thoroughly reviewed the manuscript to ensure clarity and correctness.
• Q9: Applicability to LLM training.
• A9: Applying our method to other domains, such as data selection for LLM training, is a promising direction for future research. Since our current work focuses on the image and image-text domains, extending this framework to LLM training requires adapting the definitions and estimation for both semantic consistency and sample density in a purely textual feature space. Feasible solutions may involve leveraging pre-trained language models (e.g., BERT, GPT) to assess semantic alignment and correctness, and using task-specific metrics such as token-level uncertainty or embedding-space sparsity to approximate density distributions.

Dear reviewer GA1s,

Thank you for your meticulous review and valuable suggestions on our work. We appreciate your recognition of our work's strengths, e.g., novelty, sound and valid experiments.

We provide responses to address the comments as follows:

• Q1: Universal applicability and reliance on CLIP.

• A1: Thank you for pointing this out. We agree that the applicability and effectiveness of pretrained CLIP may vary across domains, particularly in some highly specialized fields. However, we note that this concern is not unique to our work but is shared by many state-of-the-art works that leverage pretrained VLM such as CLIP. Since our work focuses on dynamic data selection in general-purpose vision domains, fully addressing CLIP’s applicability across all domains and tasks is, in our opinion, beyond the scope of this work but of interest for our future work. Some feasible solutions may include leveraging fine-tuned VLM tailored to the target domain, or incorporating domain-specific adapters or alignment modules to improve semantic consistency estimation in more specialized scenarios.

  We will add this discussion to the main paper on page 8 before the Conclusion section.

• Q2: Extreme noise conditions.

• A2: Thank you for the suggestion. To assess the robustness of our method under extreme noise conditions, we conducted experiments on Tiny-ImageNet with high label, input noise, and both (with 50% and 70% noise ratio).

  As shown in Table C-1, while the overall performance naturally degrades under severe noise, our approach consistently achieves significantly higher accuracy, e.g., over 10% accuracy improvement in the presence of input noise. Moreover, we further analyze the average proportion of noisy samples retained during training (Table C-2). It shows that the proportion of noisy samples in our selected datasets remains considerably low even in high-noise conditions. These results underscore the benefits of our approach: even in highly noisy environments, the multimodal semantic consistency used for sample selection helps retain informative, high-quality data and maintain model robustness.

Table C-1: Comparison of accuracy with Tiny-ImageNet in high-noise conditions.

Table C-2: Further analysis of the label noise proportion (%). We report the average introduced noise ratio in the selected datasets through the entire training process.

• Q3: More suggested related works.

• A3: We will include the suggested works in Sec. 2.2 of the revised version. Specifically,

  • Sec 2.2 line 152: add references "Beyond these, generative data augmentation (Moreno-Barea et al., 2020) ..." and "Recent studies also emphasize representation consistency (Atienza, 2022) and address distribution gaps between clean and augmented data (He et al., 2019) ...".

• Q4: Minor Typos.

• A4: We have corrected all the mentioned issues and thoroughly reviewed the entire manuscript to ensure clarity and correctness in the final version.

• Q5: Explanation of $p_{sel}$.

• A5: As defined in Eq. (3) of Sec. 3.4, $p_{sel}$ is computed as the product of $p_\rho$, which reflects the density distribution, and $p_{con}$, which captures semantic consistency. Specifically, $p_\rho$ assigns higher values to low-density samples, promoting coverage of underrepresented regions in the feature space. $p_{con}$ prioritizes samples with strong semantic alignment, ensuring their informativeness and relevance By combining the two, $p_{sel}$ encourages samples that are both structurally important and semantically meaningful.

  We appreciate your suggestion and will revise Sec. 3.4 to include this clarification.

• Q6: Sensitivity to the choice of pre-trained model.

• A6: Insightful question. To assess the sensitivity of our method to the choice of pre-trained models, we conduct experiments replacing CLIP with another pretrained multimodal model LanguageBind (LB)[a], which focuses on aligning diverse modalities (e.g., video, audio, infrared, and depth) into a shared language space. While CLIP is highly optimized for image-text alignment estimation, our framework still achieves consistent performance when using LB, with only a marginal drop (< 0.7%) in Table C-3.

Table C-3: Sensitivity analysis on the pretrained model.

[a] Zhu, Bin, et al. Languagebind, ICLR'24.

Dear Reviewer bQXA,

We sincerely thank you for the careful review and insightful comments/questions. We appreciate your recognition of our work's strengths, e.g., novelty and empirical effectiveness.

For the comments and questions, we provide our responses here:

• Q1: The impact of data selection on generalization.

• A1: Thank you for your insightful comment. To assess the impact of data selection on generalization, we have evaluated models trained on the selected data across various benchmark datasets (Tab. 1/3/4/5/6, Fig. 3,), which are commonly used to assess generalization performance. The results present a consistent trend across all baselines, as the selection ratio decreases, model test performance degrades, indicating reduced generalization. Despite this overall degradation, our method effectively alleviates this degradation by integrating data augmentation (DA) and selection.

• Q2: Clarification on the data used for training.

• A2: We would like to emphasize that only the augmented data is used for training. This follows common practice in many widely used DA and selection methods (e.g., TrivialAug, AutoAugment, InfoBatch, Moderate, etc.), where only the augmented version of selected data is used for training. Retaining the original and augmented versions would double the number of training data used for model forward and backward passes, thereby doubling the computation costs. This would substantially undermine the training efficiency that our method is designed to achieve.

  While only augmented data is used for training, our augmentation strategy is carefully designed to preserve semantic consistency. Please refer to A3 and A4 for details.

• Q3: Semantic alignment before and after augmentation.

• A3: Thanks for the insightful comments. While our method does not explicitly verify each augmented data post-augmentation, our DA strategy is designed to prioritize semantic alignment. Specifically, each selected data is augmented using only one operation with its magnitude contained within a moderate, predefined range (Sec. A.1). In contrast, widely-used methods, such as AutoAug, Fast-AA, and RandAug, typically apply 2-3 operations per sample with stronger transformation strengths, emphasizing more diverse training data rather than maintaining semantic consistency.

  To further address your concern, we investigate the semantic alignment with a vanilla pretrained CLIP model using cosine similarity. Tab. B-1 shows that the augmented data maintains a strong semantic consistency with the original data.

  While our current method does not include explicit post-augmentation verification to maintain high efficiency, we agree that incorporating a lightweight verification mechanism, without compromising efficiency, is a promising direction for future work, which is also a key challenge in DA research.

Table B-1: Semantic alignment between original and augmented data points.

• Q4: Clarification on local structure after augmentation.

• A4: To further assess the local structural stability, we investigate changes in each sample's local nearest neighbors before and after augmentation. As shown in Tab. B-2, the proportion of altered neighbors is extremely low (<= 0.3%), validating that our DA introduces only minimal changes to local structure.

  However, it is important to note that the feature space evolves continuously during training as the model updates. Thus, even if the augmented data remains semantically and structurally stable, it is necessary to update the embedding space to capture the model's current training state for dynamic data selection. This is also the core principle of dynamic data selection approaches. Table B-2: The ratio of changes in local nearest neighbors after augmentation. ||Change Ratio| |-|-| |C-10|0.2%| |C-100|0.3%|

• Q5: Clarification on the embedding generators.

• A5: As clarified in Sec. 3.1, only the CLIP model is used to derive the semantic consistency distribution for filtering, which captures the intrinsic representativeness of training data. Meanwhile, as introduced in lines 207-213, to adapt CLIP to the target domain, we use a lightweight adapter to enable domain-specific knowledge transfer while preserving CLIP's strong alignment capabilities. On the other hand, the task model (e.g., ResNet or ViT series) is used to estimate the evolving density distribution during training. Thus, our framework minimizes the inherent biases.

• Q6: Details of augmentation operations.

• A6: For each sample, we apply only one random augmentation operation from Sec A.1. Following common practice in data augmentation (e.g., AutoAug, TrivialAug), we control the number of applied operations and the applied strength via a predefined, bounded magnitude range, ensuring consistency and avoiding overly distorted augmentation.

Dear Reviewer JbtQ,

We sincerely thank you for the comments and constructive suggestions. We appreciate your recognition of our work's strengths, e.g., reasonable experimental designs and well-supported claims.

For the comments, we provide our response as follows.

• Q1: Comparison with Data-Efficient Augmentation (DEA) [a].
• A1: Thank you for suggesting additional references and comparisons. We acknowledge the theoretical significance of Data-Efficient Augmentation, particularly its principled approach to rigorously determining which samples are most useful for training.

In response to your suggestion, we compared our method with the suggested data-efficient augmentation using the same amount of training data and number of training iterations, across standard benchmarks (Table A-1), and under challenging noisy and corrupted scenarios (Table A-2). For ImageNet-1k, we utilized the reported results from [a].

The results show that our method consistently achieves higher accuracy than DEA across selection ratios and datasets, achieving at least 3% higher accuracy on ImageNet-1k, and notable gains on CIFAR-10 and Tiny-ImageNet. Moreover, as shown in Table A-2, our method demonstrates stronger robustness to both noisy and corrupted conditions.

Table A-1：Comparison with [a] on CIFAR-10 with ResNet-18 and Tiny-ImageNet and ImageNet-1k with ResNet-50.

Table A-2: Comparison with [a] under noisy and corrupted conditions using Tiny-ImageNet with ResNet-50.

[a] Liu, Tian Yu, and Baharan Mirzasoleiman. "Data-efficient augmentation for training neural networks." NeruIPS 2022.

• Q2: Suggested References.

• A2: Thank you for suggesting more related works. Since the revised manuscript can not be updated at the current stage, we will include these references in Section 2.1 in the final version. Specifically,

  • Sec 2.1, paragraph 3: add references "The work (Liu & Mirzasoleiman, 2022) proposes a theoretically rigorous approach to determine impactful data for training. SAS (Joshi & Mirzasoleiman, 2023) improves data efficiency in SSL by proving and selecting the most beneficial data for contrastive training."

We hope these additions and clarifications address your comments.