Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection

Thank you for your insightful feedbacks! We address your questions in the following responses.

---

W1: A detailed analysis of why FM as IE underperform on coarse-grained datasets with noisy labels.

A1: We sincerely appreciate the reviewer's insightful comments. Due to the character limit in the rebuttal, we kindly refer the reviewer to our response to Reviewer aGx6's "W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels."

---

W2: Limited sampling rate evaluation in ablation studies.

A2: We sincerely appreciate the reviewer’s insightful feedback and acknowledge the importance of conducting ablation studies across diverse sampling rates.

To address this concerns, we have conducted additional Ablation Studies at 10% and 30% sampling rates. The revised Tables 1, 2, and 3 are provided below.

Table 1. Ablation study based on Pet.

Across various sampling rates, RAM consistently outperforms MIN (CLIP as IE), while RAM-APL further improves performance, reaching levels comparable to DINOv2. Though RAM-APL (CLIP+DINOv2) demonstrates overall superior performance, its effectiveness at 70% sampling can be improved. In future work, we aim to enhance our method’s effectiveness at high sampling rates to further improve its practical utility.

Table 2. Comparison of the performance of our method using different numbers of foundation models as information extractors.

We observe that leveraging multiple foundation models outperforms using a single model. The optimal balance of computational efficiency, memory usage, and performance is achieved with DINOv2 + CLIP. For the highest overall accuracy, DINOv2+CLIP+EVA-CLIP is recommended. These findings validate the benefits of multi-model selection, and the results will be included in the supplementary material.

Tabel 3. Comparison of feature fusion strategies.

We observe that Our strategy outperforms Concatenate, especially at higher sampling rates, which are crucial for practical applications. To maximize the performance of multi-model method at high sampling rates, we adopt the Ours fusion strategy. The findings and new results will be included in the supplementary material.

Thank you for your positive feedbacks! We address your questions in the following responses.

---

W1: Evaluation on fine-grained image datasets with noisy labels.

A1: We sincerely appreciate the reviewer’s insightful suggestion. We acknowledge the importance of evaluating the effectiveness of our approach with other selection methods on fine-grained image datasets with noisy labels.

To address this concerns, we conducted additional experiments (as detailed in the tables below) on the Oxford-IIIT Pets dataset with 20% symmetric label noise and with 40% symmetric label noise. Subsets are sampled following the same experimental setup described in the manuscript.

Dataset: Oxford-IIIT Pets dataset with 20% symmetric label noise

Dataset: Oxford-IIIT Pets dataset with 40% symmetric label noise

("IE" means information extractor, "Model-TD" denotes the model trained on the full set for 10 epochs.)

We observe that RAM-APL consistently outperforms all baselines across different sampling rates on each noisy fine-grained dataset, demonstrating its effectiveness.

Your suggestion has been highly valuable. Through experimental analysis, we have identified the significant advantages of designing selection algorithms based on foundation models for noisy datasets, which motivates us to explore more effective foundation model-based denoising approaches in future work. The above experimental results and discussions will be included in the supplementary material.

---

W2: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.

A2: We sincerely appreciate the reviewer's insightful comments. Due to the character limit in the rebuttal, we kindly refer the reviewer to our response to Reviewer aGx6's "W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels."

---

S1: Code release.

A3: We thank the reviewer for this suggestion. We will release the full implementation code in a public repository upon paper acceptance to ensure reproducibility.

Thank you for your positive feedbacks! We address your questions in the following responses.

---

W1&Q1: Cross-architecture generalization of RAM-APL.

A1: We sincerely appreciate the reviewer’s insightful question regarding the cross-architecture generalization of RAM-APL. We acknowledge the importance of evaluating whether our selected subsets remain effective across different model architectures beyond ResNet.

To address this concerns, we conducted additional experiments on the Oxford-IIIT Pets dataset (Pets) using MobileNet-V3 as the target model. The results, presented in the table below, compare RAM-APL against five strong baselines that maintain identical architectures for their information extractors (IE) and target models.

MobileNet-V3 (MBV3)

We observe that RAM-APL consistently outperforms all baselines across different sampling rates, indicating its strong cross-architecture generalization ability.

Your suggestion has been highly valuable, inspiring us to further explore multi-model subset selection in broader cross-architecture settings in future work. The above experimental results and discussion will be included in the supplementary material.

---

W2&Q2: A deeper discussion of why FMs struggle with coarse-grained datasets containing noisy labels but perform well on fine-grained image datasets.

A2: We sincerely appreciate the reviewer's insightful comments. Due to the character limit in the rebuttal, we kindly refer the reviewer to our response to Reviewer aGx6's "W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels."

---

W3&Q3: How do RAM and APL influence the distribution of representations in selected subsets?

A3: We sincerely appreciate the reviewer’s insightful question regarding the influence of RAM and APL strategies on the distribution of representations in the selected subsets. We acknowledge the importance of analyzing how these strategies shape the feature space and their impact on sample diversity and representativeness.

To address this concern, we conducted additional experiments and analyzed the feature distributions of different selection strategies. Specifically, we examined the average cosine distance between data pairs within the selected subsets, which provides insights into intra-class and overall diversity. The results are summarized in the table below:

Table. Average cosine distance of data pairs in the subset

From these results, we observe that RAM and RAM-APL lead to a more diverse feature distribution in the selected subset compared to Min-based selection. The whole-subset average cosine distance is highest under RAM-APL (0.2787), indicating that it selects more diverse samples overall, improving coverage of the feature space. Moreover, the per-class distances suggest that RAM-APL encourages a balance between inter-class and intra-class diversity, with slightly higher values in harder-to-distinguish classes.

Furthermore, the t-SNE visualizations in Figures 9-11 (https://anonymous.4open.science/r/RAM-APL-DED5/README.md) further confirm these findings. Compared to Min-based selection, which tends to concentrate samples within certain regions of the feature space, RAM and RAM-APL distribute samples more broadly across the space, ensuring better representational coverage. This suggests that our approach enhances model performance by capturing a more comprehensive representation of the dataset.

Your suggestion has been highly valuable in strengthening our analysis. The above results and discussions will be included in the supplementary material to provide a clearer understanding of the impact of our proposed selection strategies.

Thank you very much for your positive feedback! We greatly appreciate your insightful questions, which have deepened our analysis of the findings and inspired further exploration for future work. Below, we address your questions in sequence.

---

W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.

A1: We sincerely appreciate the reviewer's insightful comments regarding the theoretical understanding of foundation models (FMs) on noisy datasets. We have conducted extensive additional analyses to explain this phenomenon, with key findings visualized in Figures 1-8 (https://anonymous.4open.science/r/RAM-APL-DED5/README.md).

Our empirical investigation reveals:

In coarse-grained datasets (CIFAR-10N-worse, Figures 1-4)

• FM-extracted features exhibit:
  • Weak inter-class separation for visually similar categories (e.g., dog/cat in CIFAR-10N-worse);
  • Substantial overlap between clean and noisy samples' feature distributions.

This explains FMs' limited effectiveness as information extractors for coarse-grained noisy data.

By contrast, in fine-grained datasets (Oxford-IIIT Pet with 40% symmetric label noise, Figures 5-8):

• FM-extracted features exhibit:
  • Compact clustering of correctly-labeled samples;
  • Strong inter-class separation for visually similar categories;
  • Smaller overlap between clean and noisy samples in feature space.
• Features from models trained on full noisy set show:
  • Loose clustering of correctly-labeled samples;
  • Significant overlap between clean and noisy samples in feature space.

This leads to the selection of more noise samples (visible in dark red points) and substantially weaker performance of traditional information extractors (IEs) compared to FMs.

Key Inference:

The comparative analysis reveals that FMs serve as superior information extractors when their features demonstrate:

• Tighter clustering of correctly-labeled samples;
• Reduced overlap between clean and noisy distributions.

We will incorporate these analyses in the supplementary material to strengthen our empirical analysis and contributions.

---

W2&Q2: Theoretical Analysis of Multi-FM Complementarity in RAM-APL.

A2: We thank the reviewer for this important question. RAM-APL's effectiveness in leveraging multiple FMs stems from two fundamental principles:

1.Feature Space Orthogonality:

Our analysis reveals that different FMs learn nearly orthogonal feature representations (Figure 6 in the Suppl.), with:

$$\text{cossim}⟨M_i(x), M_j(x)⟩ ≈ 0 \quad \forall i \neq j$$

This orthogonality demonstrates that each FM (e.g., $M_i, M_j$) captures distinct, complementary aspects of the data $x$.

2.Bias Reduction via Ensemble Consensus:

The ensemble mechanism could mitigate individual FM biases and preserve robust cross-model agreements. Table 2 in the manuscript demonstrates RAM-APL's performance gains when combining CLIP and DINOv2 versus individual FMs, confirming the benefits of multi-FM integration.

---

W3: Expanded Discussion of FM Literature.

A3: We sincerely appreciate the reviewer for this constructive suggestion. We will significantly strengthen our discussion of FM literature by incorporating CLIPCleaner (Chen et al., ACM MM 2024). The key insights are:

Both our work and CLIPCleaner leverage CLIP's zero-shot capabilities. Differently, CLIPCleaner, a single-FM method, focuses on noisy label cleaning via prediction probabilities. Our RAM-APL, a Multi-FM approach, specializes in subset selection for clean and noisy fine- rained data using visual deep features.

We'll discuss CLIPCleaner in Section 2 of the revised manuscript.

---

Q3: Extensions to Few-shot Learning and Noisy Data.

A4: We sincerely appreciate the reviewer's insightful question regarding the broader applicability of RAM-APL. By leveraging the strong feature discriminability of foundation models and mitigating biases through ensemble consensus, RAM-APL shows a strong ability to distinguish noisy datasets. While our current work focuses on standard subset selection for fine-grained datasets, its theoretical framework and algorithmic design naturally extend to:

• Noisy Few-shot Learning: Enhances robustness in few-shot scenarios by effectively identifying label-feature mismatches in small support sets.

• Noisy Label Scenarios: Particularly effective for fine-grained noisy data (as demonstrated in our experiments). Moving forward, we plan to develop more effective denoising strategies tailored to such datasets within the RAM-APL framework.

We will include this extended analysis in the supplemental materials to better position RAM-APL's broader applicability.