Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models

We would first like to express our genuine appreciation for the time and effort the reviewer has dedicated to reviewing our manuscript. We believe these comments and questions provide valuable feedback and will greatly help us improve the overall quality of our work. Please let us know whether our response has sufficiently addressed your concerns. We are more than happy to engage in further discussion.

Q1.  Clarification of the association between worst-case selection and robustness gains

• We appreciate the opportunity to elucidate the connection of our ARDS and robustness goal and will incorporate the following discussion into our revised manuscript.
  • To begin, we would like to humbly highlight our key idea is to customize targeted instruction selection (e.g., LESS) for improving the robustness against specific dataset biases while avoiding gradient-based calculation.
  • Targeted instruction selection improves robustness.
    • Theoretical Analysis. Built on TracIn[1], LESS[2] quantifies the influence of a training datapoint $z$ on the loss of a test data $z^{\prime}$ via $\ell\left(z^{\prime} ; \theta^{t+1}\right)-\ell\left(z^{\prime} ; \theta^t\right) \approx -\eta_t\left\langle\nabla \ell\left({z} ; {\theta}^t\right), \nabla \ell\left(z^{\prime} ; \theta^t\right)\right\rangle$. Let $m$ denote vulnerable samples selected by our worst-case evaluation subgroups and $m^{\prime}$ denote perturbed test sample for robustness evaluation.
      • If the gradient inner-products $\left\langle\nabla \ell\left(m ; \theta^t\right), \nabla \ell\left(m^{\prime} ; {\theta}^t\right)\right\rangle$ is notably non-negative, then the loss of perturbed test samples $\ell\left(m^{\prime} ; \theta^{t+1}\right)-\ell\left(m^{\prime} ; \theta^t\right)$ decreases.
      • The decreased loss mirrors adversarial training for enhancing adversarial robustness [3].
    • Empirical Validation.
      • We first compute this gradient inner product using 100 training batches (batchsize=32) and 128 perturbed ScienceQA samples. The results of training batches from our robust training mixture show the average $\left\langle\nabla \ell\left(m ; {\theta}^t\right), \nabla \ell\left(m^{\prime} ; {\theta}^t\right)\right\rangle=25.76$, while that of training batches from random selection is $1.95$. This strongly supports our theoretical premise.
      • We kindly refer to Table 6 for the contribution of worst-case evaluation subgroups.
  • Alignment between ARDS and targeted instruction selection. To avoid the heavy cost of gradient-based calculations and sidestep the need for access to downstream data, we leverage the rich hidden representation of LMMs and construct worst-case evaluation subgroups on the training corpus.
    • Theoretical Analysis. In Appendix M, we provide a theoretical analysis under a linear model to demonstrate that the cosine similarity discrepancy between feature embeddings $r$ and gradients $g$ of two data samples $\left|\Delta_{i j}\right|=\left|\cos \left(r_i, r_j\right)-\cos \left(g_i, g_j\right)\right|$ can be upper-bounded.
    • Empirical Validation. We utilize the established gradient datastore in LESS and conversation vector datastore in our ARDS to compute the overlap between gradient-based selection and our feature-based selection for ScienceQA. The results, in the table below, demonstrate high selection overlap.
  • The transitive relationship thus substantiates why ARDS enhances robustness, supported by both theoretical insights and empirical evidence.

Selection ratio	30%	40%	50%	60%	70%
Selection Overlap	58.87%	69.00%	75.94%	79.97%	83.60%

[1] Estimating training data influence by tracing gradient descent. In NIPS, 2020

[2] Less: Selecting influential data for targeted instruction tuning. In ICML, 2024

[3] Towards deep learning models resistant to adversarial attacks. In ICLR, 2018

Q2. Why does full-data training perform worse than a random selection on pure-text tasks?

• We appreciate the reviewer's great feedback, and delve into a meticulous analysis for the decreased performance of full-data trained model on ARC-e and BoolQ.
  • We hypothesize that this phenomenon arises from the catastrophic forgetting driven by imbalanced training data across modalities, which aligns prior work that identified text-only forgetting during visual instruction tuning [1, 2] For example, WINGS [1] reports up to 13.33% text-only degradation after visual modality expansion.
  • In LLaVA-665K, only around 40688/665298 $\approx$ 6% of samples are pure-text instructions. During large‑scale multimodal tuning, the optimizer focuses overwhelmingly on vision‑centric tasks and gradually forgets language knowledge acquired from text‑only instruction tuning.
  • We have incorporated additional analysis by measuring text-only forgetting with random subsets of increasing size for LLaVA‑1.5 (7B), owing to compute limits. Larger subsets are expected to include more image-text pairs and thus amplify modality mismatch between multimodal and text-only inputs. The results, shown in table below, confirm that adding vision‑heavy data helps model excel in multimodal tasks while accentuates forgetting on text‑only tasks.

Task Type	Training Size	5%	10%	30%	50%	70%	90%	100%
text-only task	BoolQ Acc.	64.10	58.44	55.93	53.43	55.66	48.35	37.77
multimodal task	A-OKVQA Acc.	72.05	74.32	78.25	79.04	80.00	80.79	80.52

• We would like to humbly highlight that after once data selection, all subsequent results (e.g., ScienceQA, ARC and BoolQ) were obtained with the identical checkpoint trained with the selected data under the same evaluation setup.

[1] Wings: Learning multimodal llms without text-only forgetting. In NIPS, 2024

[2] Vila: On pre-training for visual language models. In CVPR. 2024

Q3. More discussions concerning 30% ratio on other tasks

• We appreciate this great suggestion, and in response, we have incorporated additional experiments to assess the generalization of our curated robust training mixture with 30% ratio.
  • Specifically, we have, following your advice, included a comparison with other mentioned methods on another text-only benchmark (SocialIQA[1]) and two additional math benchmarks (MathVista[2] and DynaMath[3]). These two mathematical benchmarks can be viewed as more difficult Out-of-Domain benchmarks, as the original LLaVA-665K does not explicitly involve math training data [4].
    • MathVista[2] is a mathematical reasoning benchmark introduced to analyze reasoning capabilities in visual complex scenarios (e.g., tables, function plots).
    • DynaMath[3] is a dynamic visual math benchmark designed to systematically analyze the robustness of mathematical reasoning capabilities across different topics (e.g., arithmetic, graph theory) under diverse condition changes (e.g., numerical value change). We report the average clean and robust accuracy on each benchmark.
  • The results, shown in the table below, demonstrate our curated robust training mixture with the 30% consistently boosts the generalization and robustness of visual instruction tuning, compared to full-data training and previous state-of-the-art selectors.

Method	data percentage	SocialIQA-Avg	MathVista-Avg	DynaMath-Avg
Full		48.70	23.38	24.48
Random	30%	49.25	19.86	21.06
LESS	30%	50.93	23.38	25.19
COINCIDE	30%	52.30	18.98	19.98
ARDS(ours)	30%	58.56	25.00	26.14

[1] Social IQa: Commonsense reasoning about social interactions. In EMNLP, 2019.

[2] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In ICLR, 2024

[3] DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. In ICLR, 2025

[4] Improved Baselines with Visual Instruction Tuning. In CVPR, 2024

Q4. More in-depth discussions concerning the motivation

• We appreciate the reviewer's invaluable feedback.
  • We would like to humbly highlight our primary motivation is to improve the robustness of visual instruction tuning against specific dataset biases.
    • We agree with the reviewer that full training data-LLAVA665k are assumed to be clean, but individual clean samples do not imply they will not reinforce spurious correlations or detrimental bias [1]. Large multimodal models tend to memorize common concepts or prioritize stereotypes present in the training corpus [2]. For example, both "roses are red" and "banana is yellow" image-text pairs are clean samples, but training corpus dominated by these samples may make the model disregard the actual user input that deviates from those patterns, and thus harm generalization and robustness under distribution shift.
    • As shown both theoretically and empirically in Q1 discussion, our curated robust training mixture improves not only the robustness of visual instruction tuning but generalization capabilities as well—sometimes even exceeding the full-data baseline.
  • Beyond robustness, our smaller training mixture favorably enjoys data efficiency, which is also targeted by prior selection methods such as LESS and COINCIDE. The smaller training mixture is appealing for its lightweight storage requirements and reduced training cost. This makes our approach practical for real-world deployment.

[1] An investigation of why overparameterization exacerbates spurious correlations. In ICML, 2020

[2] Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287, 2023.

We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.

W1 & Q1: More analysis of selected samples across worst-case evaluation subgroups (clusters).

• We sincerely appreciate the reviewer's invaluable suggestion. Following your advice, we have incorporated the statistical analysis on the loss distribution of our worst-case evaluation subgroups.
  • Specifically, we measure balance through Shannon entropy given as $e(\ell _ {\mathcal{S}})=-\sum _ {m=0}^M p\left(\ell _ {\mathcal{S} _ m}\right) \log p\left(\ell _ {\mathcal{S} _ m}\right)$, where $p\left(\ell _ {\mathcal{S} _ m}\right)$ is the weight of the $m$-th cluster after $\text{Softmax}(\ell _ {\mathcal{S} _ m})$. As entropy depends on the number of subgroups, we further normalize entropy w.r.t. the number $M$ of possible clusters: $\bar{e}(\ell _ {\mathcal{S}})=\frac{e(\ell _ {\mathcal{S}})}{\log (M)}$, where $\log (M)$ is equal to the entropy of a uniform distribution of size $M$. We obtained the normalized entropy $\bar{e}(\ell _ {\mathcal{S}})=0.92$. The result empirically verifies that selected samples remain well dispersed under the loss-based weighted-sum aggregation strategy.
• We would like to humbly highlight that we only perform clustering for building the worst-case subgroups and the importance score of each training sample is aggregated from all subgroups in a weighted-sum manner. To validate that this choice mitigates imbalance, we replaced it with a max operator (i.e., the importance score of each datapoint is determined only by its closest cluster). The results, in Appendix Table 10, show that the weighted-sum aggregation strategy yields a better clean and robustness trade-off and avoids the skew that the max variant exhibits. We appreciate the reviewer for prompting this analysis and will highlight these results in our main paper.

W2 & Q2: Clarification on the ablation settings in Table 6

• We apologize for any potential confusion and misunderstandings.
  • For the first row in Table 6, we randomly sample the same number of samples $MB$ as the final total size of the worst-case subgroups and the importance score of each training data is then aggregated with the sample-to-sample weighted-sum rule ( $\mathcal{I}\left(x_i\right)=\frac{\sum _ {j=1}^{MB} \exp \left(\ell_{j}\right) \cdot d _ {i j}}{\sum _ {j=1}^{MB} \exp \left(\ell _ j\right)}$ ). For the second row, we apply Equation 4 to retrieve top-$MB$ samples from the training dataset with the largest loss difference, and the importance score is aggregated in a sample-to-sample weighted-sum manner. The third row denotes our final construction strategy of worst-case subgroups, where we first perform clustering and use Equation 4 to retrieve top-$B$ candidates from each cluster. The importance score of training data is then aggregated with the sample-to-cluster weighted-sum scheme (Equation 6).

We would first like to express our genuine appreciation for the time and effort the reviewer has dedicated to providing such a detailed and in-depth review of our submission. We believe these comments and questions will contribute significantly to improving the overall quality of our work. We are more than happy to engage in further discussion.

Q1. More discussions concerning the relationship to active learning and data selection.

• We sincerely thank the reviewer for the constructive comments and will incorporate the following discussion into our revised manuscript.
  • Both data selection and active learning aim to identify a subset of data that yields the best possible model, while the distinction lies in the focus on settings. Active learning defaults to that data is unlabled and emphasize on reducing annotation cost [1-5]. In contrast, data selection methods usually have full access to all labels when selecting [6], and the goal is to improve training efficiency [7] or robustness[8].
  • Our method ARDS is a data selection method with no extra annotation required. Our focus is to boost robustness against specific dataset biases for visual instruction tuning while avoiding computation-intensive gradient-based calculation in [7, 8].

Q2. More comparisons with other related work and on a wider range of standard benchmarks.

• We appreciate the reviewer's great suggestion, and in response, we have conducted supplementary experiments to compare with D3M [8], which is a gradient-based data selection method designed for improving image classification robustness. Specifically, we reproduce D3M in our setting using the same warmed-up proxy model for extracting gradient vectors. We follow its standard procedure to project gradients, estimate the coefficient for each training sample and calculate the group alignment as the final score.
  • The results show that compared to this gradient-based method, the proposed ARDS achieves better robustness improvement while avoiding computation-intensive gradient calculation.

Method	data percentage	ScienceQA-Clean	ScienceQA-PA	ScienceQA-SA	ScienceQA-SA+PA	ScienceQA-Avg.
Random	30%	69.76	52.60	59.44	23.75	51.39
D3M	30%	69.86	56.87	62.52	27.17	54.10
ARDS(ours)	30%	69.26	59.40	68.57	47.60	61.21

• Additionally, we have included more standard benchmarks for a more comprehensive comparison. The results, shown in the table below, demonstrate that our method achieves comparable generalization capability and consistently boosts robustness across various benchmarks. This further demonstrates the effectiveness of the proposed ARDS.

Method	data percentage	TextVQA	GQA	SocialIQA-Avg	MathVista-Avg	DynaMath-Avg
Full		57.23	61.94	48.70	23.38	24.48
Random	30%	55.90	59.61	49.25	19.86	21.06
LESS	30%	49.45	56.49	50.93	23.38	25.19
COINCIDE	30%	56.09	59.15	52.30	18.98	19.98
D3M	30%	54.41	56.73	51.28	23.69	20.55
ARDS(ours)	30%	56.97	60.32	58.56	25.00	26.14

Q3. More computational analysis with baselines

• We appreciate the reviewer's valuable suggestion, and in response, we have incorporated additional computational analysis.
  • We would first like to humbly emphasize that we use the standard LLaVA-1.5 training configurations when comparing all baseline methods, and thus the wall-clock time of visual instruction tuning on the same size of training mixture should be the same for all baseline methods.
  • While ARDS, together with prior competitor COINCIDE, extract sample features for information measure to perform data selection, which can indeed increase the computational cost compared to random selection, this trade-off is justified by our results significantly enhancing the robustness—a key motivation of our work.

Method	Selection Ratio	Wall-clock time of data selection	Wall-clock time of visual instruction tuning	SciPA-Avg.	SEED-Avg.
Random	10%	<1min	3.6h	46.27	33.31
COINCIDE	10%	100 min	3.6h	47.30	33.91
ARDS (ours)	10%	39 min	3.6h	61.42	45.08
Random	30%	<1min	6.9h	51.39	37.97
COINCIDE	30%	100 min	6.9h	52.27	39.58
ARDS (ours)	30%	39 min	6.9h	61.21	46.80

Q4. Split Table 2 into separate tables and add more results to ensure fair comparisons

• We appreciate the reviewer’s invaluable suggestion.
  • To begin, we would like to humbly emphasize that our primary motivation is to improve the robustness of visual instruction tuning against specific dataset biases. We also humbly emphasize that our method does not require access to downstream few-shot examples used in LESS or a reference model well-trained on the full data required by COINCIDE.
  • We will follow your advice to divide clean and robust accuracies in Table 2 into separate tables to make comparisons clearer.
  • Following your advice to ensure fair comparison, we have conducted additional experiments by adding our constructed worst-case evaluation subgroup as the selection source for the previous state-of-the-art selector COINCIDE. The results, shown in the table below, demonstrate the effectiveness of our ARDS to achieve superior robustness across benchmarks.

Method	Data Percentage	ScienceQA-Avg	SEED-Avg	MMBench-EN-Avg	A-OKVQA-Avg
COINCIDE	30%	52.27	39.58	59.54	64.97
COINCIDE+worst-case subgroup	30%	53.21	40.97	60.445	67.77
ARDS (ours)	30%	61.21	46.80	65.26	72.95

Q5 Clarification and more results on the worst-case perturbation.

We deeply appreciate the reviewer’s feedback. To begin, we would like to humbly highlight that perturbations are introduced solely to build worst-case evaluation sub-groups, which are not applied during training. To build the worst-case subgroups, we inject task-specific visual and textual perturbations (i.e., symbol plus permutation attacks) simultaneously, and thus only one augmented sample is generated. We further explored the effect of generating multiple samples. The results, shown in the table below, demonstrate that the performance of the proposed method is not very sensitive to the number of augmentations.

#Aug. in Worst-case	Computational Cost	data selection ratio	ScienceQA-Clean	ScienceQA-PA	ScienceQA-SA	ScienceQA-SA+PA	ScienceQA-Avg
1	39min	20%	69.26	59.40	68.57	47.60	61.21
2	50min	20%	68.67	58.85	68.07	48.34	60.98

Q6. Justification for our analysis of more evaluation benchmarks.

• We appreciate the reviewer's invaluable suggestion and we have incorporated additional, more detailed ablational analysis using A-OKVQA, MMMU, MMBench-EN, and MMBench-CN benchmarks. We report the average clean and robust accuracies on each benchmark. The results, in the table below, demonstrate that each component of the proposed method is consistently effective across benchmarks.

Worst-case Perturbation	Evaluation Subgroup Clustering	Data Percentage	ScienceQA-Avg.	SEED-Bench-Avg.	A-OKVQA-Avg.	MMMU-Avg.	MMBench-EN-Avg.	MMBench-CN-Avg.
$\times$	$\times$	30%	52.39	40.43	66.03	20.92	60.53	55.42
$\checkmark$	$\times$	30%	55.65	45.40	71.57	21.99	64.09	58.79
$\checkmark$	$\checkmark$	30%	61.21	46.80	72.95	22.88	65.26	59.80

Q7 More results on Vision-FLAN

• We sincerely appreciate the reviewer's invaluable suggestion and have conducted additional experiments on another training dataset Vision-Flan using the proposed ARDS. The results, shown in the table below, demonstrate that the robust training mixture created by ARDS on Vision-Flan achieves superior robustness, highlighting the effectiveness of our robustness-aware data selection method in mitigating the model's biased behaviors.

Method	Data Percentage	Sci-Clean	Sci-PA	Sci-SA	Sci-SA+PA	Sci-Avg.
Full	100%	64.06	39.71	53.59	13.73	42.77
Random	50%	61.38	34.41	53.59	18.89	42.06
ARDS (Ours)	50%	62.87	37.98	60.54	33.56	48.74

Q8. More discussion concerning limitations.

• We appreciate the reviewer's valuable feedback and will incorporate the following discussion of limitations in the revised manuscript. Although the efficacy of the proposed method has been confirmed through empirical experiments for mitigating several dataset biases, such as position bias, symbol-content spurious correlations, and visual spurious correlations, opportunities for refinement persist. One avenue for improvement involves a more diverse investigation into the potential dataset biases and their interaction effects.

[1]Coresets for Scalable Bayesian Logistic Regression. In NIPS, 2016

[2]Consistency-Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost. In ECCV, 2020

[3]Not All Labels Are Equal: Rationalizing the Labeling Costs for Training Object Detection. In CVPR, 2022

[4]Box-level active detection. In CVPR, 2023

[5]A Survey on Deep Active Learning: Recent Advances and New Frontiers, In TNNLS, 2024

[6]A Survey on Data Selection for Language Models. In TMLR, 2024

[7]Less: Selecting influential data for targeted instruction tuning. In ICML, 2024.

[8]Improving subgroup robustness via data selection. In NIPS, 2024.

Thank you sincerely for your thoughtful feedback on our work. Below, we have provided a detailed explanation for your concerns as follows. Please do not hesitate to let us know if you have any further questions.

Q1. More results in scenarios where the whole training dataset is not available.

• We appreciate the reviewer bringing up the concern in a scenario where one cannot obtain the whole training dataset at one time.
  • We would first like to highlight that our experimental setting follows existing static visual instruction selection methods, where the selectors are aware of all training data to build a gradient database [1] or perform clustering on the whole dataset [2]. The key idea of our worst-case evaluation subgroups is to find representative samples that are most vulnerable to model-biased behavior while not needing the downstream few-shot examples required in [1].
  • We totally agree with the reviewer that a selection method supporting dynamically coming data is desired. To further address the concern about the training data requirement for building our worst-case evaluation subgroups, we have conducted supplementary experiments. We randomly sample 10% training data as the obtained small subset of the original training corpus and treat the remaining 90% as new incoming training data. We denote the variant as ARDS$^*$. Specifically, to build worst-case evaluation subgroups, we perform clustering on these 10% data with the same number of clusters $K$ and subgroup budget $B$ and then apply task-specific perturbations (see Appendix Line 424-428 for more details). The results indicate that despite seeing only a tenth of the corpus during subgroup construction, ARDS$^\*$ achieves robustness close to the full-information ARDS—and still outperforms both LESS and COINCIDE. This highlights the potential and applicability of our method to more dynamic data selection scenarios.

Method	Selection Ratio	Sci-Clean	Sci-PA	Sci-SA	Sci-PA+SA	Sci-Avg.
LESS [1]	30%	68.42	55.63	64.70	34.95	55.93
COINCIDE [2]	30%	67.72	52.21	61.08	28.06	52.27
ARDS	30%	69.26	59.40	68.57	47.60	61.21
ARDS$^*$	30%	68.86	58.30	67.13	42.39	59.17

[1] Less: Selecting influential data for targeted instruction tuning. In ICML, 2024.

[2] Concept-skill transferability-based data selection for large vision-language models. In EMNLP, 2024.

Q2. More results on further data selection.

• We appreciate the reviewer's insightful suggestion regarding performing data selection a second time.
  • To begin, we would like to humbly emphasize our current 30% ratio is determined by the trade-off between data efficiency as well as both clean and robust performance across varying tasks (kindly refer to Appendix Figure 5)
  • Following your advice, we have incorporated additional experiments to explore the impact of performing data selection again. The total selection ratio equals to the product of the two selection ratios. The results, in the table below, demonstrate that iterative selection, which first chooses 30% and then selects that subset to 33.3%, outperforms a single 10% selection. This indicates the great potential of our ARDS method to be extended to other scenarios. We will add the results and the discussion to our revised manuscript.

Method	Total Selection Ratio	1st Selection Ratio	2nd Selection Ratio	Sci-Clean	Sci-PA	Sci-SA	Sci-PA+SA	Avg.
Random	10%	10%	-	66.48	45.31	55.68	17.60	46.26
COINCIDE	10%	10%	-	66.24	48.14	56.57	18.24	47.30
ARDS (ours)	10%	10%	-	68.72	55.73	68.86	52.35	61.42
ARDS (ours)	10%	30%	33.33%	71.10	60.14	71.15	56.42	64.70

Q3. Typos correction

We appreciate the reviewer for pointing out these typos. We have revisited and corrected the typos in our manuscript.

Q4. Clarification of limitation discussion

We appreciate the reviewer's feedback, and we kindly refer to Appendix A for the limitation discussion.