NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric

We thank the reviewer for highlighting the importance of our work on data efficiency in instruction tuning and for finding the paper easy to understand. Below are clarifications to address your concerns:

1. Since discrepancy between NTP loss & eval. metrics has been shown in previous works (lines 380-1 L for refs.), it explains why we only consider a single experiment to further validate this claim. Nevertheless, we conduct experiments on 3 other datasets with different reward functions (eval. metrics) to show that the discrepancy between loss & metrics can be generalized. The results in link are similar to that in Fig.1: checkpoints with minimal loss (highest negative loss) do not correspond to checkpoints with the best performance (measured by the eval. metric) (lines 394-6 L); the performance can continue to increase even if the negative loss decreases.
2. Our work can be motivated using two limitations of loss-based data selection: (a) discrepancy between NTP loss & eval. metrics as discussed in point 1 (lines 19-43 R) & (b) dependency on high-quality labels which can be unavailable in practice (lines 149-51 L). In contrast, NICE can select training data to directly optimize commonly used eval. metrics of generation tasks (lines 46-9 R) & alleviate the dependency on validation labels when the eval. metric is independent of labels (lines 64-8 L).

2.1. High-quality labels (ground-truth responses) are often unavailable or available only in small quantities for generation tasks. So, a common practice is to use an LLM judge or train a reward model based on the small quantity of high-quality labels, and assume they are reliable, high-quality metrics that generalize to other inputs. Thus, this does not contradict the importance of high-quality labels. In fact, prior works have shown that these eval. metrics can track the response quality & align well with humans (Bai et al. 2022; Dubois et al., 2024; Stiennon et al., 2020; Zheng et al., 2023) and are hence widely adopted for evaluating generation quality.

2.2. Tab.D below shows the performance of LESS using GPT-generated labels (as suggested, 2nd row), which is generally worse than LESS + true labels (1st row) and our approaches. Hence, the performance of loss-based data selection is hurt by the unavailability of true labels, highlighting their importance. Furthermore, using GPT-generated labels alone does not address the issue discussed in point 1.

3. Advantage of NICE (policy gradients) over LESS (loss-based gradients) is supported by experimental evidence in Tab.3: NICE generally outperforms LESS (lines 321-9 L). About the concern on MC sampling & the eval. metric $r$, note that the policy gradient is used to optimize $r$ with data selection and approximated by MC sampling. So, it won’t be meaningful if we consider MC sampling as a standalone component and remove the eval. metric (e.g., for an ablation study). To see this, not using the eval. metric is equivalent to treating all sampled responses equally, i.e., $r$ is a constant function, resulting in a policy gradient of 0, as derived in link.

4. We use Johnson-Lindenstrauss random projections (see lines 275-8 L).

5.1. In Tab.D above, we have provided the results of LESS + GPT-generated labels, making the comparison fair. However, as mentioned in point 2.2, LESS is using the true labels to outperform that with GPT-generated labels and thus has an unfair advantage (yet poorer performance) over our approach in TLDR, RLHF & HumanEval (lines 278-81 R). Moreover, NICE (without AMC) already outperforms LESS without the help of additional models (Tab.3).

5.2. To clarify, our “task-agnostic” & “task-specific” settings pertain to the data preparation stage (to form the initial pool of training data without & with the knowledge of the target task, resp.) prior to data selection (lines 245-8 R, 255-7 R). For data selection, this work focuses on targeted (task-specific) data selection (lines 102-5 R). Hence, all our baseline choices are fair and valid. We will clarify this in the revised paper. In addition, we have included 2 more baselines TSDS & DSIR in Tab.C of reviewer d8As’s rebuttal.

6. While NICE incurs more computational cost due to MC sampling, this trade-off is justified by not needing validation labels—a key motivation of this work—and its improved performance over other methods (Tab.3). We defer the computational analysis to reviewer waTE’s rebuttal.

We will include the above discussions in the revised paper. We hope we have addressed your concerns and improved your opinion of our work.

We thank the reviewer for the positive feedback on our writing and the thoroughness of our methods and evaluations. We wish to make the following clarification.

Claims And Evidence:

To clarify, our “task-agnostic” & “task-specific” settings pertain to the data preparation stage (to form the initial pool of training data without & with the knowledge of the target task, resp.) prior to data selection (lines 245-8 R, 255-7 R). For data selection, this work only focuses on targeted (task-specific) data selection (lines 102-5 R).

However, methods based on DPP [https://arxiv.org/pdf/2402.02318] and submodular optimization [https://arxiv.org/abs/2401.06692] focus on non-targeted (task-agnostic) data selection without validation data, which is a different problem setting that falls outside the scope of our work. We will improve the clarity in the revised paper.

To improve the comprehensiveness of the experiments for targeted data selection, we add TSDS [https://arxiv.org/abs/2410.11303] (as suggested) and DSIR (highlighted in the suggested survey [https://arxiv.org/pdf/2402.16827] [1] ) for the setting where the initial pool of training data does not have the knowledge of the target task (our "task-agnostic" setting). The results are shown in Tab. C below.

Across multiple datasets and models, our method generally continues to outperform the newly added baselines. Although DSIR performs the best on AlpacaEval with the LLaMA2-7B model, it selects data based on n-gram lexical feature matching (similar to BM25) between training and target distributions, independent of model-specific signals. As a result, it selects the same subset regardless of the model used for instruction tuning, which can be sub-optimal. We can see it is no longer the best with Mistral-7B on the same task. This highlights the advantage of our model-aware data selection strategy. We will incorporate these baselines and citations in the revised paper.

Experimental Designs Or Analyses:

We would like to clarify that for three tasks—TLDR, RLHF, and HumanEval—the ground-truth labels of the validation dataset are not used by our method, although they are available to the baselines (lines 305–18, L). For AlpacaEval, labels are provided to all approaches. Specifically, NICE relies solely on the probability of generated responses and the score from a reward function, as enabled by the policy gradient mechanism (lines 130–132, R). In these three tasks, the reward function does not require ground-truth labels. Specifically:

• For RLHF and TLDR, the reward function is a learned reward model that outputs scores based on the prompt and generated response (lines 160-162, L).
• For HumanEval, the reward is defined by whether the generated code passes unit tests, not requiring reference solutions (lines 245-250, L).

We will make this distinction clearer in the experiment section to avoid confusion.

Essential References Not Discussed:

We have checked the survey paper in https://arxiv.org/pdf/2402.16827 and added a related work DSIR as a baseline for comparison. We will incorporate the citations of the survey paper and works that are not directly comparable but related like DPP, SKILL-IT[2] in the related work section.

We thank the reviewer for the valuable comments. We hope the additional baselines make our comparison more thorough and improve your impression of our work.

[1] Xie, Sang Michael, et al. "Data selection for language models via importance resampling." Advances in Neural Information Processing Systems 36 (2023): 34201-34227.

[2] Chen, Mayee, et al. "Skill-it! a data-driven skills framework for understanding and training language models." Advances in Neural Information Processing Systems 36 (2023): 36000-36040.

We thank the reviewer for recognizing our insight and our RL-based data selection approach. Please find our responses below.

Essential References Not Discussed:

We have included TSDS (as suggested) and an additional baseline, DSIR in Tab. C of our response to Reviewer d8As. Our method outperforms the new baselines across diverse datasets and models.

Weaknesses:

1.1 Computation cost

Although Monte Carlo sampling increases the computational cost, this trade-off is justified by NICE not needing validation labels - a key motivation of our work. NICE fills a gap left by existing loss-based baselines by supporting data selection with unlabeled validation data in cases where the evaluation metrics are label-independent. Furthermore, we can observe the performance improvement over other methods in Tab.3.

Nevertheless, we provide a comparative analysis of the computational costs between NICE and LESS, showing NICE remains within a practical computational range: Tab. A lists the asymptotic complexity and wall-clock runtime (training is measured on a single H100 hour, others are on L40 GPU hours) for each stage in the data selection procedure. Tab. B highlights the validation gradient computation where NICE differs from NICE. Let $E, d, M$ denote number of epochs (saved checkpoints), dimension of the projected gradients, number of MC samples. Let $D_W, D_N, D_V$ denote the warmup, training, validation set. When the sizes of the validation set and the MC samples are small, NICE adds only marginal overhead to LESS (e.g., AlpacaEval).

1.2 Stability

We provide an ablation study with results in https://postimg.cc/Mf0N6b76, varying MC samples from 5 to 20 on the RLHF dataset for a more in-depth discussion on stability. Results show that increasing the number of MC samples $M$ generally lowers the standard deviation across runs with different seeds, indicating better stability. The benefit of reduced standard deviation diminishes as $M$ increases. This validates that our chosen $M$ provides a good trade-off, offering sufficient stability without excessive computation.

2. We ensure there is no data leakage in our experimental setup, since the test data has no overlap with the validation data. Additionally, we clarify that similar to LESS and TSDS, our work focuses on targeted data selection (lines 102-5 R), which explicitly assumes access to a validation set that is distributionally aligned with the target task. Our “task-agnostic” & “task-specific” settings in the experiments pertain to the data preparation stage (to form the initial pool of training data without & with the knowledge of the target task, resp.) prior to data selection (lines 245-8 R, 255-7 R).

We believe that selecting training data without the knowledge of the downstream evaluation data falls under a different setting—task-agnostic data selection—where alternative approaches based on perplexity or coreset are applicable. However, these directions are outside the scope of this work, focusing on targeted data selection.

3. NICE_AMC generally performs well under two conditions: (a) when a stronger model is available for the target task—one that can generate higher-quality Monte Carlo samples, and (b) when the training pool is sufficiently large, allowing better gradient alignment with NICE_AMC (lines 313–5, R). This explains its success in our task-agnostic (data preparation) setting with abundant data.

As for guidance on when to use NICE vs. NICE_AMC:

• NICE_AMC: a stronger assistance model is available + a large, diverse training pool can be leveraged.
• NICE: resources or training data are limited or when the base model is already powerful enough.

In the revision, we will include additional baselines and analyses, and clarify the use case of NICE_AMC. We hope the discussion above will address your concerns and improve your impression of our work.

We thank the reviewer for acknowledging the effectiveness of NICE and the soundness of our methods and evaluation.

Claims And Evidence:

We defer the computational analysis between LESS and NICE to our rebuttal for Reviewer waTE. For BM25, it is indeed an efficient retrieval method based on lexical matching, ranking training data by relevance to the validation data. However, BM25 is model-agnostic (i.e., it selects the same data for different models). Thus, a subset selected may perform well with one model but not necessarily be optimal with another (lines 286-94, R). While being model-agnostic contributes to BM25's efficiency, it also limits its performance. In contrast, NICE enables model-aware data selection, optimizing training data specifically for the target model’s validation performance.

Theoretical Claims:

The theoretical suitability of policy gradient for influence estimation can be justified as follows. Loss-based influence estimation methods (e.g., TracIn, influence function), which our method builds upon, estimate the influence of a training point on validation loss. In particular, these methods measure the influence via the “gradient of the validation loss” (the change in validation loss w.r.t. the model parameters / policy), which by the chain rule, can be combined with the the first-order gradient or the Hessian of the training loss (w.r.t. the model parameters). In contrast, we estimate the influence on validation performance, measured by evaluation metrics. However, since the evaluation metrics are non-differentiable, the policy gradient, which measures the change in validation performance caused by the corresponding training data point, is a direct replacement of the “gradient of the validation loss”. It also allows the estimating the influence of the training data via chain rule.

Moreover, we show elaborate the derivations for NICE (based on TracIn) and NICE_IF (based on influence functions) in lines 141–52 R and lines 201-19 L, respectively. Both of them also relies on the derivations shown in App A.5.1 and App A.5.2, respectively.

Therefore, our NICE is theoretically-grounded since it uses the principled influence estimation framework from prior work and extends it to calculate the influence of data on the non-differentiable metrics using policy gradient.

Experimental Designs Or Analyses:

Cost of GPT-4.

Note that NICE AMC is an optional enhancement—NICE itself does not require GPT-4. We list the projected GPT-4 cost for NICE AMC in the table below. The costs are low for the majority of the tasks, except for RLHF due to its large validation set (which can be addressed as below).

Use of open-source/smaller LLMs.

As suggested, to reduce cost, we can use high-performing open-source models. On the RLHF dataset, we use Qwen 2.5-7B/3B-Instruct for AMC. Both outperform NICE. Notably, even Qwen 2.5-3B performs better due to its better alignment training, despite its smaller size. These models offer comparable performance to GPT-4 without the API cost.

500 Monte Carlo samples on HumanEval.

We use 500 samples because HumanEval is challenging—correct generations are rare, and we aim to estimate pass@100, whose evaluation requires 100 generated code pieces. In contrast, other tasks are evaluated using one sample, and we use 20 samples (i.e., a multiplier of 1) to approximate expected performance under the current policy.

MC sample size vs. performance trade-off.

We perform experiments on HumanEval with MC sample sizes ranging from 200 to 500 (see Fig. in link), showing a clear performance improvement with more samples. We also analyze this trade-off in Sec. 4.5 (lines 374-84, R). Both HumanEval and RLHF results show performance can improve with sample size. On RLHF, 20 samples already yield strong performance, with additional gains as the size increases.

Other Comments Or Suggestions:

We provide qualitative examples in App. A.10 (pages 23–24). These examples illustrate that “harmful” data can include paraphrased versions of the questions or fail to provide useful information.

We will include additional experiments in the revised paper and hope our justifications will address your concerns and improve your opinion of our work.