DELIFT: Data Efficient Language model Instruction Fine-Tuning

Thank you for your detailed review and for highlighting the strengths of DELIFT. We also appreciate the suggestion to discuss the "Entropy Law" paper. This paper provides an interesting perspective by leveraging compression measures to identify optimal subsets, which is somewhat analogous to using semantic embeddings with the Facility Location function in our method.

However, our approach differs fundamentally in several ways:

1. Utility vs. Compression-Based Measures: While the entropy-based method identifies representative subsets using bit compression metrics, such measures focus primarily on the compactness and redundancy of the data. Similarly, semantic embedding-based approaches capture surface-level diversity but often fail to account for the utility of a sample in enhancing model performance. In contrast, DELIFT directly employs a utility-based kernel to evaluate the contribution of a sample to downstream performance, resulting in empirically superior outcomes (e.g., as demonstrated in our results comparing DELIFT to DELIFT (SE)).

2. Theoretical and Empirical Strength of Facility Location: The Facility Location function used in DELIFT has well-established theoretical guarantees, including provable approximation bounds for submodular maximization, ensuring the selection of diverse and representative subsets. Empirically, our results further validate that the subsets selected using DELIFT outperform those chosen by methods relying on compression or semantic embeddings in terms of both efficiency and efficacy.

3. Versatility Across Fine-Tuning Stages: Another key distinction is the scope of application. While the entropy-based approach targets the selection of representative subsets from a dataset, it does not offer the flexibility to address specific needs such as selecting new, complementary samples for continual fine-tuning. DELIFT, on the other hand, is specifically designed to optimize subset selection across diverse fine-tuning stages, including instruction tuning, task-specific fine-tuning, and continual fine-tuning, ensuring both theoretical robustness and practical applicability.

We have incorporates these clarifications into the revised related work section to better position DELIFT in the context of related approaches.

We thank the reviewer for their time. Please let us know of any more questions and concerns you might have - we are happy to answer them. If we addressed your concerns sufficiently, we would appreciate to see this reflected in an increase of rating scores. Thank you again!

We thank the reviewer for their thoughtful feedback and for taking the time to review our work. If there are any remaining concerns or additional points you'd like us to address, we would be happy to do so. If our revisions have sufficiently addressed your feedback, we would appreciate your consideration of this in your evaluation. Thank you again for your time and effort!

Q3: "analysis of diversity-based selection methods"

A3. To address the need for additional diversity-based comparisons, we included DEFT-UCS as a baseline, which uses sentence embeddings with centroid-based clustering to select diverse subsets. As shown in Tables 10 and 11, DELIFT significantly outperforms DEFT-UCS (LAJ scores: 2.98 vs 2.59), highlighting that our utility-based kernel is more effective at identifying informative samples than purely embedding-based clustering approaches.

To better understand how different methods select examples for in-context learning, we analyze the top-2 examples chosen for the query "What are popular street foods from Maharashtra in India?. The experimental setting is the same as Table 11: (Llama-3.2-3B, Use Case 1, Mix-Instruct). Due to the space limitation, just the instruction/inputs are shown.

Table 13: Top-2 ICL examples from each baseline given the input "What are popular street foods from Maharashtra in India?"

The analysis reveals several key patterns:

1. Task Alignment: Most methods except Random and DEFT-UCS select a recommendation-focused first example, but vary significantly in relevance:

   • DELIFT maintains food-domain focus through restaurant recommendations and recipes
   • DEFT-UCS drifts to general tourism and language tasks
   • DELIFT (SE) and SelectIT stay with recommendations but lose food context in second example

2. Complementary Information: Methods differ in second example selection:

   • DELIFT combines restaurant recommendation with vegetarian recipes, covering both general and specific food queries
   • DELIFT (FL) maintains food context but with less recommendation focus
   • Other methods either repeat recommendation structure (SelectIT) or select topically unrelated examples

This analysis demonstrates how DELIFT's utility-based selection achieves better domain relevance and information diversity compared to both traditional clustering (DEFT-UCS) and embedding-based approaches. We will include this comprehensive comparison in the revised paper.

Q4: "analysis of different Facility Location variants"

A4. Thank you for this insightful suggestion. We conducted a thorough ablation study comparing vanilla Facility Location (FL) with our specialized variants across different fine-tuning stages. As shown in Table 11, for task-specific fine-tuning, DELIFT with FLMI (Facility Location Mutual Information) achieves a 2.61% improvement over DELIFT with FL only (LAJ scores: 2.71 vs 2.63). The qualitative analysis in Table 13 further illustrates this difference:

• FL Only: Selects generally food-related examples ("vegetarian recipes", "cooking-related words")
• FLMI: Selects more targeted examples combining recommendations and food context ("restaurant recommendation", "vegetarian recipes")

This improvement validates our design choice of using specialized FL variants for different stages:

• FL: Promotes general diversity
• FLMI: Adds mutual information term to align with target task
• FLCG: Incorporates conditional gain for continual learning

We will expand the ablation analysis in our revision to demonstrate the benefits of each specialized variant across all fine-tuning stages.

Q5: "What is $s_{ij}$?"

A5. In our paper, the term $s_{ij}$ can refer to two different kernels, depending on the context:

1. Utility-based kernel ($UF_{ij}$): This is the primary kernel used in our proposed DELIFT method. It quantifies the utility gain from selecting a sample, based on its contribution to improving the model’s performance.

2. Pairwise similarity kernel: This kernel is used in our baseline method, DELIFT (SE), as discussed in Section 4.2, point (4). It measures the semantic similarity between samples.

The current version of DELIFT employs a static subset selection process, where uninformative data examples are pruned at the beginning of the fine-tuning stage. The goal of this approach is to maximize data efficiency by minimizing the number of different samples the model encounters during training. To achieve this, the utility-based kernel $UF_{ij}$ is computed once, before fine-tuning begins.

However, we acknowledge that a dynamic subset selection approach—where utility values and subsets are updated regularly throughout training—could potentially improve performance. This dynamic approach, however, would increase the number of samples the model processes during training, as well as the computational costs associated with subset selection. We view this as an interesting direction for future work, as it would allow the selection process to adapt to the evolving model, potentially improving its ability to learn from the most informative samples at each stage of fine-tuning.

Q6: "How is the ground truth distribution determined?"

A6. Thank you for the insightful question. The objective of our fine-tuning process is to align the model's predicted response with the ground truth. One effective way to achieve this is by matching the token distribution of the language model to that of the ground truth.

In an ideal scenario, where the language model generates the exact correct response, the token distribution would assign a probability of 1 to the correct token and 0 to all other tokens in the vocabulary. Therefore, our "ground truth token distribution" is represented as a vector of 1’s, which reflects the desired probabilities for the correct token.

We will provide further clarification on this in the revised version of our paper.

Q7: "The template is modified, lacks line numbers"

A7. Thank you for pointing that out. I inadvertently used an older template without line numbers. We have since updated the manuscript to include the latest template with line numbers. Apologies for any confusion or inconvenience caused.

Q8: "Why instruction-finetuned models instead of pre-trained models?"

A8. This is a valid concern. For more details, please refer to Tables 10 and 11 in our general comment, where we present experiments using Llama-3.2-3B and Mistral-7B-v0.1 models. The results from these experiments are consistent with those presented in the paper, highlighting the effectiveness of DELIFT over all considered baselines.

We thank the reviewer for their time. Please let us know of any more questions and concerns you might have - we are happy to answer them. If we addressed your concerns sufficiently, we would appreciate to see this reflected in an increase of rating scores. Thank you again!

Thank you for your response and the appreciation of our solution towards a timely research problem.

Q1: "ICL gains and fine-tuning improvements"

A1. Our utility metric $UF_{ij}$ is grounded in a well-established framework based on the distance between probability distributions. Specifically, for a ground truth distribution $GT_i$ and the model's predicted probabilities $p(y_i | x_i)$, we define the utility metric as:

$UF_{ij} = d(GT_i,\, p(y_i \mid x_i)) - d(GT_i,\, p(y_i \mid x_i, x_j, y_j)),$

where $d(\cdot, \cdot)$ represents the distance metric between the distributions; $GT_i$ assigns probability 1 to the sequence $y_i$ and 0 to all other sequences; $p(y_i \mid x_i)$ is the model's predicted probability for $y_i$ given only the input $x_i$; and $p(y_i \mid x_i, x_j, y_j)$ is the predicted probability for $y_i$ when the model is provided with $(x_j, y_j)$ as an in-context example along with $x_i$.

Theoretically, when using Kullback-Leibler (KL) divergence as the distance metric $d$, we prove that $UF_{ij}$ is equal to the conditional pointwise mutual information (PMI) between $y_i$ and $(x_j, y_j)$ given $x_i$. Specifically, we have:

For the detailed proof and formal derivation, please refer to Theorem 1 in the revised manuscript. This theoretical connection directly links our utility metric to an information-theoretic measure, demonstrating that $UF_{ij}$ quantifies the informational gain from in-context examples.

For practical implementation and computational efficiency, we adopt the Euclidean distance as the distance metric in place of KL-divergence:

where $\mathbf{1}$ is a vector of ones corresponding to the ground truth distribution $GT_i$, and $\left\| \cdot \right\|_2$ denotes the Euclidean norm.

In-context learning (ICL) provides an efficient means to estimate these probabilities without incurring the computational costs associated with gradient-based methods. Despite using a simpler distance metric in practice, the utility metric retains its connection to information theory and serves as a robust measure of the impact of in-context learning on model behavior. Our experimental results across three diverse use cases empirically validate that the ICL-derived utility metric correlates with fine-tuning gains, confirming that the utility derived from ICL can be a reliable predictor of improvements in fine-tuning.

Q2: "utility gain reflecting sample diversity and model changes"

A2. Let us clarify two key aspects of DELIFT's utility-based selection:

Sample Diversity: Rather than relying on semantic similarity, our utility kernel captures information-based diversity - selecting samples that provide complementary information for improving model performance. This is evidenced by DELIFT consistently outperforming DELIFT (SE), which uses only semantic embeddings, demonstrating the advantage of model-based diversity over purely semantic diversity.

Static vs. Dynamic Selection: DELIFT currently performs one-time subset selection before fine-tuning, focusing on data efficiency through initial pruning. While the utility values of samples may change as the model is fine-tuned on the selected subset, adapting the selection during training introduces additional challenges: (1) recomputing utilities after each update is computationally expensive, (2) frequent changes to training data may impact convergence, and (3) maintaining consistent performance across training requires careful balance between exploration and exploitation. We leave the development of efficient adaptive selection strategies as future work, particularly for continual learning scenarios where the model's capabilities evolve significantly over time.

Q4: "task-specific fine-tuning on reasoning datasets"

A4. Thank you for this valuable feedback about evaluation methodology. We acknowledge your concerns about MT-Bench's small size and the MMLU performance drop. To address this, we conducted extensive experiments with GSM-8k, a comprehensive mathematical reasoning benchmark. As shown in Table 11, DELIFT demonstrates strong performance - achieving 2.71 LAJ score with only 30% of the data, compared to 2.85 for full data, while consistently outperforming all baselines across both ICL and QLoRA settings. Importantly, even when starting from a relatively low baseline performance (LAJ: 1.36), DELIFT-selected subsets enable significant improvements, validating our approach for task-specific fine-tuning on structured reasoning tasks. We will expand our evaluation to include additional reasoning benchmarks like MATH Hard and ARC-Challenge in the final version.

Q5: "probability vectors from all predicted tokens?"

A5. Thank you for this clarifying question. For each token position in the sequence of length x, we compute the model's probability distribution over the entire vocabulary. Then, for each position, we extract only the probability assigned to the ground truth token, resulting in an x-dimensional vector. Using teacher forcing during generation ensures accurate probability estimation at each step. This approach is computationally efficient as it avoids working with full vocabulary-sized distributions while capturing how well the model predicts the correct tokens in sequence. The distance metric in equation 2 then compares this x-dimensional probability vector against a vector of ones (representing perfect prediction where each ground truth token should have probability 1).

Q6: "typo in Figure 2"

A6. Thank you for catching this error. We have corrected "LJA scores" to "LAJ scores" (LLM-as-Judge) in Figure 2 and verified all other instances in the paper for consistency.

Q7: "additional baselines comparison"

A7. Thank you for suggesting these relevant works. We have incorporated DEFT-UCS as a baseline in our experiments - see Tables 10 and 11. While DEFT-UCS uses sentence embeddings to select diverse subsets through centroid-based clustering, our results show that DELIFT significantly outperforms it (LAJ scores: 2.98 vs 2.59 in Table 10). This highlights that DELIFT's utility-based kernel is more effective at identifying informative samples than purely embedding-based approaches. Regarding the Text Quality paper, while it focuses on filtering grammatically incorrect samples for pre-training, our work addresses a fundamentally different aspect of data quality - selecting informative examples that improve model performance during fine-tuning. While grammatical correctness is important, DELIFT optimizes for examples that actively enhance the model's capabilities. We will include a detailed comparison with both approaches in our related work section.

We thank the reviewer for their time. Please let us know of any more questions and concerns you might have - we are happy to answer them. If we addressed your concerns sufficiently, we would appreciate to see this reflected in an increase of rating scores. Thank you again!

Thank you for your feedback, and recognition of our novel utility metric and improvement over existing sentence embedding-based metrics.

Q1: "effect of DELIFT when starting from base models"

A1. This is a valid concern. Please refer to Tables 10 and 11 in our general comment. We used DELIFT on Llama-3.2-3B and Mistral-7B-v0.1 base models and find a similar trend in performance as in the paper with the instruction-tuned models. This showcases the effectiveness of DELIFT to select informative subsets even for base models that are not instruction fine-tuned.

Q2: "diminishing gains with full model fine-tuning?"

A2. Thank you for raising this important question. While our original experiments focused on QLoRA due to computational constraints, we have now conducted additional experiments with full fine-tuning using the Facebook opt-125m model (see Table 12). Our choice to primarily use QLoRA was motivated by three key considerations:

1. Method Independence: DELIFT's selected subsets demonstrate consistent effectiveness across a spectrum of use cases - from parameter-free approaches like in-context learning to parameter-efficient (QLoRA) and full model fine-tuning. This versatility highlights that our subset selection strategy captures inherently valuable training examples regardless of how they are ultimately used.

2. Current Best Practices: Parameter-efficient fine-tuning (PEFT) methods like QLoRA represent the current state-of-the-art approach for adapting large language models, offering a practical balance between computational efficiency and performance gains.

3. Empirical Validation: Our new results with opt-125m show that DELIFT consistently outperforms all baselines for both QLoRA and full fine-tuning settings. Notably, the performance gap between QLoRA and full fine-tuning is minimal, suggesting that in many cases, QLoRA may be the more practical choice given its significantly lower computational requirements.

These findings demonstrate that DELIFT's benefits are robust across fine-tuning approaches while supporting our focus on QLoRA for large-scale experiments.

Table 12: opt-125m on Use Case 1: Mix-Instruct (equivalent to Table 1).

Q3: "computational cost of data selection algorithm"

A3. Thank you for this important question about computational efficiency. While DELIFT's utility computation requires O(n²) forward passes through the language model (where n is the total number of training examples), this is a one-time preprocessing cost that can be amortized across multiple use cases - the selected subsets can be stored and reused for different fine-tuning approaches (ICL, QLoRA, full fine-tuning), future model updates, and continuous dataset refinements. The actual subset selection process using submodular optimization is highly efficient, requiring only O(kn) computations (where k is the desired subset size) with pre-computed utility scores. In practice, the total computational cost of DELIFT (utility computation and subset selection) takes about 50% of the time needed for QLoRA fine-tuning on the full dataset, making it particularly cost-effective when considering full fine-tuning (which requires significantly more compute than QLoRA) or when maintaining and updating training sets over time. This makes DELIFT a valuable tool for teams looking to maintain high-quality, reusable training sets while optimizing their computational resources.

Thank you for taking the time to review our work. We note that your feedback aligns with some of the concerns raised by other reviewers, which we have addressed through additional experiments detailed in our revised manuscript and initial response. If you have any further questions or concerns, we would be happy to address them. If our revisions sufficiently address your feedback, we would appreciate your consideration of this in your evaluation. Thank you again for your efforts in reviewing our work!

Given my own experiences with supervised fine-tuning, I would always prefer full fine-tuning compared to data-efficient fine-tuning techniques, because you have to consider performance across all benchmarks at once and not look at just each benchmark separately. However, given hard resource constraints, the paper's methods, results, and additional results during the rebuttal stage have convinced me of the effectiveness of their approach, and therefore, I am raising my score to a 6.

Thank you Junyu for taking the time to read our work, and the pointer to your paper. Your two-step methodology of using the labeled data for ICL/PEFT and using the unlabeled data for selecting confident responses from an ensemble of models for PEFT is interesting!