Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning

We want to thank Reviewer fypD for their positive feedback and comments. We will address individual comments below.

Weakness 1: the effectiveness of this approach

Response: Thank you for your insightful suggestions！We present the average results from three independent trials (using three different random seeds) across two base model configurations. Clearly, our proposed methods still outperforms all baselines. Additionally, the corresponding t-tests results confirm that the performance improvements are significant.

The t-test results indicate that Self-Evolving Cleaning significantly outperforms RHO, showing statistically significant improvements with a p-value less than 0.05.

Weakness 2: Computational cost

Response: Thank you for highlighting this important issue. The computational costs associated with the token cleaning pipeline primarily consist of the training costs, akin to those of standard supervised fine-tuning (SFT), along with two types of additional inference costs. These additional costs stem from one base model and another reference model, both of which are used to calculate token-level influence scores.

Compared to the Rho-1 baseline, our two proposed methods do not incur any additional inference costs since the total data size for inference remains unchanged. For the Naive Fixed-Model Cleaning, we perform a one-shot inference on all samples simultaneously, mirroring the process used in Rho-1. For the Self-Evolving Cleaning pipeline, we simply segment the data pool into several partitions for independent inference using different reference models, i.e., (50k samples -> 10k (reference model 1), 10k (reference model 2), …, 10k samples). Consequently, the inference cost for the Self-Evolving Cleaning pipeline is also equivalent to that of Rho-1.

Weakness 3: technical contribution clarification

Response: Thank you for raising this concern. We would like to clarify that although the concept of masking unimportant tokens, initially introduced by Rho-1, is not new but remains very timely. Based on this, our work makes several novel contributions:

• Self-Evolving Cleaning: Our iterative approach that progressively refines token selection is entirely new and shows superior performance.
• Theoretical Framework: We provide a rigorous analysis of when and why token cleaning outperforms full-token training, with error upper bounds that explain the observed Matthew effect.
• Noisy-Label Perspective: We reframe token selection as a noisy-label problem, providing a new theoretical lens that connects token cleaning to a rich body of work on learning with noisy labels. These contributions extend beyond prior work (Rho-1) on token selection, offering both theoretical insights and practical improvements.

We want to thank Reviewer wKJy for their positive feedback and comments. We will address individual comments below.

Weakness 1: how accurate is using influence function to assess the quality of token itself

Response: Thank you for raising this important question. We'd like to clarify with the following points:

• The mentioned work made the claim because of the violations of required theoretical assumptions (e.g., positive-definite Hessian matrix) for language models. These assumptions are proposed for approximation. However, in this work, the method involves directly working with the difference in loss values rather than making approximations or Taylor expansion. Thus, the accuracy of the influence function would not be affected.

Here is a more detailed comparison between our implementation of the influence function and the standard one who needs approximations. The standard influence form is defined as loss($\theta$) - loss($\theta’$), given that the $\theta$ model is the current model and $\theta’$ is the model by removing a token, which requires some approximations to avoid model re-training. However, in our scenario, we do not have to counter-factually remove a token to calculate the influence since the training procedures are naturally adding tokens, which also have a form of loss($\theta$) - loss($\theta’$) but $\theta’$ can be seen as the current model (for example) and $\theta$ can be seen as the model after adding one token. Specifically, in the self-evolving clearing pipeline, we are iteratively using the influence function to identify informative tokens for the next-round training. Therefore, we can obtain those two models naturally with the original base model as $\theta$ and the current model as $\theta’$. Therefore, the only difference is we are adding tokens actively [1-2] while the standard approach is deleting tokens (i.e., re-training) [3].

• Besides, our theoretical analysis is based on a unified and analytical framework, which mainly explains the when and why the SFT with cleaned tokens outperforms with full tokens, regardless of the specific token quality evaluation metric used. As such, our theoretical insights are independent of the precise accuracy of the influence function, focusing instead on the overall reduction in token noise rates.

Finally, to address the reviewer's concern, there are more insights here. Intuitively, we hope that the base model, denoted as $\theta$, will perform similarly to the reference model, $\theta'$, on downstream tasks after fine-tuning. To approximate this criterion, we use the token loss calculated by the reference model, expressed as $\\ell(x_{i,j} | x_{i,:j}, \theta')$. If the token loss calculated by the base model, $\\ell(x_{i,j} | x_{i,:j}, \theta)$, is significantly higher than that calculated by the reference model, it indicates that this particular token requires fine-tuning to better align the base model with the reference model. Relying solely on $\ell(x_{i,j} | \boldsymbol{x}_{i,:j}, \theta)$ as a metric for token quality—by ranking and selecting tokens based on this loss—would only consider token-level information and neglect the task-specific insights provided by the reference model.

[1] Estimating Training Data Influence by Tracing Gradient Descent, NeurIPS’20.

[2] Fairness without harm: an influence-guided active sampling approach, NeurIPS’24.

[3] Understanding black-box predictions via influence functions, ICML’17.

Weakness 2: larger-scale model performance evaluation

Response: Thank you for your valuable feedback. Given our constraints with GPU resources and time, we utilized LLaMA-2-13B-hf as the base model to assess the efficacy of our proposed methods. All experimental settings are consistent with the original settings. The results demonstrate the superiority of our methods.

We would like to thank Reviewer dD3A for the time and effort. We will address individual comments below.

W1: evaluation benchmarks

Response: The benchmarks we used are standard in SFT studies (see Related Work) and assess a broad spectrum of capabilities, including question answering, reasoning, and multilingual comprehension, not just classification. While some benchmarks use multiple-choice formats, they test the model's generative abilities and knowledge without converting it into a classification model.

For your concern, we use two popular LLM-Judge benchmarks, MT-Bench and Vicuna-bench, to evaluate our model generations (the higher numbers indicate the better results). These results highlight the superiority of our methods.

W2: Additional baseline STAFF

Response: We replicated the STAFF baseline with pruning rates of 40%/60%. Due to the absence of a comparable smaller model for Mistral-7B, we assessed STAFF using two LLaMA base models, using LLaMA-3.2-1B to compute speculative scores. While STAFF remains a strong baseline, yet Self-Evolving Cleaning shows competitive results. For instance, with LLaMA-3.1-8B, our method surpasses STAFF in average performance (59.20 vs. 57.0/56.4). Although STAFF performs better on BoolQ, Self-Evolving Cleaning consistently improves across various tasks. In our revised manuscript, we will cite this reference and discuss.

W3: the necessity of manually selecting uninformative tokens

Response: We clarify the distinction between uninformative and noisy tokens as follows. In the SFT phase, every response token is labeled as 1, making it difficult to identify which tokens should be included for task-specific knowledge and which excluded for higher knowledge density. As a result, traditional labels are considered noisy, consisting of informative tokens (correctly labeled as 1) and uninformative tokens (incorrectly labeled as 1 but should be 0), with the latter corresponding to label errors.

We acknowledge that while uninformative tokens have low loss, their impact on model convergence cannot be ignored. These tokens may lead the model to make trivial, non-task-specific predictions. Our findings, supported by curriculum learning literature [1], show that while "easy" patterns are learned first, "difficult" patterns require more focus. Therefore, adjusting the weights of these tokens is crucial for optimizing downstream task performance.

[1] A Survey on Curriculum Learning, TPAMI'21.

Q1: Computational cost

Response: There are two types of additional inference costs in our method: one associated with the base model and the other with the reference model. Besides, there is also training cost for the reference model on a small subset (10k). Compared to the baseline Rho-1, our approach does not incur any extra inference costs. This is because the Self-Evolving Cleaning pipeline merely divides the data pool into several partitions for independent inference, i.e. (50k samples -> 10k, 10k, …, 10k samples).

Q2: Influence function

Response: The standard influence form is defined as loss($\theta$) - loss($\theta’$), given that the $\theta$ model is the current model and $\theta’$ is the model by removing a token. However, in our scenario, we do not have to counter-factually remove a token to calculate the influence since the training procedures are naturally adding tokens, which also have a form of loss($\theta$) - loss($\theta’$) but $\theta’$ can be seen as the current model (for example) and $\theta$ can be seen as the model after adding one token. Specifically, Self-Evolving Cleaning iteratively use the influence function to identify informative tokens for the next-round training. Therefore, we can obtain those two models naturally with the original base model as $\theta$ and the current model as $\theta’$. Therefore, the only difference is we are adding tokens iteratively [1] while the standard approach is deleting tokens (re-training) [2].

[1] Estimating Training Data Influence by Tracing Gradient Descent, NeurIPS’20.

[2] Understanding black-box predictions via influence functions, ICML’17.

We want to thank reviewer W8kk for the positive feedback! We will address individual comments below.

While the theoretical analyses is not fully precise, e.g in Sections 5.1 & 5.2 it makes a few trivial observations such as the challenges caused by the noise in tokens...

Response: We agree that LLM learning dynamics and convergence are much more complicated than our analytical framework, but our analyses in Section 5, form a complete theoretical framework based on error upper bounds that provides valuable support for our claims. Our main claim is that the two proposed token cleaning methods can effectively improve LLM SFT performance. Our theoretical analyses support this claim in two important ways:

Explaining why the proposed methods work:

• Theorem 5.1 and Corollary 5.2 establish an analytical framework that precisely describes when token cleaning outperforms using all tokens through the trade-off between data quality (η) and data quantity (M);
• Section 5.2 analyzes the characteristics of Fixed-Model Cleaning, explaining why this method provides stable but bounded improvements
• Section 5.3 analyzes the dynamic behavior of Self-Evolving Cleaning through the Matthew Effect, explaining why this method can produce more significant improvements in certain scenarios

Provide practical guidance for method selection and application:

• Clearly identifying when Self-Evolving methods might lead to performance degradation (G2 group data)
• Providing theoretical basis for selecting appropriate cleaning methods in different scenarios
• These alerts and guidance are validated in our experiments, as seen in Table 2 where performance changes across different tasks align well with theoretical predictions

We agree that while some analyses provide foundational observations, our theoretical framework offers deep insights into method behaviors, supporting our claims and guiding practical applications. In the revised paper, we will clarify the links between theory and experiments to explicitly highlight the importance of the theoretical aspects.

However, I would like to note that the theoretical claims do not explicitly support the proposed approach...

Response: We agree that our theoretical analyses primarily serve as intuitive explanations rather than explicit proofs for our proposed approaches, and they are indeed based on certain simplifying assumptions. We acknowledge that the experimental results constitute the primary evidence supporting our claims about the effectiveness of our token cleaning methods. The theoretical analyses were intended to complement these empirical findings by:

• Providing a conceptual framework to understand when and why token cleaning can be beneficial
• Offering explanations for the observed performance patterns across different methods
• Helping practitioners make informed decisions about method selection in various scenarios

We appreciate the reviewer's perspective that this does not diminish the overall support for our claims. In the revised manuscript, we will clarify the role of the theoretical analyses as providing intuitive explanations and insights, rather than positioning them as rigorous proofs of our methods' effectiveness.

How do you think token cleaning effects the supervised fine-tuning efficiency, both with respect to the data requirements and number of training iterations?

Response: Token cleaning affects supervised fine-tuning efficiency in several important ways:

• Regarding data efficiency, our approach selects only 60% of the most informative tokens rather than utilizing all tokens. This effectively reduces the data requirement by focusing the model's learning on tokens that contribute most significantly to performance. For our self-evolving cleaning method, we divide the total data samples into several chunks, each representing one iteration phase, which allows for progressive refinement of token selection.
• Despite this data partitioning for self-evolving cleaning, the number of training iterations remains consistent with standard SFT. Therefore, token cleaning does not impact the efficiency of SFT in terms of training iterations.

However, it's important to note that compared to standard SFT, token cleaning does not achieve computational speedup because the next-token prediction $x_{ij}$ still relies on the context formed by previous tokens $x_{i,:j}$. The current implementation simply masks out uninformative tokens to ignore their token loss, which is simple and compatible with all training paradigms. Existing token-level approaches, including RHO and our methods, mainly focus on performance efficiency rather than data (token) training efficiency. In practice, the GPU memory usage of these methods remains the same as when using full tokens. Investigating and enhancing token training efficiency represents a promising direction for future research, such as exploring ways to skip uninformative token memory occupation altogether.