Understanding Data Influence in Reinforcement Finetuning

Response to Reviewer rGmp

We truly value your acknowledgment of our efforts! Your time and contributions to the academic community are greatly appreciated. We are more than willing to answer any questions you might have!

---

Weakness-1: On the computation cost of RFT-DA

In the supplementary material (Table 3), we include a comparison with HVS on the MATH-500 dataset. Notably, HVS incurs virtually no additional time cost during the scoring phase. LIMR demonstrates comparable computational cost, as both rely solely on reward statistics collected during training and do not require any additional computation at inference. However, both methods exhibit significantly lower performance (Pass@1 of 83.6) compared to full-set training (86.2), limiting their practical utility in performance-critical scenarios.

In contrast, our proposed approach achieves substantially higher performance: 89.0 with RFT-DA and 88.3 with the faster variant, RFT-DA Fast. Despite these gains, the total computation time—including surrogate training, scoring, and subset training—is significantly lower than full-set training (192.2 hours for RFT-DA, 134.2 hours for RFT-DA Fast vs. 277.6 hours for full-set training). These results will be added to the main paper for clarity. It’s worth noting that the scoring time is measured using 8 GPUs which can be significantly reduced by using more GPUs. Next, we will introduce a revised version of RFT-DA that reduces the scoring phase time from several hours to 32 seconds.

RFT-DA Dynamic: A Faster and More Efficient Variant: To further reduce computation time for scoring, we introduce RFT-DA Dynamic, which eliminates the need for fixed model checkpoints by computing scores dynamically during surrogate training. Specifically, at iteration $t$, given a batch $B_t$ and a sample $z \in B_t$, we compute and store the inner product between the gradient of $z$ and the average gradient of the batch $B_t$. After training, we aggregate these stored inner products across all iterations in which each sample was involved to estimate its final score:

$D(z) \approx \sum_{t \in \{t^z_1, t^z_2, \ldots\}} \frac{2\eta_t}{N} \langle G_z^t, G_{B_t}^t \rangle,$

where $t^z_i$ denotes the iteration index of $i$-th time in which sample $z$ was used for training. This approach allows RFT-DA Dynamic to accumulate all scoring information on-the-fly during training, thus incurring negligible additional computation. The table below summarizes the comparison across methods in terms of scoring time using 8 GPUs and performance:

While RFT-DA Dynamic yields slightly lower accuracy than the original RFT-DA, it achieves comparable computational efficiency to HVS and LIMR, requiring less than one minute for scoring. Importantly, it still significantly outperforms HVS, LIMR and full-set training in terms of final model performance, making it a highly effective and scalable solution for data selection under budget constraints. We will include the above in the paper.

---

Q1: On the Selection of Settings and Sensitivity of the Budget.

That's a great question. This issue has also been raised by other reviewers. Here, I've restated a unified response, which I hope will address your concerns.

How was the 90% budget chosen? Rather than applying a fixed threshold on the RFT-DA score, which varies significantly across models and datasets, we opted for a more robust, generalizable strategy based on score summation rate. Specifically, we found through extensive experiments on GSM8K that selecting samples whose cumulative scores account for 90% of the total positive score sum yields the highest data efficiency, defined as the ratio of performance gain to the amount of data used. Please refer to the table below for detailed results.

This heuristic proved to be consistently effective across different tasks and model sizes, including experiments on MATH-500 and the Puzzle dataset. As a result, we adopted this 90% summation rate as a unified selection criterion throughout all experiments reported in the paper.

Sensitivity to budget selection. We also evaluated the sensitivity of our method to different data selection budgets. The results are provided in the supplementary material (Table 2 and the table below), under the following experimental settings:

• Setting 1 (MATH-500): We use the DeepSeek-R1-Distill-Qwen-1.5B model and evaluate Pass@1 accuracy on the MATH-500 benchmark.
• Setting 2 (GSM8K): We use Qwen-2.5-1.5B-Instruct as the base model and evaluate Pass@1 accuracy on the GSM8K validation set.
• Setting 3 (Puzzle dataset): We follow the Logic-R framework to generate 5,000 puzzle-style questions varying in difficulty and number of roles (2–5 roles). We then test generalization on 6–8 role puzzles using the same Qwen-2.5-1.5B-Instruct model.

Across all three settings, we consistently observed that setting the summation threshold to 90% yielded the best trade-off between performance and data size, and generalized well across datasets and tasks. This empirical consistency motivated our use of the 90% threshold in all main results reported in the paper.

---

Q2: Can RFT-DA be used on dynamic or continually evolving datasets?

This is a great suggestion, and thank you for raising it!

RFT-DA is indeed well-suited for dynamic or continually evolving datasets. As shown in Equation 5, the RFT-DA score measures the gradient alignment of a sample with respect to the gradient of the overall data. This formulation naturally supports incremental updates. When the dataset changes, for example, through the addition of new samples or the removal of existing ones, we can simply update the aggregate gradient accordingly by incorporating the gradients of new samples or removing the contributions of those no longer present. This allows the RFT-DA scores to be recomputed efficiently without retraining from scratch, making it highly adaptable to evolving datasets.

Response to Reviewer wKRB:

We sincerely appreciate your recognition of our work!. We are more than happy to address any questions you may have!

---

Weakness-1: Evaluation on More Benchmarks and Larger Models.

Experiments on More Benchmarks. We further conducted performance tests on GPQA and Arena-Hard! GPQA is a benchmark focused on assessing large models' expert-level knowledge comprehension and complex reasoning abilities, covering fields such as biotech and chemistry. Arena-Hard is an automatic evaluation tool for instruction-tuned LLMs designed to reflect whether the models meet human preferences. The model used for the experiments was DeepSeek-R1-Distill-Qwen-7B, and the datasets involved included DeepScaleR and a 50K subset of the Tulu-3-Preference dataset. The RL algorithm employed was Reinforce++, and the reward model used was Tulu-3-8B-RM. To ensure fairness, all methods were filtered to include only the top 90% based on summation rate.

Notably, RFT-DA continues to maintain its leading advantage! We will update these results in the revision.

Experiments on Larger Models. Thank you for your suggestion! We present the training results on the 32B (DeepSeek-R1-Distill-Qwen-32B) and 70B (DeepSeek-R1-Distill-Llama-70B) models. Here, we directly apply the subsets chosen in the 1.5B model experiments for RFT on these larger-scale models. The performance advantages of RFT-DA remain consistent with those observed on the 1.5B and 7B models, outperforming both HVS and LIMR, as well as the training results on the full set!

---

Weakness-2: On the typos.

Thank you for pointing that out! We have conducted a thorough proofreading of the paper and corrected the typos in the revision.

---

Q1: On the Selection of Settings and Sensitivity of the Budget.

That's a great question.

How was the 90% budget chosen? Rather than applying a fixed threshold on the RFT-DA score, which varies significantly across models and datasets, we opted for a more robust, generalizable strategy based on score summation rate. Specifically, we found through extensive experiments on GSM8K that selecting samples whose cumulative scores account for 90% of the total positive score sum yields the highest data efficiency, defined as the ratio of performance gain to the amount of data used. Please refer to the table below for detailed results.

This heuristic proved to be consistently effective across different tasks and model sizes, including experiments on MATH-500 and the Puzzle dataset. As a result, we adopted this 90% summation rate as a unified selection criterion throughout all experiments reported in the paper.

Sensitivity to budget selection. We also evaluated the sensitivity of our method to different data selection budgets. The results are provided in the supplementary material (Table 2 and the table below), under the following experimental settings:

• Setting 1 (MATH-500): We use the DeepSeek-R1-Distill-Qwen-1.5B model and evaluate Pass@1 accuracy on the MATH-500 benchmark.
• Setting 2 (GSM8K): We use Qwen-2.5-1.5B-Instruct as the base model and evaluate Pass@1 accuracy on the GSM8K validation set.
• Setting 3 (Puzzle dataset): We follow the Logic-R framework to generate 5,000 puzzle-style questions varying in difficulty and number of roles (2–5 roles). We then test generalization on 6–8 role puzzles using the same Qwen-2.5-1.5B-Instruct model.

Across all three settings, we consistently observed that setting the summation threshold to 90% yielded the best trade-off between performance and data size, and generalized well across datasets and tasks. This empirical consistency motivated our use of the 90% threshold in all main results reported in the paper.

---

Q2. What are the characteristics of samples that receive very high positive scores versus those that receive negative scores?

That's an excellent question! We found that after sorting the data in descending order based on the RFT-Score, the lower-ranked instances (indicating a negative impact) are primarily those where the model is consistently either completely correct or completely incorrect across all eight responses. In contrast, the higher-ranked entries are those where the model occasionally answers correctly but also makes mistakes.

This underscores the importance of including questions that the model can occasionally answer correctly during the RFT process. By reinforcing learning on such data, the model can thoroughly explore potential reasoning paths, both correct and incorrect. Then, the feedback from rewards helps the model quickly identify and learn these correct reasoning abilities! We showcase several such data cases on Pages 3 and 4 of the original supporting materials.

Dear Reviewer wKRB,

We sincerely thank you for your participation throughout the entire review cycle and for your valuable contributions to the academic community.

During the rebuttal period, we have made our best efforts to address your concerns, including conducting additional experiments on new data combinations and evaluating our method on benchmarks such as GPQA and Arena-Hard. We hope that these efforts have adequately addressed your questions and clarified the strengths of our work.

If possible, we would be deeply grateful if you could consider updating your evaluation to a higher score based on our responses and the new results.

Thank you again for your time and thoughtful feedback.

Sincerely,

The Authors

Response to Reviewer L7kr's Comments:

---

We sincerely thank the reviewer for the insightful feedback. We appreciate the constructive criticism and have carefully addressed the concerns raised regarding the computational complexity analysis and the overall presentation of the paper.

---

1. On the computational complexity analysis.

We will revise this! Regarding the definitions of NE, N, and C, please refer to Line 212 of the manuscript. Following your suggestion, we will clarify the definitions of N, C, and E to improve clarity.

Specifically:

• $N$ denotes the total number of training samples,
• $E$ is the number of training epochs,
• $C$ is the number of checkpoints selected for score computation.

Accordingly, the computational complexity of the surrogate training phase is $O(NE)$, while the scoring process incurs a complexity of $O(NC)$. Therefore, the overall computational complexity is $O(NE) + O(NC)$.

---

2. On the overall presentation:

We appreciate the feedback. We have carefully revised the manuscript to improve clarity, flow, and readability. Specifically:

(a). Line 121: This was a typo caused by a compilation error. We have corrected it: the correct expression should be "Z = {z_i = (x_i, y_i)|_{i=1}^N }". The {} notation is not properly rendered in math mode.

(b). The function $J(\cdot)$ denotes the expected advantage, and we will ensure it remains consistent with the definition provided in Equation (1).

(c).Equation (5) indeed does not involve $Z \setminus z$, and we will remove the reference to $Z \setminus z$ in Line 180. Additionally, we will explicitly include the time horizon $T$ in the summation notation for clarity.

(d).The incorrect section reference in Line 265 was due to a compilation error, and we will correct it in the revised version. And we have updated the reference to OlympiadBench: Chaoqun He, Renjie Luo, Yuzhuo Bai, et al., OlympiadBench, ACL 2024.

(e).Line 82 in the Supplementary Material should be $D^0(z) = 0$. Thank you for catching this typo, and we will correct it accordingly.

---

3. Speed Comparison with LIMR:

In the supplementary material (Table 3), we include a comparison with HVS on the MATH-500 dataset. Notably, HVS incurs virtually no additional time cost during the scoring phase. LIMR demonstrates comparable computational cost, as both rely solely on reward statistics collected during training and do not require any additional computation at inference. However, both methods exhibit significantly lower performance (Pass@1 of 83.6) compared to full-set training (86.2), limiting their practical utility in performance-critical scenarios.

In contrast, our proposed approach achieves substantially higher performance: 89.0 with RFT-DA and 88.3 with the faster variant, RFT-DA Fast. Despite these gains, the total computation time—including surrogate training, scoring, and subset training—is significantly lower than full-set training (192.2 hours for RFT-DA, 134.2 hours for RFT-DA-Fast vs. 277.6 hours for full-set training).

These results will be added to the main paper for clarity. It’s worth noting that the scoring time is measured using 8 GPUs, which can be significantly reduced by using more GPUs.

---

RFT-DA Dynamic: A Faster and More Efficient Variant:

To further reduce computation time for scoring, we introduce RFT-DA Dynamic, which eliminates the need for fixed model checkpoints by computing scores dynamically during surrogate training. Specifically, at iteration $t$, given a batch $B_t$ and a sample $z \in B_t$, we compute and store the inner product between the gradient of $z$ and the average gradient of the batch $B_t$. After training, we aggregate these stored inner products across all iterations in which each sample was involved to estimate its final score:

$D(z) \approx \sum_{t \in \{t^z_1, t^z_2, \ldots\}} \frac{2\eta_t}{N} \langle G_z^t, G_{B_t}^t \rangle,$

where $t^z_i$ denotes the iteration index of $i$-th time in which sample $z$ was used for training. This approach allows RFT-DA Dynamic to accumulate all scoring information on-the-fly during training, thus incurring negligible additional computation.

The table below summarizes the comparison across methods in terms of scoring time using 8 GPUs and performance:

While RFT-DA-Dynamic yields slightly lower accuracy than the original RFT-DA, it achieves comparable computational efficiency to HVS and LIMR, requiring less than one minute for scoring. Importantly, it still significantly outperforms HVS, LIMR, and full-set training in terms of final model performance, making it a highly effective and scalable solution for data selection under budget constraints.

Response to Reviewer s3Rt:

We sincerely appreciate your time and feedback! We hope our response addresses your concerns. If you have any further questions, please don't hesitate to let us know. We are more than happy to discuss further.

---

1. On the theoretical justification for the gradient alignment.

Great question! The use of gradient alignment, as formulated in Equation 5, is supported by both theoretical justification and empirical evidence.

First, as detailed in Section 3 of the supplementary material, Equation 5 provides a first-order approximation of the leave-one-out influence of a sample z on the RFT process. This influence, formally defined in Equation 4, quantifies how the removal of a specific sample affects the training outcome, thus enabling us to assess the importance of individual samples.

We further analyze the estimation error of the gradient alignment score in Proposition 1, where we show that our approximation of leave-one-out influence is more accurate than many existing influence estimation methods in supervised learning literature. This theoretical result underpins the validity of using gradient alignment as a proxy for sample importance.

Intuitively, in RFT, a sample whose gradient is well-aligned with the aggregate update direction (i.e., the average gradient of other samples) is more representative and contributes more constructively to training. This idea is not only theoretically motivated but also strongly supported by our empirical results. As shown in Tables 1, 2, and 3, our approach consistently outperforms baseline methods, including those using the full dataset—demonstrating notable performance gains with significantly reduced training data.

In summary, both the theoretical formulation and the experimental validation provide solid support for our gradient alignment assumption as a meaningful and effective criterion for sample selection in reinforcement fine-tuning.

---

2. More evaluation results on GPQA and ArenaHard.

During the rebuttal period, we conducted experiments on GPQA and ArenaHard. The model used for the experiments was DeepSeek-R1-Distill-Qwen-7B, and the datasets involved included DeepScaleR and a 50K subset of the Tulu-3-Preference dataset. The RL algorithm employed was Reinforce++, and the reward model used was Tulu-3-8B-RM. To ensure fairness, all methods were filtered to include only the top 90% based on summation rate.

Notably, RFT-DA continues to maintain its leading advantage! We will update these results in the revision.

Moreover, for larger model settings, we present the training results on the 32B (DeepSeek-R1-Distill-Qwen-32B) and 70B (DeepSeek-R1-Distill-Llama-70B) models. Here, we directly apply the subsets chosen in the 1.5B model experiments for RFT on these larger-scale models. The performance advantages of RFT-DA remain consistent with those observed on the 1.5B and 7B models, outperforming both HVS and LIMR, as well as the training results on the full set!

---

3. Comparison with O1-mini.

The results presented in Table 1 indicate that, at the 7B model scale, our method achieves performance comparable to that of o1-mini on AMC23 and Olympiad (89.4 vs. 90.0 and 65.4 vs. 67.2). Notably, on AMC23, our method even surpasses the performance of o1-mini (91.0 vs. 90.0). We will update our statement to reflect that our method can match or even exceed the performance of o1-mini on certain benchmarks.

---

4. On the necessity of a separate surrogate training and scoring phase.

This is a great question! We appreciate your suggestion! We will include this new version with improved speed in our revision. The separate surrogate training and scoring phases are the main contributors to the time consumption in our method when compared to HVS and LIMR. Following your suggestion, we have also implemented a dynamic version of conducting scoring during surrogate training as below.

RFT-DA Dynamic: A Faster and More Efficient Variant: To further reduce computation time for scoring, we introduce RFT-DA Dynamic, which eliminates the need for fixed model checkpoints by computing scores dynamically during surrogate training. Specifically, at iteration $t$, given a batch $B_t$ and a sample $z \in B_t$, we compute and store the inner product between the gradient of $z$ and the average gradient of the batch $B_t$. After training, we aggregate these stored inner products across all iterations in which each sample was involved to estimate its final score:

$D(z) \approx \sum_{t \in \{t^z_1, t^z_2, \ldots\}} \frac{2\eta_t}{N} \langle G_z^t, G_{B_t}^t \rangle,$

where $t^z_i$ denotes the iteration index of $i$-th time in which sample $z$ was used for training. This approach allows RFT-DA Dynamic to accumulate all scoring information on-the-fly during training, thus incurring negligible additional computation.

The table below summarizes the comparison across methods in terms of scoring time using 8 GPUs and performance:

RFT-DA Dynamic maintains comparable performance as RFT-DA while being much more efficient, making it a highly effective and scalable solution for data selection under budget constraints.