Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning

Dear Reviewer aFno,

We sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.

1. Ablation Study (MCTS, SPRE, DPO)

Component-wise contribution unclear: The proposed system integrates several components—MCTS, SPRE, and DPO—but their relative importance remains unclear. An ablation study removing or replacing each module (e.g., substituting MCTS with naive sampling) would help clarify which elements are essential to performance.

Process vs. outcome comparison not controlled: Search-R1 differs in model scale and data quantity. To isolate reward type effects, would it be possible to train a DPO model on outcome-based preferences from the same 5k questions?

Thank you for the valuable suggestion. As mentioned in Table 3, we conducted a comprehensive ablation study to evaluate the contributions of MCTS, reward design, and training methods, as detailed below:

1.1 Ablation study settings

Following your valuable advice, we added a controlled experiment replacing our process-level reward with a simplified final-answer-based reward (Outcome reward DPO). The results are as follows:

1.2 Performance Comparison

1.3 Key findings

(1) SPRE enhances performance compared to outcome-only reward designs. (2) MCTS-guided data generation contributes to stronger reward signals and sample efficiency, which boosts performance especially when combined with process-level reward supervision. (3) Base and SFT variants show limited gains, highlighting the importance of both reward quality and training method.

2. Penalty Analysis

Penalty coefficient \alpha is under-analyzed: Equation 1 includes a tunable penalty weight \alpha to encourage shorter paths. However, the paper does not analyze its effect on learning dynamics or final performance. Could the authors provide experiments or sensitivity analysis on different values of \alpha?

Thank you for your comment. Our reward design follows two key principles: (1) reasoning chains with higher accuracy should receive higher rewards, and (2) among chains with similar accuracy, shorter reasoning paths should be rewarded more. To achieve this, we set the penalty coefficient α within the range (0,1], choosing a value close to 1 to effectively balance these objectives.

3. Sample Efficiency

Effect of data selection on sample efficiency: The method achieves strong results using only 5k queries, compared to 90k used in Search-R1. This raises the question: to what extent does the performance stem from careful data selection rather than reward formulation? How were these 5k examples chosen, and could biased or curated selection influence the outcome?

Thank you for the thoughtful question. As mentioned on Page 5, Lines 165–167, we randomly sampled 3k examples from each of the three datasets, applied MCTS-based filtering, and retained 5k high-quality trajectories. Therefore, the performance gains are not due to curated data, but rather the effectiveness of our automated data generation pipeline. This pipeline adaptively selects diverse, high-reward trajectories through search and reward modeling, enabling strong performance with limited supervision.

4. Reward reliability

Does SPRE provide sufficiently discriminative reward signals for complex queries where multiple reasoning paths may be valid? Are there failure cases where SPRE assigns similar scores to semantically different strategies?

Thank you for your comment. SPRE provides sufficiently informative reward signals for training. While semantically different reasoning paths may receive similar SPRE scores, we address this by filtering preference pairs based on their reward gap, as described in Figure 5 and Lines 172–175: “To ensure high-quality preference data, we perform post-processing to remove duplicates and uninformative comparisons: (i) we discard preference pairs with identical response sequences, and (ii) pairs with a reward difference less than 0.01. After filtering, the final dataset consists of 4,603 questions and 13,289 distinct preference pairs.".

This ensures that retained comparisons exhibit clear differences, enhancing the discriminative strength of the supervision.

5. Label Annotation

How reliable are the preference labels used for DPO? Were the "better" and "worse" trajectories chosen purely via SPRE scores?

5.1 SPRE Annotation

Thank you for the comment. Preference labels are assigned purely based on SPRE scores, which prioritize both correctness and shorter reasoning paths. To ensure reliability, we filter out preference pairs with reward gaps below 0.01, and report the distribution of reward gaps in Figure 3(d).

5.2 Open-Source Dataset and Manual Inspection

The full dataset is available via a link in Appendix Table 5, and manual inspection confirms that SPRE-based rewards reliably reflect reasoning quality.

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Best regards,

The Authors

Dear Reviewer 2RBG,

We sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.

1. ReasonRAG vs. AirRAG

From a workflow perspective, an MCTS‑based RAG system has already been proposed in AirRAG (https://arxiv.org/pdf/2501.10053). The main contribution of this paper—using a UCB‑based stepwise selection of positive and negative examples for DPO FT offers limited novelty.

Thank you for the valuable comment. We agree that MCTS has previously been explored in RAG contexts. Our approach differs fundamentally from AirRAG in goals, methodology, and the role of MCTS. A comparison is summarized below:

While both systems utilize MCTS, ReasonRAG leverages it during training to enable reward-driven supervision, whereas AirRAG applies MCTS at inference to aggregate final answers via self-consistency. We sincerely appreciate the reviewer for highlighting this connection, as it helped us better position our contributions relative to existing work.

2. Step-Level vs. Outcome-Level Supervision

In terms of training loss, although the paper’s focus is on comparing PRM and ORM for agentic RAG FT, it applies identical DPO weighting to both procedural decision correctness and final‑output correctness. The manuscript does not explore how varying these weightings affects training outcomes, nor does it investigate which aspect has a greater impact on RAG system performance.

Given that the primary focus of your paper is the comparative impact of PRM versus ORM on LLM FT, it is advisable to enrich the PRM experiments, for example, by assigning distinct weights to procedural correctness and outcome correctness and systematically evaluating how these weightings affect downstream performance metrics.

Thank you for raising this important question. We agree that identifying the relative contribution of step-level (procedural) and answer-generation(outcome) supervision is critical to understanding what drives performance in agentic RAG systems.

Following your valuable suggestion, we conducted a controlled ablation by reducing the amount of training data used for either (1) step-level rewards or (2) answer-generation rewards to 50%. The variant settings are: (1) 50% Step-level Data: 4.5 step-level data + 4k answer-generation data (2) 50% Answer-Generation Data: 9k step-level data + 2k answer-generation data. The results are presented below:

2.1 Key findings

The experiment results indicates that reducing step-level data causes a more substantial drop. In contrast, cutting final-answer data leads to milder degradation. These results demonstrate that step-level supervision plays a more critical role in agentic RAG training.

3. Clarification of Figure 3

Figure 3 contains incorrect annotations, and its caption does not clearly explain the figure’s intended meaning.

The axis labels and their semantic descriptions in certain figures (e.g., Figure 3) lack sufficient clarity. A careful revision of these annotations is necessary to ensure that the figure’s intent and interpretation are unambiguous.

Response: Thank you for pointing this out. We apologize for the lack of clarity in the original caption and presentation. While we provided a description in the original manuscript (Page 5, Lines 180–189), we realize it may have been insufficient. We clarify the intention of Figure 3 below:

3.1 Pair Type Distribution

Figure 3(a) shows the distribution of preference pair types. The x-axis denotes combinations of accepted vs. rejected action types (A: answer generation, Q: query generation, E: evidence extraction), and the y-axis indicates the number of preference samples per type. This demonstrates that our dataset spans diverse comparison scenarios across multiple reasoning stages.

3.2 Reasoning Step Distribution

Figure 3(b) illustrates the distribution of reasoning iteration counts per query. The x-axis is the number of MCTS steps, and the y-axis is the number of samples. This reflects the varying difficulty of different multi-hop questions.

3.3 Token Counts Distribution

Figure 3(c) shows the distribution of response token lengths, where the x-axis is the number of tokens in generated responses and the y-axis is the sample count. This confirms our dataset captures both concise and lengthy trajectories.

3.4 Reward Gap Distribution

Figure 3(d) presents the distribution of reward gaps between preferred and rejected paths. The x-axis indicates the reward difference, and the y-axis shows probability density. Larger gaps indicate clearer supervision signals for preference learning.

We will correct them in the revised version. Thank you again for your careful reading and helpful suggestion.

4. Training on more LLM variant: Qwen2.5-3B

The paper does not specify which large language models were used at each baseline; if different LLMs were employed, the reliability of the results would be compromised.

Thank you for the valuable comment. As mentioned in Page 6, Lines 241–243, all baselines in our main experiments use Qwen2.5-7B-Instruct as the backbone model to ensure a fair comparison. Implementation details for each baseline are provided in Appendix F.2, and the necessary artifacts are listed in Appendix Table 5 (Page 20) for full reproducibility. To further assess the scalability of ReasonRAG, we additionally conducted experiments using a smaller model (Qwen2.5-3B-Instruct). Results are shown below:

4.1 Key findings

ReasonRAG consistently outperforms baselines even with smaller LLMs, demonstrating that our method generalizes well across model sizes.

5. Results of Additional Baseline: Search-O1

The set of baselines should concentrate on RAG frameworks that have undergone comparable ORM or PRM FT. Incorporating additional relevant baselines, such as RAG‑Gym, would further bolster the robustness and persuasiveness of your evaluation.

Response: Thank you for your suggestion. Following your advice, we have included the Search-O1 baseline from RAG-Gym, and the results are presented below:

5.1 Key findings

The experimental results demonstrate that ReasonRAG consistently outperforms both Search-R1 and Search-O1, further confirming the effectiveness of our approach.

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Best regards,

The Authors

Dear Reviewer WaUS,

We sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.

1. Gains from Sampling Number or Reward Design?

While the results are strong, the use of MCTS introduces additional sampling overhead. It remains unclear whether the performance gains stem primarily from the increased number of generated samples or from the specific reward computation method itself. A more controlled analysis would help clarify this distinction.

Thank you for your valuable comment. We conducted two controlled experiments to disentangle the effects of MCTS iteration count and reward design. The results are summarized below:

1.1 Key findings

(1) Reducing MCTS iterations from 64 to 32 per query results in a clear performance drop, indicating that sufficient sampling iteration improve data quality. As also observed in Luo et al. (2024), increasing the number of MCTS samples generally improves reasoning quality, but the marginal gain diminishes beyond a certain point. We adopt 64 iterations as a practical trade-off between performance and efficiency.

(2) We replaced our step-level MCTS reward with a final-answer-only reward and trained a variant (64 iters(ORL)). The notably lower performance of this variant confirms that the SPRE reward design yields greater gains.

[1]. Luo, Liangchen, et al. "Improve mathematical reasoning in language models by automated process supervision." arXiv preprint arXiv:2406.06592 (2024).

2. Stepwise Scaling on Cost and Accuracy

Moreover, the datasets used in their experiments involve no more than five steps per episode. It remains unclear how the computational cost and accuracy would scale as the number of steps increases.

How do computational cost and accuracy vary with the number of steps in a sample? Since the current datasets contain at most five steps, it would be valuable to understand how the method scales with longer reasoning trajectories.

We conduct a step-wise cost and accuracy analysis across both single-hop (PopQA) and multi-hop datasets (HotpotQA, 2WikiMultiHopQA). As shown in the table above:

2.1 Key findings

We observe consistent performance gains across both single-hop and multi-hop datasets as the number of reasoning steps increases. For single-hop QA, most gains are achieved in early steps, and further steps bring marginal improvements. Despite longer total inference times, our method mitigates latency by executing each reasoning step in parallel across all queries using a batched vLLM engine.

3. Computational Cost Comparison

What are the computational costs of the different algorithms listed in Tables 2 and 3? A comparison in terms of runtime or sample efficiency would help contextualize the performance gains.

Thank you for the helpful suggestion. We have added a runtime and efficiency comparison using 1,000 queries from the 2WikiMultihopQA dataset. The runtime analysis are shown as below:

3.1 Key findings

ReasonRAG demonstrates the best overall performance (Avg. EM/F1) among all evaluated methods, while maintaining lower latency than baselines with comparable or higher accuracy.

4. Training on more LLM variant: Qwen2.5-3B

Have you tried using different base models? It would be helpful to know whether the observed improvements are consistent across models of varying sizes or architectures.

Thank you for the valuable question. We conducted additional experiments with all baselines and our method implemented using Qwen2.5-3B-Instruct as the base model. The results are presented below:

4.1 Key findings

These results demonstrate that ReasonRAG generalizes well to smaller models, maintaining strong performance across tasks and outperforming several strong baselines. Thank you for your valuable feedback.

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Best regards,

The Authors

Dear Reviewer WaUS,

We hope this message finds you well. We would like to gently follow up regarding our submitted rebuttal. In our response, we have made every effort to thoroughly address the reviewers’ comments, clarify our contributions, and thoughtfully respond to the concerns raised.

If time permits, we would be truly grateful if you could kindly take another look at our submission and consider whether our clarifications may warrant a reevaluation. Your insights are highly valued, and we are more than happy to provide any further information if needed.

Thank you sincerely for your time and consideration.

Dear Reviewer ATvT,

We sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.

1. Training on more LLM variant: Qwen2.5-3B

Experiments use a Qwen2.5-7B-Instruct model only. It is uncertain whether ReasonRAG’s benefits translate to smaller or less capable LLMs.

To prove the generality of the Method it will be interesting if the authors can perform experiment using a simple llama-3.1 or 3.3 model against search-o1 and search-r1 on 2 unseen datasets only.

Thank you for the valuable comment. We conducted additional experiments by re-implementing all baselines using Qwen2.5-3B-Instruct as the backbone. The results are reported below:

1.1 Key findings:

(1) ​ReasonRAG remains effective when applied to smaller language models​, achieving strong performance with Qwen2.5-3B-Instruct and outperforming baseline methods across most datasets. (2) The performance gains are especially notable on multi-hop QA tasks such as HotpotQA and 2WikiMultihopQA, suggesting that ReasonRAG is particularly effective in complex reasoning scenarios.

2. Inference Efficiency vs. Efficacy

Although MCTS is used offline for annotation, the inference algorithm still involves iterative reasoning loops which may incur latency in real-time applications. An analysis of runtime cost per query would strengthen practicality claims.

Giving the run time cost, speed , searches done and tokens for the inference will significantly help the paper.

Thank you for your suggestion about runtime cost analysis. We have added a runtime comparison across methods based on 1,000 queries from 2WikiMultihopqa dataset. All experiments use the Qwen2.5-7B-Instruct model with the vLLM inference engine for efficient batched decoding, and baseline implementations follow the open-source FlashRAG framework:

2.1 Key findings:

ReasonRAG demonstrates the best overall performance (Avg. EM/F1) among all evaluated methods, while maintaining lower latency than baselines with comparable or higher accuracy.

3. OOD Generation

While the authors claimed that they used 2 OOD datasets (unseen) it is not very convincing see the gains are marginal over the search-r1 in those 2 datasets (Bamboogle, MusiQue).

Thank you for your valuable feedback. We agree that gains on OOD datasets are an important indicator of generalization. While the performance improvements on Bamboogle and ​MusiQue over Search-R1 are relatively smaller than on in-domain datasets, it is worth noting that ReasonRAG is trained with only ​5k queries​, whereas Search-R1 is trained on ​90k queries​. Despite this 18× smaller training set, ReasonRAG achieves ​stronger overall performance​, including notable improvements on Bamboogle and competitive results on MusiQue, demonstrating both generalization and data efficiency​.

4. Results of Additional Baseline: Search-O1

And No recent baselines like search-O1.

Q1: Take care of another baseline (i.e search-o1).

We have included the performance of ​Search-O1​ as shown in the table below:

4.1 Key findings:

ReasonRAG outperforms both Search-R1 and Search-O1 across nearly all datasets and metrics, further validating the effectiveness and robustness of our approach.

5. Clarification of ReasonRAG Base and SFT

The base & SFT performance of the ReasonRAG is closer or same as standard RAG (table-3 row 1-2 and table 2 zero shot section) this requires further analysis on what exactly is happening.

Thank you for your insightful question. To clarify the differences, we present the comparison results below:

Why ReasonRAG-Base falls short: Unlike standard RAG which passively uses retrieved documents, ReasonRAG requires active agentic capabilities—planning retrieval strategies, issuing targeted queries, and orchestrating multi-step reasoning. The base model lacks these learned behaviors, making it more similar to untrained agentic baselines like Search-O1 than to standard RAG, resulting in degraded performance across tasks. Why ReasonRAG-SFT fails to generalize: SFT training on high-reward trajectories from PopQA, HotpotQA, and 2WikiMultihopQA creates task-specific overfitting. While it improves performance on seen datasets (HotpotQA, 2WikiMultihopQA), it fails on unseen tasks like Bamboogle and MuSiQue. This aligns with recent findings that "SFT memorizes, RL generalizes" [1]—the model learns surface patterns rather than transferable reasoning strategies. Why ReasonRAG-PRL succeeds: PRL leverages both positive and negative feedback with fine-grained reward signals based on model preferences, encouraging flexible reasoning behaviors rather than memorized patterns. This approach enables the model to develop generalizable agentic strategies that transfer across diverse QA tasks, achieving consistent improvements (42.3% avg F1) compared to both base (32.8%) and SFT (32.0%) variants.

[1] Chu, Tianzhe, et al. "Sft memorizes, rl generalizes: A comparative study of foundation model post-training." arXiv preprint arXiv:2501.17161 (2025)

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Best regards,

The Authors

Dear Reviewer ATvT,

We would like to kindly remind you that we have submitted our rebuttal. We have carefully addressed the reviewers’ comments and provided detailed responses to clarify our contributions and resolve the raised concerns.

If possible, we would greatly appreciate it if you could take a moment to review our responses and consider the possibility of raising the score of our submission. Your feedback is very valuable to us, and we are happy to provide any further clarification if needed.

Thank you very much for your time and consideration