Thank you for your thorough review and helpful remarks. Below, we address each of your suggestions in detail.

W1: The design of the reward function may be too rigid, which could limit its generalization or learning efficiency. The reward is defined as the product of two binary indicators: an exact match (EM) between the generated and ground-truth final answers, and a format-based check ensuring the presence of intermediate reasoning steps. This sparse signal could make policy optimization challenging. An ablation or justification of this reward choice would help strengthen the argument.

Q2: Have you tried other reward functions other than the proposed one?

A: Thank you for raising this important point. To assess this, we experimented with both EM and F1 score as reward signals across multiple QA benchmarks.

The result demonstrates that the simple EM reward yields very robust performance, while performance drops a little when using F1 as the reward.

Q1: Given that both roles are trained jointly via preference optimization, did you observe any instability where improvements in one degrade the other? Did you try strategies other than randomly shuffle the preference pairs for different components, e.g. first train the solver given the correct context, and then the decomposer?

A: Thank you for the question. As shown in Figure 2, with reasonable hyperparameter settings (e.g., lr=5e-7, 5% warmup for Llama-8b), we observed stable average performance across QA and reasoning datasets, without significant fluctuations. We also tried sequentially training the solver and decomposer in separate stages, but encountered forgetting, where previously learned skills were lost in the second stage. In contrast, our joint training strategy using shuffled preference pairs enables both components to learn more effectively and maintain stability.

Q3: How diverse are the decompositions sampled by the model during reinforcement fine-tuning, and how sensitive is performance to decomposition redundancy? Do you apply any techniques to encourage diversity in z during sampling?

A: Thank you for the thoughtful question. To encourage diverse decompositions during reinforcement fine-tuning, we set the sampling temperature to $t=1.0$, which promotes variability in the generated decompositions. In practice, we observe that most sampled decompositions differ meaningfully from one another. Additionally, to reduce redundancy and ensure training efficiency, we explicitly filter out duplicate or highly similar decompositions before training. This helps maintain a diverse set of training signals and supports more robust learning.

Thank you once again for your insightful review. We appreciate your feedback on our work. Feel free to let us know if you have any further questions, and we are happy to discuss further.

We are grateful for your careful assessment and insightful comments on our paper. Our responses to your suggestions are provided below.

W1: The training pipeline (SFT + RFT) is somewhat intricate, potentially limiting reproducibility despite good documentation.

A: Thank you for the valuable feedback. We would like to mention that it is a de facto standard to do SFT + RFT for post-training LLMs. We also want to clarify that our training pipeline is built on top of the Llama-Factory codebase, which provides streamlined support for both SFT and RFT, making it easy to use and reproduce. To further ensure accessibility, we will open-source our code and data along with detailed documentation and ready-to-use scripts. Users will be able to reproduce our experiments or fine-tune models on their own datasets with minimal effort.

W2: The assumption that reward signals from final answers sufficiently guide decomposition quality could be risky in more ambiguous domains.

A: Thank you for the insightful comment. The tasks considered in our work primarily involve questions with short-form, verifiable answers, where reward signals from final answers provide reliable and sufficient supervision. In more ambiguous domains involving long-form or non-verifiable answers, we could adopt an LLM-as-a-judge approach to generate reward signals. However, since such methods are less scalable and unnecessary for our tasks, we do not employ them in this work.

W3: The decomposition and multi-step retrieval introduce additional inference overhead compared to the standard RAG pipeline, which could limit applicability in real-time scenarios.

A: Thank you for the question. We would like to point out that most strong baselines in our comparison also adopt multi-step retrieval, and AceRAG does not introduce additional inference overhead beyond those methods. While AceRAG does have higher latency than standard single-step RAG due to its decomposition and multi-step reasoning components, this overhead is justified by the performance gains. As shown in Section 5.5 and Figure3(c), AceRAG achieves substantial performance gains – even outperforming 32B models with comparable inference time. We believe this trade-off is a reasonable compromise for the gains in effectiveness.

Thank you for your insights. We appreciate your feedback and are happy to address any further questions you may have.

Thank you for your detailed review and constructive suggestions. We address your feedback point by point below.

Q1: What is the justification only exploring fairly outdated retriever models such as E5, Dragon and Contriever (Appendix H)?

A: Thank you for raising this point. We selected these retrievers due to their widespread use in prior works, such as Search-R1 [1], InstructRAG [2], CORAG [3], Plan-RAG [4], etc. This choice ensures fair and meaningful comparisons with existing methods. We agree that evaluating with more recent retrievers (e.g., Qwen3-embedding [5]) could further improve performance, and we will consider this in the future work.

Q2: What is the justification for comparing to Qwen3-32B Reasoning when it does not have access to retrieved contexts.

A: Thanks for the question. We would like to clarify that all the baselines, including Qwen3-32B reasoning model, have access to retrieved contexts by default. We also keep the same retriever and retrieval corpora as our method during evaluation to ensure fair comparison.

References

[1] Jin et al. "Search-r1: Training llms to reason and leverage search engines with reinforcement learning." arXiv preprint arXiv:2503.09516 (2025).

[2] Wei et al. "Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales." ICLR 2025.

[3] Wang et al. Chain-of-Retrieval Augmented Generation. arxiv preprint 2501.14342.

[4] Verma et al. "Plan-rag: Efficient test-time planning for retrieval augmented generation." arXiv preprint arXiv:2410.20753 (2024).

[5] Zhang et al. "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models." arXiv preprint arXiv:2506.05176 (2025).

Thank you once again for your insightful review. We appreciate your feedback on our work. Feel free to let us know if you have any further questions, and we are happy to discuss further.

Thank you for your thoughtful feedback and for carefully reviewing our work. Please find our responses to your suggestions below.

W1: Related Work is sparse and perhaps undersell the work in the paper: For instance, many of existing work on fine-tuning LLMs for RAG focus primarily on decomposition, and use a standard/off-the-shelf generator, unlike AceRAG that solves them jointly by repurposing the same LLM.

A: Thank you for the appreciation and helpful suggestion. While our initial intention was to provide a broader discussion of related work, we recognize that including too many references may distract readers from our main contribution. We agree that a clearer positioning of AceRAG in contrast to prior methods would help clarify the novelty of our joint training framework. We will revise the Related Work section to better highlight these distinctions and more effectively emphasize our contributions.

W2: I didn't find LeReT [26] compared in the experiments. LeReT has a similar strategy for training the decomposer, but uses a standard SLM for generator, but I recall it did show strong results in some of the RAG benchmarks. It would be good to contrast the results of AceRAG with LeReT.

A: Thank you for pointing this out. Following your suggestion, we provide a comparison between LeReT and AceRAG on HotpotQA and Hover below. We report EM scores and include only the 8B version of AceRAG for fair comparison with similarly sized LeReT models.

It's important to highlight that LeReT leverages direct supervision by comparing retrieved documents against ground-truth retrievals, and uses a reader (Llama-3.1-70B) with much more parameters (8x). In contrast, AceRAG uses indirect supervision by deriving reward signals from final answer correctness. While direct supervision may yield stronger alignment with decomposition quality, it requires access to oracle retrieval labels, which limits scalability. We choose to use indirect supervision in this work to enable broader applicability across real-world settings. It is also worth noting that many existing works such as [1,2] also adopt the setting where only the final ground-truth answer is available. That being said, the results show that AceRAG can outperform LeReT on HotpotQA and achieve comparable performance on Hover under this setting, further verifying the efficacy of our design. We will add the comparison and discussion of LeReT in the next version of the paper.

W3: I was somewhat puzzled by the need for the first SFT phase (Sec 4.1) when I read it. My thinking was that if you are starting with models like Qwen instruct versions, SFT on standard datasets with next-token prediction might not be as valuable as the RL-based training in the next phase. But the ablations in Table 3 shows that SFT is so crucial! Could you comment on the overlap of datasets between SFT and RFT phases?

A: Thanks for the insightful question! There is some overlap between the datasets used in SFT and RFT, especially for context-rich reasoning tasks. As described in Section 4.1, the SFT phase uses a curated mixture of context-rich QA data, question decomposition data, and chain-of-thought examples from a wide range of open-source datasets. This provides direct supervision to help the model learn how to answer questions accurately and generate effective decompositions. In contrast, the RFT phase fine-tunes the model primarily on QA pairs from HotpotQA, 2WikiMultiHopQA, HOVER and reasoning datasets such as GSM8K, ConvFinQA, StrategyQA (see Line 152 and 160), which teaches the model to refine its own response based on the final answer supervision.

Although the base LLM is already instruction-tuned, the SFT phase is critical for explicitly teaching the model the specialized behaviors required by AceRAG – specifically, decomposition and multi-step reasoning grounded in retrieved context. General instruction tuning supports general language capabilities but does not provide sufficient supervision for RAG applications. By incorporating additional context-rich QA and question decomposition datasets in SFT, we ensure the model reliably acquires these targeted behaviors. This targeted supervision in SFT establishes a strong foundation for RFT to further refine model performance. We will incorporate this discussion into Section 5.4 to clarify the motivation behind our two-stage training design.

Q1: What's the exact configuration you use for the two CORAG baselines (Greedy & Inference Scaling).

A: Since the implementation and training datasets for CORAG are not publicly available, we report the numbers directly from their paper for comparison. CORAG is an iterative approach that generates sub-queries and answers step by step using tree-search algorithms.

• CORAG Greedy: It uses greedy decoding to generate one sub-query and its corresponding sub-answer at each step, proceeding sequentially. This has similar computational complexity to AceRAG.
• CORAG Inference Scaling: At each step, it applies breadth-first search (BFS) with multiple retrieval chains. It generates 10 sub-questions per step and uses best-of-N sampling (N=8) to select the final answer. This setting is significantly slower at inference, as noted in their paper.

References

[1] Jin et al. "Search-r1: Training llms to reason and leverage search engines with reinforcement learning." arXiv preprint arXiv:2503.09516 (2025).

[2] Song et al. "R1-searcher: Incentivizing the search capability in llms via reinforcement learning." arXiv preprint arXiv:2503.05592 (2025).

Thank you for taking the time to review our work so carefully. Your feedback is invaluable, and we welcome any additional questions or comments you might have.

We appreciate your valuable insights and the time you dedicated to evaluating our paper. Our responses to your comments are as follows.

W1: The contribution is more on the engineering side than in theoretical novelty. Most of the key components in the proposed framework can be directly traced to existing literature, including self-play [1, 2], RL-based RAG optimization [3, 4], "decomposer-solver" strategy [4], and two-stage training pipelines [5, etc.]. While AceRAG introduces its own implementation refinements, it shares a similar underlying idea with prior work. This limits its novelty from a conceptual standpoint. That being said, I do not consider that this detracts from the fundamental contribution of the work, but view it as a missing citation ([1], [3], and [4])

A: Thanks for providing those related works! We will add them to the references in the next version of the paper.

Q1: Since the "decomposition-solve" strategy is well calibrated for complex problems, I would like to confirm the scope of this work. Specifically, is AceRAG designed for multi-hop problems only, or is it a general RAG solution? If the latter, I would like to see the performance of AceRAG on a simple single-hop RAG benchmark to understand the impact of this training strategy on the model behavior.

A: Thanks for raising this question. AceRAG is mainly designed for multi-hop problems, as it benefits from the joint learning of the decomposer and question solver. For simple single-hop RAG problems, as they typically do not require such decomposition, standard single-round retrieval can already achieve very strong performance [1, 2]. As a result, we do not expect AceRAG to offer significant advantages in this case. We will clarify the scope of AceRAG in the next version.

Q2: According to Line 142-144, different retriever is used for different tasks. Does this apply to inference time only or both training and inference time?

A: The use of different retrievers applies only at inference time. During training, we use E5 as the sole retriever across all tasks. At inference time, E5 remains the default retriever, with the only exception being the document-level reasoning task, where we follow the original benchmark setup [3] and use OpenAI’s Embedding-3-Large to ensure a fair comparison.

Q3: In Figure 5b, the reranking baseline (blue line) is based on RankLLM. Given that this is an LLM-based reranking method, I am curious why it appears as a flat line, with no scaling in recall at all as model size increases. Could the authors provide more details about the experiment setup for Figure 5b, including model configurations?

A: Thank you for pointing this out. The RankLLM baseline is based on a fixed 7B model with no varying sizes. As such, it yields a single performance point rather than a curve. We represented it as a flat dashed line to facilitate visual comparison with AceRAG’s scaling behavior. To avoid confusion, we will revise the figure in the next version by adding a marker at the 7B position to explicitly indicate the model size.

References

[1] Wei et al. "Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales." ICLR 2025.

[2] Yu et al. "Rankrag: Unifying context ranking with retrieval-augmented generation in llms." NeurIPS 2024.

[3] Zhao et al. "DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents." ACL 2024.

Thank you again for your helpful review. We truly appreciate your feedback and are glad to address any further questions or engage in additional discussion.

The rebuttal confirms my original understanding. I maintain my positive recommendation and suggest the author add the above discussion into the final version.