DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation

Thank you for your positive feedback regarding "principled integration of retrieval and generation objectives", “balancing sample efficiency and task alignment” and "rich supervisory signal". We will address each point below to clarify and resolve your concerns:

1. "The focus remains on question answering": Indeed, one important scenario for RAG is open-domain QA problems. Many recent related works, such as Search O1 and Search R1, use Exact Match as the sole directly optimized reward and have achieved excellent results on QA tasks. We chose to incorporate other rewards, such as BERT-score, entirely to improve the robustness of our training. Additionally, we have demonstrated the effectiveness of our designed rewards on non-QA tasks, such as fact checking (FEVER) and long text generation (ELI5). Furthermore, we evaluated our model's performance on slot filling tasks (T-REx), as shown in the following table: | Model | T-REx | |-------|-------| | GPT-3.5-Turbo | 33.6 | | GPT-4 | 41.7 | | GPT-4o | 42.3 | | LLaMA2-7B | 22.3 | | LLaMA2-7B w/ Reranker | 25.6 | | LLaMA2-7B-SFT | 30.8 | | Ours (7B) | 41.5 | | LLaMA3-8B | 25.3 | | LLaMA3-8B w/ Reranker | 27.6 | | LLaMA3-8B-SFT | 42.4 | | Ours (8B) | 56.8 | As shown in the table, our 8B model outperforms GPT-4o on the T-REx dataset, demonstrating strong generalization capabilities beyond standard QA tasks.

2. "end-to-end latency numbers under realistic deployment conditions are not reported.": Thank you for pointing this out! In fact, our Table 4 already reports the specific runtime when deployed using vLLM on 8 A100 GPUs (we did not include retrieval time because this step is required for all methods). As can be seen, Vanilla-RAG takes approximately 0.57 seconds, LLM-based list-wise reranking takes approximately 0.75 seconds, while traditional reranking that concatenates question and document takes 13 seconds. Our method takes approximately 0.61 seconds, which is slightly slower than the Vanilla-RAG setting, but faster than all other methods.

3. “Have the authors considered applying DynamicRAG to tasks beyond question answering”: In addition to mainstream QA task datasets (NQ, TriviaQA), we also tested on the fact checking dataset (FEVER) in our main table. Furthermore, we tested on the slot filling task dataset (T-REx). The results can be found in the response to the first weakness. As can be seen, DynamicRAG is essentially an improvement to RAG, so it can be applied to all tasks that are compatible with RAG. We achieved excellent results on non-QA tasks without any modifications, which can be attributed to our consideration of non-QA rewards such as BertScore, Rouge, etc., when designing the reward functions.

4. "How robust is performance to the relative weights of the five reward components?": We detailed the impact of different reward designs on final results in Appendix Section C.3. As shown in Table 5, overall, the consistent trends across different reward function configurations highlight that the model's success is predominantly driven by the synergy of EM, SS, TF, and LP, with LLMEval serving as a supplementary component in refining performance. Additionally, we also tested the hyperparameters of these five rewards, as shown in Table 6. The results show that different parameter settings yield similar performance, indicating that our framework is robust and largely insensitive to these hyperparameters.

Once again, thank you for your thorough review. I hope these points can address your concerns.

Thanks for your feedback! The rebuttal has fully resolved my questions and I decide to keep my score positive.

Dear Reviewer mLW6,

Thank you very much for your constructive feedback on our submitted paper. We were particularly grateful to see that you keep your positive score following our responses.

We sincerely appreciate the time and effort you dedicated to reviewing our work. Your insightful comments were invaluable in helping us improve the manuscript.

Thank you again for your support and positive evaluation.

Best,

Author

Thank you for your positive feedback regarding “making the system more robust”, and "less sensitive to hyperparameter tuning". We will address each point below to clarify and resolve your concerns:

1. We conducted the following experiments, applying the currently best-performing reranker (LLaMA3-8B) and a relatively weaker one (LLaMA2-7B) to other generators. The results are as follows: | Reranker | Generator | NQ (EM) | HotpotQA (EM) | ASQA (EM) | Avg | |----------|-----------|---------|---------------|-----------|-----| | LLaMA2 | LLaMA2 | 38.7 | 29.4 | 41.1 | 36.4| | LLaMA2 | LLaMA3 | 39.4 | 30.6 | 43.5 | 37.8| | LLaMA3 | LLaMA2 | 41.6 | 32.7 | 47.3 | 40.5| | LLaMA3 | LLaMA3 | 48.4 | 36.7 | 56.3 | 47.1| The results demonstrate that using a reranker trained on LLaMA2 combined with a generator trained on LLaMA3 yields better performance than using both reranker and generator trained solely on LLaMA2, which can be attributed to the superior performance of the generator. Furthermore, when using a reranker trained on LLaMA3 paired with a generator trained on LLaMA2, the performance is even better, indicating that the effectiveness of retrieval determines the generator's performance. Therefore, we argue that there is no "lacks true model-agnostic generality".

2. "The system has been effectively optimized for token-level answer overlap": Indeed, one important scenario for RAG is open-domain QA problems. Many recent related works, such as Search O1 and Search R1, use Exact Match as the sole directly optimized reward and have achieved excellent results on QA tasks. We chose to incorporate other rewards, such as BERT-score, entirely to improve the robustness of our training. Additionally, we have demonstrated the effectiveness of our designed rewards on non-QA tasks, such as fact checking (FEVER) and long text generation (ELI5). Furthermore, we evaluated our model's performance on slot filling tasks (T-REx), as shown in the following table:

Once again, thank you for your thorough review. I hope these address your concerns.

Thank you for your positive feedback regarding "the clear motivation on balancing information loss and noise introduction", "the targeted method design with dynamic reranking and RL adaptation", and "the strong performance validation across diverse benchmarks". We will address each point below to clarify and resolve your concerns:

1. "static k vs. dynamic k": We conducted the following experiment: we processed all samples in the training set with k=12 and k≥12 (the average number of k after RL training with non-fixed k is 12), and then performed training. The results are shown in the following table:

2. "direct comparison with RankRAG": Thank you for pointing this out! RankRAG is an important baseline in this paper, but since its source code has not been made publicly available, we cannot reproduce its results. Consequently, we are unable to reduce RankRAG's training data to 150k for training and comparison purposes. One of our key contributions is training a better and more comprehensive RAG system with less data, and we conducted data scaling experiments in Table C.5 to verify that further increasing data can improve the effectiveness of our method.

3. “Key definitions“: Thank you for pointing this out! Regarding ""s_i > T" for document filtering lacks explanation," we indeed did not explicitly specify its value in Figure 3's caption. In fact, we included this information in line 720 of section D.1, and we will add it earlier in the updated version of the paper to reduce misunderstanding. Additionally, if no document satisfies this condition, we discard that sample.

Once again, we apologize for not clarifying this point. In fact, we mentioned in line 719 of section D.1 that we use MonoT5 as our expert model, and we take the probability of the "True" token from the probability distribution of its first output token as s_i. (MonoT5's output is either True or False).

We will address both of these points in the updated version of the paper. Thank you again for your feedback and suggestions.

Q1: "main experimental results appears to be missing": All baseline results in our main table are directly copied from their respective papers. Sometimes other baselines may not have evaluated on the datasets or benchmarks we selected, and some methods are difficult to reproduce, which leads to missing entries in the main table.

Q2: "What is the rationale behind the choice of hyper-parameters?": All experiments in the main table are conducted based on hyperparameters (0.2, 0.2, 0.2, 0.2, 0.2), and this hyperparameter choice was made purely for simplicity. We conducted experiments in Appendix C.4 to explicitly demonstrate the robustness of hyperparameter selection in our framework. If you need any theoretical or empirical justification, we can provide a simple explanation: As shown in Table 5, after removing Exact Match, the final model's performance degrades significantly more compared to removing other reward functions. Many other works, such as Search O1 and Search R1, use Exact Match as the sole directly optimized reward and achieve excellent results on QA tasks. This also indicates that Exact Match serves as the dominant reward, while other rewards play auxiliary roles.

Once again, thank you for your thorough review. I hope these points can address your concerns.

Dear Reviewer Mhs5,

We are writing to follow up on the response we submitted regarding your comments.

Firstly, we would like to express my gratitude for your initial review and valuable feedback. The insights provided have been instrumental in refining our paper, and we have tried our best to address each point in our response with the aim of enhancing the paper's quality and relevance.

Understanding the time and effort required for a thorough review, we greatly appreciate the commitment you have shown towards ensuring the high standards of NeurIPS. However, as the rebuttal process is concluding, we are keenly awaiting your further feedback on our responses. The comments from your expertise are crucial for the final preparation and improvement of our paper.

If our responses more closely meet your expectations for the paper, we respectfully ask you to reconsider your initial rating. If you have any further questions or require more information to raise your initial score, please feel free to let us know. We are fully committed to making all necessary adjustments to meet your expectation.

Thank you once again for your time and dedication. We look forward to your valuable feedback.

Author

Thank you for your positive feedback regarding “ straightforward”, “ interesting”, "the use of DPO make sense" and "effectiveness". We will address each point below to clarify and resolve your concerns:

1. "Have the authors tried DynamicRAG with larger LLMs": Due to resource limitations, we cannot directly train models larger than 13B (such as LLaMA 70B). However, we conducted experiments with GPT-4o through API, as shown in Table 7 in section C.4. The results show that the Dynamic Reranker module consistently improves GPT-4o's performance across all datasets, with gains of 2.3, 0.8, and 0.7 percentage points on NQ, HotpotQA, and ASQA, respectively.

2. "In addition, have the authors tried reasoning models?": We tested QwQ-32B's performance on NQ and TriviaQA with different retrieval top-n settings, as shown in the following table: | Method | NQ | TriviaQA | Avg. | |--------|----|---------|----- | | QwQ (n=5) | 32.8 | 61.3 | 47.1 | | QwQ (n=10) | 34.0 | 63.4 | 48.7 | | QwQ (n=20) | 33.6 | 62.5 | 48.0 | We can observe that reasoning models still suffer from this problem: We observe a trade-off of selecting top-k contexts: a smaller k compromises the recall, while a larger k could introduce irrelevant or noisy context and mislead the LLM generation. Instead of training based on the QwQ foundation, we followed current mainstream practices by using Qwen2.5-3B-Instruct and employing the GRPO algorithm to let it explore reasoning chains with think tags (note that we only used Exact Match as the reward function at this stage). The results are as follows: | Method | NQ | TriviaQA | Avg. | | Qwen-2.5-3B-Instruct (RAG) | 34.8 | 54.4 | 44.6 | | with Reranker | 35.2 | 55.6 | 45.4 | | Ours | 37.6 | 58.7 | 48.2 |

As can be seen, our method significantly improves performance, with an average improvement of approximately 3.6 points. This demonstrates that our approach achieves further improvements in reasoning mode.

3. "How sensitive is DynamicRAG to the choice of weights in Eq.7?": We detailed the impact of different reward designs on final results in Appendix Section C.3. As shown in Table 5, overall, the consistent trends across different reward function configurations highlight that the model's success is predominantly driven by the synergy of EM, SS, TF, and LP, with LLMEval serving as a supplementary component in refining performance. Additionally, we also tested the hyperparameters of these five rewards, as shown in Table 6. The results show that different parameter settings yield similar performance, indicating that our framework is robust and largely insensitive to these hyperparameters.

4. "what is the end-to-end training cost (compute hours) compared to static rerankers?": For example, regarding RankLLaMA (static reranker), its paper clearly states that it uses 16 V100 GPUs to train a 4k context length reranker for 8 days. In contrast, we need 8 A100 GPUs to train a dynamic reranker based on LLaMA2-7B (also with 4k context length) for 2 days. Therefore, the specific gpu-hours ratio is 3072 gpu-hours on V100 versus 384 gpu-hours on A100. However, considering that A100's actual computational power is approximately 2.5-3 times that of V100, the final cost ratio may be 3:1, meaning our training cost is one-third of theirs.

Once again, thank you for your thorough review. I hope these points can address your concerns.

Dear Reviewer tviR,

We are writing to follow up on the response we submitted regarding your comments.

Firstly, we would like to express my gratitude for your initial review and valuable feedback. The insights provided have been instrumental in refining our paper, and we have tried our best to address each point in our response with the aim of enhancing the paper's quality and relevance.

Understanding the time and effort required for a thorough review, we greatly appreciate the commitment you have shown towards ensuring the high standards of NeurIPS. However, as the rebuttal process is concluding, we are keenly awaiting your further feedback on our responses. The comments from your expertise are crucial for the final preparation and improvement of our paper.

If our responses more closely meet your expectations for the paper, we respectfully ask you to reconsider your initial rating. If you have any further questions or require more information to raise your initial score, please feel free to let us know. We are fully committed to making all necessary adjustments to meet your expectation.

Thank you once again for your time and dedication. We look forward to your valuable feedback.

Author