C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation

Dear Reviewer stM2,

Thank you for your thoughtful review and constructive suggestions. We particularly appreciate your recommendations about extending our evaluation to more challenging benchmarks and exploring alternative training strategies. These insights will help strengthen our work. We would like to address your suggestions in detail:

W1

Thank you for this valuable suggestion about testing on more challenging benchmarks. We agree that evaluation on complex reasoning tasks is crucial for validating our framework's capabilities.

We have conducted additional experiments on Humanity's Last Exam (HLE) text-only questions using Google as the retriever (an out-of-domain search engine for C-3PO):

The results show that:

• C-3PO improves performance by 1.11 % compared to vanilla RAG
• C-3PO-Planning further boosts performance by 1.49%
• These improvements demonstrate our framework's effectiveness even on highly challenging reasoning tasks with out-of-domain retrieval

We will include these results in our revised manuscript to provide a more comprehensive evaluation of our framework's capabilities.

W2

Thank you for this insightful suggestion about exploring from-scratch RL training. We agree that this direction, similar to Deepseek-R1's approach, is very interesting and could potentially lead to more general training strategies for multi-agent systems.

While our current warm-up approach helps ensure stable and smooth training in the multi-agent setting, we believe exploring from-scratch training could:

• Reduce dependency on seed data collection
• Potentially discover novel agent interaction patterns
• Lead to more generalizable training strategies

We will include this as an important direction for future research. The challenge of balancing exploration and stability in from-scratch multi-agent RL training presents an exciting opportunity for advancing the field.

We sincerely appreciate your valuable suggestions that have helped us identify important directions for both immediate improvements and future research. Your feedback about evaluation on challenging benchmarks has already led to meaningful additional results. We will incorporate these improvements in our revised manuscript.

Dear Reviewer Tndf,

Thank you for your thorough and constructive review. We would like to address each of your concerns in detail:

W1

Thank you for raising this important issue about the framework's generalizability. We would like to clarify several aspects:

1. Design Philosophy:

• Our proxy-centric alignment is inspired by human interaction patterns in knowledge-intensive tasks, where information gathering and reasoning are fundamental operations.
• The core design focuses on how proxy can align retrievers and LLMs through planning and reasoning to collect information, rather than being strictly QA-specific.

2. Preservation of LLM Capabilities:

• Importantly, our framework does not fine-tune the LLM, preserving its general capabilities (e.g., writing, summarization).
• The agents serve as information gathering and coordination proxies, which are inherently applicable to various knowledge-intensive tasks beyond QA.

We will include this as an important direction for future work, while maintaining that the current design principles are fundamentally task-agnostic.

W2

Thank you for raising this important issue. We would like to address your concerns from multiple aspects:

1. Training Efficiency:

• Our approach does not introduce significant additional training overhead compared with standard PPO.
• Traditional RL methods typically require sampling multiple independent trajectories per question in parallel, and our approach may reuse partial trajectories during rollout. In SGLang inference system, the reused context can be efficiently cached for further inference. Algorithmically, our approach just redistributes this sampling effort from the question level to the action level, maintaining a similar computational budget.

2. Inference Efficiency:

• Figure 3 shows our method does not introduce substantial inference latency.
• This efficiency is achieved through our Decision Maker, which dynamically allocates optimal strategies to balance computation and performance.
• Figure 5 shows how different strategies evolve during RL iterations, providing transparency into our method's adaptation.

3. Regarding Baseline Comparisons:

• While works [1,2] are not yet open-sourced for faithful reproduction, we have included another reranking work in Tables 2/3.
• We chose QueryRewriting as our efficiency baseline due to its parameter efficiency (1.5B) and consistent stability across scenarios.

We appreciate these suggestions and will incorporate the related works [1,2] to better position our work.

Q1

Thank you for this detailed question about computational efficiency. Let us break down the number of forward passes required for each strategy:

1. Empirical Evidence:

• Figure 6 provides detailed distributions of inference depths across different datasets
• Figure 3 shows the inference latency of C-3PO compared to baselines

2. Forward Passes by Strategy:

• [No Retrieval]: 1 proxy pass + 1 LLM pass
• [Retrieval]: 2 proxy passes + 1 LLM pass
• [Planning]: 2 LLM passes + variable proxy passes (distribution shown in Figure 6)
• Note that Proxy are lightweight (0.5/1.5B) and LLM can be 7/72B

This strategic allocation of computational resources allows us to maintain efficiency and achieve superior performance. The actual number of forward passes is optimized for each specific query rather than using a fixed number for all cases.

Q2

Thank you for this insightful observation. The [Planning] strategy, while powerful, involves collecting additional information that may introduce more noise and potentially mislead the LLM. Meanwhile, for many RAG datasets, a well-crafted query combined with effective filtering ([Retrieval] in our C-3PO) might suffice, especially when search engines can retrieve the necessary information in a single pass. This suggests that the optimal strategy depends on the alignment between dataset and proxy capabilities rather than following a one-size-fits-all approach.

Q3

Thank you for this thoughtful question about extending our framework beyond QA tasks. While our C-3PO framework is specifically designed for knowledge-intensive tasks where proxy-centric alignment between retrieval and LLM components is crucial, the tasks in [3] primarily focus on logical reasoning that may not heavily rely on external knowledge. For such pure logical reasoning tasks, our retrieval-oriented multi-agent system might offer limited benefits in its current form.

However, we believe our framework could be adapted for logical reasoning tasks by:

• Combining a reasoning verification agent
• Integrating training approaches similar to Deepseek-R1 for pure reasoning tasks

We appreciate this suggestion as it opens up interesting directions for future research.

We sincerely appreciate your detailed review and thoughtful questions. We believe addressing these points has helped strengthen our paper. We look forward to your further feedback.

Dear Reviewer B7TK

We sincerely appreciate your thorough review. We have carefully addressed each of your concerns below:

Issue 1 & W3

We appreciate the mentioned related works, and would like to clarify several important points:

1. First, we acknowledge the importance of these works. We will cite these related works and incorporate detailed discussions in our revised version.
2. Regarding the timeline and reproducibility of mentioned related works:

• [1] is published on Jan 25, 2025, 2 days later than ICML abstract submission deadline Jan 23, 2025.
• [3] is published on Dec 17, 2024, which can be considered concurrent work.
• For [2] and [3], despite their relevance, the absence of publicly available implementation makes faithful reproduction challenging.

3. Our evaluation have covered 3 major baseline categories (retriever/LLM fine-tuning and intermediate approaches) across 6 in-domain and 2 OOD datasets, demonstrating thorough effectiveness and generalization.

Issue 2

We apologize for any confusion, while the experimental setup of each row in Table 3 was presented in Line 418-426, we would like to provide further clarification:

• [No Retrieval]: Relies solely on LLM's inherent knowledge
• [Retrieval]: Employs single retrieval-filter loop
• [Planning]: Utilizes multi-step reasoning

The full C-3PO system's ability to adaptively select strategies leads to robust performance across different datasets.

Issue 3

We appreciate the reviewer's careful examination of our tree-structured rollout. We would like to provide additional clarification:

1. Regarding the performance gains:

• We observe substantial gains on challenging tasks such as Musique, HQA, and PopQA.
• While our method enhances agent decision-making instead of directly answering the question, the performance ceiling ultimately depends on the LLM (remains frozen in C-3PO). Our approach still outperforms many recent methods that fine-tune LLMs (AutoRAG) as shown in Tables 1/2.

2. On complexity concerns:

• We would like to emphasize that our tree-structured rollout does not introduce additional computational overhead compared to standard RL.
• Traditional RL methods typically require multiple independent sampling trajectories per question in parallel, and our approach may reuse partial trajectories during rollout. In SGLang inference system, reused context can be efficiently cached. Algorithmically, our approach redistributes these sampling efforts from the question level to the action level, enabling more efficient credit assignment in multi-agent systems through expectation-based reward distribution.
• We can also control the breadth/depth of the tree to balance exploration and cost (Eq. 4), making it flexible for various computational budgets.

We believe the clarifications show that our approach offers meaningful improvements and maintains computational efficiency.

Eval Criteria

We appreciate the reviewer's suggestion regarding evaluation metrics. We would like to clarify our choice of metrics and provide additional results:

1. Limitations of EM metrics:

• Through our preliminary studies, we observed that rule based metrics, such as EM, can be unreliable, especially when using frozen LLMs that may express correct answers in varied formats.
• Inaccurate rewards from strict rule based matching could potentially harm the RL training.

2. The choice of LLM-based evaluation:

• Recent benchmarks, such as FreshQA, HLE (Humanity's Last Exam), have increasingly adopted LLM-based evaluation due to its ability to capture semantic correctness beyond EM.
• In our human verification process, we found that Qwen2-72B-instruct demonstrates high accuracy in assessment, making it as a more reliable source for both evaluation and RL rewards.

3. Additional EM Results:

• To address this concern, we provide some EM results (due to limited chars, a full table can refer to the response of W4 for Reviewer TgQU).

W1

Regarding inference efficiency, we already presented a detailed analysis in Section 6.5 and Figure 3. It shows that C-3PO achieves the best performance-efficiency trade-off.

W2

We appreciate the suggestion and would like to clarify that our evaluation already covers a diverse range of LLMs across different scales and types, such as Qwen2-7B, Qwen2-72B, Llama3.3-70B, and GPT4o-mini (commercial API), as shown in Tables 1/2. While we acknowledge that testing on other commercial APIs like Claude and o1 would be interesting, the significant costs make such extensive evaluation prohibitively expensive.

We sincerely thank you for your detailed comments and hope our responses have adequately addressed your concerns.

Dear Reviewer TgQU,

Thank you for your thorough and constructive review of our paper. We would like to address each of your concerns in detail:

W1

We appreciate your concern about computational efficiency. We would like to clarify that our tree-structured rollout does not introduce additional computational overhead compared to standard RL:

• Traditional RL methods typically require sampling multiple independent trajectories per question in parallel, and our approach may reuse partial trajectories during rollout. In SGLang inference system, the reused context can be efficiently cached for further inference. Algorithmically, our approach just redistributes this sampling effort from the question level to the action level, maintaining a similar computational budget.

• The tree structure actually provides several advantages:

  • It enables more systematic exploration of the action space
  • It allows for expectation-based credit assignment
  • It reduces the variance in training compared to random sampling

• We can also control the breadth and depth of the tree to balance between exploration and computational cost (Eq. 4), making it flexible for different computational budgets.

Therefore, while our approach may appear computationally intensive at first glance, it actually offers a more structured and efficient way to explore the action space within the same computational constraints as traditional RL methods.

W2

We apologize for any confusion regarding the warm-up phase. We would like to clarify several key points:

• As mentioned in Section 5.2 and Appendix A.2, we collect seed data through rejection sampling from Qwen2-72b-instruct, specifically gathering 2 correct solutions for each question.

• The detailed training hyper-parameters are provided in Table 4.

• To further validate the effectiveness, we conducted comparative experiments between C-3PO-RL and C-3PO-ICL in Table 8. These results demonstrate the feasibility of our warm-up strategy.

We hope these clarifications address your concerns about the warm-up phase implementation.

W3

We agree that the evaluation metrics deserve more prominence in the main text. We will move a concise version of the evaluation metrics from Appendix D.2 to Section 6.1, making these important details more accessible while maintaining the paper's flow and readability.

W4

Thank you for raising this important point about evaluation methodology. We would like to address this concern from multiple aspects:

1. Additional EM Results:

We have conducted additional experiments using the EM metric. The results show that on the EM metric, C-3PO still achieves significant improvements over all baselines:

2. Limitations of Traditional Rule Based Metrics:

• Through our preliminary studies, we found that rule based metrics, such as EM, can be unreliable, especially when working with frozen LLMs that may express correct answers in unpredictable formats.
• These inaccurate rewards from strict rule based matching could potentially harm the training of reinforcement learning.

3. Adoption of LLM-based Evaluation:

• Recent famous benchmarks like FreshQA and Humanity Last Exam (HLE) increasingly adopt LLM-based evaluation to capture semantic correctness beyond exact matching.
• This trend reflects the community's recognition of the limitations of traditional metrics for complex QA tasks.

4. Reliability of Our Evaluation:

We conducted rigorous human verification of Qwen2-72B-instruct's evaluation capabilities.

We understand the importance of using standardized metrics. However, we believe that combining both traditional and LLM-based evaluation provides a more comprehensive assessment of model performance. We appreciate this feedback and have enhanced our evaluation section accordingly.

References Not Discussed

We sincerely thank you for suggesting these valuable references. We will incorporate these citations and related discussions in our revised manuscript to better position our work in RAG.

We appreciate again for your time and effort in reviewing our paper. We believe that addressing these concerns has helped strengthen our work, and we hope our responses have satisfactorily addressed your questions. We look forward to your further feedback.