ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

We sincerely appreciate the reviewer’s thoughtful comments and suggestions. Below, we address each of the weaknesses (W) and questions (Q) raised.

[W1] Comparisons with more robust baselines

While we did not directly run more robust baselines, such as DeepSeek-R1, we collected their evaluation results from other recent works [1] that follow the same benchmark datasets as ours. The exact match (EM) scores for DeepSeek-R1 and Qwen3-235B-A22B are taken from [1], while the results for ReSearch-Qwen2.5-7B-Instruct and ReSearch-Qwen2.5-32B-Instruct are from our own experiments. The comparison is shown below:

It is evident that, although our models are considerably smaller than DeepSeek-R1 and Qwen3-235B-A22B, their performance is overall comparable across these challenging multi-hop benchmarks.

[W2] Investigation of training on reasoning bases

Thank you for highlighting the potential value of evaluating ReSearch on models that already contain some form of chain‑of‑thought reasoning. Our current study deliberately starts from non‑reasoning checkpoints for two reasons:

1. Clean attribution. Starting from a “plain” model lets us see exactly how much reasoning‑with‑search our RL pipeline adds, without mixing in skills learned elsewhere.

2. Compute and openness. Reasoning‑enhanced models (e.g., DeepSeek‑R1‑Distilled‑Qwen, >30 B params) are heavyweight, use private data, and are costly to reproduce. In contrast, vanilla Qwen‑2.5 is fully open and fits within our compute budget.

This keeps the study controlled and reproducible. We agree that plugging ReSearch into QwQ‑32B‑preview, Qwen3, or distilled DeepSeek variants is a promising next step. We will release code hooks so that the community can replicate these experiments.

[Q1] Issue of growing sequence length

We did not introduce any truncation or external‑memory mechanism in the current study. As described in §3.1, the retriever returns at most the top‑5 passages per query, keeping each search block to only a few hundred tokens; in practice, the full context seldom approaches the model’s 32 k‑token window. Figure 3 further shows that the reasoning tokens remain well below this limit across multiple turns, so the computation comfortably fits within our hardware budget.

We acknowledge, however, that context growth becomes a real bottleneck for longer‑horizon tasks. Recent work suggests two complementary remedies: (i) treating the thinking content as a working memory to retain important reasoning or facts on previous steps [2]; and (ii) summarizing each retrieved passage before appending it, preventing unbounded expansion [3]. These techniques are orthogonal to ReSearch, and we plan to integrate them in future work.

References

[1] DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning, 2025.

[2] MEM1: Learning to Synergize Memory and Reasoning for Efficient Long‑Horizon Agents, 2025.

[3] WebSailor: Navigating Super‑Human Reasoning for Web Agents, 2025.

Thanks for authors' reponse. My concerns are addressed. I will maintain my positive score.

We sincerely appreciate the reviewer's efforts and insightful comments to improve our manuscript. Below, we address the weaknesses (W) and questions (Q) raised.

[Q1 & W1] On more challenging benchmarks

We share the reviewer’s interest in the performance of ReSearch on full‑web agent benchmarks such as BrowseComp.

Our current study confines itself to Wikipedia retrieval so that we can cleanly measure how RL decides when to search versus when to keep thinking without additional confounders. BrowseComp, in contrast, exposes a much richer environment: the agent must issue DOM clicks, scroll, fill forms, and render JavaScript. Although a few recent systems have reported BrowseComp results by leveraging high-level “search + summary” APIs, this action space is still well beyond the "think, search" interface used here.

The good news is that the key skill ReSearch learns—deciding when to query and when to reason internally—is orthogonal to the concrete low-level actions. In principle, the learned policy can sit atop any web-navigation layer once one is available. Nevertheless, several obstacles must be overcome before we can expect stable gains:

• Richer action repertoire. BrowseComp introduces clicks, back/forward, scrolling, and form submission. A naive flat policy would face an intractably large exploration space; hierarchical or curriculum RL is required.
• Much longer horizons with sparse rewards. Solving a BrowseComp question often involves ten or more navigation steps and yields only a single binary reward. Without intermediate shaping signals, the advantage estimates become extremely noisy.
• Context bloat. Raw HTML plus extracted text quickly exceed the model’s capacity unless working-memory or on-the-fly summarization is introduced.

We have begun addressing these issues and outline three concrete extensions for future work:

• Action layer. We are integrating a lightweight DOM-navigation wrapper so the agent can click and scroll rather than issuing only textual queries.
• Memory and compression. We plan to add working-memory and retrieval-result summarization to control context length during long-horizon interactions.
• Complex task generation. We will create harder, multi-step question sets comparable to BrowseComp to stress-test the upgraded agent.

We'll address these issues in our future work.

[Q2] Performance across different traning startegies

For an SFT baseline, we reproduced a STaR-style [1] pipeline as follows: (i) use Qwen-2.5-7B-Instruct to sample five rollouts for every MuSiQue training query (same prompt as our main experiments); (ii) keep only the trajectories whose final answer matches the ground truth; and (iii) fine-tune Qwen-2.5-7B on the resulting nearly 15k trajectories. Results show that this filtered-SFT model improves over prompt-only baselines but is consistently outperformed by our RL-trained ReSearch agent, underscoring the added value of RL.

[Q3] Scaling Laws for this task

Since our end‑to‑end RL + retrieval training is compute‑intensive, we restricted experiments to two backbone sizes (7 B, 32 B) and a fixed rollout budget, so we did not pursue formal scaling‑law analysis. We agree that understanding how performance scales with data, compute, and model size is crucial for tool‑augmented agents and plan to investigate these factors in future work; we will note this limitation in the revision.

References

[1] STaR: Bootstrapping Reasoning With Reasoning, 2022.

We sincerely appreciate the reviewer's efforts and insightful comments to improve our manuscript. Below, we address the weaknesses (W) raised.

[W1.1] How different technical details affect the effect of the method

Our method is intentionally minimalist, and all other components follow standard settings. Consequently, there are relatively few knobs to ablate. Nevertheless, we did run the most informative comparison we could devise—pure SFT vs. our RL training—to isolate the benefit of exploration. We give a STaR-style [1] SFT baseline:

(i) Use Qwen-2.5-7B-Instruct to sample five rollouts for every MuSiQue training query (same prompt as our main experiments); (ii) keep only the trajectories whose final answer matches the ground truth; and (iii) fine-tune Qwen-2.5-7B on the resulting nearly 15k trajectories. Results show that this filtered-SFT model improves over prompt-only baselines but is consistently outperformed by our RL-trained ReSearch agent, underscoring the added value of RL.

The resulting filtered-SFT model does outperform prompt-only baselines, but it is consistently weaker than our RL-trained ReSearch agent on all four benchmarks. This experiment confirms that the extra exploration and credit assignment provided by RL—not merely the presence of chain-of-thought data—drives the performance gains.

[W1.2] Study more complicated scenarios

We deliberately began with tasks whose answers admit a rule‑based F1 reward so we could cleanly test whether reinforcement‑trained reasoning + search works at all, without confounding factors from noisy evaluators. We agree that extending to open‑ended outputs (e.g., multi‑page tech reports) demands richer feedback signals. Our framework is agnostic to the reward source, and in ongoing work we are replacing the string‑match reward with (i) rubric‑guided LLM judges and (ii) learned reward models that score content quality and coherence. We will add this discussion to the revision.

[W2] Where the reasoning agent shows advantages

(1) Decomposition & planning for complex questions.

Many queries require breaking a goal into ordered sub‑goals and scheduling searches accordingly—something native RAG/prompt baselines do not do. Example (abridged):

“Name the famous bridge in the city where the composer of La costanza trionfante degl’amori e de gl’odii was born.” ReSearch explicitly: (i) finds the composer (Antonio Vivaldi), (ii) locates his birthplace (Venice), and (iii) retrieves the bridge (Rialto Bridge). A naïve RAG baseline, lacking this plan, often latches onto an unrelated bridge (e.g., Bridge of Sighs).

(2) Iterative correction when the first retrieval is wrong.

As shown in our case studies (Sec. 3.4), when an early search is off-target, ReSearch can detect the mismatch, revise its plan, and issue alternative queries. Baselines typically accept the first hit, propagating early errors with no recovery mechanism. These two capabilities—explicit multi-step planning and self-correction—explain why training a reasoning agent improves search tasks: it closes the loop between “think → search → verify → adjust,” which prior methods do not cover. We will make these points clearer in the revision.

[W3] Training and Deployment Efficiency

As specified in Appendix B, we clarify that our training experiments were conducted on 8 nodes × 8 Nvidia H800 GPUs (64 H800s in total). For deployment/inference, we use a single node with 4 H800 GPUs. We will make this information more explicit in the paper and further emphasize the practical deployment efficiency of our framework.

References

[1] STaR: Bootstrapping Reasoning With Reasoning, 2022.

Thank you for you responses. Given this is a paper relating to timely topic, I appreciate the technical details and observations provided.

It is surprising to see obvious gaps between SFT and RL results. I am curious about the computation resources required between the two settings. And whether SFT improves by scaling more data or data filtering. And how about doing iterative SFT? The author only study STaR like SFT while it would be more reasonable to compare RL to rejective finetuning.

The author attributes this to extra exploration and credit assignment provided by RL which seems ambiguous. Why RL provides extra exploration than your rejective finetuning baseline? How GRPO provides credit assignment without a value model?

We sincerely appreciate the reviewer's efforts and insightful comments to improve our manuscript. Below, we address the weaknesses (W) raised.

[W1] Comparison with simple SFT strategies

Why RL beats plain SFT

Supervised fine-tuning (SFT) can only imitate trajectories that the current model already happens to generate; it cannot move beyond the quality of those teacher samples. Reinforcement learning (RL) instead allows on-policy exploration: the model tries new tool-calling sequences, receives scalar rewards, and updates itself accordingly. This feedback loop teaches the model not only what a good trajectory looks like but also why it is better than the alternatives—crucial for tool-augmented agents, where many useful reasoning paths are so unlikely under an untrained policy that they never appear in an SFT dataset.

Our STaR-style SFT baseline

To give SFT its best shot, we reproduced a STaR-like pipeline: (i) use Qwen-2.5-7B-Instruct to sample five roll-outs for every MuSiQue training query (same prompt as our main experiments); (ii) keep only the trajectories whose final answer matches the ground truth; and (iii) fine-tune Qwen-2.5-7B on the resulting nearly 15k trajectories. Results show that this filtered-SFT model improves over prompt-only baselines but is consistently outperformed by our RL-trained ReSearch agent, underscoring the added value of RL.

We intentionally did not reuse rollouts generated during GRPO training, because those samples are themselves produced by an ever-improving policy—a that benefits inherent to RL. Using a static model (here we use Qwen2.5-7B-Instruct) keeps the comparison faithful to standard SFT practice.

[W2] Extending ReSearch with process‑level rewards

While our current design deliberately uses a simple, cheap reward (answer correctness + format) to avoid heavy annotation, we agree that ReSearch can be naturally extended to incorporate process signals. Concretely, the following heuristics can be injected into our framework without altering the GRPO loop:

• Redundancy penalty: If a retrieval step returns passages that highly overlap with previously retrieved content, assign a negative reward to discourage wasted calls and encourage informational diversity.

• Self-critique feedback: Use a lightweight verifier to flag unsupported or logically inconsistent intermediate statements, yielding sparse negative rewards.

These additions keep our RL loop intact (still GRPO) and only modify the reward function. We will add a short discussion of the above options and leave them as future work, acknowledging that richer process signals are orthogonal and complementary to our current outcome-level scheme.

[W3] Discussion of R‑Search & Search‑R1 (contemporaneous work)

We appreciate the pointer. Both R‑Search (June 2025) and Search‑R1 (March 2025) were released after March 1, 2025 and thus fall under NeurIPS’s “contemporaneous work” policy. Per the call for papers, such work should be cited and discussed but the absence of full empirical comparison is not grounds for rejection.

Methodologically, ReSearch differs along several axes:

• Reward design: We use purely outcome-level GRPO—without step labels, PRMs, or auxiliary verifiers. In contrast, R‑Search incorporates process-level rewards or external evaluators (such as cross-family reward models for evidence quality). Furthermore, Search‑R1 adopts rule-based rewards based on exact string matching, whereas our work uses F1‑score as a smoother and more informative reward signal compared to exact match.
• Model scale. We report results on larger backbones (7B and 32B), whereas R‑Search/Search‑R1 primarily train/evaluate 3B–7B models.

We will add a concise paragraph to clearly position ReSearch relative to these contemporaneous papers.

[W4] Comparison to MCTS-style test-time search

ReasonRAG [1] integrates Monte Carlo Tree Search (MCTS) into an agentic RAG pipeline to offline-generate process‑supervised data (step scores) before training. It is thus a close, tree‑search–based counterpart in this domain. Under the same evaluation protocol (Qwen2.5‑7B‑Instruct backbone, Exact Match metric), ReSearch consistently outperforms ReasonRAG on all four benchmarks:

These results suggest that reinforcement learning with a simple outcome reward can match—and even surpass—the performance of more complex MCTS-based supervision.

References

[1] Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning. 2025