Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

We sincerely thank the reviewer for the constructive feedback, which has strengthened our work. Please find our point-by-point responses below:

• Is it necessary to perform RL without any cold-start SFT data? We agree with the reviewer that incorporating an intermediate cold-start supervised fine-tuning (SFT) stage could potentially improve final performance. However, SFT relies on large-scale, high-quality annotated trajectories of search-and-reasoning interactions, which are costly and difficult to obtain—posing significant challenges to scalability. In this work, we show that such intermediate trajectories can instead be acquired automatically through outcome-only reinforcement learning. The resulting RL-trained model can then be used to generate synthetic data for future cold-start SFT. We leave the systematic exploration of this direction to future work.

• Instruct-version model may also not be trained on data invoked with a search engine. How do these two fashions perform differently? You are right. The general instructed LLMs are typically not trained on interleaved reasoning and search engine invocation data, and their performance can indeed be improved with more targeted supervision. In this work, we demonstrate that both base and instructed LLMs can learn to perform interleaved reasoning and search behavior through outcome-driven reinforcement learning. Moreover, the resulting RL-trained models can be used to generate high-quality trajectories with explicit search engine usage, which can serve as synthetic data for future supervised fine-tuning.

• Baseline which performs rejection sampling or knowledge distillation for trajectory collection is missing. In this work, our primary objective is to propose a reinforcement learning (RL)-based method for training large language model (LLM) agents capable of interleaving reasoning and search. While knowledge distillation from a larger teacher model is a possible direction, it introduces additional supervision signals that may lead to unfair comparisons. That said, we agree with the reviewer that rejection sampling offers a reasonable and relevant baseline for comparison. Accordingly, we have included this baseline using both Qwen2.5-3B-it and Qwen2.5-7B-it. Specifically, we generate five candidate responses per training prompt from the Search-R1 dataset and select those that lead to correct final answers. These selected trajectories are then used to construct a new training set that retains the same multi-turn LLM–search engine interaction rollout mechanism proposed in Search-R1. We refer to this variant as Search-R1 (Rejection Sampling). The updated results are presented below, showing that Search-R1 with RL consistently outperforms Search-R1 with rejection sampling across both model sizes.

Qwen2.5-7b-it

Qwen2.5-3b-it

We sincerely thank the reviewer for the constructive feedback, which has strengthened our work. Please find our point-by-point responses below:

• The evaluation is only on two small variants of a single LLM and just QA task. Thank you for the valuable comments. In response, we have extended our evaluation to include additional LLMs—specifically, a 32B-scaled model and LLaMA-type models—as well as long-form generation tasks beyond QA. The results show that Search-R1 consistently achieves strong performance across different model architectures and a broader range of task types, demonstrating its generalizability and robustness.

Results on other LLMs

Qwen2.5-32B

Llama3.2-3B

Long-form generation task

• No results to show that training is stable. We conducted three times-repeated experiments and observed that the training reward curves are highly consistent, exhibiting only minor variance. Additionally, the standard deviation of the final performance on the test set is only 0.008, indicating the stability and reproducibility of our training process.

• Comparison between base and instruct. Our key observations are as follows: (1) Instruct-tuned LLMs exhibit stronger initial performance and converge more quickly during training; (2) Both base and instruct models ultimately reach similar training reward levels. However, due to the lack of transparency regarding the pretraining and supervised fine-tuning (SFT) data used for Qwen2.5 3B and 7B, it is difficult to make definitive conclusions about their final performance differences. If the SFT data includes reasoning or tool-calling demonstrations, this would naturally give instruct models an advantage in reinforcement learning, potentially leading to better final performance compared to base models. We will clarify this in the revised manuscript.

• The name of R1. We will modify the name to “R1-style RL” according to your suggestion.

• Algorithm 1 (force decoding). This issue typically occurs when the LLM fails to properly enclose a query or final answer within the expected special tokens. To mitigate this, we insert a prompt that subtly suggests the previous generation may be incorrect, encouraging the LLM to engage in self-reflection and revise its output accordingly.

• Minor comments in figure 1, page 4, sec 3.3, page 6, tab 2, sec 5.4. We will modify it according to your suggestions.

• What is “valid search”? A valid search is counted when the search engine is successfully called with LLM outputted special tokens.

We appreciate your insightful feedback and believe it has significantly strengthened our manuscript. We have carefully addressed each of your comments as detailed below:

• The implementation is preliminary. Given that reinforcement learning for interleaved reasoning and search in LLM agents remains underexplored, we intentionally begin with a clean and simple setting to establish a strong foundation. It is encouraging to observe that Search-R1 proves to be a simple yet effective RL methodology in this context. We view this as a promising initial point, and future works can explore more complex instructions and advanced reward designs (e.g., neural reward models).

• Format. We are using the “submission” option, but sorry, not enabling “\ifcolmsubmission”. Will fix it accordingly.

• Q1.1. Yes, they serve as inputs for policy rollout.

• Q1.2. To clarify, when computing the token-level loss, we mask out the retrieved tokens and only include the LLM-generated tokens in the loss calculation. However, the logits for these generated tokens are still conditioned on the retrieved tokens, as the retrieved content is provided as input during optimization. This setup ensures that while the model leverages retrieved information to inform its generation, it is explicitly trained to write effective queries and perform reasoning over the retrieved content, rather than memorizing external information.

• Q2. Thank you for the comments. We will make modifications accordingly.

• Q3.1. We conducted a study on Search-R1 using both Qwen2.5-3B and Qwen2.5-7B. The ratio of such sequences is 1.32% and 4.98%, respectively. Notably, this ratio is influenced by the maximum action budget B: as B increases, the ratio tends to decrease. A larger B also improves final performance, but at the cost of efficiency, as it leads to longer rollouts.

• Q3.2. You are right. It will remove the <search> </search> tokens and extract the query in between.

• Q3.3. Yes, the prompt in Table 1 is placed at the beginning of x. The input query x will not be changed during rollout.

• Q4.1. Yes, Search-R1 conducts interleaved reasoning and search engine calling during inference.

• Q4.2. In our experiments, we focus on the challenging open-domain setting where only the question is provided, without any additional context. This requires the LLMs to actively retrieve relevant information on their own. We will clarify this setup in the revised manuscript for better transparency.

• Q5. Sure, all the resources will be open-sourced.

We appreciate your insightful feedback, which has significantly strengthened our manuscript. We address each comment below.

• Performance on bigger models (llama models). Thank you for your feedback. In response to your request, we have added results on both the Qwen2.5-32B-Base and LLaMA3.2-3B-Base models.

Qwen2.5-32B-Base

Llama3.2-3B-Base

The results demonstrate that Search-R1 consistently outperforms strong baseline methods across various model sizes (3B, 7B, 32B) and architectures (Qwen2.5 and LLaMA3.2), highlighting its robustness and generalizability.

• How performance varies based on the choice of knowledge base or search engine. This is a great point. We conduct additional experiments to study how the choice of search engine (type of retriever + knowledge source) impacts both the training and inference performance of Search-R1.

Impact during training. We evaluate four search engines: (a) Random noise, (b) BM25 + Wikipedia, (c) E5 (ANN) + Wikipedia, and (d) E5 (Exact match) + Wikipedia. Results are shown in the table below. Key findings include: (1) Training with stronger retrievers (e.g., E5 (Exact) and E5 (ANN)) leads to more stable reinforcement learning and better final performance. (2) In contrast, using weaker retrievers (e.g., Random and BM25) significantly limits the final model performance.

Impact during inference. We further evaluate how models trained with different search engines generalize across various inference-time retrievers. The results (table below) show: (1) Search-R1 exhibits strong generalization across retrievers: even when trained with a specific search engine, it performs reasonably well when tested with others. (2) More powerful retrievers at inference time—particularly Google Search (via API)—consistently yield the best results, underscoring the importance of high-quality retrieval in downstream tasks.