Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging

Thank you for your positive feedback and thoughtful questions. We appreciate your recognition of the quality and usefulness of our method and dataset. We address each point below:

---

1. Data Release

Reviewer:
The newly constructed training data and evaluation data would be useful for the community. Do the authors plan to release them?

Response:
Yes, we will release both the training and evaluation data, and will continue to improve them for broader community benefit.

---

2. Qualitative Examples and Failure Modes

Reviewer:
Table 2's case study is helpful. Could the authors provide a bit more qualitative examples? Did the authors find any failure modes?

Response:
Thank you for your suggestion. We will add more qualitative examples to the revised version to illustrate both successful and failure cases in detail. A representative failuire question is:

• Question: If we assume all articles published by Nature in 2020 (articles only, not book reviews/columns, etc) relied on statistical significance to justify their findings and they on average came to a p-value of 0.04, how many papers would be incorrect as to their claims of statistical significance? Round the value up to the next integer. (correct answer: 41)

The reasoning trajectory of Inforage-7B and Gemini2.5 pro shows here:

Both methods failed on this example, but for different reasons.

For InForage, the model ignored the specific instruction in the question to use a p-value of 0.04, and instead searched for the typical p-value threshold for significance testing (i.e., 0.05). As a result, although it correctly retrieved the number of Nature papers published, it applied the wrong statistical threshold, leading to an incorrect final answer.

Gemini 2.5-Pro, on the other hand, failed to retrieve the exact number of Nature research articles published in 2020 after several search attempts. It resorted to estimating this number based on secondary sources, and while it followed the correct calculation process, the final answer was still incorrect due to the inaccurate input.

This case highlights a specific failure mode of InForage: improper task planning. In this scenario, the model should have first identified the correct number of Nature articles and then directly performed the computation as instructed. Its additional, unnecessary planning—such as searching for a typical p-value—introduced extra bias and ultimately led to the wrong answer.

We will include this failure example in the revised manuscript to illustrate current limitations and inform future improvements.

Thank you for your thoughtful and constructive review. We appreciate your recognition of the strengths of our work and your insightful questions and suggestions. Below, we provide detailed responses to each of your comments and outline the revisions we will make to address your feedback.

1. Details and Example for Information Gain Reward

Reviewer:
It is clear that information gain is measured as cumulative coverage of the retrieved patches against the expected knowledge set. It would be useful if the authors provided more explanation, perhaps even a concrete example, to further clarify this critical component in the reward function.

Response:
Thank you for highlighting the need for a more concrete explanation of the information gain reward. We are happy to clarify both the computation and provide an illustrative example.

The information gain reward measures how much of the expected knowledge set $\mathcal{D}^*$ has been cumulatively retrieved by the agent up to each reasoning step. At each step $t$, we evaluate the coverage as:

$C \left( \bigcup_{\tau=1}^t \mathcal{K}_\tau,\, \mathcal{D}^* \right)$

where $\mathcal{K}_\tau$ is the set of evidence (e.g., retrieved web pages) obtained at step $\tau$. The information gain reward is the maximum coverage achieved across all steps:

$R_\text{gain} = \max_{t=1,…,T} C \left( \bigcup_{\tau=1}^t \mathcal{K}_\tau,\, \mathcal{D}^* \right)$

Example:
Suppose the query is:
“Do I need a visa for this year's NeurIPS conference?”

The expected knowledge set ($\mathcal{D}^*$) includes:
(1) The official NeurIPS conference website’s visa information page, and
(2) The official government immigration website for the host country (providing general visa policy and requirements).

• Step 1: The agent retrieves the NeurIPS official website visa information page. Coverage is 0.5 (1 out of 2 gold sources found).
• Step 2: The agent retrieves the host country's government immigration website detailing visa policies. Coverage becomes 1.0 (both gold sources found).

At each step, coverage is calculated as the proportion of gold sources retrieved so far, and the information gain reward is the maximum value achieved during the search process. This encourages the agent to efficiently collect all necessary evidence through stepwise search.

We will add this concrete example to the revised manuscript for clarity.

---

2. Clarification of Web Browsing Trajectories in SFT

Reviewer:
Can authors explain more what they mean by web browsing trajectories? If it means prompts used to search the web it makes sense, however, if it consists of web URLs I am not clear how they are being used in SFT.

Response:

Thank you for your question. In Section 3.2, we define “web browsing trajectories” as the full step-by-step process by which a human or agent searches for and gathers information from the web to answer complex questions. Each trajectory includes the sequence of search queries, the URLs retrieved and visited, and the key facts or claims extracted from these web pages—all represented in structured JSON format. For example:

[
  {
    "query": "NeurIPS 2025",
    "id": 0,
    "page_content": "...",
    "url": "...",
    "extracted_claim": "NeurIPS 2025 will be held in ..."
  },
  ...
]

Next, we use strong LLMs to convert these JSON-format web browsing trajectories into plain text following the agentic reasoning paradigm (defined in Eq. 4), where the model alternates between internal reasoning, issuing help requests, and integrating retrieved web evidence at each step. Note that web URLs themselves are not directly used for SFT supervision.

During RL training, URLs are included as concrete evidence for intermediate reasoning steps, supporting the computation of the information gain reward and the supervision of retrieval behaviors.

3. Limitation: Computational Cost and Debuggability

Reviewer:
Computational cost compared with other alternative methods as opposed to just a barrier to explore bigger models for empirical studies. Lower debuggability due to using a more complex optimization scheme.

Response:
Thank you for pointing out these important limitations.

• Computational Cost:
  Our framework introduces additional computational overhead compared to some traditional approaches, not only as a barrier to scaling up model size, but also as an intrinsic cost of the more complex optimization and reasoning procedures required.

• Debuggability:
  IIncorporating tool calling into the reasoning process increases the complexity of debugging, as it can be challenging to pinpoint which specific step led to an error. However, compared to models that only perform end-to-end answer prediction, our approach provides richer intermediate records—such as the expected evidence set—which help illuminate the reasoning process. These intermediate signals not only allow us to better analyze the model’s behavior but also offer improved opportunities for debugging and error analysis compared to standard end-to-end approaches.

Thank you for raising these points. We will explicitly discuss these limitations and their practical implications in the revised manuscript.

We sincerely thank Reviewer sR8g for the thorough and thoughtful assessment of our work, as well as for highlighting the technical soundness, theoretical motivation, and practical contributions of InForage. We address each of your comments and questions in detail below.

1. Baseline Comparisons

Reviewer:
It might be helpful to compare InForage with FLARE (Forward-Looking Active REtrieval). Also, many closed-source LLMs have search APIs (e.g., GPT-4o); it would be more comprehensive to include closed-source LLMs in the experiments. Scalability validation: Given that most experiments use 3B models, can you provide more comprehensive results on larger models (13B+, like LLaMa3) to validate that the approach scales effectively?

Response:

(1). Per the reviewer’s suggestion, we have added experimental comparisons with FLARE on both Qwen 2.5-3B and 7B backbones. Across the evaluation datasets, InForage consistently outperforms FLARE.

(2). Per the reviewer’s suggestion, in addition to the 3B and 7B versions of InForage discussed in the paper, we also report results for InForage-14B. Furthermore, we evaluate against commercial closed-source LLMs, specifically OpenAI’s GPT-4o and GPT-4o-mini, both equipped with search tools via API. The results demonstrate that InForage achieves superior performance compared to these commercial APIs. A potential reason is that commercial APIs typically invoke the search tool in a single pass, which may not sufficiently accumulate information for multi-hop QA tasks.

Besides, across the 3B, 7B, and 14B backbone models, InForage consistently improves its performance, verifying the scalability of our approach.

---

2. Efficiency Penalty and Reward Design

Reviewer:
The efficiency penalty shows inconsistent benefits, suggesting the reward design may need refinement.

Response:
Thank you for your observation. As discussed in our paper, the efficiency penalty is designed to balance reasoning efficiency and depth. Its optimal value may depend on the nature of the task: tasks requiring fewer retrieval steps and more sensitive to efficiency benefit from a larger penalty, while tasks demanding deeper reasoning may benefit from a smaller penalty. Our ablation study shows that most evaluation datasets benefit from the efficiency penalty, except for the SELF dataset—which typically requires more reasoning steps, as it is intentionally designed to be as complex as possible. In practical scenarios, such highly complex tasks are uncommon, so the efficiency penalty remains effective for most real-world applications.

---

3. Coverage Function Definition, Implementation, and Sensitivity

Reviewer:
Key technical details lack precision – the coverage function C in Eq. 3 is undefined, and the operationalization of “information scent” from theory to implementation is unclear.
How exactly is the coverage function C in Eq. 3 computed? What constitutes the “expected knowledge set D” in practice, and how sensitive are results to this definition?

Response:
Thank you for highlighting these important points. In our framework, inspired by Information Foraging Theory, “information scent” refers to the evolving trail of reasoning steps and subqueries that guide retrieval. The coverage function $\mathcal{C}$ in Eq. 3 quantifies how well the union of all retrieved knowledge patches matches the expected knowledge set $$\mathcal{D}^*$), typically measured as the recall (or proportion) of gold evidence covered during the reasoning trajectory. Speficially,

• Coverage function computation:
  As described in Section 2.3 (and Eq. 6), the coverage function ( C\left( \bigcup_{\tau=1}^t \mathcal{K}_\tau, \mathcal{D}^* \right) $$ measures the cumulative coverage of all retrieved knowledge patches up to step $t$ $against the expected knowledge set$ $\mathcal{D}^*$ $. At each retrieval step, we compute the overlap between all retrieved sets so far and the gold set, and the information gain reward is defined as the maximum coverage achieved throughout the reasoning process.
• Expected knowledge set $\mathcal{D}^*$:
  In practice, $\mathcal{D}^*$ is constructed from annotated gold evidence, represented by the complete set of URLs, with each URL corresponding to a gold document deemed sufficient to support the final answer for each instance. This set serves as the authoritative reference for both reward computation and evaluation throughout training and testing.
• Sensitivity analysis:
  Through ablation studies, we find that the information gain reward consistently benefits our method design, validating its effectiveness across various datasets. This demonstrates that our approach remains robust as long as $\mathcal{D}^*$ accurately reflects the core factual basis for each instance.

This design rewards not only correct final answers but also reasoning trajectories that effectively accumulate relevant information, providing a principled and practical operationalization of information scent. We will further clarify these definitions and implementation details in the revised manuscript.

---

4. Scalability and Use of Self-Constructed Dataset

Reviewer:
Ablation study shows that performance heavily relies on self-constructed dataset, which might be a scalability concern.

Response:
We appreciate this important concern. The self-constructed dataset was carefully designed to align with our training schema and task requirements. In practice, this data brought a notable performance improvement. Our dataset construction process began with manual annotation and was subsequently scaled up using strong LLMs. To promote transparency and reproducibility, we have included the annotation interface and data generation scripts in the supplementary material. We believe this approach can be efficiently scaled or adapted to other domains by leveraging these resources.

---

5. Incremental Novelty over Related Reward Shaping Approaches

Reviewer:
Reward shaping (answer + coverage + length) resembles Search-R1-GRPO and REST-style self-training; novelty is incremental.

Response:
Thank you for raising this point. Incentivizing LLMs for search-enhanced reasoning is indeed an active area of research. Search-R1 is a representative recent work, which, like most similar approaches, applies only final answer correctness as the reward signal. At the time of our submission, there was limited systematic construction of reward signals tailored specifically for search-augmented reasoning models. Our core contribution lies in proposing a theoretically motivated learning framework, integrating carefully designed and complementary reward components, and constructing the necessary data to enable and evaluate this framework. We believe this solid step provides a practical foundation for further research in this direction.

---

Once again, thank you for your constructive feedback and detailed review. We will incorporate clarifications and additional results as described above in the revised manuscript.

We greatly appreciate reviewer XiLb’s recognition of the significance, clarity, and originality of our work. We are especially grateful for the positive comments regarding the introduction of information gain as a reward signal, the comprehensive ablation studies, and the clear improvements demonstrated over prior methods.

Your recognition is greatly appreciated by our research team.

Below, we address your questions and suggestions in detail.

---

1. Exploring Datasets with Gold Retrieval Labels & the Quality of the SELF Dataset

We fully agree that evaluating on datasets with many gold retrieval labels provides meaningful comparison. However, most widely-used QA benchmarks lack gold retrieval annotations, which motivated our construction of the SELF dataset.

As described in Section 3.2, the SELF dataset was built by first manually browsing the web to collect authentic human browsing trajectories and identify key decision points. To scale up data collection while maintaining quality, we leveraged strong LLMs as autonomous agents to plan and make retrieval decisions, based on these initial human examples. We then performed strict quality control, filtering out queries that could be correctly answered without retrieval, or that could not be correctly answered even with all relevant information.

We will add more details about the construction and quality assurance of the SELF dataset in the revised version.

Regarding the performance of GRPO on the SELF dataset, although one might expect GRPO to excel given the reward signal alignment, we observed that PPO slightly outperformed GRPO, especially on SELF.

We believe this is because our InForage model is first supervised-fine-tuned on constructed reasoning trajectories from SELF, giving PPO’s critic a well-informed starting point and leading to more accurate advantage estimation. In contrast, GRPO is less able to benefit from this supervised initialization, resulting in its slightly lower performance.

---

2. The Critical Role of SFT Data Collection

Thank you for highlighting this point. We agree that collecting high-quality trajectory data for SFT is particularly valuable at the current stage. High-quality trajectories provide rich, step-by-step supervision signals that closely mirror the actual reasoning and retrieval processes required by the target tasks. This, in turn, makes it easier for the model to receive rewards during the RL training stage, thereby alleviating the sparse reward issue.

Nonetheless, we believe that both high-quality data and algorithmic advances are crucial and complementary.

Rich supervision signals from data unlock the full potential of advanced algorithms, while improved algorithms help us better utilize and generalize from the available data.

We will further emphasize this interplay in the revised version.

---

3. Direct Evaluation on Deep Research Industry Systems

We appreciate this suggestion.

Direct evaluation on commercial deep research systems would indeed be valuable; however, current industrial systems are often closed-source, use non-aligned retrieval modules, or lack transparent implementation details, making direct comparison infeasible at this stage.

---

Responses to Specific Questions

1. How extensively did you attempt to get GRPO to improve over PPO?

As noted, the SFT stage enables PPO’s critic to start from a well-informed value function, leading to more accurate advantage estimation and higher performance in our setting. We conducted extensive ablation, but GRPO was generally less able to benefit from this supervised initialization.

2. Does your comment about learned rewards imply that training was not stable?

We found that GRPO’s stability depends on the choice of group size and the reliability of verifiable reward signals. In our experiments, PPO’s critic structure contributed to more stable and effective training.

3. Was Pass@K sufficiently high to your liking?

We primarily report pass@1 accuracy, as aggregating multiple answers from different rollouts is non-trivial for QA tasks and may not reflect realistic usage.

We consider pass@1 a more practical and fair metric.

As shown in our results, InForage consistently outperforms baseline models.

4. Can models automatically discover better trajectories, or do they mainly mimic SFT data?

Thank you for raising this point. Our SFT data, synthesized from real web browsing records, provides high-quality supervision and is critical for model foundation. During RL post-training, the model is encouraged to autonomously explore and discover improved trajectories guided by reward signals. In practice, RL-based post-training further improves performance on top of SFT, suggesting the model can move beyond simply mimicking supervised data.

5. Could information gain be approximated by a judge without access to gold documents?

Information gain is a crucial signal, but many datasets lack intermediate labels. To address this, we consider two potential strategies:

• (1) Using strong models with predefined workflows to simulate and recover valid intermediate steps, retaining those trajectories that ultimately lead to correct answers as additional training data;
• (2) Employing a relevance model to estimate information gain via query-document relevance.

However, the latter’s accuracy is highly dependent on the relevance model.

We will discuss these strategies and their limitations more thoroughly in the revised manuscript.

{
  "question": "What was the verdict in the case where the Australian soccer player known for her goal-scoring prowess, who pleaded not guilty to racially aggravated harassment, appeared at the British court located in a district sharing its name with a royal palace?",
  "answer": "Not guilty",
  "predicted_answer": "Sam Kerr was found not guilty of racially aggravated harassment at Kingston Crown Court. ([url1](url2))",
  "em": 0,
}