ReasonIR: Training Retrievers for Reasoning Tasks

We thank the reviewer for the insightful and comprehensive comments. We also thank the reviewer for recognizing our motivation, the efficiency and effectiveness in retrieval and RAG, and the insights from the ablation study. We address the comments below.

Weaknesses:

Q. The paper lacks an analysis of training scaling laws and data combination ratios based on the proposed synthetic data generation techniques.

A. We agree with the reviewer that studying the scaling behavior of the size and the composition of datasets is an important research direction. Within our computational budget, we prioritized ablations on data composition in Table 2 of the main paper. However, the scaling study is computationally intensive. For example, each training run requires 640 GPU hours. Data generation at the current scale requires thousands of GPU hours. Therefore, a full scaling study is out of our budget. Thus, we briefly discussed scaling studies for future work in Section 7. Nonetheless, we provide a preliminary study to check the potential of scaling below for this rebuttal.

Specifically, we subsample existing data into 50% of the full ReasonIR training data and retrain the model on this subsampled data. As shown in the table below, reducing the data to half drastically reduces the final performance. Based on the results, we hypothesize that further scaling up the data may lead to additional performance improvements. While our work initiates efforts to generate synthetic data for reasoning-intensive retrieval tasks, we leave it to future work to study the corresponding scaling laws and optimize the ratio of the data due to computational constraints.

Table. BRIGHT performance. ReasonIR-8B is trained on our full dataset (public+VL+HQ), and ReasonIR-8B-0.5scale is trained on public data and 50% randomly sampled VL and HQ data (public+50%VL+50%HQ).

Questions:

Q1. Could the authors provide details on the training data size of competing methods, e.g., GritLM vs. ReasonIR? What is the benefit when training GritLM with the same data as ReasonIR does?

A1. GritLM differs from ReasonIR in multiple aspects, such as the size and openness of the data, as well as the training objective. Overall, GRIT uses 2,216,784 samples that are not fully open-sourced while we use 1,729,368 samples that are fully open-sourced.

Firstly, the data used by GritLM is not fully open-sourced. As shown in the table below, GritLM partially uses GPT4-generated proprietary synthetic data, which is closed-sourced. On the other hand, all data used by ReasonIR is fully open-sourced. Secondly, GritLM also performs generative modeling using the Tulu V2 dataset, while ReasonIR focuses on training only the bi-encoder retriever. The GritLM paper claims that this additional training object helps GRIT achieve better performance on embedding/retrieval tasks.

The details of the training data composition between GritLM and ReasonIR are shown in the two tables below.

Table. Data Composition for GritLM and ReasonIR.

As stated above, the key differences between our work and GRIT are the data and the objective. Since we focus on embedding training, we did not apply the language modeling objective. We follow GRIT's training setup and include their publicly available training data in our training set. As shown in Table 2, training on this public data alone results in 19.6 nDCG@10 on BRIGHT. Adding VL and HQ data increases performance to 24.4. Therefore, we believe that training GritLM with ReasonIR data should improve its reasoning-intensive retrieval performance if we had access to the full training data (including the proprietary ones). We leave studying the best combination of these training techniques to future work.

Thank you for your response. My questions have been addressed and I have increased my score. Please include these discussions in the final paper.

We thank the reviewer for providing insightful comments and for recognizing both the novelty of our work and the efficiency and effectiveness of our model for reasoning-intensive retrieval. Our responses are as follows:

Weaknesses

Q1. The performance of the paper seems to rely on in-domain "high quality synthetic data", the out-of-distribution/generation capability is not examined.

A1. We thank the reviewer for initiating this discussion. In Section 5.2, we evaluated our retriever's generalization ability on RAG tasks (MMLU and GPQA) where no documents exist for synthetic data generation. As shown in Figure 1, our retriever brings consistent gains on both reasoning-intensive RAG tasks (4.5% gains on MMLU and 7.2% gains on GPQA compared with the closed-book baseline), outperforming GRIT-7B and you.com search engine baselines.

For BRIGHT, we agree that the domains of documents used to curate HQ data align with the benchmark. The 12 domains covered by seed documents are general and diverse, including math, biology, earth science, and economics. Importantly, we did not use any BRIGHT test queries to curate synthetic training data. The generation pipeline only accessed documents created by browsing and crawling STEM websites. We are optimistic that the benefits of our data can be generalized to other STEM IR benchmarks.

Additionally, though not explored here, our work could potentially be extended as a domain-adaptation method for resource-limited domains, as noted by Reviewer 4Dkf. For example, when accessing private documents, one could curate private training data following our approach without human-annotated queries. We will edit our paper to include this discussion on our method's generalization and add our discussion on its potential application as an unsupervised domain adaptation method in the discussion section.

Questions

Q1. In Table 1, why the Rank1 method does not work with the original query?

A1. Rank1 can also work with the original query. We apologize for not including it originally because the performance was not reported in its paper. In response to this question, we evaluate rank1’s performance (using top-100 candidates from BM25) on the original query in the table below. ReasonIR, having an average nDCG@10 score of 24.4, outperforms rank1’s 21.0 by a larger margin in this scenario (compared to the case with GPT4 Reason-query, where the nDCG@10 score is 27.5 for rank1 and 29.9 for ReasonIR). We will include the additional results in the revision.

Table. Reranking performance of Rank1 on BRIGHT with original queries.

Q2. Is the gap between Rank1 smaller on non-reasoning retrieval benchmarks? In other words, does author think the reranker is not neccessary for reasoning queries?

A2. We answer the two questions separately:

To answer the first question, we compare ReasonIR with rank1 on the subset of BEIR (a non-reasoning retrieval benchmark) tasks reported by rank1. As shown in the table below, ReasonIR-8B outperforms all rank1 variants. We will include these results in our next version of the paper.

Table. Comparing ReasonIR with Rank1 on a subset of BEIR tasks. Rank1 uses top-100 candidates retrieved via BM25.

For the second question, we think it is important to study and improve rerankers for reasoning-intensive retrieval, as it can provide complementary gains to the base bi-encoder retriever. As shown in Section 4.6, our simple ReasonIR-Rerank boosts ReasonIR’s performance w/ GPT4 reasoning from 29.9 to 36.9. We advocate future work to develop more efficient and performant rerankers for reasoning-intensive retrievers to result in further gains on top of ReasonIR.

We appreciate the reviewer's thoughtful feedback and the time invested in reviewing our submission. As the discussion period draws to a close, we remain available to provide any additional clarification or address further questions that would be helpful!

We thank the reviewer for providing insightful and detailed suggestions. We have updated the draft to fix the reviewer’s suggestions on presentation clarity, including

• Adding an example reasoning-intensive query from BRIGHT to the introduction to help clarify the reasoning in the context of retrieval.
• Providing a high-level summary of the content in the appendix when referring to it for details.
• Moving the definition of EQ before Table 2. The change will appear in the next version of our submission.

We address the reviewer’s other questions below:

Q1. Lacks definition of baselines in Table 1; OpenAI, Voyage, Google are not defined, and the rest of the baselines are only defined by citation without providing training details and ensuring fair comparison with the proposed approach.

A1. We apologize for the missing baseline definitions and thank the reviewer for pointing this out. The baselines and numbers originate from the official BRIGHT benchmark. We will supplement closed-source model references and note the score sources of these baselines in our next revision.

Q2. An analysis on why the combination with the sparse retriever is beneficial would be interesting to show cases where REASONIR fails and can be improved.

A2. We agree that a qualitative analysis will help to further improve the paper, since our quantitative results (Appendix G.3) show that BM25 and ReasonIR are retrieving different sets of documents, with only a 50% overlap. To answer the question, we analyzed failure cases of ReasonIR on BRIGHT. For certain queries, critical keywords appear in both the original query and the GPT4-reason query explicitly, allowing simple BM25 keyword matching to successfully retrieve relevant documents. However, as ReasonIR encodes the full semantic meaning of queries/documents as a dense retrieval model, the target keyword's semantic meaning can get diluted by other content, causing potential failure. As a result, we hypothesize that ReasonIR may not always effectively encode rare or specialized terms for long-form inputs.

For example, in a BRIGHT psychology task, a query about "kinetic synesthesia" (a perceptual phenomenon in which people may experience colors when listening to music) is easily addressed by BM25 via keyword matching. However, ReasonIR focuses broadly on "perception"-related documents rather than "synesthesia". A sparse retriever like BM25 can effectively increase the influence of certain rare keywords that ReasonIR may fail to encode. Alternatively, improving representations of rare but important concepts could further enhance ReasonIR's performance. We will include this discussion in our next revised version.

Q3. In certain cases, generating long queries can make the task easier because of high lexical match with the relevant document. How do you control that? Is the LLM able to always avoid high lexical match? Would other, smaller LLMs be able to deal with this complexity?

A3. We would appreciate more clarification–is the question asking about the difficulty control for our HQ or VL data?

For VL data, the focus of including it is to extend the retriever’s context length. Despite potential high lexical overlap, increased length makes VL data more challenging than public data—as shown in Figure 3, both GRIT-7B and BM25 have higher error rates on VL data than on public data. If the reviewer is asking about Reason-query: we use LLM to rewrite queries to make searching for targeted documents easier. Since query rewriting aims to make retrieval easier, higher lexical overlap is desirable in this process.

For HQ data, our data-generation prompt includes "avoiding high lexical overlap with relevant documents when generating hard queries" as a requirement (Appendix A.1). The quantitative assessment of the HQ data is discussed in Section 4.5, where we show that the HQ data generated using our prompt has the highest BM25 error rate of 53%, indicating the lowest lexical overlap.
To compare small LLMs' effectiveness at generating HQ data and lexical match levels, we generated hard queries for 1200 seed documents using Llama3.1-8B with the same prompt and compared their difficulty levels (described in Section 4.5) with Llama3.1-70B queries. As shown below, the queries generated from the 8B model have significantly lower difficulty. In particular, the low BM25 error indicates high lexical match between generated queries and positive documents, meaning the smaller LLM fails to follow our data-generation instructions precisely.

Table. Comparing the difficulty of queries generated by small and large LLMs.

We thank the reviewer for acknowledging our contribution on data generation and impact on future RAG research on data-limited domains. We also appreciate the reviewer’s insightful questions with regard to data quality and scale. We address the comments below.

Q1. It is not clear the impact of the quality of the synthetic data, since there exist no studies to verify it manually. If possible, better to run an experiment by trading the quality of the synthetic data and the actual retrieval accuracies by, e.g., injecting noises into the synthetic data by an LLM.

A1. Following the reviewer's suggestion, we manually inject noise into the data. Specifically, we ask an LLM to corrupt 10% of the HQ queries using the prompt below:

System:
You are a helpful assistant.
Your task is to corrupt the queries by rewriting them in one or more of the following ways:
1. Inject small factual errors or misleading words before or after the main query.
2. Add some off-topic or unrelated information to the query. 
3. Introduce multiple grammatical errors or typos in the query.

After rewriting the query, output the new query strictly following the json format below.
```json
{
    "corrupted_query": <your rewritten query here>. 
}

User:
Here is the query you need to corrupt:

{query}

The other 90% HQ queries remain unchanged.

We train a new model, ReasonIR-8B-perturbed, on this new data using the same training configuration as ReasonIR-8B. As shown in the Table below, injecting noise into 10% of the HQ data slightly harms the averaged performance—the BRIGHT performance slightly decreased from 24.4 to 24.0. We hypothesize that it is because our query perturbation prompt, which adds minor errors and irrelevant information, only slightly harms the data quality. We will add these new results and discussions into the next version of our paper.

Table. BRIGHT performance. ReasonIR-8B-perturbed is trained on the same dataset as ReasonIR-8B, but with 10% of the data perturbed.

Q2. It is also interesting to measure the impact of the synthetic dataset sizes to see if more data will be helpful or not.

A2. We agree that studying synthetic data scaling trends would provide useful insights for future work. We provide a preliminary scaling study by subsampling 50% of the full ReasonIR training data and retraining the model (see ReasonIR-8B-0.5scale in the table). As shown below, halving the data reduces nDCG@10 performance from 24.4 to 20.8 with original queries and from 29.9 to 28.4 with GPT4 Reason-query. We conjecture that generating more training data could yield further improvements.

However, the scaling study is computationally intensive. For example, each training run requires 640 GPU hours. Data generation at the current scale requires thousands of GPU hours. Therefore, a full scaling study is out of our budget. Within our computational budget, we prioritized ablations on data composition in Table 2. We briefly discussed scaling studies for future work in Section 7 and will expand within our budget. We will add these new results to the next version of the paper.

Table. BRIGHT performance. ReasonIR-8B is trained on the full dataset (public+VL+HQ), while ReasonIR-8B-0.5scale uses public data plus 50% randomly sampled VL and HQ data (public+50%VL+50%HQ).