RARe: Retrieval Augmented Retrieval with In-Context Examples

Thank you for your review! We address your comments below.

Unclear why MS-MARCO has been excluded

We do not exclude MS-MARCO for any experiment. For LLM2Vec-Llama-3-8B-Supervised and E5-Mistral-Instruct, MS-MARCO is part of the In-Domain split of BEIR (among 4 other datasets). See Table 6 in the Appendix for individual numbers on MS-MARCO on these models.

When we train on MS-MARCO, all other datasets in BEIR are out-of-domain, which is why we report MS-MARCO separately for those experiments.

We will update Tables 1 and 2 to enhance clarity.

Given that there a dense retrieval versions of PRF, comparing to a BM25 approach is not very informative.

Below, we provide a comparison against Query2Doc [1]. This is a PRF method which uses a proprietary generative LLM to augment queries and a fine-tuned dense retriever instead of BM25 for embeddings.

We see that RARe offers higher improvements on average compared to Query2Doc over their respective baselines.

[1] https://aclanthology.org/2023.emnlp-main.585.pdf

Table 1 has a training data column, but Table 2 does not, why?

We apologize for the confusion. We aimed to highlight the main aspect that we were varying/studying in each table. All models in Table 2 use the same training dataset. In Table 1, Promptriever uses an additional synthetically augmented dataset (Line 170) from GPT-4o. We will make this clearer in the revision.

Table 2 has an "instruct" row, but Table 1 does not, why?

For Table 1, the baseline RepLLaMA is the equivalent of “Instruct”. Since RepLLaMA only trains on a single task MSMARCO, it provides null string as the as instruction (Line 171).

Latency

As shown in Table 10, the cost of constructing in-context examples is minimal, since we retrieve from the queries in the training/validation set rather than the full retrieval corpus. The primary source of latency arises from encoding longer queries by the LLM - a limitation that is inherent to any PRF-based method involving query expansion. Compared to other query augmentation approaches such as ReDe-RF [1], CSQE [2], or HyDE [3], which involve generating hypothetical documents using LLMs, RARe may also be more efficient as it only requires a single forward pass through the LLM.

In our current implementation, no additional optimizations such as prompt caching are applied. With such optimizations in place, we expect the additional latency of query encoding to also be minimized. Moreover, the cost of encoding longer queries is typically outweighed by the time required for similarity search using FAISS over large indexes. For example, the 40x latency on NFCorpus is an artifact of a very small index (ony ~3000 documents to retrieve from). We acknowledge that efficiency wise, ‘Instruct’ may be a better choice over RARe (or any other PRF or query expansion method) when the document index is very small.

[1] https://arxiv.org/abs/2410.21242

[2] https://arxiv.org/abs/2402.18031

[3] https://aclanthology.org/2023.acl-long.99/

Statistical tests would be helpful

We provide statistical significance tests for each individual dataset in the Appendix (Tables 6–9) via Fisher’s method. * indicates p<0.05.

RARe is not consistently superior to Promptriever on BEIR, which benefits from additional synthetic data generated using GPT-4o, but achieves statistically significant improvements on 8 out of 15 BEIR datasets: CQADupStack, FiQA, NFCorpus, HotpotQA, MS MARCO, Natural Questions, TREC-COVID, and Touche2020. When compared to RepLLaMA, RARe demonstrates statistically significant gains on 11 out of 15 datasets. On RAR-b, RARe significantly outperforms Promptriever on aNLI, PiQA, TempReason, and WinoGrande (Table 9).

We address additional comments below

RepLlama and Promptriever baselines

We acknowledge that Table 1 could be clearer regarding the training setup for the baselines and RARe’s use of demonstrations.

RepLLaMA was re-trained using the publicly available implementation provided by its authors, as the paper originally used Llama-2. The best performing Promptriever models (which use more recent Llama-3) were already hosted on huggingface, which we evaluated on with dataset-specific prompts provided in the official code.

As for RARe, it does use demonstrations in Table 1 (5 examples, as noted in Line 135).

ICL vs. Fine-tuning

We agree that when thousands of demonstrations are available, fine-tuning may be more effective.

The advantage of RARe is the ability to adapt in a few-shot manner to new tasks or domains at test time via a small number of examples. RARe, even when trained with the Retrieved format and evaluated with Random (Retrieved-Random), still benefits over Random-Random and Instruct (Figure 3).

This shows that unlike fine-tuning approaches, RARe does not require generating larger amounts of synthetic data or training the model on new tasks, providing the user with fine-grained control of the retrieved documents by providing representative demonstrations at inference time. Moreover, RARe does not rely on domain-specific hyperparameter tuning other than number of few-shot examples.

BM25 vs Dense

We empirically observed that BM25 is lightweight yet effective. As shown in Table 10 Appendix, the primary contributor to latency is the query encoding step performed by the dense encoder. Since BM25 performed reasonably well in our experiments, we opted to use it throughout.

RL and Joint Objective For Generation & HyDE-like Approach

This is indeed an interesting direction to pursue! Retrievers that first generate and then embed potentially trained using a reinforcement learning or joint generative objective could combine the strengths of both generative models (used in HyDE-like approaches) and dense retrievers. While exploring this approach is beyond the scope of the current work, it presents a promising avenue for future research.

Thank you for your review! We address your comments below.

Discussion w.r.t BGE-ICL

Thank you for the comment! We discuss our work in relation to this work below. We agree with the reviewer about a comparison of the method. Our Random-Random baseline in Figure 3 is the same as the training pipeline of BGE-ICL with two minor differences: (1) The training data, and (2) their approach uses a variable number of ICL examples during training.

We highlight two new contributions:

1. While both work explores in-context examples for encoding queries, our work further proposes retrieving semantically similar in-context examples. We empirically show semantically similar examples are beneficial, whether used during training, inference, or both (Figure 3).

2. Detailed analysis of the method:

   (i) We demonstrate that this can be applied to both LLM and retriever checkpoints.

   (ii) We present an ablation to study whether ICL format matters or simply query expansion helps. This is shown in Table 4, where we compare against various baselines (shuffled, queries-only, etc).

   (iii) Analogous to the role of negative examples in contrastive learning, we investigate whether incorporating negative docs into the context yields performance gains. Our findings indicate that such inclusion offers no substantial advantage over only positive docs (Table 5).

   (iv) We include an efficiency analysis, breaking down the contribution of each component (Table 10).

Our work was developed concurrently but as Li et al is already accepted to a conference, we will revise our draft to re-position our paper accordingly. While it is hard to argue using in-context examples for retrieval as our novel contribution, our work significantly deepens the study of using in-context examples in retrieval models.

Query-Expansion vs ICL & BM25

Thanks for thoughtful feedback! We agree understanding whether adding in-context examples is merely query expansion or not is important to understand.

We compare RARe with ReDe-RF [1] and CSQE [2]: both PRF (Pseudo Relevance Feedback)-based query augmentation approaches (extensions of HyDE), in Table 17 of the Appendix.

We additionally tried expanding the queries to BM25 with the same demonstrations - i.e. we augment retrieved in-context examples at test time to the query. We provide results in the table below. Note that since BM25 is a sparse retriever, it cannot be trained with our approach, rather only an inference-time modification can be made, similar to the experiments in Figure 2.

We observe that performance drops when using BM25 with retrieved in-context examples, similar to the degradation seen with inference-only modifications in dense retrievers in our experiments (Figure 2). This highlights several key points:

1. The assumption that performance degradation in dense retrievers under inference-only modification is due to longer input context may not be true, since even sparse retrievers like BM25 exhibit a similar drop.
2. If the benefits were simply due to query expansion, BM25 should have improved, as it does with ReDe-RF [1]/CSQE [2], which works out of the box for sparse retrievers.
3. This underscores the importance of our approach, where training with in-context examples is critical. It enables the model to learn how to utilize relevant signals and mappings from the Query-Document pairs in context effectively.
4. Our experiments in Table 4 additionally disentangle the effects of expanding the query from following the ICL format, by training and evaluating with alternative formats. Here, the best performance is achieved with the full ICL format, further indicating that the improvements cannot be explained by query expansion alone (shuffled setting, despite having the same content but trained and evaluated with the shuffled format does not perform as well).

[1] https://arxiv.org/abs/2410.21242

[2] https://arxiv.org/abs/2402.18031

Thank you for your review! We address your comments below.

Similarity with Li et al, ICLR 2025.

Thank you for the comment! We discuss our work in relation to this work in detail below. You are correct that our method (RARe) is very similar to Li et al (2024). Our Random-Random baseline in Figure 3 is the same as the training pipeline of BGE-ICL with two minor differences: (1) The training data, and (2) their approach uses a variable number of in-context examples during training.

We highlight two new contributions of our work:

1. While both work explores in-context examples for encoding queries, our work further proposes retrieving semantically similar in-context examples. We empirically show semantically similar examples are beneficial, whether used during training, inference, or both (Figure 3).

2. Detailed analysis of the method:

   (i) We demonstrate that this can be applied to both LLM and retriever checkpoints.

   (ii) We present an ablation study to study whether ICL format matters or simply query expansion helps. This is shown in Table 4, where we compare against various baselines (e.g., shuffled setting, queries-only setting, etc).

   (iii) Analogous to the role of negative examples in contrastive learning, we investigate whether incorporating negative documents into the in-context learning (ICL) context yields performance gains. Our findings indicate that such inclusion offers no substantial advantage over using only positive documents (Table 5).

   (iv) We include an efficiency analysis, breaking down the contribution of each component (Table 10).

Our work was developed concurrently but as Li et al is already accepted to a conference, we will revise our draft to re-position our paper accordingly. While it is hard to argue using in-context examples for retrieval as our novel contribution, our work significantly deepens the study of using in-context examples in retrieval models.

For OOD dataset, examples are synthetically generated. How does the quality of the synthetically generated questions affect RARe performance?

We clarify that among the OOD datasets: NFCorpus, FiQA-2018, SciFact, and DBPedia do not use synthetically generated examples, rather their original training splits. For the rest (ArguAna, ClimateFEVER, Covid, CQADupStack, SCIDOCS, Touche2020), synthetic examples used are provided by BEIR and were not generated by us. We will include these details in the Appendix of the updated draft.

While the quality of synthetically generated examples may influence performance, we did not observe any consistent correlation between performance and whether the examples were synthetic. Investigating this further is indeed an interesting direction but is beyond the scope of this work.

Thank you for your review! We address your comments below.

I'm wondering if how does the proposed methods compared to pre-pending the query with in-context queries instead of examples.

Thanks for the suggestion! We perform careful ablation and present the result in Table 4. The experiment you are asking for is represented as the “Queries-Only” row in Table 4. This provides only in-context queries, rather than full examples. We find this underperform base Instruct or the full ICL format on average. We copy-and-paste results here again for convenience.

Also, how does this method compare to BM25 with original query as a baseline?

All of our LLM-based dense retriever baselines outperform BM25 as a retrieval method. We provide a comparison below.

Please also refer [1] and [2], which show that our baseline E5-Mistral > Contriever > BM25 on the BEIR benchmark.

[1] https://arxiv.org/pdf/2401.00368

[2] https://arxiv.org/pdf/2112.09118