RARe: Retrieval Augmented Retrieval with In-Context Examples

Thank you for the review! We address your concerns below and in our updated draft (changes are highlighted in blue).

W1/Q2

In-context Learning vs. Query expansion

We agree that training and evaluating with different input formats will show more conclusively the difference caused by input formats, rather than a setting where we only change the evaluation input format.

We provide the results on training and evaluating with different formats (including doc-only) in Table 4 in the updated paper. We find that even in this setting, our proposed “Instruct + IC” i.e. Regular outperforms other settings, such as query-only or document-only input format.

W2

This paper lacks an in-depth analysis of the generated embeddings.

We appreciate the reviewer’s suggestion to conduct additional analyses. Prior literature has explored the impact of ICL example format [1], order [2], and diversity [3] on downstream performance in decoder-only LLMs. Although such analyses are beyond the scope of this work, future studies could extend this line of research by examining aspects like attention patterns to gain deeper insights into the role of ICL in retriever models. Please also see our general response on format-sensitivity of ArguAna.

W3

The training strategy is too simplistic & in-batch negatives

We believe that this simplicity is a strength rather than a limitation, as it allows for clarity in isolating the impact of our proposed approach. Moreover, we validate its effectiveness comprehensively across multiple large-scale retrieval benchmarks and provide detailed ablations and analytical experiments.

Our method builds upon the commonly used in-batch negatives strategy from prior work (e.g., LLM2Vec [4]), where each in-batch negative is randomly sampled from different tasks, as described in Line 3 of Algorithm 1.

The focus of our work is to investigate whether retriever models can effectively leverage in-context examples, laying a foundation for future studies. These could explore developing more sophisticated objectives, such as improved methods for selecting in-batch negatives, to further enhance training with in-context examples.

W4/Q1

Analytical experiments on all OOD

We have extended Figure 3 and Figure 4 with all OOD datasets from BeIR. We present these results in Figure 5 and Figure 6 in the Appendix due to space constraints. The results on full OOD datasets from BEIR on other analytical experiments are already provided in the Appendix. We observe similar trends as what we see in the previously selected datasets:

(i) Retrieved/Retrieved performs the best on average, and Retrieved during eval is second best on average. Using retrieved examples either in training or evaluation (or both) offers performance enhancements on 7/10 datasets. Note that TRECCOVID is the only dataset where training retriever checkpoints further with Instruct led to a sharp decrease in performance with respect to the base model.

(ii) We observe that more similar examples yield improvements in performance over the base model on 6/10 datasets, and do not offer any additional gains on the rest.

Q3

Evaluating more embedding tasks and scenarios, such as classification, reranking, and clustering

Our focus in this paper is to study whether the retrieval tasks in particular can be augmented with in-context examples. We have extensively evaluated this hypothesis on multiple large-scale retrieval benchmarks with detailed ablations and analytical experiments.

Generally, trends on retrieval tasks generalize to other embedding tasks as well [4, 5]. We did not evaluate mainly due to extensive compute costs (as this is not the focus of our work). We agree with the reviewer that exploring ICL for other embedding-based tasks is an interesting future work direction.

References

[1] Reframing Instructional Prompts to GPTk’s Language (Mishra et al., ACL 2022)

[2] Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (Lu et al., ACL 2022).

[3] How Do In-Context Examples Affect Compositional Generalization? (An et al., ACL 2023)

[4] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (BehnamGhader et al., COLM 2024)

[5] Improving Text Embeddings with Large Language Models (Wang et al., ACL 2022)

Thank you once again for your review. As the discussion period is nearing its end, we wanted to follow up to confirm that we have adequately addressed your concerns and to kindly request if you would consider reevaluating your assessment.

To summarize our updates:

1. Query Expansion: We added results showing RARe format outperforms Doc-Only when training with Doc-Only.
2. Training Strategy: Emphasized the simplicity of our strategy which facilitates clarity of isolating the impact of our proposed approach.
3. OOD Experiments: Extended Fig 3 and 4 analysis to all BEIR OOD datasets.
4. Clarified the scope of our work and highlighted analysis of embeddings and other embedding tasks as a future research direction.

We hope the additions clarify our contributions and resolve your concerns. Please let us know if you have any further questions or if there are additional issues we can address.

Thank you for the review! We address your concerns below and in our updated draft (changes are highlighted in blue).

W1

While the application of in-context learning to retrievers is new, it might not strike everyone as a groundbreaking shift.

While the application of in-context learning to retrievers may seem simple, we show that it does not function effectively out-of-the-box (Figure 3), unlike its success with generative models. We do believe that demonstrating this distinction and proposing a method for adapting in-context learning for retrievers represents a meaningful contribution to the field.

W2 & W3

Ease-of-use and real-world applications

Our approach is straightforward to use, as it involves simply pre-pending in-context examples in the form of (q, d+) pairs to the query using a lightweight retriever like BM25. However, a potential challenge in real-world applications is the availability of suitable (q, d+) pairs, similar to the requirements for in-context learning in generative models, as discussed in L520.

Efficiency-accuracy tradeoff

This aspect is discussed in our experiments in Table 6. For reviewer’s convenience, we summarize the discussion here: for very small corpus sizes (<500K documents), the performance gains from RARe may not justify the additional latency. However, in large-corpus scenarios (>4M documents), the added latency reduces, making RARe an effective solution for such retrieval tasks. In real-world scenarios, as the index size grows, the added latency of RARe will be less apparent.

Performance gain looks a bit incremental

Please see our general response.

Dear Reviewer 3GiC,

We have addressed your concerns with additional context and clarification in the “Follow-up and Additional Clarifications” post, emphasizing the significance of our work. As the rebuttal phase draws to a close, we wanted to ensure our response sufficiently addresses your points. We kindly request you to consider revisiting your evaluation. Should you have any further questions or additional concerns, we would be more than happy to address them promptly.

Thank you for the review! We address your concerns below and in our updated draft (changes are highlighted in blue).

W1

I am not sure how much value will the less than 2% improvement add...

Please see our general response.

There are other leading models on BEIR benchmark and how will the proposed methods compare to those and will those method get improved after adding in context example?

Theoretically, RARe can be applied to any retriever model, as it is a straightforward approach that involves prepending (q, d+) pairs to the original query and fine-tune with this setting. However, we find that fine-tuning with publicly released datasets often hurt performances of top retriever systems in the leaderboard. For example, we tried fine-tuning one of these models (Linq-Embed-Mistral, 6th in the leaderboard now) further with the "Instruct" format (i.e. without any in-context examples) using our public data, it led to a significant decrease in performance. To compare fine-tuning with in-context examples vs. not, it seems we might need to train with their training data mixture, which is not available.

Thus, we chose three different base architectures – LLM2Vec, E5-Mistral, and RepLLaMA, which are top-performers among models trained on publicly available training data. We do emphasize that our focus is not solely on achieving state-of-the-art performance but on developing a conceptual/empirical understanding of the potential of incorporating in-context examples into retrievers.

Q1. DBPedia vs. Quora search time

This is because we report the total time required to search for all queries in the test set, not the time required for individual examples. At an individual level, DBPedia's example takes longer than Quora, for their corpus is bigger, as you commented. DBPedia test set has fewer queries than Quora (Table 1 in [1]). To avoid confusion, we updated Table 6 by normalizing by the number of queries, reporting the latency numbers in milliseconds per query.

[1] BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (Thakur et al., NeurIPS 2021 Datasets and Benchmark)

Thank you for your thoughtful feedback.

We would like to kindly re-emphasize that while the improvements may appear modest, they are significant in the context of large-scale IR benchmarks. This is particularly evident when considering the papers that introduced task-specific instructions (Instruct) [1, 2]. These works achieved comparable ranges of improvements over their respective baselines, and the approach has since been adopted by leading models on BEIR, including those used as baselines in our work.

Furthermore, there is precedence in both Machine Learning and Information Retrieval literature (cited below) where improvements of 1-3% nDCG@10 are considered impactful, since nDCG is highly sensitive to the ordering of the retrieved results [12]. For example, concurrent work [8] https://openreview.net/forum?id=wfLuiDjQ0u explores the use of random in-context examples to improve text representations, and reports an improvement of 0.97% on retrieval and 1.25% on other representation tasks, relative to zero-shot fine-tuning (i.e. Instruct).

Given these considerations, we believe our findings contribute meaningfully towards future work in this space. We are open to further suggestions to improve the clarity or framing of this aspect.

[1] One Embedder, Any Task: Instruction-Finetuned Text Embeddings (Su et al., ACL 2023)

[2] Task-aware Retrieval with Instructions (Asai et al., ACL Findings 2023)

[3] Learning List-Level Domain-Invariant Representations for Ranking (Xian et al., NeurIPS 2023)

[4] RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses (Zhuang et al., SIGIR 2023)

[5] How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval (Lin et al., EMNLP 2023)

[6] Document Expansion by Query Prediction (Nogueira et. al, 2019)

[7] Promptriever https://openreview.net/forum?id=odvSjn416y (ICLR 2025 Submission)

[9] Fine-Tuning LLaMA for Multi-Stage Text Retrieval (Ma et. al, SIGIR 2024)

[10] Rethinking the Role of Token Retrieval in Multi-Vector Retrieval (Lee et al., NeurIPS 2023)

[11] Adversarial Retriever-Ranker for Dense Text Retrieval (Zhang et al., ICLR 2022)

[12] A Theoretical Analysis of NDCG Ranking Measures (Wang et al., JMLR 2023)

[13] Contextual Document Embeddings https://openreview.net/forum?id=Wqsk3FbD6D (ICLR 2025 Submission)

Thank you for the review! We address your concerns below and in our updated draft (changes are highlighted in blue).

W1

There are no statistical significance tests

Please see the general response.

W2

Only a basic retriever, BM25, was applied.

We appreciate the reviewer’s suggestion to explore stronger retrievers than BM25. We chose BM25 for its efficiency, which offers notable performance despite its simplicity. Using more powerful in-context example retrievers could potentially provide even further gains on our method, which can be studied in future exploration (L526).

W3/Q2

Performance on ArguAna

Please see the general response.

W4/Q1

Figure 4 appears to contradict the paper’s premise.

We apologize for the confusion. In Figure 4, we initially reported the score@Top-1 after normalizing the scores of top 5 retrieved examples. This is because the BM-25 implementation that we use returns the (un-normalized) scores of only the retrieved documents, which are not between 0 and 1. This may have inadvertently biased the x-axis—if all retrieved examples had high scores, the score@Top-1 would appear lower. We have updated the figure by computing similarity scores on the retrieved examples using an off-the shelf model (https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), and grouping by score@Top-1. Our updated figures show improvements with higher similarity on 6/10 datasets, while the gains are less pronounced on the rest.

Dear Reviewer PxAH,

Thank you for your valuable feedback and comments. With the rebuttal phase drawing to a close, we wanted to follow up to ensure that our responses have sufficiently addressed your concerns. We kindly ask that you consider revisiting your evaluation. If you have any further questions or additional concerns, we would be more than happy to address them. Additionally, please refer to our two general responses, which include some additional clarifications and remarks.