Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models

We thank the reviewer for noticing that our work "mark[s] a significant advancement in retrieval systems" and offers a "scalable and cost-effective solution"!

The use of varying amounts of training data for different models (Promptriever vs. RepLLaMA) raises concerns about the fairness and validity of the comparative performance analysis.

We show ablations with the same training data in Table 4, for our models except our final Joint version which combines all the instruction and standard data. We see that even with the same amount of training data (e.g. the “w/Instruction Negatives row”) there is a large gain in performance on instructions: +11.4 p-MRR, +3.2 nDCG@10 on FollowIR, +5.8 nDCG@10 and +13.1 Robustness@10 on InstructIR. As shown in our t-tests, we see these differences are statistically significant also.

The performance improvement on in-domain retrieval is very marginal. I guess the instruction-tuning process may affect the original retrieval performance.

We agree that we do not show gains on in-domain data without instructions (as stated on lines 250-251).

However, we show comparable performance with no instruction-based evals (e.g. it doesn’t hurt to add instructions) while also showing significant improvements on data that does have prompts/instructions (Tables 1 and 3).

As you say, it is intuitive that adding instruction-based training data doesn’t help the model more than normal when it’s not given instructions/prompts at test time. However, as we show, adding instructions does enable new abilities when they are given.

In Table 5, the selected baselines are not strong on BEIR benchmark. For example, E5-mistral has achieved 56.9 on BEIR. I think the authors should compare their methods with more advanced baselines.

We do include E5-Mistral in Table 1 for instructions, as well as in Table 12 in the appendix on BEIR.

As E5-Mistral was trained on significantly more data than just MS MARCO, including the training sets of BEIR, we cannot equally compare. BEIR was designed to be a zero-shot test set, and when models train on the training sets, we are unable to equally compare their performance as it is no longer zero-shot for them..

However, we show in Table 12 that on the few datasets in BEIR that are truly out of domain (no train data) that we score similarly to E5-Mistral despite training with much less data. This showcases the usefulness of instruction-training.

A minor suggestion: please use `` and ‘’ (two single quotes) to generate double quotes in LaTeX.

Thank you for the suggestion, we have fixed this.

Thank you for the response!

However, I maintain that the observed performance improvements are likely attributable to the training data being supplemented with instructions. This approach appears to yield a noticeable advantage over models that were not trained similarly. Therefore, the comparison is not very "fair".

The main contribution of our work is indeed the careful, scaled data curation process, which (as you noted) provides a “noticeable advantage”. Our contribution is not a new neural architecture. The comparisons we made (including the data volume ablation --- which shows that training with our instruction corpus, and not just training with more data, indeed causes performance improvements) are intended to illustrate the advantages of our data curation method.

We are the first to propose using query-specific instructions data for bi-encoders in IR and we do indeed show it provides advantages (but only if done carefully, as we describe and ablate). Many works curate new training sets as the main contribution, as we also do.

If you still feel this is unfair, would you be willing to elaborate further?

We thank the reviewer for their time and for their feedback that our work provides a "novel idea" and a "solid resource"!

For out-of-domain retrieval, what is the rationale behind selecting the best score out of 10 prompts, rather than fixing a prompt or reporting an average score?

In the first half of Table 4, we report results in a train/dev/test setting, selecting the prompt for each dataset that works best on the dev set (then reporting on the dev set). This standard ML setup simulates how a system might be used in practice.

We agree that the options you mentioned would also be interesting+valid; however, they would underestimate the model. The reason we chose the train/dev/test method is: 1) to show that you can reliably improve retrieval performance, just with prompting (like with LLMs: practitioners generally optimize a prompt for each task/dataset [1] using a dev set); and 2) to illustrate that there is no “magic bullet” prompting retrieval models on all datasets/tasks (e.g., even “let’s think step by step” only works for math and logic problems for LLMs [2]). This thus aligns IR with ML/NLP and simulates the setting that would happen in practice if someone was trying to apply Promptriever to new out of domain data.

References:

[1] “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines” by Khattab et. al. 2024

[2] “To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning” by Sprague et. al. 2024

In Table 11, simpler prompts like “Think carefully about relevance” seem to yield strong performance. Why might these simpler prompts outperform other more detailed ones that closely match the training data?

This is a great question! We agree that we were also unable to find any correlation between score and prompt. It seems, similar to LLMs, that prompting has various quirks that may be dependent on the model and data (see Limitation, lines 475-478). Unfortunately, as this is still an issue with LMs, it seems like it will carry over into IR as well.

Could the authors provide some qualitative examples illustrating how Promptriever performs across different categories in Table 6?

Unfortunately the examples in Table 6 are synthetic and are used for contrastive training. Thus, the only examples we could show are that the model contrastively prefers one over the other (there is no test set, just one positive and one negative document per instruction for training).

We would be happy to include more qualitative examples from the test sets however. We show one example from FollowIR and one from InstructIR and show it in the general comment section.

We thank the reviewer for their time and for noticing that our work has a "clear" problem definition with an "easy to follow" and "effective" method!

The dataset is generated solely based on MS MARCO. It would be great if more datasets could be considered, especially for the test. BEIR should also be considered for the zero-shot scenario.

We do evaluate on BEIR in Table 3 and have a comparison to more models in Table 12 in the appendix.

BEIR is designed to be a zero-shot test set, meaning that they do not want authors to train on the training sets, so we do not train on them. In Table 12 we compare to models which did train on the training sets, but show on datasets without training sets (e.g. truly OOD) that we approach SOTA.

In fact, this showcases the strong performance of Promptriever’s training recipe, as we achieve comparable results while only focusing on MS MARCO for training – without training on the millions of other examples (and domains) used by other models.

It would be great if the definition of instructions in IR could be clearly defined. For example, given the example "Which type of volcano eruption has not been seen?", the authors claim that the volcano types and formations can be added as additional instructions. I wonder why these are treated as additional instructions but not part of the original information need (i.e., the original queries).

Typically in IR a query is short string, to the point, and often ambiguous. We evaluate on many standard+popular text IR datasets like MSMarco, BEIR, TREC, etc that define queries this way.

FollowIR and InstructIR (our other evaluation corpora) make queries more precise with instructions. An instruction could be: specifics for what defines relevance, specifics that define not-relevance, or any other background / additional information.

We will add this definition and discussion in revision, thank you.

More details about the experimental setup should be provided. For example, are the baseline models trained using similar assessment data (with instructions in the queries)?

The baselines in Table 4 do train with instructions and are the ablations for how we picked our recipe. All of the models except the final “Joint” model have the same data, as the Joint model fully combines the full instruction and non-instruction dataset, whereas the others split it (lines 355-357).

For all the other results, the baselines are not trained with similar data. This is the main contribution of Promptriever: we are the first to suggest using query-level instruction data in dense retriever training. We show the difference these instructions make by directly using RepLLaMA’s recipe but adding instruction data.

Better baseline models should be added. For example, we can generate query rewrites via LLMs and then retrieve results using the rewritten query (converting the NLP-like instruction into keyword-like queries).

We agree that one could have an LLM re-write a query and search with that (called query expansion). Query expansion is well-studied and has been shown to help retrieval models generally. As such, it is orthogonal to our work as these query expansions could be applied to instruction-based retrievers also.

As our paper is focused on comparing the retrieval models themselves, we leave the analysis of the impact of query-expansions and their interactions with other LMs to future work.

We thank the reviewer for their time and for noticing that our paper's "motivation is strong and the story is convincing"!

Can you list out some use cases where the promptriever is used in scenarios such as RAG? How would it perform?

This is a great question - RAG crucially depends on the results gathered from the retrieval model. Promptriever, like any other IR model, could be used for RAG. The ability to prompt the retrieval model could add additional benefits to RAG, such as less reliance on keywords, ability to pass along “big picture” information, and/or additional instructions to enable RAG for very complex topics.

However, this paper is focused only on the task of retrieval itself. As such, we leave other applications of promptable retrievers, such as RAG, to future work.

You mentioned in L240 that cross-encoders perform best due to their significant compute advantage. Could you list out the compute resources required by different baselines and Promptriever?

Cross-encoders compute full attention over combinations of queries and documents at inference time and are O(Q*D) where Q is the number of queries and D is the number of documents. On the other hand, dense retrievers (a.k.a. bi-encoders) like Promptriever compute attention over queries and documents individually, and estimate relevance using a cheap dot product, yielding O(Q + D + dot-product) (the dot product is negligible, and is often implemented with a vector database in practice). We will add this discussion in revision.

As for the comparison between other dense retrieval baselines, the difference will be the same as your normal intuitions: RepLLaMA and Promptriever use 7B models (e.g. LLaMA) while other models like TART use BERT. The cost to encode all documents depends on the base model (e.g. Llama vs BERT).

It would be nice to see some qualitative examples of Promptriever compared to traditional retrievers, besides the example in the intro.

Thank you for the suggestion! We have added some qualitative examples of the various categories in the general section and will add them to the appendix. We show one from FollowIR and one from InstructIR.

InstructIR Example

Query: what is the temperature in flagstaff az?

Instruction: As a remote worker considering relocating to Flagstaff, AZ, I'm trying to compare the temperature there with other cities on my list. My search would focus on a comprehensive weather tool that can show me a comparison of temperatures along with cost-of-living indices.

Retrieved Passages comparing Promptriever and RepLLaMA

Promptriever:

Rank 1 (relevance=True): Comparison Tool for Climate and Living Expenses in Various Cities - February Insights Located at a higher elevation, this city experiences cooler temperatures in the winter months. The daytime highs around this time of year average at 7\u00b0C (45\u00b0F), while the nocturnal lows often plunge to an average of -10\u00b0C (14\u00b0F). This intuitive tool not only displays these temperature statistics for February but also allows you to juxtapose this data with that of other potential cities on your radar. Furthermore, for a comprehensive overview, this comparison can be tailored to include living expenses, presenting you with a dual-axis chart that informs both your budgeting and climate preferences.

Rank 2 (relevance=False): Real estate stakeholders now have access to a comprehensive climate profile of Flagstaff, which includes an array of meticulously crafted weather charts. Each month is represented, showcasing temperature trends with peaks in July averaging a comfortable 78.2\u00b0F, while the winters introduce a cooler demeanor with December's average resting at 29.6\u00b0F. Precipitation peaks are highlighted, with October notable for its rainfall, averaging approximately 1 inch. This climatological snapshot, accumulated from several years' data, caters to those placing importance on historical weather patterns. Additionally, curated for the environmentally conscious buyer, the health of Flagstaff's atmosphere is quantified, boasting lower air pollutant levels compared to state and national benchmarks. This information has been visualized in an easily digestible format, ready for integration in listings to guide clientele through the perks of Flagstaff's seasonal rhythms

RepLLaMA (note that the relevant doc is not in the top 5):

Rank 1 (relevance=False): Real estate stakeholders now have access to a comprehensive climate profile of Flagstaff … (same as above)

Rank 2 (relevance=False): During the past five years, Flagstaff, Arizona, has experienced varied climatic conditions pivotal for any adventurer planning an extended stay. In July, the warmest month, average highs have consistently hovered around 78 degrees, based on an in-depth analysis of the half-decade\u2019s climatic patterns. Conversely, the winter month of December has presented average lows approximating 30 degrees, a crucial detail for the correct selection of thermal gear. Historical monthly rainfall peaked in October, averaging 1 inch - a statistic that could influence the timing of hikes and outdoor activities. This data, sourced from a comprehensive review of meteorological records, ensures both the precision and reliability required for a meticulous travel guide. The air quality index and pollution levels, although not the primary concern for temperature planning, remain significantly better than statewide and national averages, thus assuring a breathable, pristine environment conducive to exploration.

Our Comment:

RepLLaMA was not able to identify the most important piece of the instruction, the comprehensive weather tool. Promptriever identified it and placed it as rank 1 whereas it did not make the top 5 for RepLLaMA.