Making Text Embedders Few-Shot Learners

Thanks for your response! Now I see that the order and the selection of few-shot examples does not make a big influence on the performance. I'd like to increase my score to 8. It is indeed a simple but effective method.

Nevertheless, I still want to discuss the LoRA part. Even though there are references showing that full-rank tuning is prune to over-fitting, but your experiments show that larger LoRA rank makes a better result. If you can show that the bigger rank (e.g. 64, 128) will make the model overfitting, it will be much more persuasive.

Dear Reviewer zzdF,

Thank you very much for your thorough review and constructive feedback! We greatly appreciate the opportunity to address your questions with the following response.

W1: Unlike zero-shot text representation, ICL is highly sensitive to the chosen demonstrations. There are countless related works discussing that the selection and order of task demonstration affects the final performance. However, in this paper, the demonstration is not discussed well. The paper only mentions "a consistent set of in-context examples is applied to each query".

Q1: Following Weaknesses 1, I believe not all of the demonstration can provide a positive influence on text representation. Therefore, I'm curious if you make a demonstration search, or the examples is just randomly sampled, or you report the average score of many demonstrations.

To explore these questions, we conducted the following pilot experiments involving the order and selection method of the in-context examples.

• The order of examples. We randomly shuffled the 3-shot examples multiple times and reported their performances on MTEB. According to our results, the order of examples did not significantly impact the overall performance.

• The selection method. We explored the following selection methods in the experiment.

  • Default selection. In our default implementation, we sampled the k-shot examples from the training set three times and chose the optimal k-shot examples with the highest accuracy on a small subset of the training set.
  • Random selection. We further explored the random selection method, where the k-shot examples were randomly sampled from the training set (we performed random selection three times, denoted as "random selection strategy -1, -2, -3"). The experiment results on MTEB are reported as follows.

  According to the experiment results, both the default selection strategy and the random selection strategy were sufficient to achieve substantial improvements over the zero-shot baseline.

Dear Reviewer uYsr,

Thank you very much for your thorough review and constructive feedback! We greatly appreciate the opportunity to address your questions with the following response.

W1: My main concern regarding the paper is that some key implementation details are missing (See Questions). Given the extra page limit and verbose Section 3.1 and Figure 2, I think the authors could have done a better job in covering those details.

W2: Other than that, NV-Embed2 results can also be included in the revision.

W3: Other comments about writing.

• Thanks a lot for pointing out the issues about writing and format. We have revised the paper accordingly based on these suggestions.
• We've also added NV-Embed-v2 to the revised paper (line 337, 346, 385).

Q1: How exactly are the query and passage representations obtained? The last line of Section 3.1 suggests that the two are obtained using <EOS> embeddings. Are there two <EOS> tokens?

We append an <EOS> token to the end of the input when encoding the query and passage. The final hidden state of the <EOS> token is used as the embedding. We apply the same <EOS> token for encoding both the query and the passage.

Q2: Why are passages/documents truncated to 256 tokens, given that Mistral has 32K tokens and there are only a maximum of 5 documents? Are the gains on Long Doc evals in line with other gains, given that nothing in the architecture/training should benefit long documents in particular?

• For the majority of our training tasks, 256 tokens are sufficient to cover a complete passage/document in the datasets.
• Excessively long inputs will severely restrict the batch size, which is unfavorable to the training quality.
• During evaluation, the examples are introduced without truncation. Despite that the training is conducted with shorter lengths, our method remains effective when it is directly applied to such longer inputs (as shown in Table 3, lines 378-390).

Q3: In Evaluation, it's said that "a consistent set" is used for ICL evals. What is a consistent set? Do you mean fixed? How many examples?

• We use a fixed set of examples for all queries within the same dataset.
• The number of examples is randomly determined for each dataset in the original paper.
• We further include additional experiments by fixing the number of examples (using E5-Mistral data).

Q4: In Section 4.4, are all the ablations models trained similarly to the baseline model?

Yes, all the ablations models are trained using the same settings as the baseline model.

Thanks for answering all my questions in detail and for sharing the additional results.

I wanted to follow up on a few things from the response.

The number of examples is randomly determined for each dataset in the original paper.

What do you mean by randomly determined?

We further include additional experiments by fixing the number of examples (using E5-Mistral data).

The result suggests an almost monotonic performance between the number of examples and performance which is quite encouraging.

Thanks a lot for your reply! Here are our answers to your following-up questions.

What do you mean by randomly determined?

We randomly sample the value of k between 1~8 for each dataset. Then, we further sample the k-shot examples as our demonstration for this dataset.

The result suggests an almost monotonic performance between the number of examples and performance which is quite encouraging.

Based on our experiment result, we can have the following observations:

• Within certain ranges, the empirical performance of different tasks improves consistently as the number of examples increases.
• Beyond these ranges, the performance becomes stabilized where additional examples do not lead to further gains.
• It empirically suggests that 5 examples are sufficient for most of the tasks .

W3: Similarly, it's not clear how much of the performance comes from increasing the number of tokens (and consequently FLOPs) that the model can use at test-time vs. actually incorporating new information into the model. A useful ablation would be training a model with the same number of tokens as a typical ICL example, but with a single token substituted for all ICL tokens (a-la the "let's think dot-by-dot" work). I would suspect that most of the performance comes simply from increased computation at test time.

To explore the impact of simply increasing the number of tokens, we conducted a pilot experiment in which we replaced the in-context examples with an equivalent number of periods ("."), and re-trained the embedding model using E5-Mistral data. We got the following result from our pilot experiment:

As we can observe, the substitution does not improve the performance of the ICL-embedder (zero-shot). Instead, the performance declines from 64.67 to 62.43 due to the introduction of unrelated information.

W4: Little analysis is shown: beyond questions at the systems level, another clear demonstration of utility would come from comparing model performance to number of shots in context. If increasing the number of in-context examples also increases performance, that would provide a compelling case for practitioners.

Q1: Does performance get better by using more examples in context?

• Following this suggestion, we conducted an extra experiment where the number of ICL examples is increased from 0 to 5. The experiment results are reported as follows:

Based on the above results, the following observations can be made:

• Within certain ranges, the empirical performance of different tasks improves consistently as the number of examples increases.
• Beyond these ranges, the performance becomes stabilized where additional examples do not lead to further gains.
• It empirically suggests that 5 examples are sufficient for most of the tasks .

Q2: At test time, are few-shot examples selected per-example or per-dataset?

We use a fixed set of examples for all queries within the same dataset.

Q3: Do you pool over instruction tokens?

Q4: Do you pool over ICL tokens?

Following the setting in GRITLM [4], the mean pooling only involves the query's tokens, where the instruction's tokens and the examples' tokens are omitted from this operation.

[4] Generative representational instruction tuning, Muennighoff et al.

Dear Reviewer mLW8,

Thank you very much for your thorough review and constructive feedback! We greatly appreciate the opportunity to address your questions with the following response.

W1: In context examples have been shown to be most useful when a language model needs to learn a template or format for doing a task; in many cases, this is their only contribution (as opposed to actually teaching semantic information about the task at hand). This is not useful for embedding models because embedding models always have the same task format (outputting an embedding).

• For our problem, the in-context examples are introduced to adapt the models for different embedding tasks. For example, by giving "example question: example answer", the model is adapted for QA retrieval tasks; by giving "example document: example label", the model is adapted for classification tasks.
• In other words, the in-context examples are used to prompt the model to specialize the embedding for the user's interested task. This approach aligns conceptually with the principles of prompt tuning, which applies for both generation and representation tasks [1, 2, 3].
• The impact of adaptation from in-context examples is demonstrated by our experimental studies. Please continue to check our detailed analysis in the following discussion.

[1] The power of scale for parameter-efficient prompt tuning, Lester et al. [2] P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks, Liu et al. [3] Making Pre-trained Language Models Better Few-shot Learners, Gao et al.

W2: A major weakness is that this obviously makes the embedding process slower, and it's not clear by quite how much or what the tradeoff is. The performance gains are quite marginal (71.2 -> 71.7 on MTEB) and practitioners would need to balance this with the

W5: Finally, does this help more out-of-domain? One would expect in-context examples to be most useful (or perhaps even only useful) when the test task is most different from the training task. Is this true in your case?

• In-context examples are particularly useful to adapt the embedding models for unseen tasks (i.e., out-of-domain scenarios), which aligns with your assumption.
• This point is illustrated by the experiment with the E5-Mistral data (64.67 -> 66.08), where most of the MTEB tasks are held out of the training data.

• This point is further emphasized by AIR-Bench, where unseen tasks are introduced for the embedding models. Specifically, it demonstrates improvements from 53.60 to 55.92 on top of the few-shot examples.

• Although it remains effective for in-domain applications, its improvement effect is mitigated due to in-domain fine-tuning (71.2 -> 71.7).
• Although in-context examples introduce additional encoding costs, they only affect query processing. Given their significant impact on out-of-domain applications, this extra cost can still be affordable in practice. (Besides, knowing that our in-context examples are fixed for each task, the encoding result can be cached and shared by all queries. Therefore, the extra cost can be significantly reduced even further.)

Dear Reviewer mLW8,

We highly appreciate your valuable feedback to us! In case our previous response is too wordy to comprehend, we would like to highlight its key points as follows.

• Our method achieves a much more significant improvement in the out-of-domain evaluations: 64.67->66.08 on MTEB and 53.60->55.92 on AIR-Bench (the detailed result is presented in our previous response). In fact, the training data of E5-Mistral is mainly made up of retrieval task, where other tasks, like classification, clustering, pairwise, are not included in fine-tuning. Such a result indicates that the improvement is resulted from the embedding model's in-context learning capability, rather than multi-task learning.
• The extra latency is moderate given that 1) it only influences query processing, 2) the in-context examples are fixed for each task, thus can be significantly accelerated by KV cache. To demonstrate the acceleration effect of KV cache, we perform the following experiment, where the query encoding time can be reduced by more than 4 times in 3-shot setting (35.75->8.07). While there is an additional time cost compared to the zero-shot setting, it is acceptable in many scenarios, allowing users to trade off embedding quality and inference latency effectively (i.e., performance improvement through scaling-up of runtime computation, which is in the same spirit of GPT-o1).

Please feel free to let us know any further questions about these issues. We are looking forward to engaging with the reviewer for more discussions.

Thanks,
The authors

Dear Reviewer 5eiG,

Thank you very much for your thorough review and constructive feedback! We greatly appreciate the opportunity to address your questions with the following response.

W1: It would be nice to see an experiment on how the number of few-shot examples impacts performance.

Following the above suggestion, we investigated the effect of the number of examples (denoted as "k-shot" examples) on performance. The experimental results on the MTEB dataset are summarized below.

Based on these results, the following observations can be made:

• Within certain ranges, the empirical performance of different tasks improves consistently as the number of examples increases.
• Beyond these ranges, the performance becomes stabilized where additional examples do not lead to further gains.
• It empirically suggests that 5 examples are sufficient for most of the tasks.

W2: The discussion about overlap between training data and MTEB was a bit difficult to follow.

Q1: The "more comprehensive" dataset that you train your model on still consists of public datasets, doesn't it? I find the way you chose to name these two different experimental settings a bit misleading.

• To avoid ambiguity in the original draft, we've updated the names of the datasets to "E5-Mistral dataset" and "Augmented E5-Mistral dataset".
• The E5-Mistral dataset is the one originally used by E5-Mistral [1]. It primarily consists of some public data and MTEB in-domain datasets for the retrieval task. In contrast, the Augmented E5-Mistral dataset is a new dataset introduced in our work, which includes additional in-domain datasets for other MTEB tasks, such as STS and classification. Detailed specifications for both datasets are provided in lines 270–302 of the revised paper.
• In-context examples enhance the empirical performance on both datasets. However, the improvement is more noticeable on the E5-Mistral dataset, indicating that in-context learning is particularly effective in zero-shot applications.

[1] Improving text embeddings with large language models, Wang et al.