Repetition Improves Language Model Embeddings

Thanks for your review, we appreciate the feedback!

It is possible that the specific tasks and datasets used in the evaluation may favor the proposed approach.

We hope to convince you that our evaluation is very broad! We evaluate on the Massive Text Embedding Benchmark (MTEB), which is a standard benchmark to evaluate the broad quality of embeddings. It includes 56 different datasets from 7 different categories—classification, pair classification, clustering, reranking, retrieval, sentence similarity, and summarization [1]. We include a detailed breakdown of the performances across categories in Table 1&5, and by specific dataset in Appendix F. Currently, this is the gold-standard benchmark in the field for embeddings.

And also, does this technique helps some generation tasks? For example, if echoes some important information in the prompt, will it boost the performance on downstream generation tasks?

Great question! Our primary focus is on constructing high quality embeddings, and thus we do not evaluate the generation capabilities. However, commonly, methods to convert large language models into embedding models require modifying the architecture by converting causal to bidirectional attention and also additional training. In our setting, we require no changes to the architecture, which means that a base language model, with strong generation capabilities, could be used to construct high quality embeddings while also retaining its ability to perform well on generation tasks. We believe echo embeddings are thus an important step toward unified machine learning models, but investigating generation tasks in the context of echo embeddings is a promising avenue for future work.

Can you provide more details on the impact of different prompts on the performance of echo embeddings? Have you explored variations in the prompt structure, and how sensitive are the results to these variations?

Absolutely! We find that as long as we adopt the echo embeddings structure from Section 3.2—that is, repeat the input twice and instruct the model to perform a repetition-like task—the model is fairly robust to changes in the exact wording or structure. Based on [2], we evaluate echo embeddings with randomly generated prompts which vary in the instruction (we vary the language in the instruction, including using: “repeat”, “rephrase”, “fill in the blank”, and “rewrite”, along with minor variations in the phrasing), and also the exact formatting details, such as the separators between the instructions and each input (e.g., newlines, colons, other characters) and capitalization. We find that echo embeddings have similar performance for most prompts—the variance is low (Figure 3). See Appendix D for further details.

We perform the same analysis for our baselines—classical embeddings, and PromptEOL. Classical embeddings, while also robust to changes in the exact prompting format, have much lower performance than echo embeddings, and PromptEOL is highly sensitive to the exact structure of the prompt.

Have you tested the echo embeddings method on a wider range of autoregressive models beyond those reported in the paper? How does the method perform on models with different architectures and training paradigms?

Our primary focus, at the moment, is on autoregressive transformer-based models, as transformer-based models are currently the models that perform best on benchmarks on general NLP tasks (e.g., [3]). We do evaluate over two different transformer-based architecture classes (Mistral and LLaMA). Evaluating on, e.g., Mamba-style models would be an interesting avenue for future work!

[1] Muennighoff, Niklas, et al. "MTEB: Massive text embedding benchmark." arXiv preprint arXiv:2210.07316 (2022).

[2] Sclar, Melanie, et al. "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting." arXiv preprint arXiv:2310.11324 (2023).

[3] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

Thanks for your review, we appreciate the feedback!

The motivation regarding causal attention seems questionable. Although LLM2Vec utilizes causal attention, it still performs exceptionally well in extracting text embeddings.

Good question. We want to clear up a possible misconception here: LLM2Vec does not utilize causal attention. LLM2Vec is based on Mistral, but modified to have bidirectional attention (see Figure 1 left in their paper [1]).

This is what makes echo embeddings exciting—the requirement to modify the architecture to be bidirectional, which is common among recent autoregressive large embedding models [1, 2], is unnecessary—echo embeddings can be effective even without changing their architecture

After fine-tuning the model, the performance gap between "echo embedding" and other models is minor. However, "echo embedding" requires the input sentence to be repeated twice, increasing computational costs.

We find that echo embeddings can outperform classical embeddings even in an entirely compute-matched setup. In particular, we can achieve a compute-matched setup by only encoding the first half of each input. The improvement of embedding quality is enough, with echo embeddings, that even when only encoding the first half of the input, performance is still improved over classical embeddings (Table 5, “compute matched” row). In addition, even during training, we can train for fewer steps to match the compute cost of classical embeddings. Even when combining both of these tricks so that the training- and inference-time compute are matched, echo embeddings still outperforms classical embeddings.

I am curious about the influence of number of repetition times

Great question (also asked by Reviewer gzEy)! We ran this experiment, and found no improvement past the first repetition; results are below. Note: For this rebuttal, to save on compute, we evaluate on a 37-dataset subset of MTEB (typically 56 datasets). For this reason, the numbers themselves are not comparable to any other numbers in the paper.

At least one illustrative example should be included in the main article, rather than relegating all examples to the appendix. Typo: line 122, "encode can" -> "can encode" line 283: "embedding 4" -> "embedding in Table 4"

Thanks for pointing these out, we will make these changes.

[1] BehnamGhader, Parishad, et al. "Llm2vec: Large language models are secretly powerful text encoders." arXiv preprint arXiv:2404.05961 (2024).

[2] Muennighoff, Niklas, et al. "Generative representational instruction tuning." arXiv preprint arXiv:2402.09906 (2024).

Thanks for your review, we appreciate the feedback!

The echo embedding method will inevitably double the input length. Although experiments show that reducing the input length and training steps by half still yields good results, this approach may not be suitable in cases where important information is located in the latter half of the input context.

As you point out, there are cases where observing the entire sequence is helpful, such as S2 data (early redundant; late discriminatory). However, we find that classical embeddings can perform poorly on S2 data, too, especially when using mean-pooling (Section 3). Since mean-token is pooling is essential in the zero-shot setting (Table 4), echo embeddings may be necessary to obtain strong performance, anyways.

Because self-attention has a computational complexity of O(n^2) with respect to input length, training for only half as many steps may not necessarily equate to the training cost of the original baseline.

Good point! We also report echo embeddings performance after ~⅓ of an epoch of training, and find that it still outperforms classical embeddings (Table 17, step 280). Alternatively to training for fewer steps, we can also reduce train-time compute by halving the input length, as we do to match inference-time compute. For completeness, we will add a comparison to the final manuscript (the compute requirement makes it infeasible to train and evaluate additional models during the rebuttal period).

Have you tried tripling or increasing the input length even further?

Great question (Also asked by Reviewer HXgu)! We ran this experiment, and found no improvement past the first repetition; results are below. Note: For this rebuttal, to save on compute, we evaluate on a 37-dataset subset of MTEB (typically 56 datasets). For this reason, the numbers themselves are not comparable to any other numbers in the paper.

Thanks for your review, we appreciate the feedback! To respond to your main concern: We believe we have a fair and sound comparison to PromptEOL—to verify we exactly reproduce the results of their paper below. Thanks for the great questions, in addition to our responses we will edit the paper to address everything.

The results of PromptEOL appear significantly different from those reported in the original paper.

We agree that the difference between our evaluation and [1] (PromptEOL) is strange, and so we went ahead and triple-checked that our evaluation is correct. There are two primary reasons for the difference: As you mentioned, we evaluate on three additional STS datasets that are part of MTEB; We use Mistral-7B-Instruct-v0.1, which performs slightly worse than OPT-6.7B. This is consistent with the PromptEOL paper, where they also find, counterintuitively, that larger models sometimes have worse performance on the STS tasks.

Upon closer inspection of the code from [1], we noticed that they perform a data preprocessing step to add punctuation to each example, and to replace double quotes with single quotes. To achieve identical results (“our reproduction + preprocessing” below), we include this preprocessing step. Surprisingly, we find that this preprocessing slightly harms performance.

First, we plot a comparison between the original paper, and our reproduction:

Now, comparing OPT to Mistral:

Here are the results from the other three tasks:

I found the prompt used in this paper differs from that in the original paper.

Another good point of clarification, thank you for pointing this out. We use the exact prompt provided by the PromptEOL paper for the results in Table 1 (please see above for proof of replication), but we forgot to include this detail in our paper. We will add it in the revision.

In addition to replicating their results, we also investigated the prompt sensitivity of the PromptEOL method (Figure 3). Since the original paper did not provide other variants of their prompt, we included our own systematically generated variations of the original prompt (Appendix D) in order to measure the variance in performance over the exact prompt.

For a prompt like "rewrite S, rewritten S'", does it mean halving both S and S' to extract embedding?

Yes, in this situation we halve both S and S’. Perhaps surprisingly, we find that the benefits in embedding quality from echo embeddings still outweigh the drawback we incur from halving the inputs on the MTEB benchmark, which consists of a broad set of evaluation datasets. In cases where halving the number of encoded tokens harms performance, echo embeddings would require additional compute, but this additional compute improves the quality of embeddings beyond the classical methods!

How about also halving the input for PromptEOL?

This is a good point. We find that there are (small) compute budgets where classical embeddings outperform echo embeddings after limiting the number of tokens seen by the model (Figure 4) and we suspect the same would be true for PromptEOL. If compute is available, however, echo embeddings outperforms PromptEOL by ~5% (Table 1). This means that in order to achieve strong performance in this setting, echo embeddings (and the additional compute) may be necessary.

[1] Jiang, Ting, et al. "Scaling sentence embeddings with large language models." arXiv preprint arXiv:2307.16645 (2023).