Scaling Sentence Embeddings with Large Language Models

We sincerely thank you for your helpful feedback and insightful comments. We address your comments and questions below.

W1: While it is helpful to the reader to see the entire distribution of Spearman correlations, it may be relevant to give more details on how the two sources for ICL data impact the quality of downstream representations. The 1/3-2/3 mix of STS sentences vs Oxford definitions would benefit from an explicit ablation.

We would like to clarify that in our method, only one example from the ICL data is used to perform sentence embeddings. To select this example, we evaluate the performance of each example on the STS-B development set and choose the one that performs the best. This process is described in detail in Section 3.2 of our paper. The proportion of STS-B data and the Oxford dictionary in our method is not a crucial factor; we use a 1/3-2/3 mix simply because obtaining data from the Oxford dictionary is much easier compared to generating words from ChatGPT.

W2: The value of ICL and CSE is demonstrated only for OPT. Indeed, table 1 is missing results that would demonstrate the added value of ICL and CSE on Llama.

In this work, we focus on scaling up sentence embeddings with various model sizes. Compared to OPT, LLaMA only has four model sizes. We use OPT as the main model to show the results. Due to the page limit, we only present the ICL results on OPT. However, we have included the CSE on LLaMA in Table 2.

To address the concern about the robustness of our method, we evaluated the performance of PromptEOL and PromptEOL+ICL on current popular LLMs, varying in size from 3B to 70B. The evaluated LLMs include LLaMA, LLaMA-2, open-LLaMA, MPT, and Mistral. Our method shows significant improvement across above 13 models. PromptEOL achieves an average Spearman correlation score of 70.27 on STS tasks among the above models, while PromptEOL+ICL achieves an average score of 77.14. Comparing to the average pooling baseline, PromptEOL and PromptEOL+ICL achieve average improvement of 22.27 and 29.14, respectively. Furthermore, when compared to the previous prompt representation method, PromptEOL and PromptEOL+ICL exhibit an average improvement of 15.55 and 22.42. We have included detailed results in Appendix F of our paper.

W3: The proposed methods (in-context learning and possibly fine-tuning) are performing worse than simple explicit-one-word-limit prompting on transfer tasks.

Benefiting from our PromptEOL method, LLMs can capture the semantics of sentences well for transfer tasks. In Figure 4(c), we demonstrate that PromptEOL significantly outperforms other representation methods like prompt and average pooling on transfer tasks, achieving state-of-the-art results without fine-tuning. For in-context learning, we can flexibility modify the prompt to generate embeddings that fit the task following [1]. For example, by using a sentiment classification example, we can improve the results of PromptEOL 6.7B OPT from 91.34 to 91.78 average accuracy on seven transfer tasks. For fine-tuning, due to limited computational resources, we only use efficient fine-tuning LLMs with QLoRA, which may harm the original transfer task performance through 4-bit quantization.

References:

[1] Su H, Kasai J, Wang Y, et al. One embedder, any task: Instruction-finetuned text embeddings (ACL findings 2023)

Dear reviewer oEkP,

Thanks very much for your time and effort. As the discussion phase is going to end today, could you review our responses to ensure they have addressed your concerns? We would also appreciate any additional feedback you might have for improvements.

Best regards,

Authors of 596

We sincerely thank you for your helpful feedback and insightful comments. We address your comments and questions below.

W1: In Table 1, only the results based on OPT are presented. Why not also including the results based on LLaMA?

In this work, the main focus is on scaling up sentence embeddings with various model sizes. While LLaMA only has four model sizes, we use OPT as the primary model to showcase the results.

To address the concern about the robustness of our method, we evaluated the performance of PromptEOL and PromptEOL+ICL on current popular LLMs, varying in size from 3B to 70B. The evaluated LLMs include LLaMA, LLaMA-2, open-LLaMA, MPT, and Mistral. Our method shows significant improvement across above 13 models. PromptEOL achieves an average Spearman correlation score of 70.27 on STS tasks among the above models, while PromptEOL+ICL achieves an average score of 77.14. Comparing to the average pooling baseline, PromptEOL and PromptEOL+ICL achieve average improvement of 22.27 and 29.14, respectively. Furthermore, when compared to the previous prompt representation method, PromptEOL and PromptEOL+ICL exhibit an average improvement of 15.55 and 22.42. We have added results in Appendix F of our paper.

W2: In Table 1, the best configuration PromptEOL+ICL+ OPT (6.7B) does not show clear advantages than PromptRoBERTa (123M)

PromptRoBERTa is fine-tuned using contrastive learning, whereas PromptEOL+ICL does not require any fine-tuning. It may not be fair to directly compare these two methods. However, PromptEOL+ICL also achieves similar performance to PromptRoBERTa, which demonstrates that a general language model can achieve such performance without relying on contrastive learning or fine-tuning. We also introduce other advantages benefit from PromptEOL+ICL in W5.

W3: For the in-context setting, why only use one demonstration? In Table 1, comparing PromptEOL+ICL + OPT with baselines models is not fair since the baseline models do not use the development set.

We found that one demonstration is sufficient for LLMs to generate good sentence embeddings. Increasing the number of demonstrations does not further improve the performance. We believe the comparison is fair. The ICL examples used in PromptEOL+ICL are taken from STS-B training set and Oxford dictionary. We only use development set to evaluate the performance of ICL examples. In comparison, other baselines such as SimCSE and PromptBERT also utilize STS-B development set to select the best checkpoint during training. Moreover, BERT prompt also utilizes the STS-B development set for prompt searching.

W4: When the model size increases, the performance does always not increase. Especially, the 13B, 30B, and 60B models do not perform better than smaller models such as 1.3B and 6B models.

In this work, we aim to investigate the capability of LLMs on sentence embeddings by scaling up the model size. As we mention in our paper, we find that continuing to scale up to over ten billion parameters does not always improve the performance of sentence embeddings for STS tasks. We believe that our work can provide valuable insights for future research on leveraging LLMs for sentence embeddings.

W5: The overall method is a little bit heavy. It is worth to discuss whether we should improve the sentence embeddings using large language models.

First, our work not only works for LLMs (7B, 13B, 60B models), but also smaller models like OPT 1.3B or 2.7B. Furthermore, we significantly reduce the computation cost of fine-tuning high-quality sentence embeddings compared to the previous SOTA methods like ST5. We can exceed the previous 4.8B ST5 with 2.7B OPT fine-tuned on less training data. Compared to 4.8B ST5, it uses two-stage training with 2 billion extra question-answers pairs and 275k NLI datasets by fine-tuning all parameters, PromptEOL+CSE can leverage efficient fine-tuning method QLoRA on 275k NLI datasets with one epoch. We can train 2.7B OPT with PromptEOL+CSE for around one hour on four 3090 GPUs. However, it will take more than hundreds of hours with the same hardware environment to train 4.8B ST5.

Second, PromptEOL+ICL also provides a lightweight method that allows LLMs to generate sentence embeddings without the need of fine-tuning. We can use LLMs themselves to embedding sentence to perform retrieval augmented generation, without additional embedding models. Moreover, we can control the embedding behaviors by modifying prompt and ICL example to suit specific application, such as cross-lingual representation. For instance, we can utilize an ICL example with a Chinese sentence and an English summary word to project the semantics of Chinese sentences onto an English word. This can help bridge the language gap in sentence embeddings, which we leave for future work.

Thanks for responses. We greatly appreciate your positive comments.

However, we would like to clarify above points:

• First, our method does not compress a sentence directly into a single word, but rather uses hidden states as sentence embeddings. This reflects the probability of words that best summarize the sentence, rather than a single word. This ensures that our method also works on long sentence tasks like SUBJ in transfer tasks.

• Second, scaling up works well on transfer tasks and STS tasks with fine-tuning (PromptEOL+CSE). We noticed that the model performance doesn't scale up as effectively only on STS tasks without fine-tuning (PromptEOL+ICL). We also discussed the reason for this issue in Q1 for reviewer dT5b. Regarding the performance concerns of our method:

  • Our method with LLMs achieves the state-of-the-art results on transfer tasks and STS tasks, demonstrating the potential of using LLMs to generate sentence embeddings. Our method significantly improves the performance over small models on Table 2 and 3.
  • Since contrastive learning can alleviate the anisotropy problem of sentence embeddings[1], comparing a fine-tuned model with PromptEOL+ICL might seem unfair. Nonetheless, PromptEOL+ICL still achieves comparable performance. When considering baselines without fine-tuning, LLMs can benefit from in-context learning with PromptEOL+ICL, showing strong advantages over other methods. We also discuss the benefits of in-context learning in the second point of W5.
  • We provide PromptEOL+CSE for the fine-tuning setting, which outperforms small models by more than 3 points on the Spearman correlation, even with limited computational resources. For instance, we can efficiently fine-tune a 2.7B OPT with PromptEOL+CSE for approximately one hour on four 3090 GPUs.

Thank you again for your time and effort. Please let us know if you have further questions.

Reference

[1] Gao T, Yao X, Chen D. SimCSE: Simple Contrastive Learning of Sentence Embeddings (EMNLP 2021)

Dear reviewer ViaW,

Thanks very much for your responses. As the discussion phase is going to end today, could you review our recent responses adequately addressed your concerns?

We sincerely thank you for your positive feedback.

Best regards

Authors of 596

Your prompt is This sentence: “xi” means in one word:, which intends to compress the whole semantics into a single word. The hidden state vectors represent the intention of your prompt. I do not know why you are now claiming you are not compressing it. Ideally, this prompt will compress any input sentences into a class in the output vocab, which might be too aggressive.

SUBJ is a simple task for identifying subjectivities of user's movie review. The performance on SUBJ can not directly and fully show the embedding quality.

For scale, I mean your 66B model give much lower performance than your 1.3B model in Table 1 for the PromptEOL OPT setting. Similarly, for the PromptEOL+ICL+OPT setting, the 66B model does not show clear advantage of the 13B model.

I have clearly expressed my score will be 7. And this is my final score. Thanks.

We sincerely thank you for your helpful feedback and insightful comments. We address your comments and questions below.

W1: While the methods proposed are sound, they consist of previously suggested and widely implemented techniques, which diminishes the novelty aspect of the work.

The primary focus of our research is the scaling up of sentence embeddings using LLMs, which remains unexplored. We also proposed three methods to leverage the capacity of LLMs for sentence embeddings: PromptEOL, PromptEOL+ICL, and PromptEOL+CSE.

While our work is inspired by earlier studies such as PromptBERT, we have identified that PromptBERT does not extend its applicability to LLMs for sentence embeddings. To address this shortcoming, we have proposed a solution named PromptEOL. This method introduces an implicit single-word limitation to the prompt, thereby facilitating the generation of sentence embeddings. It is necessary to mention that, as the output of sentence embeddings results in a singular vector, in-context learning cannot be applied directly. To overcome this challenge, we propose PromptEOL+ICL. To the best of our knowledge, we are the first to show that in-context learning can be applied to sentence embeddings without fine-tuning.

W2: Contrary to SimCSE, the in-context learning approach depends on the use of the STS-B dataset, including its training and validation components, which could potentially confer an unfair advantage to the method.

We think the comparison with SimCSE and other similar methods is fair. First, SimCSE also uses STS-B validation set for selecting the best checkpoint during training. Second, we do not use any label information from the STS-B training set. Instead, we only utilize 100 sentences from the STS-B training set, ensuring that none of these sentences come from the same sentence pairs. In fact, our approach outperforms SimCSE even when solely leveraging the Oxford dictionary. Finally, our ICL method does not rely on fine-tuning or contrastive learning, which are considered essential for achieving better representation in current sentence embeddings methods[1,2].

W3: There appears to be no direct link between the two proposed methods; that is, the approach based on in-context learning and the one utilizing contrastive learning.

Both in-context learning and contrastive learning methods are built upon our proposed representation method, namely PromptEOL. This method focuses on two settings in sentence embeddings: with and without fine-tuning. In the case of in-context learning, we demonstrate that combining PromptEOL with in-context learning yields high-quality sentence representations without the need for fine-tuning. Additionally, in the context of contrastive learning, we provide evidence in Figure 4(b) that PromptEOL can benefit from contrastive learning, outperforming other representation methods.

Q1: I'm curious whether the authors have any insights or hypotheses as to why (much) larger models (over 10B) do not excel as expected in computing sentence representations, which contrasts with their effectiveness in other standard applications.

We have also observed that the scaling law performs well in transfer learning tasks, as shown in Table 3, but does not yield the same success in STS tasks, as shown in Table 1. We think that the main reason for this discrepancy may be related to the issue of anisotropy in sentence representation [1,3], which harms the performance on STS tasks. To validate this, we measure the anisotropy of sentence embeddings across different model sizes following [4].

Through our analysis, we have observed that the anisotropy of sentence embeddings increases as the model size increases. While larger models may possess greater overall capacity, the issue of anisotropy may limit their performance.

References:

[1] Gao T, Yao X, Chen D. SimCSE: Simple Contrastive Learning of Sentence Embeddings (EMNLP 2021)

[2] Yan Y, Li R, Wang S, et al. ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (ACL 2021)

[3] Ethayarajh K. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (EMNLP 2019)

[4] Jiang T, Jiao J, Huang S, et al. PromptBERT: Improving BERT Sentence Embeddings with Prompts (EMNLP 2022)

Dear reviewer dT5b,

Thanks very much for your time and effort. As the discussion phase is going to end today, could you review our responses to ensure they have addressed your concerns? We would also appreciate any additional feedback you might have for improvements.

Best regards,

Authors of 596