Do We Need Domain-Specific Embedding Models? An Empirical Investigation

Thank you sincerely for taking the time to review our paper and for providing such valuable feedback. We are delighted to hear that you recognize the development of the FinMTEB benchmark and our rigorous experimental design as significant strengths of our work.

While the necessity for domain-specific models might align with reviewer's intuitive expectations, our work still distinguishes itself.

Why this is surprising:

1. current SOTA embedding models are built upon powerful LLMs (e.g., NV-Embed based on Mistral-7B).
   1. These models are pretrained on extensive, diverse datasets that include domain content.
   2. Their comprehensive training suggests an inherent capability to understand domain contexts.
2. Our testing datasets only contain traditional NLP embedding tasks using financial-related text, rather than specialized financial tasks such as stock prediction. Interestingly, merely changing the domain of texts leads to a performance drop, even when we control for the complexity of the text.

This is why many studies explore domain adaptation for LLMs in generation tasks, such as BloombergGPT (Wu et al., 2023), although it has also been examined in smaller models like FinBert (Yang et al., 2020).

Why this is useful:

1. Answer the question, "Do we still need domain-specific embedding models?" Given that general-purpose embedding models are becoming the backbone of NLP applications, companies like OpenAI and Cohere offer embeddings that potentially serve a wide range of industry applications.
2. Training a high-performance domain-specific embedding model costs significant time and money. But there is no strong evidence suggesting that general-purpose SOTA models cannot grasp domain-specific languages or linguistic patterns.

To conclude, our study provides empirical evidence on whether SOTA LLM-based embedding models can effectively handle domain-specific tasks without further specialization. This exploration into embedding models addresses a critical area that has been underinvestigated, thereby contributing valuable insights to the field.

Once again, we sincerely thank you for your valuable feedback and remain available for further discussion if you have any additional questions or require more clarification.

Dear Reviewer xpNJ,

We deeply appreciate the time and effort you have dedicated to reviewing our work.

If you have any specific aspects of our work that would benefit from further clarification, we would be more than happy to provide additional details.

Thank you once again, and we wish you all the best!

Authors of paper #9365

Thank you for your thoughtful and constructive feedback on our paper. We sincerely appreciate the time you've dedicated to providing insights that help us clarify and enhance our work. Below, we address each of your comments and questions.

For Weaknesses:

1. Title Clarity Inspired by the comments, we have revised our title to "Do We Need Domain-Specific Embedding Models? An Empirical Investigation on the Finance Domain and a Domain-MTEB Benchmark," which better captures our contribution.

2. Analysis Logic Your concern that "MTEB and FinMTEB are not directly comparable" aligns with our motivation for further analysis. This is precisely why we controlled for text complexity before comparing performance statistically, given that we do not have a domain-specific embedding model.

The controlled experiment makes the comparison valid because:

1. We focus on four different aspects of text complexity across 7 SOTA models, 7 tasks, and 35 datasets.
2. We use different regression analysis to statistically show a significant performance gap between embedding models on MTEB and FinMTEB, even when controlling for the same level of dataset complexity.
3. We also find that general embedding models struggle with the highest complexity domain datasets, while they can handle the highest level of general text. This could be an interesting point for improving the domain-specific embedding training process.

---

For Questions:

1. Semantic Diversity

Mean Cosine Similarity of 0.4: We apologize for any confusion caused by our previous statements. The threshold of 0.6 mentioned in the paper is not the mean value of inter-dataset cosine similarities. It represents the maximum observed cosine similarity value.

For the FinMTEB benchmark, the mean cosine similarity across all datasets is 0.3746. This value was computed by embedding 1000 samples from each dataset and then calculating the pairwise cosine similarities between these embeddings. This low mean cosine similarity suggests that, on average, the datasets in our benchmark do not have large semantic overlap. We appreciate your feedback and will revise the statement in the paper to avoid any misleading information.

2. Division of Datasets into Low, Medium, and High Complexity

We divided the data samples into three equal-sized groups (tertiles) based on the range of complexity scores shared by both benchmarks. For example, the readability score range of FinMTEB is [5:15), and for MTEB it is [1:12.5). Therefore, we divided the readability scores into the following levels: [5:7.5), [7.5:10), and [10:12.5).

3. ChatGPT Error Rate as a Complexity Metric

We appreciate the opportunity to elaborate:

• Methodology: We reformatted dataset examples into instruction-question-answer (IQA) pairs, as shown in Appendix F. Here, ChatGPT functions as an LLM-based retriever or classifier, utilizing question-answering prompts to locate relevant information.

This metric primarily captures the complexity in query-document relationships, serving as a complementary tool to other traditional metrics. Traditional metrics often struggle to capture the nuanced relationship between queries and documents, making the ChatGPT error rate a valuable addition.

4. FinMTEB's Dataset Sources

Most datasets without explicit citations in Tables 7-13 were cited in earlier sections. However, two types of datasets were newly constructed for this paper:

1. For the finance terminology retrieval task: We constructed TheGoldman dataset from the Goldman Sachs Financial Dictionary.
2. For the clustering task: We constructed FinanceArxiv-s2s, FinanceArxiv-p2p by titles and abstracts from the arXiv.

We apologize for any confusion regarding dataset citations and will clarify this in the revision.

5. Interpretation of p-values in Section 5

You are absolutely correct in your observation. We acknowledge the misinterpretation of p-values in our initial submission. We have revised our interpretations to accurately reflect that a large p-value indicates insufficient evidence to reject the null hypothesis, ensuring the statistical analyses are correctly presented.

From "no relationship" to "a lack of evidence for a relationship."

6. Evidence of Challenges in Capturing Domain-Specific Patterns

We understand the concerns, and to clarify the evidence:

• Performance Gap with Controlled Variables: SOTA embedding models consistently underperform on FinMTEB compared to MTEB across various tasks, even after controlling for dataset complexity.

• Complexity-Dependent Performance Degradation: These models show greater performance drops on high-complexity financial tasks, highlighting difficulties in capturing complex, domain-specific language and semantics.

Once again, we sincerely thank you for your valuable feedback and remain available for further discussion if you have any additional questions or require more clarification.

Thanks for your prompt feedback! To better address your concerns, we reorganized our response to your questions as follows:

1. Q6 & Weakness 2

First, in our work, domain-specific patterns represent two distinct aspects:

• Terminology: This includes domain-specific terms and expressions.
• Domain Document Format: This pertains to the structure and word usage patterns typical of financial documents. For example, in the finance domain, the text in annual reports often follows specific formats.

The FinMTEB datasets contains these patterns, for example:

• TheGoldman: This is a finance terminology retrieval dataset.
• FinSTS: This is an STS (Semantic Textual Similarity) dataset that compares annual reports from the same company over different years.
• FinSent: This sentiment classification dataset uses financial news to determine sentiment.
• WikiCompany2IndustryClustering: Clustering the company description by industry. (Collected from Wikipedia)

These datasets do not require extensive financial knowledge such as stock prediction or asset recommendation. They are categorized as domain-specific tasks primarily because they involve financial texts. But the objectives of these tasks are still traditional NLP embedding tasks.

Thus, a domain-specific embedding model is one adapted using a target domain corpus.

For example, if a medical company wants to develop a RAG system, is it necessary for them to train an embedding model using medical documents, such as Medrxiv, given the recent progress in LLM-based embeddings?

The pre-training process of LLMs may have already included such documents, and training a new model requires money and time.

2. Weakness 1

To better address your concern, we fine-tuned e5-mistral-7b-instruct (Wang et al., 2023) using a syntenic finance retrieval dataset (no overlap with FinMTEB).

• Overall Performance:

• Continue with the examples in Q6:

It is clear that using domain-specific documents to fine-tune the general model increases performance on the domain datasets, but slightly decreases performance on the general datasets. We will provide these experiments in detail in the final revised paper to supplement our argument.

---

3. Q1

You are right that mean cosine similarity may not fully represent semantic diversity. In section 3.2, we calculate cosine similarity to align with MTEB (Muennighoff et al., 2022), which also presented cosine similarity in a heatmap. The FinMTEB heatmap is provided in Appendix A. From the heatmap, we observe that datasets with the same task objective may show high cosine similarity, but the overall similarity is not necessarily very high.

We will revise the statement in Section 3.2 for clarity, deleting the statement about 'high semantic diversity'. Since our benchmark comprises 64 datasets that already represent the overall diversity, we will showcase our diversity from this perspective.

4. Q3

The role of GPT isn't the judge. It is an LLM embedder. For example, for a retrieval task, we will provide the GPT with the following prompt:

Given a financial question, retrieve user replies that best answer the question. Return the index.
Query: What is ’Obligor ’?
Corpus: {A List of 20 different context}

We will judge the answer returned by GPT for all tasks. The error rate is calculated at the task level in a bootstrap way, as illustrated in Section 3.3.1. We will emphasize this in the revised paper.

It's inspiring to discuss with you! We remain available for further discussion if you require more clarification.

Dear Reviewer WW39,

We sincerely appreciate your insightful comments on FinMTEB, and we would like to make one last effort to address the misunderstandings raised regarding our paper. We understand that our rebuttal may require some of your valuable time, so we have provided a brief summary of the key points addressed in our response:

Main Contributions:

1. FinMTEB Benchmark: We introduced FinMTEB, a finance embedding benchmark with 64 datasets and 7 tasks. We appreciate that reviewers AfJc and xpNJ have noted it as a valuable and helpful resource. This benchmark aims to advance research in domain-specific embeddings.

2. Exploring the Necessity of Domain-Specific Embeddings: To our knowledge, our study is the first to explore whether domain-specific embeddings are still necessary, given the rise of general-purpose LLM-based embedding models. We used a very comprehensive benchmark and conducted an empirical analysis to address this question.

Necessity of this Research Question:

1. Limited Research on LLM Embeddings: While domain adaptation in smaller models like RoBERTa has shown improvements, there's limited research on domain adaptation for LLM embeddings.

2. Strong Generalization Ability of LLM Embeddings: Modern LLM embeddings, built on diverse text corpora, have strong generalization ability. They can be further enhanced through in-context learning (ICL) (as noted by the reviewer DoPN). Their strong generalization ability allows them to handle a variety of embedding tasks, which underscores the importance of our research question.

3. Practical Implication: Training a high-performance domain-specific embedding model requires significant time and financial resources. However, there is no strong evidence suggesting that general-purpose state-of-the-art (SOTA) models cannot effectively grasp domain-specific languages or linguistic patterns.

We hope these updates address your concerns and wish you all the best.

Thank you!

Kind regards, The Authors

Thank you for your insightful and constructive feedback on our paper. We greatly appreciate the time you’ve taken to help us improve our work. Below, we address each of your comments and suggestions.

1. Title Alignment

Inspired by the comments, we have revised our title to "Do We Need Domain-Specific Embedding Models? An Empirical Investigation on the Finance Domain and a Domain-MTEB Benchmark," which better captures our contribution.

2. Lack of novelty

We understand your concern that the novelty may seem limited given existing studies on domain adaptation. However, our work still distinguishes itself. The previous work [1] [2] [3] focuses on smaller models like DistilRoBERTa; however, the conclusion cannot be applied to the current context. Because current SOTA embedding models are built upon powerful LLMs (e.g., NV-Embed based on Mistral-7B).

1. These models are pretrained on extensive, diverse datasets that include domain content.
2. Their comprehensive training suggests an inherent capability to understand domain contexts.

This is why many studies explore domain adaptation for LLMs in generation tasks, such as BloombergGPT (Wu et al., 2023), although it has also been examined in smaller models like FinBert (Yang et al., 2020).

2. One Domain

We select the finance domain because financial NLP is a critical area within the research community, with a wealth of established financial NLP datasets. While our current study concentrates on the finance domain, we believe it effectively illustrates the limitations of general-purpose embeddings in specialized contexts. But we plan to include preliminary results from additional domains in future work, which will strengthen the evidence supporting our claims.

---

Questions

1. Chinese Tasks

All the experiments in the paper only use the English datasets within FinMTEB (35 English datasets across 7 tasks). We apologize for any confusion and will highlight this point in the paper.

2. Benchmarking Time Reporting & Expanding Related Work on General-Purpose Embeddings

We agree that reporting benchmarking time is essential for reproducibility and resource assessment. Therefore, we will add a new section detailing the computational resources utilized and the time required to benchmark models of varying scales.

We also tested the mentioned models and presented the results, including benchmarking times using a batch size of 512. Since Gecko is unavailable, we tested four other methods.

The average performance does not surpass the other models reported in the paper.

3. Compliance with ICLR Style for Captions

Thanks for pointing out this! We have revised the placement of table and figure captions to adhere to ICLR style guidelines, ensuring that table captions precede the tables and figure captions follow the figures.

We appreciate your thorough review and remain available for further discussion if you have any additional questions or require more clarification.

Thank you for your thoughtful and constructive feedback! We appreciate the recognition of the value of our work in addressing the important challenge of financial domain embeddings. Below, we address each of your concerns:

1. Should Current General-Purpose Text Embedding Models Be Expected to Perform Well on Financial Embedding Tasks?

Before conducting this research, we hypothesized that state-of-the-art (SOTA) general embedding models would perform well on financial texts. Because current SOTA embedding models are built upon powerful LLMs (e.g., NV-Embed based on Mistral-7B). Thus:

1. These models are pre-trained on extensive, diverse datasets that include domain content
2. Their comprehensive training suggests an inherent capability to understand domain contexts

Our testing datasets only contain traditional NLP embedding tasks using financial-related text rather than specialized financial tasks such as stock prediction. Interestingly, merely changing the domain leads to a performance drop, even when we control the complexity of the text.

Besides, to our knowledge, many works prove the necessity for domain adaptation in generation tasks, but no research has systematically kept up with the recent progress in the field of embeddings.

2. Evaluation Language Problem All the experiments in the paper only use the English datasets within FinMTEB (35 English datasets across 7 tasks). We apologize for any confusion and will highlight this point in the paper.

3. Dataset Complexity Measurement

A) Pairwise Complexity in Retrieval Tasks:

You are correct that retrieval tasks involve the interplay between queries and documents, and complexity can arise from their relationships, that's why we use ChatGPT's error rate as one of the metrics. ChatGPT acts as an LLM-based retriever or classifier, functioning similarly to embedding models but utilizing QA prompts to find the relevant document. This process also involves reasoning.

B) ChatGPT Error Rate as a Complexity Measure:

We acknowledge that ChatGPT’s high error rate may result from its knowledge limitations. However, by categorizing all scores into three difficulty levels using a common scoring range for both general and domain-specific datasets, we ensure that the model’s knowledge limitations are represented consistently across both. Therefore, comparing error rates reliably reflects the relative complexity of the questions, as the model’s limitations equally impact both sets.

While these are valid points, our approach remains robust because:

• Multiple Metrics: We don't rely solely on ChatGPT's error rate. We also use semantic entropy, readability, and dependency distance to capture various aspects of complexity.
• Task Versatility: Our metrics apply to a range of 7 different tasks; only the retrieval and reranking depend on pairwise complexity.

4. Concerns about the Argument of Linguistic Pattern

Source of Argument: We acknowledge your concern regarding the complexity of the linguistic pattern argument. However, this argument is based on multiple factors, including sentence length, token length (average number of characters per token), syllables per token, and mean dependency distance. All metrics statistically show that FinMTEB is more complex than MTEB, as illustrated in Table 1.

Token Length: Besides, the higher average number of characters per token means longer words in FinMTEB. Longer words are generally more challenging to process for embedding models.

5. Semantic Diversity Thank you for pointing this out. While our initial sample size of 100 was chosen to align with MTEB (Muennighoff et al., 2022), we've expanded our analysis to 1,000 samples per dataset (up from 100). The results remain consistent:

• Mean: 0.3743
• Std: 0.2635

We also updated the heatmap in Appendix A of the paper using data from 1000 samples.

1. Should Current General-Purpose Text Embedding Models Be Expected to Perform Well on Financial Embedding Tasks?

(i) Previous works, such as BEIR [1]:

• The BEIR only focuses on the retrieval task, and the definition of domains may be different.
• Different Model Scales: The BEIR highlighted issues with dense retrieval models (e.g., ANCE, TAS-B) for out-of-distribution data but did not address the performance of LLM embedding models in expertise domains.

Recent studies show that domain adaptation, such as training on finance corpora, improves LLM generation tasks [1]. However, no work has focused on domain adaptation in LLM embeddings.

---

(ii) There are two questions.

Is there any evidence that pre-trained models can perform well without in-domain data? 

If we understand correctly, 'pre-trained models' refers to embedding models trained on data that is unrelated to the evaluation corpus. Using only pre-trained models without adaptation to specific embedding tasks will likely result in poor performance.

• Generality Performance in General Domain

For general tasks, pre-trained models can indeed perform well. For example, the e5-mistral-7b-instruct[2] model, a well-known LLM-based embedding, has demonstrated superior performance in various tasks such as STS and classification within the MTEB benchmark. The training data for this model included some retrieval tasks from a general corpus, yet it managed to achieve high performance across multiple general-domain tasks.

The training dataset of e5-mistral-7b-instruct[2]: ELI5, HotpotQA, FEVER, MIRACL, MrTyDi, NQ, SQuAD, TriviaQA, NLI, MS-MARCO, Quora Duplicates, DuReader, T2Ranking

• Generality Performance in Special Domain

For domain-specific tasks such as finance, the answer is not obvious. To our knowledge, prior to this work, there were no state-of-the-art finance embedding models or extensive domain benchmarks across different embedding tasks available to evaluate the performance on domain-specific tasks. This is also one of the motivations behind this work. With FinMTEB, we can better evaluate the current domain embedding models.

---

A recent study [2] introducing more in-domain MTEB data (e.g., classification data) has been shown to significantly improve MTEB performance.

We appreciate the idea of ICL in LLM-based embedding models, confirming that these models are more powerful than previous dense retrievers. This also motivated the creation of our work. We also tested the 'bge-en-icl' model in our work.

Except for the mentioned difference in the '1. Clarification on Domain', in the above study, the performance of general tasks using ICL, such as wiki, web, and news, also improved, only confirming the effectiveness of the ICL approach. They did not compare the differences between different domains. Additionally, we cannot directly compare their results across domains due to variations in datasets, such as sentence complexity.

In our paper, we focused specifically on domain comparison. We compared benchmark performance across different domains while controlling for text complexity and concentrating on domain patterns. We found that under the same text complexity, especially at higher complexity levels, the performance of domain-specific tasks was worse. We believe that the conclusion drawn from this comparison is more robust.

---

2. In Tables 8, 9, 10, 11, 12, 13, and 14 of Appendix D, many of the datasets used in this work are in Chinese. Could you clarify this?

These Chinese datasets are not used in the experiments. They are included in the benchmark to provide more options for future research. In the experiments conducted for this study, only the English datasets shown in Figure 1 were used.

For example, the MTEB benchmark includes many different languages, but only English datasets were used in the experiments of the paper.

---

We would also like to clarify, as reviewer AfJc and reviewer xpNJ mentioned, that the FinMTEB proposed in this work is a 'valuable tool' and a 'helpful resource.' We hope to pave the way for future research in domain embedding.

Once again, we sincerely thank you for your valuable feedback and remain available for further discussion if you have any additional questions or require more clarification!

---

[1] Ling, Chen, et al. "Domain Specialization As the Key to Make Large Language Models Disruptive: A Comprehensive Survey." arXiv preprint arXiv:2305.18703.

[2] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023.