Adder: Adapted Dense Retrieval

The review correctly points out that our approach uses some well-known ideas, like residuals and adapters. The key novelty is the combination of these ideas in the right way and applying them on embeddings. We are thus able to tune retrievals to solve a problem that the review agrees is well motivated. In short, the contributions include the strict heterogeneous retrieval problem definition that arises from new RAG-like applications, the idea of adapting embeddings, the idea of using residual low-rank adaptations for embeddings and the evaluation showing that the idea works.

The review also correctly points out that evaluation is limited in the paper. We performed extensive experiments with various modeling choices, such as not using residuals, not using contrastive loss, not using global hard negative, etc., before we arrived at the right solution presented in this paper. Unfortunately, we did not include those results in the paper. We plan to fix this oversight and include the results for these variants: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss. We also experimentally observed that Adder and Adder2 adapters over SBert embeddings show an even greater relative gain in nDCG metrics than what we report for OpenAI embeddings. Please see the table in the response above to Reviewer Y6UA.

Regarding the limited corpus size, the choice is guided purely by our end use case for RAG. Also in our target application, we have limited training data. We wanted to design a solution that can work with limited training data, even if it required small corpus sizes, and used limited computational resources. That was the goal of our work. We agree that for larger scale datasets and enough training data, other approaches, such as finetuning, could come into play, but they would also incur a significant cost and time. ADDER only takes 8secs for 1 epoch (668 batches for a batch size 16) on a single gpu (k-80) machine.

Q1: It is a good question about why attention + residual. Indeed, we started with a linear adapter and an MLP adapter. We observed that their performance was often much worse than the raw openAI embeddings, especially when there was limited training data. We realized that OpenAI embeddings are actually very good, and we wanted to keep their “goodness” and only perturb/tweak them slightly for a particular retrieval task. If the reviewer would find it useful, we can rerun those early experiments and report the results.

Q2: The story is the same for global hard negatives. We started with not using contrastive loss at all, then we moved to batch negatives, and finally to global negatives because we saw improvements. For Adder2, where we also adapt the embedding of the corpus, we refresh corpus embeddings at each epoch. In Adder, we do not adapt the corpus embeddings, so this is not an issue.

W1, W2: We realize now that although we did a lot of experiments with other modeling choices, we only reported the numbers with the best architecture we found. We will include performance numbers for following ablations in the final version: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss.

Regarding using other base embeddings, thank you for the suggestion because we ran experiments with sBert and noticed that the relative gains provided by Adder are huge (compared to the relative gains we get over OpenAI embeddings). Although, the absolute numbers are poorer than what we get with OpenAI embeddings.

Here are the results:
For the same BEIR benchmarks reported in Table 2, here are the results with using SBert as base. (nDCG@k values)

For the same task reported in Table 3, here are the results when using SBert as base: (nDCG@k values)

We assume black-box access to the base model. In other words, we do not access to the weights. Hence, adapter-based approach is the only possibility and fine-tuning is not an option.

Regarding comparison to the state-of-the-art IR methods, we note that the general-purpose OpenAI embeddings are a very strong baseline. Looking at the leaderboard on BEIR benchmarks (https://eval.ai/web/challenges/challenge-page/1897/leaderboard/4475), which reports ndcg@10, we see that OpenAI embeddings (our baseline) is very close to the leaders, and even beats the top leader on one dataset (Arguana). For SciFact, we improve over openai embeddings, and thus beat the state-of-the-art (on the ndcg@10 metric). As we discussed in Q1 above, we get higher relative gains on ndcg@1, which is our focus. Further, we explained in the paper why we do not see gains across all benchmarks, and when we don’t, we do not deteriorate the performance of openai by much (and still beat the leaders on some benchmarks, such as Arguana).

The specific solution proposed in the paper is the idea of learning a low-rank residual adapter for embeddings.

We agree with the reviewer that “[our] method can only improve on some of the datasets”. The important point is that we understand when it improves and when it does not. The point is that we should use an adapter only when it is needed. We are not claiming that one should always attach an adapter for every kind of retrieval task. Only when the base embeddings perform poorly, should we attach an adapter. Our explanation for when adapters are likely to improve, and when not, can guide this choice. The fact that the adapter is not improving the metrics for some benchmarks should be viewed as a learning contribution of our paper. We further note that the improvements due to ADDER are dependent also on the base model, and OpenAI embedding model is a particularly powerful model that is already proficient on retrievals over several different classes of entities. This fact is not true for other base models, such as SBert. ADDERT over SBert consistently improves retrieval performance because SBert embeddings are not “trained” for diverse retrieval tasks; please see the table in the response to Reviewer Tp6G.

Regarding the discussion on large vs small corpus, we only wanted to make the point that retrieval from small corpus is also of interest in certain applications. Of course, one can use retrieval methods that work on large corpus also on small corpus. Similarly for the issue with small versus large K. We wanted to improve metrics @k for k=1,3. As out results show, when Adder improves over baseline, we usually get a higher relative improvement for small k than we get for large k. We want to point out that our Adder approach is almost at par with the state-of-the-art, and even better than the state-of-the-art for SciFact based on the leaderboard for BEIR.

We will be happy to replace the use of “heterogeneous” since it seems it has a connotation different from what we meant by it.

We understand the reviewer's concern regarding the OpenAI embeddings not being tested on larger IR datasets. We would like to draw their attention to the following papers:

• Text and Code Embeddings by Contrastive Pre-Training: The previous generation of OpenAI embeddings already outperformed ANCE as shown in Table 5. Table 7 also shows that they generalize well with other large scale IR datasets.
• Evaluating Embedding APIs for Information Retrieval: This evaluates the predecessors mentioned in the first paper, the latest OpenAI embeddings (ada-002 which we use in our experiments), and other Embeddings available as a service. The paper finds that the ada-002 embeddings outperform others in almost all IR tasks on a very wide range of datasets.
• Vector Search with OpenAI Embeddings: This work builds upon both the previously mentioned works and further extends the evaluation to more datasets like MSMARCO & TREC-DL19/20.

We hope this addresses the concerns about extensive evaluation of OpenAI Embeddings and their generality over IR benchmarks. An important point to note in all these papers is that the OpenAI embeddings are almost as good as the other leading baselines usually exceeding or trailing them by a very small margin. We would be happy to include results over other baselines in the final version, if the reviewer believes that it would reasonably add more value to this work.

We also want to address the concern that the reviewer points out regarding why we perform well on smaller datasets. We discuss this in Section 4.1 of the paper. The proposed technique is more effective in strict retrieval settings given the limited amount of data available and used in the training process of adapters (e.g. Retrieval Augmented Generation). In the limited data setting, we tend to model the training of our adapters with a hard global negatives optimization which helps pick the ideal corpus element for a given query by contrasting it against other corpus elements. This exercise learns better to differentiate between top elements; however, as you go down in the ranking, the relations are not ordered well enough to learn fine differences to prefer one against another. Hence, resulting at better retrievals at smaller k.

If the training data was not a constraint, we would be able to finetune/retrain white-box models as well as training dense layers on top of the black-box model which could perform better at all k and be able to learn a new embedding distribution for your task. ADDER employs residual adaptation to embeddings; this allows us to massage the inherent distribution of data and not transform it completely. Over larger datasets, we suspect that transformation of the embeddings would be of a much higher degree and denser to learn.

We also obtained experimental results that show the gains from using ADDER and ADDER2 for NL2Code tasks performed using LLMs for where we (a) use ADDER to retrieve few-shot examples and (b) use ADDER to rerank the candidates returned by the LLM. This application was one of our main motivations for designing ADDER and ADDER2. On the Excel's PowerQuery Mashup dataset, we observe that using only OpenAI embeddings gives us a sketch match performance of 0.72, whereas using ADDER gives us a performance of 0.74. As part of ongoing work, we continue to perform experiments on NL2X tasks using ADDER.

Thanks for pointing out the contradictory description of our experimental setup. Our initial experiments were done on CPU machines, but later when we had to perform multiple runs to generate numbers for our tables, we switched to a virtual machine with GPU. We will fix this error. To provide evidence for the fact that the computational requirement is low, we report here that the time taken to train our ADDER for 1 epoch (containing 886 batches with batch size 16) is 8.97s on GPU, whereas the same took 71.73s on CPU. So, the experiments are doable on CPU, they just take a little longer but still reasonable time.

Thanks also for the suggestion to rearrange the columns of the tables. We will do so.

Regarding details of the datasets used and hyperparameters, we make use of the train datasets given as a part of BEIR. In case the associated training set was not available, we sampled 20% of the data from the test set for training and used the remaining 80% for test. This also highlights once again that our method does not need a lot of data to learn a general solution. Our hyperparameters were chosen by performing a grid search. Further, a smaller initial learning rate was found to be better for larger datasets. The $\gamma$ found most optimal was 0.1 using a step size of 30 epochs.

Regarding batch local negatives vs global negatives, we observed that using local negatives reduces performance over the base embeddings by over 10% or more over the datasets. Since we saw this in initial experimentation, we did not record the exact numbers. We would be happy to perform those experiments and report the numbers.

The loss function is as follows: if q, c, and c’ are the transformed embeddings of the query, true positive corpus element, and the global negative corpus element, then the loss l(q,c,c’) = max(0, dot(q,c’) - dot(q,c)), where dot is the cosine similarity function.

We thank the reviewer for their comment on SBert, and include the following results for SBERT for both the default and adapted embeddings scenarios:

For the same BEIR benchmarks reported in Table 2, here are the results with using SBert as base. (nDCG@k values)

For the same task reported in Table 3, here are the results when using SBert as base: (nDCG@k values)

Regarding comparison to competing methods, we observed that OpenAI embeddings were giving state-of-the-art results in most cases; in some cases, OpenAI embeddings performed even better than the leader on the BEIR leaderboard. That was the reason to start with OpenAI embeddings. The ADDER adapter works over those embeddings, and even improves over them in some cases.

We also realize that although we did a lot of experiments with other modeling choices, we only reported the numbers with the best architecture we found. In a revised version, we will include performance numbers for following ablations: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss.

We also thank the reviewer for the interesting [Seidl97] reference that we will discuss in the revised version.

The reviewer is correct when they state that the adaptation is tied to the specific blackbox model. When the blackbox embedding model changes, the adapters also need to be retrained. We would like to point out that even though this is true, there is minimal effort in retraining the adaptors as compared to other techniques like finetuning. For 1 epoch consisting of 668 batches (batch size 16) our technique only takes ~71 seconds on the largest dataset FiQA-2018 (corpus size 57k), that too on a CPU only compute (on a single K-80 GPU this takes only 8 secs). Hence the effort in training adapters is very minimal and not compute-intensive.

We agree with the reviewer that working on only small corpus sizes is restrictive. We do report numbers for FiQA and NFCorpus, where the corpus sizes are larger, and the number of good retrievals is less. There are applications where corpus sizes are small and training data is sparse, which serve as motivation for this work.

We agree that we cannot bring about large improvements since our technique employs a residual approach. It is limited to adapting the embeddings and not changing them entirely. Further, we envision our method to be employed in cases specifically where there is a scarcity of training data and larger models might be overfit to this. In case there is sufficient training data one can finetune a whitebox model or learn a denser layer on top of the blackbox embedding and would see better improvements since the inherent quality of dense embeddings would improve. Another benefit of our approach is the low computational requirement, which allows one to do online learning to tune ADDERs to handle new data/ outliers. Finally, the improvements in retrieval efficacy are dependent on the base model -- we observed significantly more pronounced gains SBert; please see the table in comment to Reviewer LpK4.

Thank you for asking about results on RAG. We report results on the M dataset proposed in [1], we would be happy to include these results in the final version of the paper as well. We observe the using only OpenAI embeddings gives us a sketch match performance of 0.72 on the M dataset. Whereas when we use ADDER to select the fewshot prompt and perform reranking over generated candidates, we obtain a Top-1 performance of 0.74.

[1] Anirudh Khatry, Joyce Cahoon, Jordan Henkel, Shaleen Deep, Venkatesh Emani, Avrilia Floratou, Sumit Gulwani, Vu Le, Mohammad Raza, Sherry Shi, Mukul Singh, Ashish Tiwari. From Words to Code: Harnessing Data for Program Synthesis from Natural Language (arXiv:2305.01598)

Q1. We claimed that our approach, Adder, improves strict retrieval based on the evidence that the relative improvement on ndcg@1 metric is mostly greater than the relative improvements we get on ndcg@10 (and ndcg@k for larger k). This can be verified for the data presented in Table 2 and 3. For example, in Table 3, Row 1 for smcalflow, we see a 50% relative improvement on ndcg@1 and a smaller 30% relative improvement on ndcg@10. In general, we observed that whenever there are gains, the relative gains on ndcg@k are higher for a smaller k. That is why we said that our approach is more suited for strict retrieval tasks. We will suitably justify our claims in the revised version.

Q2: We treat h, which is the rank of the LoRA transformation matrix, as a hyperparameter in our experiments. We selected the value of h from the set [16, 32, 64, 128, 256, 512, 768, 1024] through validation over the train/dev sets. We observed that the best values for h across the datasets we reported emerged to be around 256 (for ADDER) & 512 (for ADDER2). We also observed some robustness to the choice of h in our experiments – the reported metrics did not change very significantly when varying h in a range centred around the ‘best’ h determined by validation over the available data.

Weakness1: General-purpose OpenAI embeddings are very strong baselines. Looking at the leaderboard on BEIR benchmarks (https://eval.ai/web/challenges/challenge-page/1897/leaderboard/4475), which reports ndcg@10, we see that OpenAI embeddings (our baseline) is very close to the leaders, and even beats the top leader on one dataset (Arguana). For SciFact, we improve over openai embeddings, and thus beat the state-of-the-art (on the ndcg@10 metric). As we discussed in Q1 above, we get higher relative gains on ndcg@1, which is our focus. Further, we explained in the paper (and Weakness 3, below) why we do not see gains across all benchmarks, and when we don’t, we do not deteriorate the performance of openai by much (and still beat the leaders on some benchmarks, such as Arguana).

Weakness 2 about strict retrieval is the same point as Q1.

Weakness 3: It is true that our approach does not uniformly improve performance across all benchmarks. However, there is good reason for it, as we discuss in the last paragraph of Section 4.1 and the last two paragraphs of Section 4.2. We believe that knowing when our approach improves over the baseline, and when it does not, is a strength of our work. It informs us on when to use an embedding adapter or when not. So, while the first impression from our results (Tables 2,3) may be that our approach does not work consistently, if we look deeper at the class of benchmarks where it does not improve over the base openai embeddings, one realizes that the results validate our hypothesis – use adapters when the retrieval task has nuances (code, semi-structured text) that the base model is not likely to exploit.

Given that the OpenAI embeddings are widely trained on public data, including language and code, it is possible that they were also trained on some of the tasks/data from the BEIR dataset. That can make it difficult to further improve upon the OpenAI embeddings by tuning them on the same data. On the other hand, a weaker embedding model, like sBERT, provides more opportunity for improvement by tuning it for the task. In the results presented below for SBert + Adder, we do see a consistent gain across all IR tasks ranging from 2x to 10x in nDCG@1 across datasets. Thus, the overall gains that we can expect from our approach depends on the initial pretraining of the underlying model.

For the same BEIR benchmarks reported in Table 2, here are the results with using SBert as base. (nDCG@k values)

For the same task reported in Table 3, here are the results when using SBert as base: (nDCG@k values)

To illustrate why we need to adapt or tune a general-purpose embedding, consider the Natural Language (NL) utterance Select all unique rows and the utterance Remove duplicates from the table. As NL sentences, these two are different: one is talking about picking elements, while the other is talking about removing elements. However, both these NL utterances map to the same code in any NL2Code task. When a model trained on sentence similarity is used to compute similarity, we are likely to get a very low similarity score for these two sentences; in fact, OAI embeddings give a 0.1 score here indicating that the two sentences are not similar. In contrast, Adapted Embeddings (ADDER) show that these two sentences are very similar by producing a score of 0.9. Thus, ADDER is able to adapt the embeddings so that it corresponds to the code similarity intent when we are dealing with NL2Code tasks.

Motivating example from Few-Shot Selection

Let us say we are given the task of generating the code given below from the following NL.

NL: only keep the 10 longest questions

Code: Table.FirstN(Table.Sort(Table1,{"Length"},Descending),10)

The user wants to retain the 10 longest questions in the table. Default openai embeddings fixate on the word question and the number 10. As a result if we use OpenAI embeddings to retrieve few-shot examples, we see retrieved samples such as

NL: Combine the two questionnaires into one table.

Code: Table.Combine({#"Questionaire 1", #"Questionaire 2"})

NL: Retrieve the first 10 rows from Table1.

Code: Table.FirstN(Table1, 10)

NL: Select all rows from Table1 where the difference between the EndDate and StartDate is 10 days.

Code: Table.SelectRows(Table1, each Duration.Days([EndDate]-[StartDate]) = 10 )

ADDER rather gives the following example:

NL: Sort the table1 by columns C%, d%, and r% and return the first 10 rows.

Code: Table.FirstN( Table.Sort(Table1,{"C%","d%","r%"}),10 )

The sample retrieved using ADDER captures the code similarity intent much better because the code here first sorts the column in a given order and then picks the top 10 from the sorted list, which is what the code for user's NL query should also do.

As a second example, consider the task where the user provides the following NL utterance, and expects the model to generate the subsequent code.

NL: if column "URL" does not end with ".html" then I want to append the text ".html" to the column

Code: Table.TransformColumns(Table1, {{"URL", each if Text.EndsWith(_, ".html") then _ else _ & ".html", type text}})

Open AI Embeddings selects samples that contains string constants within quotes, as in the NL utterance:

NL: _Add a new column to Table1 called "End String" and fill it with the text rel="nofollow">" followed by the value of the API column and ending with "</a></td>".

Code: Table.AddColumn(Table1, "End String", each """ rel=""nofollow"">" & [API] & "</a></td>", type text)

NL: Replace the value of the URL column in Table1 with the text "https://ir.tevapharm.com/news-" whenever the value contains "/news-".

Code: Table.ReplaceValue(Table1,"/news-", "https://ir.tevapharm.com/news-", Replacer.ReplaceText,{"URL"})

ADDER gives the following examples:

NL: _Add a column to Table1 called Right, and for each row, if the Header ends with the letter W, then the value in the Right column should be W, otherwise it should be null and the type should be text.

Code: Table.AddColumn(Table1, "Right", each if Text.EndsWith([Header],"W") then "W" else null, type text)

NL: _Transform the columns of Table1 by replacing the href column with the text before the delimiter ">".

Code: Table.TransformColumns(Table1, {{"href", each Text.BeforeDelimiter(_, ">"), type text}})

Note how EndsWith and Table.TransformColumn both have been covered in the samples selected by ADDER.