Multi-Field Adaptive Retrieval

We thank the reviewer for their comprehensive feedback. We detail our responses below:

W1: Although the paper focuses on multi-field ranking, classic methods like BM25F [1] and its subsequent extensions [2,3] are not included in the baselines. Including at least BM25F would provide valuable context and facilitate a more comprehensive comparison.

Thanks for the suggestion, and we will include a BM25F baseline. However, we’d like to emphasize that in BM25F, we would need to pre-select (or search) the weights for BM25F and these weights would be constant for each query. And, we’ve found that without query conditioning, scores are considerably worse (Table 3). Nonetheless, this week we implemented BM25F [1] on the documents, and as a starting point, picked weights of 1 for each field on Amazon, MAG, and Prime, and obtained the following scores (we repeat the BM25 and mFAR_all scores below too), just to get a sense of how it compares. Since we have no prior understanding for what weights may be best, we adopted a uniform prior across fields by setting each weight to 1. This is another limitation of using BM25F: prior works had very few fields [1] and could hand select weights (mainly HTML webpages, so 3 or fewer fields); in our use case, the weight selection must be made in advanced, and do not know the importance of each field ahead of time (the best we can do would be intuition based on a cursory look).

BM25F fails to perform well, and possibly in the case of MAG, uniform weights happen to do quite well. Furthermore, BM25F does not see gains over BM25, making it a weaker baseline.

Going further, one may try to learn the BM25F weights end-to-end in combination with the query adaptation in mFAR, but that would require a GPU-friendly version of BM25F, which we are not aware of, and we did not have time to implement it this week. We think it could be interesting to try, however.

[1] Simple BM25 extension to multiple weighted fields, https://dl.acm.org/doi/10.1145/1031171.1031181, 2004

Thanks for authors response and additional experimental results in a such short period of time. Table retrieval is just one example of the multi-field ranking tasks extensively studied by the information retrieval community. Regarding datasets, there are several public datasets used for table-based QA [1-2], where table retrieval is a crucial step. In general, there are prior methods addressing multi-field ranking/learning, but none of them have been discussed or compared in this context [3-5].

[1] Open Question Answering over Tables and Text, ICLR 2021

[2] ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering. EMNLP 2022

[3] Neural Ranking Models with Multiple Document Fields, WSDM 2018

[4] GLOW: Global Weighted Self-Attention Network for Web Search, BigData 2018

[5] Multi-field Learning for Email Spam Filtering, SIGIR 2010

Yes, I asked for one additional, but optional experiment where BM25 scores are learned using coordinate ascent. Regarding the combined field ask: What I suggest for this experiment is to have a combined field as one more field in the mix. In fact, one can experiment with inclusion/exclusion of this field from the mix.

I would like to emphasize that I consider this experiment as a "nice-to-have" rather than "must-have".

PS: the rationale for adding a combined field to the mix: With enough training data, the results will never be worse than using only a single combined field. Why? Because, the learning algorithm can assign weight zero for any other field except the combined field. This would permit diagnosing issues with the learning algorithm and what not. However, in my experience with multi-field ranking is that additional fields usually improve outcomes at least a bit. Check out, e.g., an MS MARCO document ranking leaderboard, which has both single-field and multi-field BM25 runs.

Based on the discussions, I have decided to slightly increase the score.

I encourage the authors to consider adding additional baselines, such as BM25 or Contriever, that integrate scores from multiple fields with weights as hyperparameters (I am very interested in the correlation between the tuned weights and the learned weights produced by your method). This could further strengthen the study.

Once again, I thank the authors for their active engagement and for conducting additional experiments during the rebuttal phase.

We thank you for the very specific comments, and we are happy to hear that the motivation is clear to you and that the paper is easy to read. Below, we address what was mentioned:

The main weakness, and kudos to the authors for discussing this, is what they mention in Section 4 "Multi-field vs Single-field": "A side-by-side comparison of the single field models against their multi-field counterparts shows mixed results". The difference in averages between mFAR_2 and mFAR_all doesn't appear statistically significant. The primary gains (looking at mFAR_2 vs mFAR_all) seem to be in the prime data set which has a lot of fields. Should the paper instead focus on adaptive hybrid retrieval, and choose hybrid retrieval baselines?

We believe the paper’s current focus (as stated in the abstract) is on both adaptive retrieval and hybrid retrieval, although we do lean into the adaptive part more, as even in the mFAR_2 run, we are conditioning on the query to determine how to weight the (single) dense score against the (single) BM25 score.

We looked for and did not find hybrid retrieval baselines that could be directly compared to our setting. Some past works (which we will discuss more in the paper) [1, 2] typically find complementary features of dense and lexical scorers as part of a full retriever stack, which include re-ranking or alternating between lexical and dense components. Meanwhile all of our baselines are focused on obtaining a single score for document ranking. [3, 4] combines lexical and semantic features together into a single score, but their lexical features are also dense and trained end-to-end (not sparse like our BM25 scores). Due to these differences, we did not make a comparison within the paper.

Furthermore, many of these papers are not easily reproduced or adaptable as they do not release their code. So, we thought of a simpler hybrid model baseline ablation that we can run that still uses only a semantic encoder and BM25. This is a mFAR_2 model without the adaptive query conditioning. Instead, it learns a single weight for the dense score and a single weight for the sparse score, to be used for all examples. In a preliminary run, it scored 0.550 for Hit@1 on Amazon, which is a little lower than the 0.574 that we report in the paper. This is early evidence both that a simple hybrid baseline is actually quite strong too – it alone would be state-of-the-art – and that our adaptive weighting can still provide benefits on top of that.

We also agree that the “mixed results” confused us as they don’t lead to crisp conclusions about separating into multiple fields. To investigate further, we trained another variant of mFAR named mFAR_{1+n}. This one consists of a single sparse score and multiple dense scores (one per field) – we can view this as somewhere in between mFAR_{all} and mFAR_2. This model performs at or above the level of mFAR_2 across the 3 datasets, which suggests a conclusion that there is a benefit to decomposing the dense scorer into fields. We show the results of both models in the table below, and will also update our preprint with this new information.

Are "Dense only" in Table 4 and "mFAR_dense" in Table 2 the same (the numbers are close but different). Were mFAR_dense and mFAR_lexical trained separately or trained together and one component ablated?

In Table 2, mFAR_dense and mFAR_lexical are mFAR models trained with only dense scorers or lexical scorers, respectively. In Table 4, “Dense only” refers to a post-hoc masked version of mFAR_all, where all the weights associated with lexical scorers are set to 0, which would leave behind only dense scores.

In other words, if $|F|$ is the number of fields in a dataset, mFAR_dense has $|F|$ weights, mFAR_lexical also has $|F|$ weights. Meanwhile, mFAR_all has 2$|F|$ weights (|{dense, lexical}| * $|F|$). Dense-only and Lexical-only makes inference-time adjustments to mFAR_all. All 2$|F|$ weights are predicted, but half of them are clamped, at inference time, to 0. We can clarify in Section 5.2.

[1] Complementing Lexical Retrieval with Semantic Residual Embedding, https://arxiv.org/abs/2004.13969, 2020

[2] On Complementarity Objectives for Hybrid Retrieval, https://aclanthology.org/2023.acl-long.746/, 2023

[3] A Dense Representation Framework for Lexical and Semantic Matching, https://arxiv.org/abs/2206.09912, 2022

[4] UnifieR: A Unified Retriever for Large-Scale Retrieval, https://arxiv.org/abs/2205.11194, 2022

Thank you for the detailed and thoughtful response. In general, I agree that while learning adaptive retrieval across dense and sparse scorers, taking multiple fields into account seems reasonable. Perhaps future work will fully bear out the potential for multi-field retrieval. The time boundedness makes this hard, but I agree with other reviewers that it is rather important to compare against strong baselines for multi-field retrieval, some of which predate neural models. Perhaps the authors can acknowledge the difficulty in including some baselines like BM25F, referencing them in the Related Work and Future Work sections.

Overall, I think the community will benefit from the publication of this work. I will raise my score to 8.

We thank you for your concerns, as it helps us clarify the design decisions for our framework.

Though several recent retrievers [1] and [2] have been proposed, Contriever is used as a representative of dense retrievers without sufficient discussion. I am concerned that the conclusion may change if the authors use the recent retrievers. Especially, since FollowIR is instruction-tuned, it may be more robust against negation in the query, which the authors pointed out as a possible issue of the single-field method. Regarding Weakness 1 in the previous section, did you try to use other dense retrievers and do you have any insights about experiments with better retrievers?

We had tried GTR-T5 [1] initially and found that Contriever-MSMARCO [2] was performing significantly better in initial experiments for the single-field finetuning baselines, across multiple hyperparameters. In [6], they also report multiple models of various sizes, like roberta, ColBERTv2 [7], GritLM (instruction-tuned, 7B-size) [3], LLM2Vec [4], and ada-002 [5], many of these models recent and the larger models are not fine-tuned. Yet the scores we obtained by fine-tuning Contriever-MSMARCO (110M params) (in Table 2, Contriever-FT) is comparable or surpasses most of their baselines. Thus, we believe that Contriever-MSMARCO is a strong starting point to demonstrate modeling contributions, especially as we are interested in fully fine-tuning the model for end-to-end training, and that many of the other models are too large.

We did not see promising preliminary results without the ability to fine-tune the encoder, and so we did not try directly applying mFAR to the frozen LLMs. Given more compute capacity, we expect that fine-tuning any of the larger competitive models (like GritLM-7B or ada-002) could yield higher-scoring baselines, and further applying mFAR on top of that with fine-tuning could lead to similar gains/trends.

We did not consider treating instruction fine-tuned methods, like FollowIR, differently, as our work was focused on document decomposition and that most of the queries in STaRK are relatively straightforward (additionally, as noted above, GritLM is an instruction-tuned model that performs similarly to other models). Your comment raises a valid point that there are other solutions on the encoder-side to robustness – negation specifically. We do not think the point invalidates our method, which demonstrates a (separate, unintended) solution to negation on the document-side. However, we thank you for the suggestion, and we will make it clearer that there are other solutions, like FollowIR/GritLM, and that the figure presented in our error analysis is not meant to represent an explicit goal of solving negation/robustness.

The adaptive weighting mechanism in MFAR relies on training within specific domains and structured datasets, making it potentially sensitive to domain-specific data characteristics. This might lead to suboptimal field selection or scoring when the document structure is inconsistent across the corpus.

We do agree that training is domain-specific. However, this is also the case for any single-field fine-tuning setup where a retriever must be finetuned on the dataset for best performance.

Regarding inconsistent structure: entries in the Prime dataset are inconsistent. Only some entities (genes/proteins) have “interacts_with” field, while others (drugs) do not have that but instead have “side_effects.” No entity has every field. To address this, we artificially zero out nonexistent fields for each sample during training, which still allows the model to learn with all fields, even though no document contains all possible fields.

[1] Large Dual Encoders Are Generalizable Retrievers, https://arxiv.org/abs/2112.07899, 2021

[2] Unsupervised Dense Information Retrieval with Contrastive Learning, https://arxiv.org/abs/2112.09118, 2021

[3] Generative Representational Instruction Tuning, https://arxiv.org/abs/2402.09906, 2024

[4] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders, https://arxiv.org/abs/2404.05961, 2024

[5] ada-002, https://openai.com/index/new-and-improved-embedding-model/

[6] STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases, https://arxiv.org/abs/2404.13207, 2024

[7] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction, https://arxiv.org/abs/2112.01488, 2021

We thank you for your questions, which will clarify our paper, and we are glad that you find our paper well written.

What is exactly used for dense/vector retrieval?

In Eq. 4, what is exactly q in G(q, f, m): I assume it should be a dense query encoder, but it's not clear from the text (at least for me).

Thank you for catching these details. For dense/vector retrieval, we use an off-the-shelf embedder (Contriever-MSMARCO [1]) to encode the query $q$ to $\bf{q}$ and encode the text from fields $x_f$ to $\bf{x}_f$, decomposed from document $d$. For the dense retrieval, we can then compute similarity (unnormalized inner product) between $\bf{q}$ an $\bf{x}_f$. The same $\bf{q}$ is used in $G$.

We will make this notation clearer, as presently we’ve left out the definition of $\bf{q}$.

How did you decide on the selection of top-100 records for each field (L885)?

We followed the experimental setting of k=100 in the original Contriever paper and past work. To efficiently decide which documents belong in the top-k, we only fully-score documents if one of its fields belongs in the top-k for at least one of the $s_f^m$ functions. This subset of documents is sorted and the top-k (k=100) is used as the document ranking for evaluation.

[1] Unsupervised Dense Information Retrieval with Contrastive Learning, https://arxiv.org/abs/2112.09118, 2021

We thank you for your comments and appreciation of our approach. We are glad that you find the paper well-written and well-structured. Below, we address the weakness(es) and question that you brought to our attention:

The fine-tuning approach makes the approach specific to a set of fields from a dataset.

We agree with this. However, this still gives us a lot of flexibility. The main appeal of our modeling approach with a specific set of fields denoted is the fact that finetuning can be personalized for a specific dataset. If we do not know what queries will ask about, we can be more inclusive and include everything during fine-tuning (as computation budget allows), and our framework will eventually learn the important fields; the unimportant fields would anyways be assigned lower weights. As a future extension, we could even automatically learn to prune the unimportant fields either post-hoc or during training.

Information overlap in fields (see lines 416-424) might intrudice some redundancy to the retrieval process.

We do agree that there is redundancy with the tokens and phrasings that might be retrieved, and we do not claim that fields are completely orthogonal. In fact, the redundancy is warranted as fields that highlight the same proper noun or phrasing should be promoted as our scoring (especially with a lexical scorer, such as BM25) benefits from repeated instances of a specific token or repetitions in different contexts.

How much robust is the framework to variations in the number of fields, e.g., regarding field information overlap?

This is an interesting question. In some preliminary ablations, we attempted to train a model without certain fields. Specifically for MAG, we tried experiments with only 1, 2, or 3 of the 5 fields. We found that the score would monotonically increase as we increased the number of fields, and that all the fields were needed for best performance.

However, this might not be the case for all datasets. As we can see from the full results of masking out fields (Sec 5.3; Appendix D) for Amazon, some fields can be entirely removed from a fully-trained model without affecting the score, which suggests there is some degree of robustness. In particular, there are cases (like “qa” for Amazon, Table 5) where masking out either the lexical or dense scorer results in no drop, but masking out both results in a substantial drop in performance. This suggests that the model can be robust to redundant information being removed.

We appreciate your comments on our paper. We are glad that you find our paper easy to read and the quality good. Here, we lay out some thoughts regarding your comments:

It is not clear to me whether the fields may be considered indepent, so that the summation of the field-related scores suffices for determining the overall score. That is, it seems intuitive that there are correlations among the field instances and they may bias the result, as was extensively researched in the information retrieval area.

Q1: Are the scoring dimensions really orthogonal, enabling summation as the summarization metric?

In our setup, we do not guarantee a notion of independence for the fields. We use different scorers for each field, so for models that use both lexical and dense scorers, both scorers score the same underlying text, resulting in likely dependence.

We would be happy to include the references of needing independence if you could give us examples of these references showing that correlations may bias the result. Do note that we do achieve good performance with the assumption that there are potential dependencies between fields. This result suggests that the model is appropriately weighting the fields, even though there may be overlapping information in the field texts. We will add this discussion regarding independent and dependent fields to the paper (as EjWS also brought it up).

Another field-related issue is regarding the process of selecting the fields that will be considered in the whole process.

Our goal is to provide a flexible framework to automatically select any and all fields that are most salient to an individual use case. Our query-conditioning approach allows the on-the-fly weighting of fields, lowering the weight for fields that are deemed less useful and increasing the weight for those fields that are more relevant to the query. This way, we don’t need to pre-select specific fields (we can use all of them in the dataset). This is relatively lightweight, adding a small number of parameters to the model (only 768 per field) which is minor compared to the 110M-size of the full model.

Overall, the proposal is simple and basically consists of combining scoring functions associated with fields, without considering their correlations and other characteristics that may either characterize the task or explain problems or failures. For instance, although the paper focuses on the information carried by the fields, it seems intuitive to mix the aggregated value of the fields with the remaining text, exploiting eventual information there.

Our framework aims to do retrieval with a generalizable approach. Thus, finding correlations for each dataset between each field may be impractical. As for mixing with the remaining text, given a single field, we are including information under that field – for instance, if our fields of interest are “Book Title”, “Author”, and “Book Text”, we are including the texts “Moby Dick”, “Herman Melville”, and “Call me Ishmael. Some years ago… [truncated at context size]” respectively. So, we are mixing the aggregated value of the field with the full text already, to our understanding. We apologize if we have misunderstood your comment, and it would be illustrative if you could give an example or clarify what we do not already mix into the summation.

The experimental result also needs to be improved, as detailed next. First of all, the two experimental hypothesis seem to be too simple, thus quite easy to demonstrate. The advantages of using document structure are expected, in particular considering the additional information given to the models. The expected gains of hybrid approaches are also quite predictable. In both cases, it would be interesting to somehow derive upper bounds on the gains, so that the results go beyond benchmark-based evidence.

For many research problems, simple solutions are preferred over complex solutions, so we do not believe simple solutions and experimental designs (in our case, query conditioning) are a weakness. For example, zero-shot chain-of-thought [2] is also a simple method with immense impact for LLMs. Though it may seem intuitive that keeping document structure improves performance, we show that in settings where we do not utilize query conditioning, performance is far worse than simply training a retriever on entire documents. Thus, it is not trivial to take advantage of document structure.

Furthermore, it is also not necessarily obvious that hybrid approaches will always produce better results. Despite Amazon obtaining relatively high performance for lexical-only and dense-only scorers, we find that the combination does not always achieve the best performance (in the case of mFAR with multiple fields).

On upper bound: could you clarify what type of experiment you would like to see? Or are you looking for something theoretical, and can you explain what specifically?

On the other hand, it is also necessary to clear outline the limitations of such hybrid approaches and how they may be addressed. I really missed confidence intervals or similar statistical significance metrics on the results. For instance, on Table 2, the results may be too close and, despite the bold highlight, it is not clear how far the results are from the remaining of the baselines.

Thank you for the excellent suggestion. As each (model, dataset) configuration requires training a full model on multiple GPUs in parallel, we will not have significant metrics completed in time during this rebuttal period. However, we will run each of the main mFAR models across multiple random seeds and compute confidence intervals for each model and will update the preprint.

Anecdotally, we saw fluctuations of as much as 0.01-0.02 for each metric, but we agree this is not a substitute for more rigorous testing across all (model, dataset) combinations.

Q2: Aren´t there additional baselines and data sets that may be used in the experiments?

Of course, there are always additional models and datasets. Could you be specific about which dataset or model you think would improve the paper?

We already report several baselines, and [1] reports even more in their study on this dataset. Reviewer Hpub has brought up table retrieval as a task, which we also describe in related work. We will work on that for the final version, but may not complete them in time during this rebuttal period. Note that not all datasets are suitable for multi-field retrieval – standard datasets like MSMARCO or NQ do not have semi-structured documents, and so we would not expect mFAR to perform well on them (nor do we claim it would).

One terminology issue is regarding the difference between structured and semi-structured and where the paper fits. It seems to me that semi-structured is the proper jargon.

Thank you for this suggestion; we will change the terminology to “semi-structured.”

[1] STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases, 2024, https://arxiv.org/abs/2404.13207

[2] Large Language Models are Zero-Shot Reasoners, 2022