EnrichIndex: Using LLMs to Enrich Retrieval Indices Offline

Thank you for your review. Please note that there appears to have been an error on the reviewer's side regarding the evaluation setting on BRIGHT (please see (*) below). In addition, we addressed the additional comments: adding new results of a stronger baseline, a thorough analysis of our weight tuning approach, and we will be sure to add an extended Related Work section containing all of the works referenced by the reviewers. Given the initial misunderstanding and our efforts to thoroughly address the concerns raised, we would sincerely appreciate it if the reviewer would consider revising their assessment and raising their score.

(*) First of all, we would like to make two factual clarifications regarding the summary of the review.

The approach is evaluated on the coding part of the Bright dataset.

Finally reporting on a subset of Bright makes it appear that the dataset was cherry picked for the best subset of the technique. The remaining Bright numbers should at least appear in the appendix for the interested reader.

As mentioned in Section 3.1 (line 139), Table 1, Table 4, and Table 5, we evaluated on the Bright dataset across all three top domains: StackExchange, Coding, and Theorems.

While demonstrating that the added information is about to move documents closer to the query using Wasserstein distance in Section 4.1 is somewhat interesting, this fails to capture the interaction between relevant and non-relevant information that is key to ranking relevant information above non-relevant information. It is not sufficient to move relevant information closer to the query if non-relevant information also moves closer to the query.

Section 4.1 and Table 6 present the Wasserstein distance between the similarity distribution of (query, non-gold objects) and that of (query, gold objects), both before and after applying EnrichIndex—not just the distribution of (query, gold objects) alone. Our results show that EnrichIndex increases the distance between gold and non-gold similarity distributions, assisting retriever models in better distinguishing gold objects from non-gold ones. We believe this analysis reflects the interaction between relevant and irrelevant information with respect to the query, rather than considering only one side in isolation.

Lack of context of this work relative to prior work on document expansion

Thank you for pointing out the related work— we will ensure to include and compare it thoroughly in the related work section of the next version.

Specifically, in contrast to document enrichment methods like doc2query, our approach expands enrichment across multiple modalities (e.g., documents and tables) and incorporates various enrichment types—including purpose, summary, and synthetic queries—while using off-the-shelf models rather than fine-tuned ones. Additionally, our method diverges in the online retrieval process: whereas doc2query concatenates generated queries with original documents for similarity search, we index each enrichment type separately and compute a weighted sum during retrieval. This design offers greater flexibility in storing vector embeddings, as each index operates independently and can be expanded with additional embeddings or indices as needed.

Unfair comparison with rerankers

We used Llama-3.1-8B-Instruct as the online reranker because this was used in the original Judgerank paper and is a popular open-source model.

We further use GPT-4o-mini as an alternative online re-ranker, the same model we employed for the offline enrichment process. The results, shown in the table below, indicate that switching from Llama-8B to GPT-4o-mini improves overall performance. Nevertheless, EnrichIndex still outperforms Judgerank even when the latter uses GPT-4o-mini. Furthermore, applying GPT-4o-mini for online re-ranking on top of EnrichIndex leads to even greater performance gains.

R stands for recall @ 10, N stands for NDCG @ 10

Unclear how dependent the approach is on setting the different weights for the final ranking function

Regarding the dependency of EnrichIndex on setting the weights: we demonstrate the robustness and generalizability of our approach, and add an analysis on the weight tuning process. Specifically, we evaluated whether a smaller portion of validation data could still yield strong performance. While the default setup uses 20% of the data for validation, we also experimented with using 15% and 10%, and reported the results for stage-one retrievers below. Overall, we observed that even with reduced validation data, EnrichIndex continues to deliver significant performance gains, highlighting the robustness of our tuning methodology. We will be sure to include this analysis in the final version of the paper.

R stands for recall @ 10, N stands for NDCG @ 10

R stands for recall @ 10, N stands for NDCG @ 10

In addition, we explored whether a single set of coefficients could be tuned and applied across multiple datasets. To test this, we combined the validation sets from the three table retrieval datasets—Spider2, Beaver, and Fiben—into one unified validation set for the weight tuning process. The results, summarized in the table below, show that EnrichIndex still delivers substantial performance improvements, demonstrating the generalizability of our approach across diverse workloads simultaneously.

Q1 How well does the approach work on the remaining Bright tasks?

As detailed above, the performance is reported for all tasks in Bright.

Q2 Can you provide a generic setting of a1,a2, a3, and a4 to use as a comparison to the tuned version to help inform the reader about the robustness of the approach?

The weight tuning analysis above demonstrates that our approach maintains strong performance even when using limited data or a single set of weights across multiple datasets, underscoring its robustness and generalizability. Moreover, the tuning process is highly parallelizable, requires minimal effort, and enables easy adaptation of our general retrieval framework to various workloads.

R3: Too few cited work (<20) for such a highly researched topic, and not even a related work section. It is probably worth mentioning work that has focused on improving retrieval via summarization such as done in ALCE, RAPTOR, Decontextualization (Choi et al), among others. Also failures of neural IR have been well studied in papers like ABNIRML.

Thank you for pointing out the related work— we will be sure to include all of the referenced works (ALCE, RAPTOR, Decontextualization (Choi et al) and ABNIRML) into a more comprehensive Related Work section.

R4: JudgeRank is a complete strawman wrt cost. It is an agentic impractical method, and very few companies would consider deploying this---its main purpose is to show us that reranking on BRIGHT is possible, and perhaps can be distilled into a cheaper cross-encoder. I understand that it is current SOTA, hence is an appropriate target for comparison, but the focus on token counts as presented in the paper hides some of the real costs associated with using EnrichIndex.

We agree that Judgerank’s high computational cost can make it impractical for real-world deployment, which is precisely what motivated this work. We have shown that EnrichIndex not only outperforms Judgerank but can also operate at a significantly lower cost, as we explain below—making it a more practical and scalable solution for real-world applications.

In terms of cost, our method primarily involves two components: offline enrichment and storage.

b. Similarly, if the enriched info is meant to be used in RAG, reranking, or other applications, then the input costs of those downstream pipelines will be increased.

Importantly, during re-ranking, we only use the original object content, so there is no added cost during downstream tasks.

a. The storage cost is 4x. If we assume 1k tokens per object at gpt-4o-mini output token pricing (600 to enrich a 1M object corpus. In comparison, according to qdrant pricing, 1M vectors is about 210/month to support 4x vectors per object.

For offline enrichment, it is worth noting that in real-world scenarios, corpora are often relatively static, meaning enrichment only needs to be performed once. When new data is added, enrichment can be applied incrementally to just those new objects. While this approach may increase storage requirements due to additional vector embeddings, the overall cost becomes negligible when amortized over a large number of queries.

Additionally, latency is a critical factor in many applications. Our approach introduces only a minor increase in computation—specifically, a few extra similarity calculations—which adds minimal overhead compared to the latency introduced by LLM-based methods. This efficiency makes our method well-suited for high-throughput query environments.

We appreciate the highly positive review, noting that EnrichIndex may serve as a foundation for future research bridging IR and LLM research. We will be sure to add a more comprehensive overview of the Related Work to our manuscript. Regarding the reviewer’s comments:

R1: It was not clear to me whether it is important to use the exact similarity for each enriched info type. This could become not very practical if all similarities are required, and may be more prone to errors when using approximate nearest neighbors than retrieval that only uses one "type".

The core objective of the online retrieval phase is to identify the top-k objects based on multiple attributes. In the paper, we present a simple approach: computing the weighted sum of similarities for all objects and selecting the top-k results. However, other alternatives exist for retrieving the optimal object based on a series of rankings: for example, by combining nearest neighbor search with the classical Fagin’s algorithm (or the Threshold Algorithm). Fagin’s algorithm is designed to efficiently retrieve the top-k items from multiple sorted lists when using a monotonic aggregation function—such as the one defined in Equation 1. The partially sorted lists can be obtained by performing top-k nearest neighbor searches based on query-object similarities for each enrichment type.

R2. Hyperparameter tuning

We provide a more thorough analysis of the hyperparameter tuning below. We will be sure to add this discussion to the final version of our paper.

We note that this tuning process is highly parallelizable, requires minimal effort, and enables easy adaptation of our general retrieval framework to various workloads.

c. AFAICT the hparams per dataset were not released. Why not?

The coefficients used in the weighted sums in Equation 1 are easy to reproduce, as they are determined by the simple process of enumerating all possible coefficient combinations that sum to one, followed by selecting the combination that yields the highest recall or NDCG@10 on a 20% validation set. We will provide the detailed procedure and the values in the next version of the paper.

a. There is no analysis of hparam tuning. It is plausible that hparam tuning could be very important given the ablation shown in Fig 2.

To demonstrate the robustness and generalizability of our approach, we conducted a more thorough analysis of the coefficient tuning process. Namely, we evaluated whether a smaller portion of validation data could still yield strong performance. While the default setup uses 20% of the data for validation, we also experimented with using 15% and 10%, and reported the results for stage-one retrievers below. Overall, we observe that even with reduced validation data, EnrichIndex continues to deliver significant performance gains, highlighting the robustness of our tuning methodology.

R stands for recall @ 10, N stands for NDCG @ 10

R stands for recall @ 10, N stands for NDCG @ 10

Thank you for your review and feedback. We hope our response below bolsters the soundness of our method and effectively addresses the concerns about our paper.

The proposed approach is straightforward. Authors are suggested to add more discussion on why summary, purpose, and QA pairs help to improve performance in this task.

Following the reviewer's suggestions, we intend to add an expanded Related Work to the paper, where we will also discuss our enrichment types (summary, purpose, QA) and their relation with prior work.

Questions

Authors compare JudgeRank and the proposed approach in order to demonstrate the effectiveness of the proposed approach. But, authors haven't told what LlM is used in the implementation of JudgeRank. Does it also use GPT 4o-mini? If it uses other models, such as LLaMa models, the performance comparison will be unfair.

The LLM used by the JudgeRank baseline is Llama-3.1-8B-Instruct, also mentioned in Table 4. This is due to Llama-8B being used in the original Judgerank paper, which was the state-of-the-art on the BRIGHT benchmark.

Following the reviewer’s suggestion, we ran new experiments with a stronger version of JudgeRank using GPT-4o-mini. The results, in the table below, show that EnrichIndex continues to outperform JudgeRank, even when the baseline is powered by GPT-4o-mini. Furthermore, applying GPT-4o-mini for online re-ranking on top of EnrichIndex leads to even greater performance gains over JudgeRank.

R stands for recall @ 10, N stands for NDCG @ 10

Thank you for your review, we appreciate the positive feedback. We hope to address the comments regarding the related work and analysis of our results below:

The current submission lacks a discussion of related studies. Specifically, it needs to acknowledge and contextualize prior work on data augmentation techniques for boosting retrieval performance, including but not limited to works such as [1-5]. Given its direct relevance to pseudo query augmentation (QA pairs), a direct comparison to [1] is particularly encouraged.

We will be sure to add a comprehensive Related Work to our paper and to include and discuss all papers mentioned by the reviewer.

The models and baselines employed in the experimental evaluation appear to be dated. To provide a more compelling demonstration of EnrichIndex's advantages, comparison against more current state-of-the-art retrieval methods is recommended.

Our baselines for implicit retrieval and table retrieval were considered to be state-of-the-art when we submitted our paper (late March 2025). The dense retrieval models that were used were among the top retrievers on the MTEB leaderboard, while JudgeRank, the LLM-based re-ranker, was the state-of-the-art on the Bright leaderboard. We will be sure to emphasize this further in our revised manuscript.

In addition, we added new results, using a stronger version of JudgeRank — switching the underlying Llama-3.1-8B with the stronger GPT-4o-mini. Our results (in the table below) still demonstrate that EnrichIndex outperforms the GPT-4o-mini powered JudgeRank. Furthermore, combining GPT-4o-mini with EnrichIndex (in the online re-ranking phase) leads to an even greater performance gain compared to the GPT-4o-mini-powered JudgeRank. We will be sure to include these results, using a stronger baseline, in the revised version of our paper.

R stands for recall @ 10, N stands for NDCG @ 10

While the paper mentions the tuning of weights for combining similarity scores, crucial details regarding the methodology are absent.

As mentioned in Appendix E, for each dataset, we randomly split the questions into an 80/20 ratio, using 20% for tuning the weights. Overall, in order to optimize the four weights in Equation 1, we allowed each to range from 0.0 to 1.0 in increments of 0.1, ensuring their sum equals 1. This resulted in 286 possible combinations (calculated using the stars and bars method). We then parallelized the search across these combinations to identify the one that maximizes recall or NDCG@10 on the validation set. This tuning process is very efficient and requires minimal effort, while also allowing our general retrieval framework to be easily adapted to different workloads. We would be happy to move some of the implementation details to the main paper in the camera-ready version.

The exclusive reliance on GPT-4o-mini for data enrichment and expansion raises questions about the generalizability and replicability of the reported results across different Large Language Models.

We selected GPT-4o-mini for offline enrichment because it is a strong and widely used proprietary model that offers a good balance between performance and cost. However, to address the reviewer’s concerns, we also added results using the popular open-source Llama-3.1-8B-Instruct model as an alternative for offline enrichment. The results below show that even when using enrichment generated by Llama-8B, EnrichIndex consistently yields significant improvements across different stage-one retrievers.

Due to limited computational resources, we applied this model to all table datasets and one representative sub-domain from each major category in the Bright benchmark. The table below presents the average retrieval performance of various stage-one retrievers, both with and without EnrichIndex.

R stands for recall @ 10, N stands for NDCG @ 10

Questions

Could you please report statistics on the generated enrichments, such as average length per type (summary, purpose) and the number of QA pairs per item.

We provide the enrichment statistics (with the GPT-4o-mini model), using OpenAI's tokenizer: we found that the average token count is 65.1 for enriched summaries, 76.5 for enriched purposes, and 477.7 for QA pairs — with an average of 16.5 QA pairs per object. Overall, the token count for enriching each object is small. In addition, note that the number of QA pairs can be easily adjusted by specifying the desired amount directly in the prompt.

Can you share qualitative analysis (e.g., case studies, examples) demonstrating how the enrichments specifically improve table retrieval.

In the original paper, we included two examples—Figure 1 for tables and Figure 3 in the appendix for documents—to illustrate how our enrichment process works and how it enhances retrieval effectiveness.

What are the potential biases (e.g., factual errors, skewed perspectives from the LLM) or limitations (e.g., generation of overly verbose or irrelevant content) that might be introduced by the LLM-generated enrichments, and what methodologies could be employed to mitigate such concerns?

As with all LLM-generated text, there is always the risk of enrichments that include hallucinated data. It would definitely be interesting to incorporate fact verification techniques together with EnrichIndex to better counter hallucination during the offline process. While this serves as an exciting topic for future work, we believe it is orthogonal to our current paper.