Imagine All The Relevance: Scenario-Profiled Indexing with Knowledge Expansion for Dense Retrieval

Full Results of Figure 6 - Part2

Answers to Questions

About Scenario Generator

We only considered the Llama-3B model as the backbone for the scenario generator. This is because, due to resource limitations in our system, it was not possible to train larger models.
Also, due to resource and budget constraints, the teacher model was also GPT-4o, which is relatively less expensive compared to reasoning models. We also remain curious about how SPIKE’s performance might change if we were able to use different backbones or teacher models for the scenario generator.

Full Results of Table 2

Below is the expanded version of Table 2.

Full Results of Figure 6 - Part1

Below is the result table for the larger range of α.

Thank you for your interest in our research and for suggesting areas for improvement. We provide detailed responses to your comments and questions

About baseline

Thank you for this valuable suggestion. We agree that including more baselines, such as reranking and query rewriting methods, would help provide a more comprehensive comparison. In the revised version, we will add additional baseline results to better contextualize SPIKE’s performance relative to these widely used approaches.

Performance on Stronger Models

First, as seen in the upper tiers of actual retrieval benchmarks such as MTEB and BRIGHT leaderboards, even an improvement of 1 absolute point can lead to significant changes in model ranking. As models approach the top ranks, the performance gap between them becomes extremely narrow. Therefore, the ~1-point performance gain achieved by SPIKE on strong retrievers (e.g., GRIT, Qwen) should still be regarded as a meaningful improvement.

Furthermore, the aggregate improvements may seem limited, breaking down the results by domain highlights SPIKE’s practical value in addressing the blind spots of retrieval models.
The table below summarizes the average performance of each model on the natural language and code domains.

Looking at the domain-wise performance, we observe that all models consistently perform worse in the code domain compared to the natural language domain. Notably, even strong models (GRIT, QWEN) achieve already high performance in the natural language domain, but in the code domain, the baseline performance does not exceed 20, indicating a limitation of existing dense retrieval models.

In terms of performance improvement, for the natural language domain, we find that the relative gain from SPIKE drops to around 1% as the retrievers become stronger (GRIT). In contrast, for the code domain where all models continue to struggle, SPIKE consistently delivers an improvement of over 7% across all models.
This demonstrates that SPIKE can be highly effective in addressing blind spots such as the code domain, which are not well handled by existing dense retrieval models.

These results lead us to believe that SPIKE has practical value in consistently addressing the blind spots of future retrievers.

Additionally, SPIKE’s scenario generator is currently trained using reasoning traces from GPT-4o. Prior work (RANK1 $1$ ) has shown that substantial gains can be achieved by training LLM rerankers on the outputs of state-of-the-art reasoning models. We expect that if SPIKE’s scenario generator is further trained using even more powerful or specialized reasoning teachers, additional improvements could be realized even for strong retrievers. We plan to investigate this direction in future work.

Latency Analysis

We acknowledge that the additional scenario index introduced in SPIKE requires extra computation and may lead to a certain increase in latency. However, we would like to emphasize that the amount of additional computation incurred by SPIKE during inference does not increase proportionally with the index size (i.e., corpus size N).

In theory, introducing a scenario index would require computing scenario scores ($r_s$) for all scenarios associated with every document in the corpus. However, in practice, we do not adopt such an inefficient approach. Instead, SPIKE applies a much more optimized strategy.

Specifically, for retrieve top-k documents, SPIKE first computes the document-level scores ($r_d$) for all documents, and then, only for the top-k' documents (where k’ is hyperparameter satisfying k < k’ << N), it rapidly retrieves the corresponding scenarios using the document as a key. Scenario-level scores ($r_s$) are computed only for these retrieved scenarios (while $r_s$ for the remaining scenarios are set to zero).

As a result, the additional computation at inference is strictly limited to the top-k′ documents and their associated scenario embeddings, regardless of the overall corpus size or the total number of scenarios. Therefore, even as the corpus grows, the extra latency is affected only by k', enabling efficient system operation.

Furthermore, all the experimental results reported in our paper were obtained using this retrieval optimization. In fact, we have confirmed that applying this optimization results in the same retrieval performance as the naive approach without the optimization.

[1] Rank1: Test-Time Compute for Reranking in Information Retrieval; weller et al.

Thank you sincerely for reviewing our research and providing valuable feedback.
First, regarding Figure 1, we will revise it by highlighting the important information as you mentioned, and adding more detailed explanations to the caption.
Our responses to the additional points you raised are as follows:

SPIKE consistently outperforms the baseline across all hyperparameter settings.

While Figure 6 shows that performance varies depending on the value of the relevance weight, all configurations still outperform the baseline model without SPIKE. In fact, some configurations achieve even higher average scores than our default setting (see Appendix Table 7).
This demonstrates that SPIKE is robust to hyperparameter variations while still delivering consistent performance gains.

About the Baseline Model Used in Analysis

We selected E5-mistral-7B for our analysis (Figure 6) because it is still widely used as a strong retrieval baseline in recent work across diverse domains $1$

$2$ .
Our intention was to show that SPIKE is beneficial not only for retrieval tasks but also for downstream applications such as RAG, and we aimed to offer insights that can generalize to a wide range of practical use cases.
Hence, we conducted our analysis on a well-established and broadly applicable baseline model.

That said, we fully agree with the suggestion that SPIKE should be analyzed with a broader set of retrieval models. We are currently conducting experiments with additional retrieval baselines and will include the full analysis in the final version of the paper.

$1$ ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations; Wang et al.

$2$ MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation; Park et al.

Thank you for your valuable feedback and suggestions to improve our work! We fully agree with your suggestions and will incorporate them into our work.

Scenario Generation Quality

We fully agree that the quality of generated scenarios can substantially influence retrieval effectiveness. While we did not introduce an explicit metric for scenario quality in this work, we conducted a human evaluation (see Figure 4) to indirectly assess generated scenario quality.

However, as you pointed out, we acknowledge that our current work lacks a thorough analysis of how scenario quality directly impacts retrieval performance. In future work, we plan to evaluate scenario quality across multiple versions of the scenario generator using human evaluations, and to analyze the correlation between these human evaluation scores and retrieval accuracy. We believe this additional analysis will provide a clearer understanding of the relationship between scenario quality and retrieval effectiveness, and we appreciate your suggestion.

Latency Analysis

We acknowledge that the additional scenario index introduced in SPIKE requires extra computation and may lead to a certain increase in latency. However, we would like to emphasize that the amount of additional computation incurred by SPIKE during inference does not increase proportionally with the index size (i.e., corpus size N). In theory, introducing a scenario index would require computing scenario scores ($r_s$) for all scenarios associated with every document in the corpus. However, in practice, we do not adopt such an inefficient approach. Instead, SPIKE applies a much more optimized strategy.

Specifically, for retrieve top-k documents, SPIKE first computes the document-level scores ($r_d$) for all documents, and then, only for the top-k' documents (where k’ is hyperparameter satisfying k < k’ << N), it rapidly retrieves the corresponding scenarios using the document as a key. Scenario-level scores ($r_s$) are computed only for these retrieved scenarios (while $r_s$ for the remaining scenarios are set to zero).

As a result, the additional computation at inference is strictly limited to the top-k′ documents and their associated scenario embeddings, regardless of the overall corpus size or the total number of scenarios. Therefore, even as the corpus grows, the extra latency is affected only by k', enabling efficient system operation.

Furthermore, all the experimental results reported in our paper were obtained using this retrieval optimization. In fact, we have confirmed that applying this optimization results in the same retrieval performance as the naive approach without the above optimization.

We will include this analysis in the revision of our analysis section, and we appreciate your helpful suggestion in highlighting this important aspect.

Thank you for taking the time to review our research!

SPIKE also performs well on other document collections

Below table shows the retrieval performance of dense retrieval models on the BEIR benchmark, comparing their original performance against their SPIKE enhanced versions.
Due to time and computational constraints, we selected 5 BEIR datasets with fewer than 1,000 queries and a corpus size under 500,000 documents for our experiment.

Across these datasets, SPIKE consistently improves the average retrieval performance, demonstrating that its effectiveness is not limited to the BRIGHT benchmark.
These results indicate that SPIKE can be safely and effectively applied to diverse document collections, even in corpus involving different types of relevance.

SPIKE Complements Query-Side Expansion Approaches

While we acknowledge that query-side expansion can yield significant improvements in many retrieval scenarios, we emphasize that document-side expansion, as implemented in SPIKE, can effectively complement query-side expansion methods. Thanks to the offline nature of document expansion, combining both approaches allows retrieval systems to achieve enhanced performance and broader coverage with only minimal additional online latency, as demonstrated in our experiments (Please refer to Figure 3 and Table 4).

SPIKE addresses the limitations of existing dense retrieval models.

We would like to clarify that the point raised regarding BM25 + CoT query does not reflect not a limitation of SPIKE, but rather limitation of dense retrieval models in generalizing to LLM-generated queries.
As reported in the BRIGHT paper, BM25 with CoT query expansion consistently outperforms dense retrievers with the same expanded queries, even though dense retrievers outperform BM25 on original queries.

The following table from BRIGHT $1$ presents a comparison of BM25 and various dense retrieval models, both before and after applying GPT-4 CoT query expansion.

According to the table, dense retrieval models typically surpass BM25 in standard settings, but when CoT-based query expansion is applied, their performance drops below that of BM25.
BRIGHT $1$ attributes this result to the following reason:
"possibly because BM25 can adapt to different queries, while LLM-generated queries are out-of-distribution for trained models."

This implies that the relatively strong performance of BM25 + CoT queries does not stem from limitations of the SPIKE, but rather from the inherent limitations of dense retrievers in handling such queries.

We would like to claim that SPIKE can address this limitation. By using document indexing to generate scenario representations closely aligned with LLM-generated (expanded) queries, SPIKE enables dense retrieval models to respond more robustly to these queries.

For instance, in the BRIGHT benchmark, Qwen with GPT-4o CoT query expansion achieves an nDCG@10 of 24.8, which is notably lower than the 26.5 reported for BM25 + GPT-4o CoT in Weller et al. $2$ . However, when SPIKE is applied on top of Qwen + CoT, the performance increases to 26.2, effectively closing the gap with BM25 + CoT.

This evidence clearly demonstrates SPIKE's consistent ability to enhance retrieval performance across different query types. Furthermore, it suggests that as dense retrievers continue to evolve and are trained on LLM-generated queries, the benefits offered by SPIKE will become even more significant.

$1$ BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval; su et al.

$2$ Rank1: Test-Time Compute for Reranking in Information Retrieval; weller et al.