Self-Retrieval: End-to-End Information Retrieval with One Large Language Model

Thanks for your time and insightful comment. We would like to address your questions in turn.

About NQ@40k in our experiments

Currently, the most common and widely used retrieval method is dense retrieval, which primarily focuses on passage-level retrieval ([2, 3]). Consequently, we constructed NQ@40k to specifically evaluate performance at the passage level. Additionally, we observed that most other generative retrieval methods struggle with passage-level retrieval, whereas self-retrieval successfully overcomes this limitation.

The length of passages

In common settings for passage-level retrieval, documents are often segmented into chunks of 100 or 200 tokens ([1, 2]). Therefore, we believe that using 200 tokens as the passage length for our main experiment is reasonable. Additionally, we have reported the performance for the more commonly used 100-token segments in the Appendix. It should be noted that shorter passages may not necessarily lead to better performance, according to [1].

[1] Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering.

[2] Dense Passage Retrieval for Open-Domain Question Answering.

[3] C-pack: Packaged resources 535 to advance general chinese embedding.

The differences in baselins of NQ@40k and NQ@320k

1. NQ@40k is a passage-level retrieval setting, following the setting of previous dense retrieval works. We reproduced the latest dense retrieval methods like BGE and GirtLM. Since some previous generative retrieval works have not been open-sourced, we have only reproduced and compared some key generative retrieval baselines.
2. NQ@320K is a document-level retrieval setting, following the setting of previous generative retrieval works to ensure comparability with previous works. Furthermore, thank you for your reminder. In the subsequent versions, we will add the performance of the latest dense retrieval methods, such as BGE and GritLM, on NQ@320k for comparison.

About full KILT datasets

Due to time and computational resource limitations, we were unable to complete training and testing on the entire Wikipedia corpus from KILT during the rebuttal period. Our analysis of scaling corpus size in Figure 3 of the main paper shows that Self-Retrieval maintains a relatively parallel performance relationship with BGE-FT as the document size increases. This trend indicates that Self-Retrieval has substantial scaling potential. Furthermore, due to the scaling characteristics inherent to LLMs, we also believe that their capacity to handle documents has a very high upper limit. However, due to the entire Wikipedia corpus comprsing around 5.9M documents, it is challenging to complete training within the effective rebuttal period. We appreciate your insightful comment and will add more scaling experiments in the future.

Experiments on different training strategies

Different training strategies indeed have a certain impact on Self-Retrieval's performance. Our choice of the "1+2" training strategy was confirmed through early preliminary experiments. To compare different strategies, we conducted experiments on NQ40k. We found that jointly training on all three datasets yields overall performance (79.26 Hits@5) comparable to the "1+2" strategy(79.28 Hits@5). However, the "1+2" training method allows multiple different training sets with the same corpus to share a single indexing process. This significantly reduces the overall training time.

About wrting

Thank you for your suggestion. We will refine it in the next version.

Dear Reviewer uLuX,

As the discussion deadline is nearing, we wanted to gently follow up on our recent submission. We've carefully addressed your feedback and incorporated it into our revised manuscript. We provided clarifications on our choice of NQ@40K for alignment with standard passage-level experiments in dense retrieval, and explained why the "1+2" strategy is preferable to joint training in our context. We hope our rebuttal have addressed your concerns regarding our experimental and training settings.

Your feedback is highly valuable to us. We welcome any additional feedback or questions you may have about our revisions.

Thank you for your time and expertise.

Thank you for your suggestions on our work. Here is our response to your concerns.

Analyze the latency

We compared the efficiency of Self-Retrieval with SEAL in the General Response Table 1. Our generation process has the following characteristics:

1. Early Stop Mechanism: Our method includes an 'early stop' mechanism where the Self-Retrieval model ceases generation once the LLM has produced sufficient content to determine the current passage. As shown in Attachment Figure 1, the average decision length of the trie is approximately 13 tokens, which is similar to the decision length in Term-Set Generation. Consequently, the total number of tokens generated remains relatively small.
2. No Additional Post-Processing: Unlike SEAL, our Self-Retrieval method does not require extra time for post-processing or retrieving passages. This means that even when using a larger LLM, the overall efficiency remains comparable to that of SEAL.

About reranker in RAG

We have augmented our experiments with the incorporation of a BGE reranker in the RAG process (BGE-FT + BGE reranker-FT + StableLM-FT). EM scores on the NQ and TriviaQA datasets were recorded as 41.98 and 50.16, respectively. As can be seen from the results, there was no observed improvement over the BGE baseline that does not employ a reranker. This outcome is likely due to the BGE's substantial proficiency developed during the training phase. Consequently, as shown in Attachment Table 1, additional training of the reranker does not bring extra benefit.

Experiments on non-wikipedia datasets

Please refer to the General Response.

Shared Model v.s. Separated Models

We separately trained individual retrieval and ranking models, achieving Hit@1 and Hit@5 scores of 60.56 (compared to 62.16 with joint training) and 78.21 (compared to 79.28 with joint training) on NQ@40K. These results show a slight decline compared to joint training in Self-retrieval. This indicates that Self-Retrieval can synergize the retrieval and ranking tasks during training to achieve better performance. Additionally, using shared models can reduce deployment costs.

Thanks for your time and valuable comment.

Efficiency Analysis Details:

The latency/H@5 results in Table 1 of Global Response include the re-ranking step. We have updated the table to include results without re-ranking as well. These results demonstrate that without re-ranking, Self-Retrieval further improves efficiency while maintaining competitive performance.

Early Stopping Mechanism Details:

When the Early Stopping Mechanism is triggered, we have typically decoded only a few tokens. Nevertheless, these tokens are sufficient to identify a unique passage. In the prefix tree, each leaf node corresponds to a specific passage ID, allowing us to map the decoded prefix directly to a passage ID. Subsequently, we extract the full passage content from the corpus and append it to the previously decoded content for re-ranking.

Thank you for your constructive comments. We appreciate your feedback and will address each of your points in turn.

Experiments on non-wikipedia datasets

Please refer to the General Response.

Experiments on untitled documents

With untitled documents or titles of poor quality, Self-Retrieval can use the generated model from other models. We conducted experiments on MS MARCO. Specifically, we prompted LLaMA3 8B in a zero-shot manner to generate appropriate titles for the documents missing the title. We tested the settings following GenRet on MS MARCO. The experimental results, shown in the General Response Table 2, indicate that titles generated by the LLM can effectively meet Self-Retrieval needs.

About retrieval+reranker baselines

Thank you for your feedback. We conducted experiments using a 2-staged retriever+reranker approach. Specifically, we chose strong retrieval baselines such as BGE, GTR, GritLM, and DSI-XL as the foundation, and then applied three different rerankers: BGE reranker, BGE reranker FT, and RankGPT. (We did not use RankT5 because the relevant code or model has not been open-sourced). As shown in Attachment Table 1, our Self-Retrieval approach outperforms most retriever+reranker combinations, demonstrating the effectiveness of our method.

Efficiency discussion

Please refer to the General Response.

Scaling up to full NQ/TriviaQA

In Figure 3 of our paper, we demonstrate the performance of Self-Retrieval and BGE as the corpus size increases. Our experiments included a maximum of approximately 200k documents and 3M passages. We are currently conducting larger-scale NQ/TriviaQA experiments to further validate the scaling effects of our model. However, due to the full NQ/TriviaQA datasets involving the entire Wikipedia corpus (approximately 21M passages for the DPR version and 5.9M documents for the KILT version), it is challenging to complete training within the effective rebuttal period. We appreciate your insightful comment and plan to include more scaling experiments in the future.

Number of candidates in the reranking stage

In the reranking stage, we use 50 passages as candidates.

We sincerely appreciate your time and valuable feedback. We are pleased that our additional experimental results have addressed one of your primary concerns.

Regarding the scaling capabilities of Self-Retrieval:

1. As shown in Figure 3 of the main paper, our current experiments encompass a substantial dataset of approximately 200k documents and 3 million passages, demonstrating Self-Retrieval's capability with large-scale collections.
2. It's worth noting that in generative information retrieval research, using partial document collections is common due to training efficiency constraints. For instance, Ultron and GenRet utilized subsets of 300k documents from MS MARCO, while UniGen employed a subset of approximately 100k passages from the NQ dataset for evaluation. In contrast, Self-Retrieval conducted main experiments on 1 million passages of NQ and further expanded it to 3 million on the scaling experiment.
3. We are actively conducting experiments on the full NQ dataset. However, due to time and computational resource limitations during the rebuttal period, we were unable to complete these extensive experiments for this response.

Concerning efficiency, as mentioned in our Global Response, Self-Retrieval achieves comparable efficiency to SEAL while maintaining high performance. Moreover, Self-Retrieval requires only a single LLM to manage both retrieval and downstream LLM tasks. This unified architecture enhances deployment flexibility and potentially reduces computational overhead.

We hope these points can partially alleviate your concern about the scaling and efficiency aspects of our method. We remain committed to further large-scale experiments and are open to addressing any additional questions or concerns you may have.

Thanks for your helpful comments! We are very glad to address your concerns one by one.

Details of Trie

The trie is pre-built based on the corpus before retrieval. Most documents/passages only share a small common prefix. The LLM stops generating once it has produced enough tokens to determine the current document. As shown in Figure 1 of the attachment file, the average decision length of the trie is around 13 tokens. For most documents, the retrieval process can decide on the current document after generating less than 20 tokens, at which point we "early stop" and manually append the document to the context.

In this case, the depth of the trie depends more on the number of documents/passages rather than the length of each document/passage. Therefore, even when dealing with long documents, the common prefix length between documents may not necessarily be long, allowing Self-Retrieval to handle long documents effectively. We will include these details in the next version.

Space and Time Complexity

Please refer to the General Response.