Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting

We thank the reviewer dkh1 for the valuable feedback. We address individual questions carefully below.

Different Volume of Training Data

We acknowledge the importance of evaluating our framework's performance across different training data volumes. Our primary experiment utilized 40,059 instances to train our drafter model. To thoroughly assess scaling effects, we conducted additional experiments using incremental subsets of 10,000, 20,000, and 30,000 training instances. The results of these systematic evaluations are detailed below.

From the table, we can conclude that increasing the volume of training data leads to improved performance. Specifically, the model's accuracy continues to rise as more instances are included, with the highest performance observed at 40,059 instances. This suggests that larger training datasets contribute positively to the performance of our drafter-verifier framework, indicating that scaling up data size could enhance the robustness of the model.

Ablation study to assess the effectiveness of P(Yes)

Thank you for pointing this out! We believe this ablation study will further clarify the design of Speculative RAG. Below are the results of our ablation study, which we will include in our future draft.

In the table, we use various score combinations to verify the generated drafts. We find that P(Yes) is effective in identifying the best draft candidate (superior to the random choice), and its performance improves further when combined with other evaluation scores. As noted by the reviewer, we believe that the P(Yes) score could contribute even more significantly to the final performance if the verification model undergoes sufficient instruction tuning.

Reproducing the results of CRAG and Self-CRAG

We have thoroughly investigated the problem framework presented in [1] and successfully generated additional results for both CRAG and Self-CRAG implementations on the new benchmark datasets. We plan to incorporate these findings into the main results table of our paper.

We thank the reviewer niGd for the valuable feedback. We address individual questions carefully below.

Contribution of Speculative RAG

Inspired by Speculative Decoding [1], Speculative RAG is the first to apply the drafting paradigm to RAG problems. We share the core idea of using a smaller language model (LM) for efficient draft generation and a larger LM for verification and selection. As noted by reviewer GVM9, our approach is "both novel and appealing," and our empirical results demonstrate significant improvements in efficiency while maintaining high performance across benchmarks. We believe Speculative RAG has the potential to significantly impact the field of RAG by offering a practical solution for high-performance, low-latency retrieval-augmented generation.

• [1] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. "Fast inference from transformers via speculative decoding." International Conference on Machine Learning. PMLR, 2023.

About the Benchmarks

To evaluate Speculative RAG, we followed the standard RAG evaluation protocol established in prior work like Self-RAG [1], CRAG [2, 3], SAIL [4], InstructRAG [5, 6], and FlashRAG [7]. We used five well-known RAG benchmarks: TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge. Each benchmark sample includes a knowledge-intensive question, its corresponding answer, and retrieved documents provided by the benchmark authors. We believe this comprehensive evaluation across diverse datasets provides strong evidence for the effectiveness of our method.

• [1] Asai, Akari, et al. "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." The Twelfth International Conference on Learning Representations.
• [2] Yan, Shi-Qi, et al. "Corrective retrieval augmented generation." arXiv preprint arXiv:2401.15884 (2024).
• [3] Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.
• [4] Luo, Hongyin, et al. "Sail: Search-augmented instruction learning." arXiv preprint arXiv:2305.15225 (2023).
• [5] Wei, Zhepei, Wei-Lin Chen, and Yu Meng. "InstructRAG: Instructing Retrieval-Augmented Generation with Explicit Denoising." arXiv preprint arXiv:2406.13629 (2024).
• [6] Asai, Akari, et al. "Retrieval-based language models and applications." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts). 2023.
• [7] Jin, Jiajie, et al. "FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research." arXiv preprint arXiv:2405.13576 (2024).

We thank the reviewer GVM9 for the valuable feedback. We address individual questions carefully below.

Access to these external resources

In our paper, the baseline methods resort to external resources in different ways to enhance the base model’s capabilities for the RAG scenario. We compare our proposed Speculative RAG with the baseline method in terms of the external resources as follows.

We list the following new task settings:

(1) Also using synthesized data from GPT models as the baseline models

We’ve conducted experiments with GPT for data synthesis, detailed in Appendix A in the submission. We observed that Speculative RAG consistently outperforms all baselines. We are happy to relocate these results to the main text in the final version for better clarity.

(2) Fine-tuning Mistral on the same synthesized data as our Speculative RAG

We further tune Mistral-7B with the same synthesized data from Gemini as our Speculative RAG but conduct the standard RAG. We can see our Speculative RAG maintains its performance advantage.

The memory overhead in the parallel verification

Yes, verification is conducted in parallel. We will clarify this in the revised draft. This design choice enables batched input for parallel verification, converting sequentially retrieved documents in the prompt. As shown in Figure 2, generated rationales are significantly shorter than retrieved documents, mitigating the memory burden during parallel verification.

Comparison with InstructRAG

We appreciate your bringing InstructRAG to our attention. It is indeed a relevant and noteworthy concurrent work, which we will incorporate into our related work discussion. The use of rationale in instruction-tuning seems to be an independent discovery in both of our works. Here, we compare Speculative RAG and InstructRAG in detail.

(1) The Use of Rationale

InstructRAG and SpeculativeRAG share the use of rationales in instruction-tuning. InstructRAG focuses on denoising retrieved documents and justifying answers, while Speculative RAG leverages smaller and larger language models to balance efficiency and accuracy. The rationale in Speculative RAG primarily serves as a concise reference for verification. We believe these methods are complementary, and incorporating denoising-style rationales into Speculative RAG could further enhance performance.

(2) Different Problem Setting

A major difference between InstructRAG and SpeculativeRAG is the problem settings. InstructRAG instruction-tuned the base model with the supervised data from the target dataset while Speculative RAG follows the problem setting of Self-RAG and CRAG and fine-tunes the base model with the general instruction-following dataset. We believe the performance of Speculative RAG can be further improved by incorporating task-specific training data into the instruction-tuning stage.

Thank the authors for their response and additional results! My concerns under Weaknesses are partially address; however, to the best of my knowledge, the two questions I raised under Questions are not answered by the authors. Collectively, I will maintain the score of my original rating.

We sincerely thank Reviewer GVM9 for the thoughtful feedback and valuable questions. We have addressed the concerns raised and provided detailed clarifications below. We kindly ask the reviewer to revisit our responses and consider our submission. We are also open to further discussions to improve our work.

For the comparison of the average number of tokens in the generated rationale and the retrieved documents in Figure 2: What is the number of retrieved documents? Is it before or after subsetting to drafters?

Answer: In Figure 2, the number of retrieved documents refers to the subset of documents submitted to the drafter. As described in Section 4.2 (Experiment Settings), this includes 2 documents for TriviaQA, PopQA, PubHealth, and ARC-Challenge experiments, and 6 documents for MuSiQue experiments.

The contribution and result of the paper could be further strengthened if newer generation of models (e.g., gemma-2, llama-3) are adopted in the experiments.

Answer: We thank the reviewer for raising this great point! Below is the performance analysis of Speculative RAG using the instruction-tuned Gemma-2-2b as the drafter and the frozen Mixtral-8x7B as the verifier:

These results suggest that Gemma-2-2b provides a promising avenue for future work. We believe this aligns with the reviewer’s suggestion to explore the capabilities of newer-generation models and opens up opportunities for further optimization.

Is the Multi-Perspective Sampling stage also considered in the latency analysis in Section 4.5?

Answer: Yes, the latency analysis in Section 4.5 includes the time cost of the Multi-Perspective Sampling stage.

We thank the reviewer RYjP for the valuable feedback. We address individual questions carefully below.

More baseline methods

Thank you for highlighting these relevant works! We will incorporate them into our final version.

In [1], the authors introduce the ReClaim framework, which addresses knowledge-intensive questions with fine-grained, sentence-level citations. This enables models to provide detailed references for each sentence in question-answering tasks. In [2], the retrieval-augmented thoughts (RAT) framework is proposed, where each step of a chain of thought (CoT) is iteratively revised using retrieved information after generating an initial zero-shot CoT. These two related works focus on refining the rationale of generated answers, either by incorporating detailed citations or iteratively improving the reasoning chain. While these papers focus on refining the rationale for answering knowledge-intensive questions, Speculative RAG introduces a novel approach to efficiently generating multiple answer drafts in parallel using smaller LMs. These drafts are then rigorously verified by a larger LM, ensuring both efficiency and accuracy.

We are particularly excited about the potential for integrating these approaches into future iterations of Speculative RAG. Combining the strengths of rationale refinement with our parallel generation and verification process could lead to even more robust and comprehensive answers to knowledge-intensive questions.

• [1] Ground every sentence: Improving retrieval-augmented llms with interleaved reference claim generation.
• [2] RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation.

Two-stage RAG

While two-stage methods have been explored in various domains, our approach uniquely applies the draft-then-verify paradigm to RAG in a map-reduce fashion. This is a novel contribution that hasn't been thoroughly investigated in the context of RAG.

Speculative RAG specifically addresses the need for a balance between efficiency and effectiveness in RAG. By employing a smaller, specialized LM to generate multiple drafts from distinct document subsets in parallel, we significantly reduce the computational burden. Subsequently, a larger, generalist LM verifies these drafts, selecting the most accurate one. This not only improves efficiency by offloading computation but also reduces input token counts for generation, leading to faster response times.

Existing two-stage methods can be broadly categorized into those that generate a chain of thought before answering [1, 2, 3] and those that use a self-evolving process to refine answers iteratively [4, 5]. While sharing a two-stage structure, Speculative RAG's unique draft-and-verify design, inspired by Speculative Decoding, sets it apart and enables superior performance with reduced latency.

• [1] Wang, Yile, et al. "Self-Knowledge Guided Retrieval Augmentation for Large Language Models." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
• [2] Wei, Zhepei, Wei-Lin Chen, and Yu Meng. "InstructRAG: Instructing Retrieval-Augmented Generation with Explicit Denoising." arXiv preprint arXiv:2406.13629 (2024).
• [3] Wang, Zihao, et al. "Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation." arXiv preprint arXiv:2403.05313 (2024).
• [4] Khattab, Omar, et al. "DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines." The Twelfth International Conference on Learning Representations. 2024.
• [5] Jiang, Zhengbao, et al. "Active Retrieval Augmented Generation." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Q1: in Line 207-209, is the answer A obtained by human labeling or generated by LLMs given documents D? What if the D does not provide evidence for generating A?

The QA pairs are both from the open-source instruction-following datasets. In the cases where the retrieved documents are not relevant to the question or the answer cannot be derived from the documents, the LLM is instructed to generate a tag for us to filter this case out.