In-Context Learning with Retrieval Augmented Encoder-Decoder Language Models

Additional Questions:

Thank you for your questions. We respond to them one by one below:

1. Here, we report the best observed few-shot performance with more than 8 shots.
2. For decoder-only LMs, the computational cost typically increases quadratically with the input length, as well as with the number of retrieval passages. In contrast, for encoder-decoder LMs with a fusion-in-decoder architecture, the computation cost grows linearly with the number of retrieved passages, as they only perform self attention over one passage at a time (Izacard et al., 2021). Therefore, from the perspective of computational overhead increasing with the number of retrieved passages, retrieval-augmented encoder-decoder language models are superior to decoder-only language models.
3. This work was released after we completed our draft. The differences include: 1) As depicted in their Figure 1 (middle), they encode each example/input separately. In contrast, RAVEN encodes multiple examples along with the target input and each retrieved passage. 2) Their study does not focus on retrieval-augmentation. Although the title appears relevant to our proposed FiCL method, the content of the work significantly differs from our work.

Thank you for your detailed comments. We hope our response addresses your questions. Please let us know your thoughts, and we are more than happy to answer any further questions.

We thank the reviewer for highlighting the motivation and soundness of our study, as well as the writing of our paper. We greatly appreciate your thorough review and constructive feedback. We are pleased to address the points you raised.

Scope of In-Context Learning:

Thank you for your comment. We test on both open-domain question answering and MMLU, as well as providing case studies on long-form question answering. These are common tasks used for testing the performance of retrieval-augmented LMs in other studies, such as Shi et al., (2023).

As our training objectives (masked and prefix language modeling) are not directly optimized for question answering, they should generalize to various tasks.

Technical Contribution:

Our work offers contributions from both analytical and technological perspectives:

• From an analytical standpoint, we provide a thorough analysis of the in-context learning capability of state-of-the-art retrieval-augmented encoder-decoder language models. Our insights also highlight potential avenues for improvement. This analysis serves as the foundation for our proposed RAVEN training strategy and Fusion-in-Context prompting approach.
• From a technological perspective, we introduce a confluence of retrieval-augmented masked and prefix language modeling, coupled with our Fusion-in-Context Prompting and In-Context Example Retrieval strategies. These techniques not only enhance the base model's capabilities (by mitigating the mismatch between pretraining and testing) but also maximize the potential of in-context learning for these models (either by scaling the number of in-context examples with FiCL or by retrieving better in-context examples).

It's crucial to emphasize that our methodologies weren't spontaneously conceived; they stem from an in-depth understanding of the limitations of current models and a careful consideration of how to overcome these issues as discussed in Sections 3 and 4. While the technique's design might appear simple, its conception was far from obvious without our analytical groundwork. Moreover, simplicity (when effective) is a commendable attribute of a technique.

Contradictory Points on Encoder-Decoder Architecture:

Here, we would like to correct a misconception of the reviewer: prefix language modeling is not exclusive to decoder-only language models. The distinction between encoder-decoder LMs and decoder-only LMs lies in their architecture, not in their pretraining objectives (the reviewer may refer to UL2 (Tay et al., 2023) for more details). However, a significant feature that decoder-only LMs lack is the bidirectional architecture. As noted in the GPT-3 paper: “GPT-3 has several structural and algorithmic limitations ... as a result, our experiments do not include any bidirectional architectures or other training objectives such as denoising ... our design decision comes at the cost of potentially worse performance on tasks which empirically benefit from bidirectionality …” Our method capitalizes on the strengths of both masked and prefix language modeling with the incorporation of a bidirectional architecture.

Comparison with Decoder-Only Models:

Thank you for your comments. We have compared our model with RePlug (based on Codex) and Retro, which are decoder-only language models augmented with retrieval. Retro, having a parameter count similar to our models, performs significantly worse than ours. We have included the results of LLaMA 7B in Table 3. We did not find its performance superior and would be interested in knowing the sources of the papers that suggest otherwise.

Here, we also want to emphasize that this comparison may not be very meaningful, as the models are trained on different data. It is unrealistic to expect a model developed with an academic budget to match the performance of larger-scale models (where "larger-scale" refers to both the number of parameters and the extent of training data), although our results are impressive considering the scale and budget.

Furthermore, our aim is not to build the SOTA model, as this is impossible considering the budget, and no model stronger than GPT-4 truly exists yet. Our main goal is to demonstrate the possibility of in-context learning with retrieval-augmented encoder-decoder language models as a research practice. We provide comprehensive analysis and design techniques that enable these models to perform better in in-context learning.

Thank you for acknowledging our response! We are glad that our response addressed many of your concerns, and we answer your follow-up questions below.

1. MMLU

The experiments in MMLU demonstrate the effectiveness of FiCL by comparing the few-shot results with and without FiCL. This aspect is significant, as we can observe a ~4% improvement.

It’s true that the selection of in-context examples also affects question answering (Table 4 of our paper demonstrates this a bit). However, the bias is more pronounced and negative in multiple-choice problems, which is why FiCL here is very effective, as it mitigates bias without requiring longer context. The 2-shot setting would be interesting. However, as the discussion period is ending soon, we hope the reviewer can understand that we may not have enough time to run the experiments. The reason we report 5-shot rather than 2-shot results is simply because previous works use the 5-shot setting for MMLU and do not include results for 2-shot.

Since previous works report only 5-shot results, we follow them to report the 5-shot results. For GPT-3 and T5, the 5-shot performance is not strong, so reporting the additional 0-shot and 1-shot performance is not very meaningful. Moreover, as mentioned in our previous response, the comparison is not very meaningful, and our primary goal is not to build the SOTA model, where scaling and data engineering is usually the best approach. We regard the results of these models more as references to gauge our model's position. We treat this work as a pioneer effort to demonstrate the possibility and potential, and we encourage future work to scale up the model in our conclusion section.

2. Configurations in Table 2

For results without FiCL, we explored shots from 1, 2, 3, 4, 5, 8, 16, 32, 64, and reported the best observed results; for results with FiCL, based on few-shot results, we explored three significant combinations: [64-1] (or [40-1]), [64-5], [64-8].

This is a common practice for LLM papers reporting few-shot performance, as observed in Table 3 where the best results of other models are based on different shots (e.g., For PaLM, the reported results include 1, 5, 64-shot). And in fact, RAVEN's performance across various shots is stable. The reviewer may refer to Figure 6 again, which consistently shows RAVEN's few-shot performance significantly outperforming that of ATLAS across different shots.

3. LLama 7B

Yes, we added the results of LLama 7B to Table 3. We would also be happy to report the results of LLama 7B with retrieval augmentation, but we did not find it in existing literature. Instead, we report results of RePlug and Retro, which are retrieval-augmented models based on decoder-only language models.

Here, the scale is not just about model parameters but also about the number of training data and the quality of training data. The comparison between different base models is not very meaningful. A better base model can no doubt achieve better performance. Whether it outperforms or underperforms, it does not diminish the contribution of the paper. The results serve more as a reference, related to engineering effort rather than research effort, as scaling and data engineering is usually the best practice if the goal is to achieve better performance. And it’s somewhat meaningless to care about this kind of comparison, as GPT-4 is always a stronger baseline, lol.

We hope the reviewer can regard this work more as a research practice that shows the possibility and potential, focusing on the analysis, design, and comparison between RAVEN and ATLAS.

We hope our response addresses your questions, and we are more than happy to answer any further questions.

Thank you for recognizing that our work is interesting, well-written, and that the performance improvement is significant. We appreciate your insightful feedback. Below, we address your comments and questions:

Weakness 1: “Though the proposed method has a signiìcantly better few-shot performance than ATLAS…”

In the aspect of modeling, the core architecture of RAVEN builds upon ATLAS. This point also holds true for most LLMs, as they typically build upon similar architectures. However, it's crucial to emphasize that our methodologies weren't spontaneously conceived; they stem from an in-depth understanding of the limitations of current models and a careful consideration of how to overcome these issues as discussed in Sections 3 and 4. While the technique's design might appear simple, its conception was far from obvious without our analytical groundwork. Moreover, simplicity (when effective) is a commendable attribute of a technique.

In summary, our work offers contributions from both analytical and technological perspectives:

• From an analytical standpoint, we provide a thorough analysis of the in-context learning capability of state-of-the-art retrieval-augmented encoder-decoder language models. Our insights also highlight potential avenues for improvement. This analysis serves as the foundation for our proposed RAVEN training strategy and Fusion-in-Context prompting approach.
• From a technological perspective, we introduce a confluence of retrieval-augmented masked and prefix language modeling, coupled with our Fusion-in-Context Prompting and In-Context Example Retrieval strategies. These techniques not only enhance the base model's capabilities (by mitigating the mismatch between pretraining and testing) but also maximize the potential of in-context learning for these models (either by scaling the number of in-context examples with FiCL or by retrieving better in-context examples).

Weakness 2: Match Decoder-Only LLMs:

Our results indicate that RAVEN, despite having substantially fewer parameters, achieves comparable performance to much larger decoder-only models in certain scenarios. This is a notable achievement, considering RAVEN's smaller scale. We acknowledge that there is still a performance gap with larger models like PaLM 540B and Codex 175B in some cases, but given RAVEN's size and the training budget, its performance is competitive.

Question 1: Fairness of Comparison:

This is an excellent point. Other LLMs are trained on much larger data, so we cannot expect a model trained on an academic budget to match the performance of these models. However, our model, despite having substantially fewer parameters, achieves comparable performance in certain scenarios. The comparison between RAVEN and ATLAS can be considered fair since they are trained on the same data.

Furthermore, our aim is not to build the SOTA model, as this is impossible considering the budget, and no model stronger than GPT-4 truly exists yet. Our main goal is to demonstrate the possibility of in-context learning with retrieval-augmented encoder-decoder language models as a research practice. We provide comprehensive analysis and design techniques that enable these models to perform better in in-context learning.

Question 2: Inference Cost and Efficiency:

This is a great point. While ATLAS is known for having fewer parameters, the incorporation of an external corpus for retrieval in RAG does contribute to the overall computational load. This is also true for RAG when used with decoder-only language models.

For decoder-only LMs, the computational cost typically increases quadratically with the input length, as well as with the number of retrieval passages. In contrast, for encoder-decoder LMs with a fusion-in-decoder architecture, the computation cost grows linearly with the number of retrieved passages, as they only perform self attention over one passage at a time (Izacard et al., 2021). Therefore, from the perspective of computational overhead increasing with the number of retrieved passages, retrieval-augmented encoder-decoder language models are superior to decoder-only language models.

Dear Reviewer y5kL, we hope our response addresses your concerns. Please let us know your thoughts, and we are more than happy to answer any further questions.

I thank the author for providing detailed responses to the weaknesses and questions from my review.

Firstly, I think the analysis is good but not surprising. Then I still think the technique here is not really new. Prefix language modeling is an already known technique to improve performance, thus I won't count it much as a contribution (for instance I won't count data augmentation as a contribution especially if I propose some special data augmentation for my pipeline). Fusion-in-Context Prompting is new but it's kind of natural to the RAG pipeline, I would count it much for the contribution.

Overall I think the technique contribution is weak. This is my major concern which leads to my slight negative score.

Thank you for highlighting the innovation and significance of our methods, and for your insightful feedback.

The Fusion-in-Context Learning (FiCL) strategy is a natural advantage of encoder-decoder language models. For decoder-only language models, FiCL may be adapted as follows: we can feed different in-context examples to the decoder and then ensemble the outputs.

Another promising future direction is exploring how to combine the fusion-in-decoder architecture with the most powerful decoder-only language models. By doing so, we can harness the advantages of both architectures – employing a fusion-in bidirectional architecture to effectively encode retrieved passages for the most powerful decoder-only LLMs.