Two Heads are Better than One: Retrieval Augmented LLM for Question Answering with External Knowledge Attention

1. While the paper presents a novel approach to integrating external knowledge into LLMs, it may lack a broader discussion on the generalizability of this method across different LLM architectures.
2. It is unclear whether the proposed attention mechanism and memory module would require significant retraining or adaptation for each LLM, which could limit its practical applicability.
3. Although the authors explain that this embedding method has advantages (as mentioned in lines 050-085), the article does not explicitly address the robustness of the model to noisy or irrelevant data during retrieval, nor does it provide a comparison with these methods.

• Line 109: There are some errors in the figure references.

1. The usage of notations and symbols in the paper is sometimes misleading, and some of them are not consistent throughout the paper. For example,

• $s_{l,u}$ in equation (7) becomes $s_{l,n}$ in L173
• $f_{K}$ in equation (1) and $f_{KA}$ in L101, L124
• The $H^S$ in equation 3 is different from in L145
• It should be $H^{X}_{l-1,2}$ in equation (9)

2. The datasets and benchmarks used for the general domain QA experiments are somewhat outdated and are not commonly used for evaluating QA in the era of LLMs. This makes the proposed method incomparable to more recent and advanced methods. For example, the community usually uses the following datasets/benchmarks:

• Natural Questions: A Benchmark for Question Answering Research, Kwiatkowski et al., 2019
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, Joshi et al., 2019
• MuSiQue: Multihop Questions via Single-hop Question Composition, Trivedi et al., 2022
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, Mallen et al., 2023

3. The experimental settings are not entirely fair. In Table 4, the backbone models for the baselines are not the same, and some methods are optimized using the training data, while others are not. Furthermore, there is a lot of missing data in Table 4, and the improvement appears marginal compared to Tables 2, 3, and 4, especially considering that the proposed method requires additional training and inference costs. For example, it’s only about a 1-4% improvement compared to the base LLaMA in Table 2.
4. The paper overstates some of its findings and results. For example, it only experiments with the LLaMA model (7B/13B) and BERT as the encoder, yet claims "our approach works with various pre-trained LLMs" (L314). In addition, it states in L297 that each model was run three times and that the average and standard deviation were reported, but this information is not shown in the paper.

1. The architecture’s complexity, involving multiple components like memory modules and additional attention mechanisms, might pose challenges in practical implementations or scaling.
2. The multi-hop datasets are simple and small, except for GrailQA.

The approach's intuition is not fully explained. The authors argue that standard RAG suffers from "less converged integration of knowledge and LLMs" due to concatenation-based methods, yet this claim lacks citation and evidence. Moreover, the experiments do not directly validate that the proposed approach addresses this issue.

The paper is poorly written, with the methodology lacking sufficient detail. Specific aspects remain unclear, as noted in the questions below.

Certain methodological choices lack justification and intuitiveness, such as the attempt to align the BERT and LLAMA latent spaces without clear post-training evidence.

The approach seems closer to fine-tuning than instruction tuning. Given that LLMs are designed for multiple tasks, the authors should demonstrate that the approach preserves the model's performance on tasks beyond RAG.