GasketRAG: Systematic Alignment of Large Language Models with Retrievers

Concern 3: The performance improvement in experimental results is not significant, especially comparing with naive RAG.

Response:

We need to point out some important misunderstandings here. As shown in Table 2, GasketRAG demonstrates significant improvements over NaiveRAG across all test sets and metrics, which is something that no other RAG method has achieved.

---

Concern 4: Furthermore, I think the baselines are too weak and the results are not convincing, the authors should at least compare their methods with: 1) the long context models which can identify and leverage information in long contexts; 2) RAG methods with denoising and iterative query rewriting components; 3) the same model of GasketRAG, but replacing the Gasket model with a strong LLM (e.g., GPT 3.5, Llama 3, etc.) without any training.

Response:

In fact, the baselines we selected are all state-of-the-art and representative RAG methods, including different types such as question rewriting, iterative RAG, adaptive RAG and CoT-based RAG. Our experiments were conducted under strictly fair settings.

We will address your three suggestions one by one.

For 1), the average passage length in the corpus we used is 100 words. We configured the retriever to return the top-10 passages, which is far smaller than the maximum context length of the LLM generators we used (LLaMA-3 and GPT-3.5). Therefore, none of the baseline methods experienced performance degradation due to context length limitations. We understand your interest in exploring the application of GasketRAG in long-context RAG scenarios, and we agree that this is a fascinating and important direction for future research. However, it falls outside the scope of this study. In this paper, we focus on addressing the preference gap between retrievers and LLMs, and we propose a practical and straightforward preference training method to align the components of the RAG pipeline effectively.

For 2), the baselines we selected already include these two types of RAG methods: RRR and Iter-RetGen. Both of these methods serve as very strong baselines. However, as shown in the results, although these methods exhibit strong performance on specific datasets, their generalization and stability are relatively poor due to the lack of preference alignment between the retriever and the LLM in their approaches.

For 3), in Table 3, we present a comparison between the gasket model and its base model, LLaMA-3.1-8B-Instruct, which has not undergone additional training. It can be observed that our proposed preference data collection method and the Weighted KTO training approach significantly enhance the model's performance within the GasketRAG framework. Additionally, we also tested GasketRAG using GPT-3.5 as the gasket model, with the results (correctness) shown below:

It can be observed that using a stronger model did not result in a substantial performance improvement. This highlights the importance of eliminating the preference gap between the retriever and the LLM through preference learning. Furthermore, this underscores that GasketRAG is not merely a simple refinement or rewriting approach.

We hope our clarifications address your concerns and answer your questions. Thank you once again for your valuable feedback.

Thank you for your thoughtful review and feedback on our submission. We appreciate the time and effort you invested in understanding our work. We would like to address the concerns you raised:

Concern 1: Novelty of our method.

Response:

We would like to address potential misunderstandings about our work and provide clarity on the unique contributions of GasketRAG.

First and foremost, GasketRAG is not merely a combination of refining and rewriting, and we have never claimed that our primary contribution lies in proposing context selection or identification techniques. Instead, our research focuses on a critical challenge: bridging the preference gap between retrievers and large language models (LLMs) and enhancing the performance of off-the-shelf retrievers in collaboration with LLMs.

We understand the perception that the Gasket model is primarily about selecting relevant sentences. However, the crux of the issue lies in how "relevance" is defined and how to fine the optimal “relevance”. Retrievers are typically optimized based on human preferences and are designed to retrieve and rank documents in ways that align with human habits. However, the preferences of LLMs differ from those of humans, leading to a misalignment.

The Gasket model addresses this misalignment not through simple selection or rewriting but by manipulating the content and noise in inputs to both the retriever and the LLM. This manipulation modulates their behavior, effectively bridging the preference gap between the two systems.

Traditional approaches to addressing this challenge often involve jointly training the retriever and LLM using techniques such as PPO and reward models. While effective, these methods significantly increase the complexity and instability of the framework, creating barriers to research and scalability in this area.

Our innovation lies in taking a simpler yet highly effective approach:

1. No reliance on human annotations, which may not adequately address the preference gap.
2. No need for complex joint training frameworks involving all components of the RAG pipeline.
3. Introduction of a straightforward and efficient method for collecting preference data.
4. Utilization of a novel Weighted KTO method to optimize the interaction between the retriever and LLM.

Our experimental results demonstrate that GasketRAG outperforms traditional iterative retrieval, chain-of-thought (CoT) reasoning, and question rewriting methods. Moreover, it offers greater stability across different datasets and LLMs, further solidifying its utility in real-world scenarios.

---

Concern 2: I don't think LLMs and Retrievers can be systematically aligned by only rewriting queries and selecting sentences

Response:

Our experimental results demonstrate the significant effectiveness of the GasketRAG method. As shown in Table 2, GasketRAG (LLaMA-3) outperforms other methods by a large margin on the PopQA, TriviaQA, HotpotQA, and WikiMultiHop test sets, and is very close to the best-performing method on the PubHealth and StrategyQA datasets. Moreover, it exhibits consistent performance on untrained OOD (Out-of-Domain) test sets. We observed that, unlike GasketRAG, other RAG methods perform significantly worse than NaiveRAG on some datasets, highlighting the superior stability of GasketRAG.

It is worth noting that our gasket model was trained using preference-aligned data specific to LLaMA-3. However, even when the generator LLM was replaced with GPT-3.5, GasketRAG still achieved superior performance on multiple datasets, demonstrating the excellent generalization capability of our approach.

Dear reviewer,

I hope this message finds you well.

We wanted to kindly follow up to see if you have had a chance to review our response to your comments. We would greatly appreciate any feedback regarding whether we have adequately addressed the concerns you raised in your review.

Your insights are invaluable to us, and if you have any additional questions or suggestions, we would be more than happy to address them.

Thank you once again for your time and thoughtful consideration.

Best regards,

All Authors

Thank you for your thoughtful review and feedback on our submission. We appreciate the time and effort you invested in understanding our work. We would like to address the concerns you raised:

Concern 1: Preference of the retriever.

Response:

It seems there may have been some misunderstanding of our method. Let us clarify, starting from the definition of preference learning. Preference learning refers to the task of predicting an order relation over a collection of objects.

In the RAG pipeline:

• Learning the LLM's preference involves understanding how to craft prompts that guide the LLM to generate the desired answer.
• Learning the retriever's preference focuses on determining how to enhance queries so that the retriever produces high-quality retrieval results.

Now, let’s return to GasketRAG. In GasketRAG, the LLM's desired answer is well-defined because we have access to the ground truth. However, the retriever’s output lacks explicit labels. Therefore, what constitutes high-quality retrieval results is indirectly labeled based on whether these results enable the LLM to generate the correct answer. Thus, the preferences of the retriever are implicitly learned. During training, the gasket model learns how to enhance queries by observing the impact of modified input of the retiever on the final result (LLM's answer). This process helps the gasket model guide the retriever to retrieve the desired documents. Therefore, this is a result-oriented approach. We do not focus on how the gasket model improves the retriever; we only reward the gasket model when it performs the correct actions (selected the right sentences).

---

Concern 2: Novelty of our method.

Response:

We understand that you believe the gasket model simply selects relevant sentences to help the retriever find more pertinent documents and assist the LLM in generating responses. However, the issue lies in how "relevance" is defined. Retrievers are generally trained based on human preferences, designed to retrieve and rank documents in a way that aligns with human habits. However, the preferences of LLMs do not completely align with those of humans.

As pointed out in [1], when LLMs are presented with both LLM-generated and retrieved documents, they tend to favor content generated by LLMs. This demonstrates subtle differences in how LLMs process contexts from different sources. Furthermore, [2] validates that LLMs are not particularly sensitive to the ranking of documents but are highly sensitive to irrelevant information (noise) within documents. We define the divergence between the retriever's output preferences and the LLM's input preferences as the preference gap.

Thus, the gasket model does more than merely re-ranking. Its underlying logic involves manipulating the content and noise in the inputs to both the retriever and the LLM to modulate their behavior and bridge this preference gap.

Existing methods typically focus on specific components of the RAG pipeline. For instance, some use refinement techniques to help the LLM better understand the content returned by the retriever, while others employ question rewriting to improve the retriever's relevance in retrieving documents. While these approaches have achieved certain improvements, they often overlook the preference gap between the LLM and the retriever. Bridging this gap offers additional potential for enhancement.

The traditional solution to this issue involves jointly training the LLM and the retriever using algorithms such as PPO, with reward models incorporated. However, this significantly increases the complexity and instability of the framework, limiting the scope of research in this area.

The innovation of our approach lies in the fact that it does not rely on human annotations (which, in fact, may fail to align with the preference gap) nor on building a complex joint training framework involving all components of the RAG pipeline. Instead, our method introduces a straightforward and effective way to collect preference data and utilizes a Weighted KTO method. Our experiments demonstrate that GasketRAG outperforms iterative retrieval, chain-of-thought (CoT) reasoning, and question rewriting methods in terms of performance and exhibits greater stability across different datasets and various LLMs.

[1] Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?

[2] Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Bridging the Preference Gap between Retrievers and LLMs.

Concern 3: Simple SFT could also allow for the "gasket" model to identify which textual chunks are relevant for the query.

Response:

Research on SFT and preference training is already well-established [1] [2]. Compared to preference training, SFT represents a more coarse-grained optimization approach. SFT provides an unbiased estimation of the target preference, capturing general trends in labeled data. However, it offers a biased estimation for the model, as it does not account for the model's inherent limitations or specific dynamics.

This limitation means that SFT cannot enable the model to precisely adapt to subtle differences in preferences. In contrast, preference training, by focusing on fine-grained adjustments through direct feedback, allows the model to better capture nuanced variations, leading to improved alignment with desired behaviors in specific scenarios.

We trained the gasket model using SFT under the same configuration and conducted evaluations. The result (correctness) is as follows:

As expected, the gasket model trained with SFT exhibited significant performance degradation. Compared to the base model (LLaMA-3.1-8B-Instruct), the model trained with SFT performed even worse.

[1] Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). Kto: Model alignment as prospect theoretic optimization.

[2] Hua, E., Qi, B., Zhang, K., Yu, Y., Ding, N., Lv, X., ... & Zhou, B. (2024). Intuitive Fine-Tuning: Towards Unifying SFT and RLHF into a Single Process.

---

Concern 4: Why is the "Direct" performance of both models in Table 2 sometimes stronger than the NaiveRAG implementation?

Response:

We were also quite surprised by this result. However, we observed similar outcomes when testing on multiple models, such as GPT-3.5. Additionally, we noticed that similar results were also reflected in [1]. We hypothesize that the possible reason for this phenomenon is that these LLMs were likely exposed to relevant information during pretraining and may even have encountered the test sets (data leakage). As a result, they can answer some questions correctly even without context. However, when NaiveRAG provides retrieved context, a masking effect occurs [2], causing the LLM to rely more on the contextual information rather than its parameter knowledge. Since the context retrieved by the retriever often contains a significant amount of noise, this interferes with the LLM's ability to provide accurate answers.

[1] Zhang, X., Song, Y., Wang, Y., Tang, S., Li, X., Zeng, Z., ... & Wen, Q. (2024). RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation.

[2] Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?

We hope our clarifications address your concerns and answer your questions. Thank you once again for your valuable insights.

Thank you for taking the time to address my concerns.

• I am satisfied with the SFT analysis, thank you for providing it.
• I appreciate the thorough description of the "preference gap" and how your framework addresses it. I believe training a model to optimize what a retriever returns and what is ultimately presented to an LLM based on the LLM's own preferences is an important direction to explore.

However, I believe it is still not ready for acceptance based on 2 issues:

1. The example provided in response to the 6th concern of reviewer m95q makes it seem like the main improvements for GasketRAG come from its iterative retrieval ability, not the "preference gap" that motivates its design. I think more explicit examples of such a gap would be necessary to provide a more convincing story.
2. Similarly, I mentioned before that NaiveRAG's underperformance against direct prompting on 2Wiki was unexpected. However, upon further inspection, your experiments show that direct prompting outperforms all iterative retrieval methods such as Iter-RetGen and Self-Ask even though they have been shown to work in their own papers and others [1][2][3]. Why is it that simpler iterative retrieval methods provide no improvements in your setting? Considering its limited complexity, I think understanding this is a necessary step in truly understanding the benefits of GasketRAG.

I have changed my overall score from 3 to 5 and the Contribution score from 2 to 3.

[1] https://arxiv.org/pdf/2212.10509

[2] https://arxiv.org/pdf/2405.14831

[3] https://arxiv.org/pdf/2410.04343

{
        "Query": "When was the director of film Jinpa born?",
        "Iter-1 sentences ": [
            "[s_14] \"Jinpa Jinpa () is a 2018 Chinese Tibetan-language film written and directed by Pema Tseden.",
            "[s_17] Truck driver Jinpa (Jinpa) accidentally runs over and kills a sheep as he traverses on an isolated road on the Kekexili Plateau.",
            "[s_23] Jinpa Jinpa () is a 2018 Chinese Tibetan-language film written and directed by Pema Tseden."
        ],
        
        "Iter-1 answer": "The director of film Jinpa was born in 1970.",
        "Iter-2 sentences": [
            "[s_1] \"Jinpa Jinpa () is a 2018 Chinese Tibetan-language film written and directed by Pema Tseden.",
            "[s_10] Jinpa Jinpa () is a 2018 Chinese Tibetan-language film written and directed by Pema Tseden.",
            "[s_12] It made its world premiere and won Best Screenplay in the Horizons section at the 75th edition of the Venice Film Festival."
        ],
        "Iter-2 answer": "Pema Tseden was born in 1969.",
        "NaiveRAG answer": "May 25, 1955",
        "Iter-RetGen answer": "1955",
        "Direct answer": "The director of film Jinpa, Pema Tseden, was born in 1970.",
        "true_answer": "1969"
 }

Prompt: "When was Pema Tseden born?"
LLM Response: "1969"

{
        "Query": "Did the movies Passer L'Hiver and In My Skin, originate from the same country?",
        "Iter-1 sentences ": [
            "[s_1] \"Skin (2018 film) Skin is an American biographical drama film written and directed by Guy Nattiv, based on his short film of the same name.",
            "[s_2] It follows the life of former skinhead group member Bryon Widner, and stars Jamie Bell, Vera Farmiga, Danielle Macdonald, Mike Colter, and Bill Camp.",
            "[s_29] In The Skin I Live In () is a 2011 Spanish psychological horror film written and directed by Pedro Almod\u00f3var, starring Antonio Banderas, Elena Anaya, Marisa Paredes, Jan Cornet, and Roberto \u00c1lamo.",
            "[s_46] In The Skin I Live In () is a 2011 Spanish psychological horror film written and directed by Pedro Almod\u00f3var, starring Antonio Banderas, Elena Anaya, Marisa Paredes, Jan Cornet, and Roberto \u00c1lamo."
        ],
        
        "Iter-1 answer": "No, the movies Passer L'Hiver and In My Skin did not originate from the same country.",
        "Iter-2 sentences": [],
        "Iter-2 answer": "Yes, both movies originated from France.",
        "NaiveRAG answer": "The movies Passer L'Hiver and In My Skin did not originate from the same country.",
        "Iter-RetGen answer": "No, the movies Passer L'Hiver and In My Skin did not originate from the same country.",
        "Direct answer": "Yes, both movies originated from France.",
        "true_answer": "yes"
 }

    {
        "query": "Which film has the director who was born earlier, Retalhos Da Vida De Um Médico or The Shooting?",
        "Iter-1 sentences": [
            "[s_1] \"\"\"Fogo na Noite Escura\"\" (1943), at the collection \"\"Novos Prosadores\"\" (1943), by \"\"Coimbra Editora\"\".",
            "[s_2] Besides over 30 titles, along his fifty years of intensive literary life, not only wrote “neo-realistic” novels, as \"\"Casa da Malta\"\" (1945), \"\"Minas de S. Francisco\"\" (1946), \"\"Retalhos da Vida de um Médico\"\" (1949 and 1963), \"\"A Noite e a Madrugada\"\" (1950), \"\"O Trigo e o Joio\"\" (1954), but also “urban themes”, contemporary fiction, as in \"\"O Homem Disfarçado\"\" (1957), \"\"Cidade Solitária\"\" (1959), \"\"Domingo à Tarde\"\" (1961, José Lins do Rego Prize), \"\"Os Clandestinos\"\" (1972), \"\"Resposta a Matilde\"\" (1980) or \"\"O Rio Triste\"\" (1982, Fernando Chinaglia\"",
            "[s_3] Title: \"Autran Dourado\".",
            "[s_4] \"in the Portuguese \"\"sprachraum\"\".",
            "[s_5] In 2001, Brazilian filmmaker Suzana Amaral released the film \"\"Uma Vida em Segredo\"\".",
            "[s_6] It was based on the novel of same title by Autran Dourado."
        ],
        "Iter-1 answer": "The Shooting",
        "Iter-2 sentences": [
            "[s_2] Besides over 30 titles, along his fifty years of intensive literary life, not only wrote “neo-realistic” novels, as \"\"Casa da Malta\"\" (1945), \"\"Minas de S. Francisco\"\" (1946), \"\"Retalhos da Vida de um Médico\"\" (1949 and 1963), \"\"A Noite e a Madrugada\"\" (1950), \"\"O Trigo e o Joio\"\" (1954), but also “urban themes”, contemporary fiction, as in \"\"O Homem Disfarçado\"\" (1957), \"\"Cidade Solitária\"\" (1959), \"\"Domingo à Tarde\"\" (1961, José Lins do Rego Prize), \"\"Os Clandestinos\"\" (1972), \"\"Resposta a Matilde\"\" (1980) or \"\"O Rio Triste\"\" (1982, Fernando Chinaglia\"",
            "[s_1] \"\"\"Fogo na Noite Escura\"\" (1943), at the collection \"\"Novos Prosadores\"\" (1943), by \"\"Coimbra Editora\"\"."
        ],
        "Iter-2 answer": "Retalhos Da Vida De Um Médico",
        "NaiveRAG answer": "The director of The Shooting was born earlier than the director of Retalhos Da Vida De Um Médico.",
        "Iter-RetGen answer": "The Shooting",
        "Direct answer": "The Shooting",
        "true_answer": "Retalhos Da Vida De Um Médico",
        "Explain": "Noise Manipulation"
    },

Indirect Hint 4
Partial Background Hidden 4
Iterative Retrieval 4
Reranking 1
Retrieval Ablation 2
Noise Manipulation 2

    {
        "query": "Which film was released earlier, Moment Of Danger or The Ballad Of Josie?",
        "Iter-1 sentences": [
            "[s_17] \"Moment of Danger Moment of Danger (also known as Malaga) is a 1960 crime drama film starring Trevor Howard, Dorothy Dandridge and Edmund Purdom.",
            "[s_1] \"The Ballad of Josie The Ballad of Josie is a 1967 Technicolor American comedy western film directed by Andrew V. McLaglen and starring Doris Day, Peter Graves and George Kennedy.",
            "[s_10] The Ballad of Josie The Ballad of Josie is a 1967 Technicolor American comedy western film directed by Andrew V. McLaglen and starring Doris Day, Peter Graves and George Kennedy."
        ],
        "Iter-1 answer": "The Ballad of Josie",
        "Iter-2 sentences": [
            "[s_1] \"Moment of Danger Moment of Danger (also known as Malaga) is a 1960 crime drama film starring Trevor Howard, Dorothy Dandridge and Edmund Purdom.",
            "[s_4] The film proved to be the final completed film for Dorothy Dandridge.",
            "[s_5] Starting with a wordless jewel heist pulled-off by thief Peter Curran and locksmith John Bain, Curran then double-crosses his accomplice, dumps his lover Gianna and escapes with his ill-gotten gains.",
            "[s_10] It was filmed in Europe in the late months of 1959.",
            "[s_12] The film proved to be the final completed film for Dorothy Dandridge."
        ],
        "Iter-2 answer": "Moment of Danger",
        "NaiveRAG answer": "The Ballad of Josie was released earlier than Moment of Danger.",
        "Iter-RetGen answer": "The Ballad of Josie",
        "Direct answer": "The Ballad Of Josie was released earlier.",
        "true_answer": "Moment Of Danger",
        "explain": "Partial Background Hidden"
    },

    {
        "query": "Where was the director of film Ronnie Rocket born?",
        "Iter-1 sentences": [
            "[s_0] Title: \"Ronnie Rocket\".",
            "[s_12] \"Ronnie Rocket Ronnie Rocket is an unfinished film project written by David Lynch, who also intended to direct it.",
            "[s_13] Begun after the success of Lynch's 1977 film \"\"Eraserhead\"\", \"\"Ronnie Rocket\"\" was shelved after Lynch felt he would be unable to find financial backing for the project.",
            "[s_34] After releasing 1977's \"\"Eraserhead\"\", a black-and-white surrealist film and his début feature-length production, Lynch began work on the screenplay for \"\"Ronnie Rocket\"\".",
            "[s_30] The project has also suffered setbacks due to the bankruptcy of several potential backers; both Dino De Laurentiis's De Laurentiis Entertainment Group and Francis Ford Coppola's American Zoetrope were attached to the project at different times; both production companies went bankrupt before work could begin."
        ],
        "Iter-1 answer": "David Lynch",
        "Iter-2 sentences": [
            "[s_1] \"Ronnie Rocket Ronnie Rocket is an unfinished film project written by David Lynch, who also intended to direct it.",
            "[s_49] Title: \"Early life of David Lynch\"."
        ],
        "Iter-2 answer": "David Lynch was born in Missoula, Montana.",
        "NaiveRAG answer": "Ronnie Rocket was born in Bismarck, North Dakota, United States.",
        "Iter-RetGen answer": "Bismarck, North Dakota",
        "Direct answer": "The director of film Ronnie Rocket was born in New York City, New York, United States.",
        "true_answer": "Missoula, Montana",
        "explain": "Indirect Hint"
    },

Thank you for your thoughtful review and feedback on our submission. We appreciate the time and effort you invested in understanding our work. We would like to address the concerns you raised:

Concern 1: Motivation of GasketRAG and how to understand the preferences of the retriever and the LLM.

Response:

Retrievers are generally trained based on human preferences, designed to retrieve and rank documents in a way that aligns with human habits. However, the preferences of LLMs do not completely align with those of humans. As pointed out in [1], when LLMs are presented with both LLM-generated and retrieved documents, they tend to favor content generated by LLMs. This demonstrates subtle differences in how LLMs process contexts from different sources. Furthermore, [2] validates that LLMs are not particularly sensitive to the ranking of documents but are highly sensitive to irrelevant information (noise) within documents. We define the divergence between the retriever's output preferences and the LLM's input preferences as the preference gap. This means that the direct output of the retriever is not ideal as input for the LLM. Therefore, we establish a gasket model through preference training to filter information from the retrieved documents, aligning more closely with the LLM's input preferences to help it generate accurate responses. At the same time, the output of the gasket model is also used to enhance the query fed to the retriever, thereby adjusting the retriever's behavior and improving its retrieval results.

To give a straightforward example, for certain queries, an LLM might possess parameter knowledge and can provide answers without requiring additional context. In such cases, the content retrieved by the retriever might have a negative impact. The gasket model learns these preferences from training data and tends to avoid selecting any sentence from the retrieved passages to reduce interference when it believes the LLM can answer the question without assistance. Compared to the preferences of the LLM, the preferences of the retriever are actually easier to understand. The original question is not always correctly interpreted by the retriever, so the retrieval results are often suboptimal. Therefore, the gasket model also needs to enhance the input to the retriever to help it generate LLM preferred retrieval results.

The explanation above is provided for a more intuitive understanding; in reality, the input-output dynamics of retrievers and LLMs are far more complex. This complexity makes the preference gap between retrievers and LLMs resemble a black box. By designing methods to collect preference data, we enable the gasket model to learn the collaborative patterns between the retriever and the LLM, thereby understanding their preferences. Acting as a coordinator, the gasket model maximizes the performance of the entire RAG system.

[1] Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?

[2] Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Bridging the Preference Gap between Retrievers and LLMs.

---

Concern 2: Why retrieving the documents again with a combination of query and retrieved sentences could necessarily “control” the behavior of the retriever?

Response:

During training, the gasket model learns how to enhance queries by observing the impact of modified input of the retiever on the final result (LLM's answer). This process helps the gasket model guide the retriever to retrieve the desired documents. Therefore, this is a result-oriented approach. We do not focus on how the gasket model improves the retriever; we only reward the gasket model when it performs the correct actions (selected the right sentences). Thus, the preferences of the retriever are implicitly learned.

---

Concern 3: As shown in Table 4, more iterations could even harm the performance.

Response:

Although the gasket model partially bridges the gap between the retriever and the LLM, it is not perfect. During its sentence selection process, it may still introduce irrelevant information, which we refer to as noise. The introduction of noise can potentially shift the retriever's retrieval direction, and this shift may be amplified with more iterations. Therefore, more iterations do not necessarily lead to better results. Similar phenomena can also be observed in other iterative RAG methods.

Concern 4: Why the preference data could align the retriever and the LLM?

Response:

We understand that our explanation of preference training might seem complex, so here’s a simpler way to explain it:

First, let's consider only the first iteration of GasketRAG. This represents a straightforward RAG structure consisting of a retriever, a gasket (acting as a refiner), and an LLM. By applying preference training to the gasket, we align the entire pipeline with the LLM's accuracy in generating the final answer. However, this approach has limitations because the retriever's input and output remain unchanged.

To address this, we aim to adjust the retriever's input and output as well. Instead of introducing an additional model, we continue using the gasket model to enhance the retriever's input query. Consequently, the gasket model's output is constrained by two objectives:

1. Ensuring the LLM produces accurate answers.
2. Guiding the retriever to retrieve desired documents.

The question now becomes how to define "more desired" retrieval results for the retriever. We approach this in two scenarios:

1. First-Round Incorrect Answer, Second-Round Correct Answer

   If the sentence list selected by the gasket model in the first round led the LLM to generate an incorrect answer, but after a second round of retrieval and sentence selection, the LLM produces the correct answer, we consider the retrieval in the second round as "more desired." This means the first-round output of GasketRAG effectively guided the retriever to improve its results.

2. First-Round Correct Answer

   If the LLM already generated the correct answer in the first round, it will also produce a correct answer in the second round. In this case, we evaluate the average index of the sentences selected by the gasket model in the two rounds. If the average index decreases compared to the first round, it indicates that relevant information has been moved closer to the top of the retrieval results. Such a retrieval is also considered "more desired."

Using these criteria, we label the sentence lists produced by the gasket model as strong preferred, weak preferred, strong dispreferred or weak dispreferred for training. This labeling allows the gasket model to learn and optimize towards achieving the two objectives.

---

Concern 5: The comparison between GasketRAG and baseline methods.

Response:

Our experimental results demonstrate the significant effectiveness of the GasketRAG method. As shown in Table 2, GasketRAG (LLaMA-3) outperforms other methods by a large margin on the PopQA, TriviaQA, HotpotQA, and WikiMultiHop test sets, and is very close to the best-performing method on the PubHealth and StrategyQA datasets. Moreover, it exhibits consistent performance on untrained OOD (Out-of-Domain) test sets. We observed that, unlike GasketRAG, other RAG methods perform significantly worse than NaiveRAG on some datasets, highlighting the superior stability of GasketRAG.

It is worth noting that our gasket model was trained using preference-aligned data specific to LLaMA-3. However, even when the generator LLM was replaced with GPT-3.5, GasketRAG still achieved superior performance on multiple datasets, demonstrating the excellent generalization capability of our approach.

Our primary evaluation metric is Correctness. Given the free-form nature of LLM outputs, correctness as judged by GPT-4 based on semantic understanding is more convincing compared to accuracy (ACC).

Concern 6: Analyses of the sentences selected.

Response:

Below, we present an example of GasketRAG inference. In the first round of retrieval, the retriever, based on the original query, identified that Millett is the author of Sexual Politics. However, the returned documents lacked information that could truly help answer the question. As a result, the sentences selected by the gasket model could not enable the LLM to produce the correct answer.

However, these selected sentences helped the retriever locate a critical fragment, "Millett attended Oxford University," during the second round of retrieval. Additionally, we observe that in the first round, the selected sentences had indices s_8, s_10, and s_27, while in the second round, they were s_1, s_2, s_3, and s_10. The sentences selected in the second round clearly had lower indices, indicating that the edits made by the gasket model to the query effectively surfaced relevant information in the retrieval results.

{
        "Query": "The author of Sexual Politics attended which British University?",
        "Iter-1 sentences ": [
            "[s_8] \"Kate Millett Katherine Murray Millett (September 14, 1934 \u2013 September 6, 2017) was an American feminist writer, educator, artist, and activist.",
            "[s_10] She has been described as \"\"a seminal influence on second-wave feminism\"\", and is best known for her book \"\"Sexual Politics\"\" (1970), which was based on her doctoral dissertation at Columbia University.",
            "[s_27] \"\"Sexual Politics\"\" originated as Millett's PhD dissertation and was published in 1970, the same year that she was awarded her doctorate from Columbia University."
        ],
        
        "Iter-1 answer": "Columbia University",
        "Iter-2 sentences": [
            "[s_1] \"Kate   Katherine Murray Millett (September 14, 1934 \u2013 September 6, 2017) was an American feminist writer, educator, artist, and activist.",
            "[s_2] She attended Oxford University and was the first American woman to be awarded a degree with first-class honors after studying at St Hilda's College, Oxford.",
            "[s_3] She has been described as \"\"a seminal influence on second-wave feminism\"\", and is best known for her book \"\"Sexual Politics\"\" (1970), which was based on her doctoral dissertation at Columbia University.",
            "[s_10] She attended Oxford University and was the first American woman to be awarded a degree with first-class honors after studying at St Hilda's College, Oxford."
        ],
        "Iter-2 answer": "Oxford University",
        "true_answer": "Oxford"
 }

---

Concern 7: Typos and presentations.

Response:

Thank you very much for your thorough review of our paper. We will promptly correct the typos you pointed out and refine our phrasing.

We hope our clarifications address your concerns and answer your questions. Thank you once again for your valuable insights.

Dear reviewer,

I hope this message finds you well.

We wanted to kindly follow up to see if you have had a chance to review our response to your comments. We would greatly appreciate any feedback regarding whether we have adequately addressed the concerns you raised in your review.

Your insights are invaluable to us, and if you have any additional questions or suggestions, we would be more than happy to address them.

Thank you once again for your time and thoughtful consideration.

Best regards,

All Authors

Dear Reviewer,

Since we are approaching the deadline of the discussion period, we were wondering if you have had the chance to review our response to your comments. We also provided more details and case studies which may also address your concerns in our response to Reviewer kEed.

We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have.

Thank you for your time and consideration.

Sincerely,

All Authors

Thank you very much for your exceptionally thorough reading and understanding of our paper. We believe your feedback demonstrates a deep knowledge of the field, and we are more than happy to engage in a discussion on the issues you are concerned about.

Concern 1: Different gasket model architectures and resource usage.

Response:

We additionally trained two gasket models based on Qwen-2.5: a 0.5B and a 1.5B model. These were compared with our gasket model based on LlaMA-3.1. The LLM generator is LLaMA-3-8B. The following are the test results (correctness):

From the table, it can be observed that as the model size increases, the performance also shows improvement. The 0.5B model has performance limitations, indicating that regulating the retriever and LLM within the RAG pipeline requires the gasket model to possess a considerable level of text understanding. Additionally, it is evident that the 1.5B model demonstrates quite strong performance, slightly lagging behind the 8B model across datasets but still outperforming other RAG methods. This indicates that GasketRAG is more efficient compared to other RAG methods. By using a smaller model as the gasket model to filter sentences, the token sequence length input to the LLM generator is significantly reduced. As a result, despite requiring two iterations, GasketRAG has minimal resource demands.

---

Concern 2: Have the authors experimented different iterations mode of GasketRAG?

Response:

We may not have fully understood the meaning of different iterations mode in this question. In Table 4, we present the performance of GasketRAG with varying numbers of iterations. A single iteration means that the gasket model acts merely as a refiner without adjusting the retriever. We observe that increasing the number of iterations to 3 results in a decline in stability compared to 2 iterations due to the accumulation of errors. Moreover, additional iterations exponentially increase resource consumption. Therefore, we consider two iterations to be the optimal choice, which aligns with our expectations.

We also explored a variant of GasketRAG, where in each iteration, the sentence list output by the gasket model is passed to the LLM generator to produce a preliminary answer. This preliminary answer is then concatenated to the query and used as input for the next iteration's retriever. The test results (correctness) are as follows:

Using the answer to enhance the query in each iteration did not result in significant improvement. Instead, it increased the number of LLM generator calls, thereby consuming more resources. Considering that many RAG methods use draft answers to significantly improve the retriever's retrieval quality, GasketRAG can achieve similar effects efficiently through the gasket model, making it a more resource-efficient approach.

Concern 3: The impact of preference dataset size.

Response:

We used the same experimental setup and retrained the gasket model with half the preference data (8.5K). The results are as follows:

The performance gap between the gasket model trained with 8.5K data and the one trained with 17K data is minimal, demonstrating the high data efficiency of our proposed preference data collection method and the Weighted KTO training algorithm. GasketRAG requires only a small amount of preference data to achieve a high level of performance. Since GasketRAG eliminates the need for pair-wise data, it does not require meticulously crafting large data combinations for contrastive learning.

---

Concern 4: Further study.

Response:

• Other retrievers

  We are currently replacing the ColBERTv2 retriever with others such as Contriever and replicate the experiments. It involves collecting new preference data, retraining the gasket model, and running evaluations. Due to computational resource limitations, this may take some time. We will include the results in the revised version once the process is complete.

• Preference data biases

  We are not entirely sure which aspect of bias you are referring to. So far, we have not specifically explored this area. We hope you can provide further insights or suggestions that might guide us in this direction.

We hope our clarifications address your concerns and answer your questions. If there are any inaccuracies in our response, please feel free to point out. Thank you once again for your valuable feedback.

Concern 3: Using a lightweight BERT ranking model.

Response:

You’ve brought up an excellent perspective. First, we should point out that training BERT using preference data is actually more complex because BERT is an encoder-only model. To perform sentence-level ranking, it requires constructing a pair-wise training dataset for contrastive learning.

The advantage of our method lies in the simplicity of collecting preference data. It only requires labeling each record as preferred or dispreferred (with strong/weak weights), without needing pairwise comparisons or considering their distribution in the embedding space. This makes our approach more data-efficient and stable.

Additionally, as previously mentioned, LLMs are not particularly sensitive to document ranking. Therefore, what the gasket model does is fundamentally different from re-ranking. Instead of simply reordering, it edits the inputs to both the retriever and the LLM, enabling a more systematic alignment.

For efficiency concerns, we trained additional lightweight gasket models based on Qwen2.5 0.5B and 1.5B. Following is the correctness of the models tested on various datasets:

From the table, it can be observed that the 0.5B model has performance limitations, indicating that regulating the retriever and LLM within the RAG pipeline requires the gasket model to possess a considerable level of text understanding. Additionally, it is evident that the 1.5B model demonstrates quite strong performance, slightly lagging behind the 8B model across datasets but still outperforming other RAG methods. This indicates that GasketRAG is more efficient compared to other RAG methods. By using a smaller model as the gasket model to filter sentences, the token sequence length input to the LLM generator is significantly reduced. As a result, despite requiring two iterations, GasketRAG has minimal resource demands.

---

Concern 4: How is the sentence index linked to retrieval effectiveness?

Response:

In the first iteration, the sentences selected by the gasket model are used to strengthen the query, influencing the retriever’s tendency in the second round of retrieval. We hypothesize that if the useful information in the retriever's results is ranked closer to the top in the second round, it indicates that the sentences selected by the gasket model are of higher quality. Consequently, if the average index of the sentences chosen by the gasket model in the second round decreases, it implies that useful information is being prioritized in the retrieval results. This brings two benefits:

1. Preventing Cheating through Redundant Selection: By ensuring that the gasket model selects relevant sentences while avoiding irrelevant ones, it prevents the model from "cheating" by indiscriminately copying large portions of retrieved documents. This improves its ability to distinguish between golden sentences (highly relevant) and silver sentences (less relevant).
2. Promoting Discovery of Potentially Useful Documents: The overall forward shift of useful information implies that documents that were previously not included in the retrieval results, but could be relevant, are now promoted to appear in the second round's results.

To illustrate this process more concretely, consider the following example: Suppose all retrieved documents are concatenated as follows:

[1] Title: Passage 1. [2] Sentence... [3] Title: Passage 2. [4] Sentence... [5] Sentence...

After the first round of retrieval, the gasket model selects sentences [2] and [4]. We calculate their average index as (2+4)/2=3. Using these sentences to enhance the query for the second round of retrieval, a new batch of documents is retrieved. As before, sentences are renumbered independently of the first round. Suppose the gasket model then selects a new set of sentences [1], [2], and [4]. We calculate the new average index as: (1 + 2 + 4)/3=2.33. The decrease in the average index indicates that the retriever has successfully prioritized relevant information in the second round.

We hope our clarifications address your concerns and answer your questions. Thank you once again for your valuable insights.

Thank you for your thoughtful review and feedback on our submission. We appreciate the time and effort you invested in understanding our work. We would like to address the concerns you raised:

Concern 1: Alignment mechanism in GasketRAG and preference definition.

Response:

We understand that you believe the gasket model simply selects relevant sentences to help the retriever find more pertinent documents and assist the LLM in generating responses. However, the issue lies in how "relevance" is defined. Retrievers are generally trained based on human preferences, designed to retrieve and rank documents in a way that aligns with human habits. However, the preferences of LLMs do not completely align with those of humans.

As pointed out in [1], when LLMs are presented with both LLM-generated and retrieved documents, they tend to favor content generated by LLMs. This demonstrates subtle differences in how LLMs process contexts from different sources. Furthermore, [2] validates that LLMs are not particularly sensitive to the ranking of documents but are highly sensitive to irrelevant information (noise) within documents. We define the divergence between the retriever's output preferences and the LLM's input preferences as the preference gap.

Thus, the gasket model does more than merely select relevant sentences. Its underlying logic involves manipulating the content and noise in the inputs to both the retriever and the LLM to modulate their behavior and bridge this preference gap. For example, for certain queries, an LLM might possess parameter knowledge and can provide answers without requiring additional context. In such cases, the content retrieved by the retriever might have a negative impact. The gasket model learns these preferences from training data and tends to avoid selecting any sentence from the retrieved passages to reduce interference when it believes the LLM is able to answer without assistance.

Of course, the preferences of LLMs and retrievers regarding input content are complex and are treated as a black box in our approach. By focusing on the final answer as the goal, the gasket model coordinates all components within the RAG pipeline. This coordination involves simultaneously controlling the inputs to both the retriever and the LLM. We refer to this process as systematic alignment.

[1] Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?

[2] Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Bridging the Preference Gap between Retrievers and LLMs.

---

Concern 2: Novelty of our method.

Response:

Existing methods typically focus on specific components of the RAG pipeline. For instance, some use refinement techniques to help the LLM better understand the content returned by the retriever, while others employ question rewriting to improve the retriever's relevance in retrieving documents. While these approaches have achieved certain improvements, they often overlook the preference gap between the LLM and the retriever. Bridging this gap offers additional potential for enhancement.

The traditional solution to this issue involves jointly training the LLM and the retriever using algorithms such as PPO, with reward models incorporated. However, this significantly increases the complexity and instability of the framework, limiting the scope of research in this area.

The innovation of our approach lies in the fact that it does not rely on human annotations (which, in fact, may fail to align with the preference gap) nor on building a complex joint training framework involving all components of the RAG pipeline. Instead, our method introduces a straightforward and effective way to collect preference data and utilizes a Weighted KTO method. Our experiments demonstrate that GasketRAG outperforms iterative retrieval, chain-of-thought (CoT) reasoning, and question rewriting methods in terms of performance and exhibits greater stability across different datasets and various LLMs.

Dear reviewer,

I hope this message finds you well.

We wanted to kindly follow up to see if you have had a chance to review our response to your comments. We would greatly appreciate any feedback regarding whether we have adequately addressed the concerns you raised in your review.

Your insights are invaluable to us, and if you have any additional questions or suggestions, we would be more than happy to address them.

Thank you once again for your time and thoughtful consideration.

Best regards,

All Authors