UncertaintyRAG: Span Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation

Question 2: The evaluation tasks should include single-doc tasks such as NQ and Qasper. To still keep the distribution shifts, you can move these two tasks from training set to the evaluation set, and add the three few-shot learning tasks into the training data.

Response 2:

Thank you for your valuable suggestions. Based on your advice, we have reselected the training and test datasets. In the new experiments, the training set includes MultiFieldQA, Musique, QMSum, TREC, and TriviaQA, while the test set consists of NarrativeQA, Qasper, SAMSum, and 2WikiMultihopQA. The detailed experimental results are presented below:

All experiments were conducted on LLaMA-2-7B-Chat-HF. As observed, our method continues to achieve the best performance on the new datasets. This demonstrates the robustness of our method, as it does not rely on the specific selection of datasets.

Question 3:

There should be some comparisons with more baseline RAG methods, such as Self-RAG, FLARE, and RankRAG.

Response 3:

Thank you very much for your suggestion. However, we would like to first point out that our method is orthogonal to the RAG methods you mentioned. Specifically, using self-RAG or similar RAG approaches does not conflict with employing our trained retrieval model. Nonetheless, we greatly appreciate your reminder, and we have supplemented our experiments with results showing the performance of our method when combined with self-RAG.

1):In this experiment, we used the official release model self-RAG_llama2_7b for testing. It is likely that the performance of the LLaMA-2-7B model is somewhat inferior to that of LLaMA-2-7B-Chat-HF, leading to a decline in overall experimental results compared to those obtained with LLaMA-2-7B-Chat-HF. Nonetheless, we were pleasantly surprised to find that our method still outperforms other baselines.

2):Specifically, the prompt used in this experiment is as follows:

def format_prompt(input, paragraph=None):
  prompt = "### Instruction:\n{0}\n\n### Response:\n".format(input)
  if paragraph is not None:
    prompt += "[Retrieval]<paragraph>{0}</paragraph>".format(paragraph)
  return prompt

prompt = format_prompt(f"Answer the question based on the given passages.\nOnly give me the answer and do not output any other words.\n\nQuestion: {data[i]['input']}", prompt)

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.

Dear Reviewer jHwd,

Thank you very much for your comment, we will address your concerns below.

Question 1:
For measuring the span uncertainty, this work only considered one model (i.e., Llama-2-7b-chat-hf). As the proposed method is theoretically model-free, there should be experiments to demonstrate that (1) UncertaintyRAG performs well with various models to measure the uncertainty, or (2) various models can achieve an agreement on the uncertainty measurement.

Response 1: Thank you for your valuable suggestions regarding our experiments. Following your advice, we conducted additional experiments using Vicuna-7B to measure uncertainty and performed RAG experiments on Vicuna-7B. The results are presented in the table below.

1): We observe that UncertaintyRAG performs effectively in measuring uncertainty when used with other models.

2): Furthermore, we conduct RAG experiments with the model using a retrieval model trained on uncertainty measurements obtained from Vicuna-7B with LLaMA-2-7B-Chat-HF, and the results are presented in the table above.

3): We used different models (including Vicuna-7B) to estimate uncertainty, and observed consistent improvements in RAG experiments conducted on different models (Vicuna-7B, LLaMA-2-7B-Chat-HF), demonstrating that various models can achieve agreement in uncertainty measurement.

Dear Reviewer jHwd,

In our most recent response, we addressed the concerns you raised previously. However, it is possible that for various reasons, our response may not have reached you. If you find that our previous answers have resolved your concerns, would you kindly consider raising your score? We greatly appreciate your support.

Thank you very much.

Best regards,

The Authors

Weakness 4 One missing simple baseline for a proposed method is to produce labeling for semantic similarity between two text chunks by simply prompting an LLM for this task. Furthermore, the paper does not cite nor compare to a very relevant body of work on using LLMs for reranking, e.g. [2] and [3].

Response to weakness 4: Thank you for your suggestion. However, on one hand, using LLM models for re-ranking involves significantly higher computational costs compared to using retrievers. On the other hand, we have conducted some experiments and found that when dealing with extremely long texts, which need to be divided into numerous chunks, directly employing LLMs for re-ranking through prompts does not achieve satisfactory results. Below, we will present the experimental results of RankGPT along with the corresponding code.

Code

for i in range(len(data)):
    d_doc = text_splitter.split_text(data[i]['context'])
    hits = [{'content':d} for d in d_doc]
    item = {'query':data[i]['input'], 'hits':hits}
    new_item = permutation_pipeline(item, rank_start=0, rank_end=len(d_doc)-1, model_name=model_name, llm=llm)
    prompt = "\n".join([h['content'] for h in new_item['hits'][:30]])

    prompt = f"Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{prompt}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {data[i]['input']}\nAnswer:",

    completions = llm.generate(prompt, sampling_params)
    outputs.append(completions[0].outputs[0].text)

The permutation_pipeline function is based on the official code from RankGPT, available at https://github.com/sunnweiwei/RankGPT/blob/main/rank_gpt.py. We modified the run_llm function within it to include an option for using open-source models.

def run_llm(messages, api_key=None, model_name="gpt-3.5-turbo", llm=None):
    if 'gpt' in model_name:
        Client = OpenaiClient
    elif 'claude' in model_name:
        Client = ClaudeClient
    elif 'llama' in model_name or 'vicuna' in model_name:
        messages = "\n".join([f"{entry['role'].capitalize()}: {entry['content']}" for entry in messages])
        from vllm import LLM, SamplingParams

        stop_tokens = ["Question:", "Question", "USER:", "USER", "ASSISTANT:", "ASSISTANT", "Instruction:", "Instruction", "Response:", "Response"]
        sampling_params = SamplingParams(temperature=0.0, top_p=1, max_tokens=512, stop=stop_tokens)
        response = llm.generate(messages, sampling_params)[0].outputs[0].text
        return response
    else:
        Client = LitellmClient

    agent = Client(api_key)
    response = agent.chat(model=model_name, messages=messages, temperature=0, return_text=True)
    return response

Weakness 5 The method is tested on five datasets: 2WikiMultihopQA, Musique, TREC, TriviaQA, and SAMSum. The paper does not provide any motivation for this selection. At the same time, in Information retrieval there are well established benchmarks for testing retrievers in domain shift settings, e.g. BEIR [4], which should be used for corresponding evaluations.

Response to weakness 5: Thank you for the insightful question. In fact, we carefully considered the diversity of both the training and test datasets when making our selections. Specifically:

2WikiMultihopQA and Musique are tasks that require answering questions based on multiple provided documents.

TREC is a classification task that involves categorizing questions into 50 predefined categories.

TriviaQA is a single-document question-answering task with several few-shot examples provided.

SAMSum is a dialogue summarization task, also including few-shot examples.

To address your concerns, we have reselected the training and test datasets. In the new experiments, the training set includes MultiFieldQA, Musique, QMSum, TREC, and TriviaQA, while the test set consists of NarrativeQA, Qasper, SAMSum, and 2WikiMultihopQA. The detailed experimental results are presented below.

All experiments were conducted on LLaMA-2-7B-Chat-HF. As observed, our method continues to achieve the best performance on the new datasets. This demonstrates the robustness of our method, as it does not rely on the specific selection of datasets.

Weakness 6Experimental settings are given with very little detail, e.g. I was not able to find which metric is reported in the main Table 1, or what is the architecture of the proposed model (or what pretrained model it was initialised from). Other missing details include evaluation dataset sizes, how retrieval datastores were constructed, how many passages were retrieved per query, or which decoding strategy was used.

Response to weakness 6: Thanks for your suggestion! Regarding the metrics, we followed the choices made by LongBench [7]. Specifically, for 2WikiMultihopQA, Musique, and TriviaQA, we used the F1-score, for TREC, we used accuracy, and for SAMSum we used the ROUGE score. We initialize the model and configure additional settings based on EPR[8]. We will add more detailed explanations to the paper to reduce any potential confusion for readers. Once again, thank you for your reminder!

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.

Lines 293-294: “We set up two windows within M, each containing m samples. Among them, the top m are positive samples, and the bottom m are negative samples.” -> How exactly are these windows of m samples set up? Furthermore, is the “top m” computed over bm 25 scores or S^ij scores?

Response: (1) Thank you for your valuable feedback, here we followed the settings from previous works[8][9].

(2) It’s computed over S^ij scores. To clarify, we have revised the statement as follows: "We set up two windows within M, each containing m samples. These samples are ranked by S^ij scores, with the top m considered positive samples and the bottom m considered negative samples."

Regarding describing the methodology (Section 3), I would suggest showing all the steps on a particular text example. Response: Thank you for the insightful suggestion. the detailed process is presented in our response to Question 2.

Weakness 2The motivation for the proposed unsupervised retriever is unclear. The authors argue that “the lack of labeled data to determine if (query, chunk) pairs are related poses significant limitations for training retrieval models in RAG systems.” (lines 62-64). However, there is plenty of widely used high-quality data nowadays for training retrievers, e.g. MS Marco dataset [1], and also of various QA datasets, which makes unclear the motivation for developing a retriever which does not use any queries during training. This is also unclear to me how a retriever trained without any query data could work in practice; an additional analysis of query representations would be helpful to explain it.

Response to weakness 2: Thank you for the question. In fact, many influential works, such as Contriever[4], BGE-M3[5], and Dragon[6], are attempting to utilize unlabeled data. We have specifically addressed this point in detail in our response to Weakness 1 regarding line 56.

Weakness 3The proposed method brings only small and inconsistent improvements over baselines. This can be seen from Table 1. Furthermore, some rows in this Table highlight “Ours” in bold while it is not the highest score, e.g. for Vicuna-7B + TriviaQA and SamSum, or LLaMA-2-13B-Chat-HF and SamSum.

Response to weakness 3: (1) Compared to other baselines, our approach achieves better performance using significantly less data, and we also demonstrate the potential for scaling up. This indicates that with more computational resources and data, our method has the potential to achieve even greater improvements.

(2) Additionally, we would like to point out that even when comparing two baselines (e.g., Contriever and GRAGON-PLUS), the performance differences vary across different LLMs. For instance, on LLaMA-2-7B-Chat-HF, GRAGON-PLUS outperforms Contriever by an average of 2.36%; on LLaMA-2-13B-Chat-HF, this advantage decreases to 0.78%; and on Vicuna-7B, it further drops to 0.39%. This suggests that the gains achieved by retrieval models differ across LLMs. Moreover, the improvement of our model over the baselines is already substantial compared to the differences between the baselines themselves.

(3) Thank you very much for your reminder! We will correct the mistakenly highlighted portions in the table.

Line 61: “Modern RAG systems rely on complex chunking methods (Sarthi et al., 2024)” -> this is an incorrect claim, most of modern RAG systems use the same simple strategy of chunking into the fixed number of tokens or sentences, as is done in this submission. And Lines 64-65: “Recent research combines RAG with long-context LLMs to handle extended contexts and mitigate semantic incoherence in chunk processing” -> I did not understand what “semantic incoherence in chunk processing” means.

Response: (1) In fact, some recent studies have shown that using the simple strategy of chunking can disrupt the coherence of the context, thereby affecting the quality of the embeddings [7]. We also revise the presentation: Modern RAG systems may rely on complex chunking methods (Sarthi et al., 2024)

(2) Semantic incoherence in chunk processing refers to a situation where a coherent segment of information is fragmented into different chunks before its meaning is fully conveyed. For example, consider a Chain of Thought for solving a mathematical problem: if the argument is interrupted and divided into separate chunks during the explanation, it naturally leads to semantic incoherence.

Line 71: “Additionally, the complexity of these methods makes them vulnerable to distribution shifts” -> I did not understand how these two concepts are connected.

Response: As mentioned earlier, some methods require fine-tuning adapters for specific large language models (LLMs) to manage chunk representation compression. In such cases, these methods are naturally susceptible to changes in the distribution of the LLMs or the data.

Lines 77-78: “we introduce a novel uncertainty estimation technique based on the Signal-to-Noise Ratio (SNR), which stabilizes predictions and reduces biases from random chunk splitting.” -> I did not understand what “stabilizes predictions” means (and why they are “unstable” in the first place), and what are the “biases from random chunk splitting”.

Response: (1)To help you better understand, we have uploaded an anonymous code file that provides a detailed example: https://ufile.io/56d9jxyh. It can be observed that the self-information of the initial tokens is very high, indicating instability in the model's predictions. By using the SNR-based calibration method, we can effectively identify the point where the self-information stabilizes at a relatively low level. Subsequent experiments further demonstrate that using the self-information of these tokens to estimate the similarity between two chunks achieves better performance.

(2) “biases from random chunk splitting” refers to the phenomenon of semantic incoherence caused by random splitting.

Related Work:

Line 131: “However, considering that many LLMs are closed-source, fine-tuning these models specifically for a retrieval model is impractical” -> (1) I did not understand what “fine-tuning these models specifically for a retrieval model” means. (2) there are a lot of open source LLMs available nowadays.

Response: (1) Thank you very much for your suggestion. We have revised the sentence to: “However, considering that many LLMs are closed-source, end-to-end training of LLMs with retrievers is impractical.”

(2) On one hand, closed-source models cannot be trained end-to-end, and many of them do not provide access to logits. On the other hand, while there are many open-source models available, we aim to make our model adaptable to all types of models. Methodology:

Line 190: “In this section, we introduce a span uncertainty method based on SNR to obtain confidence scores between chunks” -> what does “confidence scores between chunks” mean?

Response: Thank you for the feedback. The expression should be revised to accurately reflect 'similarity between chunks' instead.

Line 244: “Additionally, we have observed a specific window in the model’s output log probabilities. The probability distribution within this window is stable, which we believe indicates better calibration” -> a “specific window” where? in the vocabulary dimension, or in the text dimension? what does it mean “the probability distribution is stable”? Why does it indicate better calibration?

Response: (1) In the text dimension, the detailed process is presented in our response to Question 2. (2) For further details, please refer to the response provided to Lines 77-78. What’s more, our subsequent experiments consistently demonstrate the effectiveness of our method.

Weakness

Weakness 1 Text needs substantial rewriting: many parts of the text are unclear and hard to understand. Below I give examples of argumentations which are unclear or incorrect. I was only able to understand the proposed method after reading the corresponding section several times. In addition, the figures in the text are too small.

Response to weakness 1: Thank you very much for your suggestions regarding the writing details in our paper. We will address the issues you mentioned one by one in the following responses.

Introduction: Line 44: “these methods are generally difficult to achieve context length extrapolation.” -> an unclear phrase “methods are difficult to achieve”, and an unclear sentence why KV cache compression methods are poor in context length extrapolation.

Response: Thank you for the suggestion. In fact, the KV cache compression method is a way to reduce memory cost, but not related to the model length extrapolation performance. And usually, the context length extrapolation is usually poor, based on previous works[1][2][3].

Line 50: “Another lightweight solution for handling long contexts is to utilize long-context Retrieval-Augmented Generation (RAG) for long-context chunking”. -> long-context RAG is used together with long-context chunking, not for it.

Response: Thank you for the suggestion. Here's the revised version of the sentence based on the feedback: "Another lightweight solution for handling long contexts is to utilize long-context Retrieval-Augmented Generation (RAG) in conjunction with long-context chunking."

Lines 52-55: “Long context Retrieval-Augmented Generation refers to a method in natural language processing where a model retrieves relevant information from large external sources to assist in generating responses. It extends the traditional RAG by handling much longer input contexts.” -> The definition for Long context RAG (sentence 1) seems to be describing the general RAG setting. Sentence 2 is unclear: handling longer input contexts in the generator LLM or in the “ large external sources”?

Response: Thank you very mush for the feedback. Here's the revised version of the paragraph to address the concerns: "Long-context Retrieval-Augmented Generation (RAG) refers to a method in natural language processing that addresses the long-context problem by retrieving relevant information from large external sources. This process helps LLMs effectively access more useful information within their limited context window, enabling them to handle much longer input contexts indirectly."

Line 56: “ Typically, RAG employs existing LLMs with limited context windows to retrieve relevant chunks, either with or without semantic truncation. Retrieval models are commonly used for this purpose; however, they require a large amount of high-quality labeled data for training, which limits their scalability and adaptability. ” -> RAG does not employ LLMs to retrieve relevant chunks, this is done by the retriever. Term “semantic truncation” is not introduced nor explained. High-quality labeled data for training retrievers is available, e.g. the widely used MS Marco dataset [3].

Response: (1) Thank you very much for your notice, we revised the presentation to "Typically, RAG helps existing LLMs with limited context windows process retrieved relevant chunks. These chunks are obtained by retrieval models, either with or without addressing semantic truncation, which refers to the phenomenon where splitting chunks results in semantic discontinuity between adjacent chunks."

(2) While it is true that high-quality labeled datasets like MS MARCO are available and widely used, it is also important to note that the vast majority of textual data remains unlabeled. Recent works, such as Contriever[4], BGE-M3[5], Dragon[6] and other methods trained on unlabeled data, have demonstrated significant progress in leveraging these abundant resources. These efforts aim to utilize the vast scale of unlabeled data effectively, as they far outweigh the availability of labeled datasets. This growing focus on unsupervised and self-supervised learning highlights the potential of using unlabeled data for training retrievers, providing broader applicability and scalability in real-world scenarios.

Dear Reviewer VFfy,

Thank you very much for your comment, we will address your concerns below.

Question 1: What is the intuition of the proposed SNR? What does it mean qualitatively when SNRj<σ ? E.g. does it mean that a window of text is predicted well by an LLM, or the opposite?

Response 1:

Thank you for the insightful question.

(1) In fact, the overall experiment is based on our observations presented in Figure 1. We noticed a sudden drop in the SNR before the end of the first chunk, followed by a stable plateau. This led us to hypothesize that this SNR behavior could reflect the stability of the model's uncertainty. Our subsequent experiments have confirmed this hypothesis.

(2) Your understanding is correct: when $SNR_j < \sigma$, it indicates that the corresponding window of text is predicted well by the LLM. To help you better understand, we have uploaded an anonymous code file that provides a detailed example: https://ufile.io/56d9jxyh.

Question 2: Formula (3): when a piece of text is input to the LLM, it outputs probabilities of tokens conditions on all their preceding tokens, till x0. However, formula (3) uses probabilities conditioned only on tokens within a specific window, till x0j. How these probabilities are computed?

Response 2:

Thank you for the insightful question. In fact, when we input a piece of text into an LLM, we can obtain the self-information(The lower the self-information, the higher its probability) for all tokens, which can be stored in a list. Subsequently, when calculating the SNR, we only use the self-information within a sliding window for computation.

To help you better understand our work, we will next demonstrate all the steps involved in measuring uncertainty using a particular text example. We also sincerely appreciate your suggestion and will include this as a case study in the appendix to facilitate future readers' comprehension. Now, assume we have two chunks:

• Chunk A: How do you do?
• Chunk B: Thank you for taking the time to review our manuscript.

Here, for the sake of convenience, we treat each word or punctuation mark as a token.

In this case, Chunk A consists of 5 tokens, and we can obtain the self-information for these 5 tokens. Similarly, Chunk B consists of 11 tokens, for which we also have their self-information. By concatenating these self-information, we get: $P_{A1}, P_{A2}, \dots, P_{A5}, P_{B1}, P_{B2}, \dots, P_{B11}$.

Assume the sliding window size is set to 3, with a stride of 1 (used here for clearer demonstration). We first calculate the SNR within the window: $(P_{B9}, P_{B10}, P_{B11})$ and find that it is less than the set threshold.

We then slide the window one step to the left and calculate: $(P_{B8}, P_{B9}, P_{B10})$ and continue this process iteratively.

For this example, let us assume we eventually find that the SNR for: $(P_{A4}, P_{A5}, P_{B1})$ exceeds the threshold. At this point, the sliding window stops.

We then take the self-information of all tokens from the previous sliding windows (i.e., $P_{A5}$ to $P_{B11}$) and compute their average. This average value is used as a similarity score between the two chunks, forming positive and negative samples for subsequent contrastive learning training.

Dear Reviewer VFfy,

In our most recent response, we addressed the concerns you raised previously. However, it is possible that for various reasons, our response may not have reached you. If you find that our previous answers have resolved your concerns, would you kindly consider raising your score? We greatly appreciate your support.

Thank you very much.

Best regards,

The Authors

Dear Reviewer rEbW,

Thank you very much for your comment, we will address your concerns below.

Question 1: Can you provide more detailed justification for why Signal-to-Noise Ratio (SNR) was chosen as the primary uncertainty metric for span-level similarity estimation? How does it compare to other uncertainty metrics？

Response 1:
Thank you very much for your suggestion!

(1) In fact, the overall experiment is based on our observations presented in Figure 1. We noticed a sudden drop in the SNR before the end of the first chunk, followed by a stable plateau. This led us to hypothesize that this SNR behavior could reflect the stability of the model's uncertainty. Our subsequent experiments have confirmed this hypothesis. To facilitate your understanding, we have uploaded an anonymized code example that illustrates a specific case: https://ufile.io/56d9jxyh

(2) Table 4 in the original text provides a comparison of our method with other uncertainty metrics. The specific formulas for these uncertainty metrics are detailed in Appendix F. We directly replicate the experiments table here.

Question 2: How does your method handle ambiguous queries where there may be multiple valid retrievals or interpretations? Does the SNR-based uncertainty provide any benefit in disambiguating between competing chunks of text?

Response 2: Thank you for the insightful question. This is a very interesting idea. We designed the following experiment: first, we use BERT for coarse ranking to select a set of candidate chunks. Then, these candidate chunks are concatenated with the question and input into an LLM. Subsequently, we apply our method to measure the similarity scores between the question and the candidate chunks. Finally, the chunks are re-ranked based on these similarity scores. The detailed experimental results can be found in the table below:

We can find that after re-ranking, the final results show a significant improvement. This could potentially be an area for further exploration in the future. Once again, thank you for your valuable suggestion!

Weakness

Weakness 1: While the paper introduces SNR-based span uncertainty as a novel metric for estimating text chunk similarity, it fails to provide a compelling motivation for why SNR is the most appropriate choice for this task. The choice of SNR seems somewhat ad hoc, and the paper would benefit from a comparative study showing why SNR outperforms these other potential metrics in various retrieval scenarios.

Response to weakness 1: Thank you for your question. For detailed information, please refer to the response to Question 1.

Weakness 2: The paper relies on a relatively simple fixed chunk size (e.g., 300 words per chunk) for dividing long-context documents. However, chunking is a critical aspect of Retrieval-Augmented Generation (RAG) systems, and the paper does not explore more sophisticated chunking methods, such as dynamic or semantic-based chunking. This is a significant limitation because fixed-length chunking can lead to semantic fragmentation, where important contextual information is split across chunks, reducing retrieval accuracy.

Response to weakness 2: In the original test set, for few-shot tasks such as TREC, TriviaQA, and SAMSum, the nature of these tasks requires full sentences to be presented in the prompt. Therefore, during retrieval for these three tasks, we used complete sentences. And it can be observed that our method also achieves excellent performance when using complete sentences.

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.

Dear Reviewer rEbW,

In our most recent response, we addressed the concerns you raised previously. However, it is possible that for various reasons, our response may not have reached you. If you find that our previous answers have resolved your concerns, would you kindly consider raising your score? We greatly appreciate your support.

Thank you very much.

Best regards,

The Authors

Thank you for your detailed response. It addresses some of my initial questions and provides clarity on certain aspects of the paper. However, there are still some remaining issues that I would like to highlight:

1. Integration of New Results and Explanations: For the points raised that could significantly impact the soundness of the paper—such as the details on the chunking procedure, the hyperparameter selection process, sensitivity analysis, experiments with alternative training/test set combinations, and the additional results in Figure 3—I would like to understand how these new results and explanations will be incorporated into the paper. It would enhance the paper’s clarity and utility for readers if these details were included directly in the main text or appendices rather than being confined to the accompanying code. Would you consider providing a revised version of the paper reflecting these updates?

2. Clarification on Q3 - Dataset Selection: Regarding my earlier question on the choice of training/test set combinations, I was seeking an explanation of the rationale behind selecting these specific combinations, rather than a description of the datasets themselves. Reviewer VFfy also raised a similar concern (Q5). Are these combinations based on specific criteria, and have you considered experimenting with a more diverse or simple mix of datasets to assess the generality of your results?

3. Further Questions on Q5 - Metric Effectiveness:

• My original concern was that the conclusion, "$x$ is an effective metric for $y$," is based on only three results without any additional quantitative analysis. This makes it challenging to substantiate the claim. Moreover, using RSA as a metric to demonstrate that the training set exhibits the most "distributional shift" among the three datasets seems to introduce a degree of circular reasoning. Could you clarify or address this concern?
• Additionally, in the new results provided, the curve for TriviaQA appears to fall below that of the training set. Do you have an explanation for this behavior, and how does it impact your conclusions?

I hope this feedback is helpful and contributes to strengthening the paper’s clarity and rigor. Thank you again for your efforts in responding to my earlier questions.

Weakness 3: Numerous hyperparameters are introduced, such as the threshold σ, the number of samples M and m, the chunk size, and the sliding window size. However, the authors do not explain how these hyperparameters were chosen or how sensitive the model is to these values.

Response to weakness 3: Thank you for the question. In the anonymous code link: https://ufile.io/56d9jxyh, it can be observed that the choice of threshold is not particularly sensitive. When adjusting the sliding window size, one only needs to observe the data accordingly and modify the threshold selection. Experiments with different chunk sizes are presented in Table 3 of our original paper. All other parameters were adopted from previous works[1][2] without modification.

Weakness 4: It is challenging to conclude that the method can be integrated with any LLMs, given that uncertainty is only calculated using Llama 2 7B. The tested models also use Llama 2 as their base, which shares the same pretraining data. From Table 1, it is apparent that when the uncertainty score model does not match the LLM, the performance gain from the proposed method decreases. This raises doubts about whether the retriever must match the LLM used.

Response to weakness 4: (1) Thank you for the insightful question. Firstly, since our measuring of the uncertainty inherently reflects the model's perplexity, it is naturally most effective when the retriever and LLM are well-matched. However, this does not mean that our method cannot be integrated with other LLMs. To demonstrate this, we conducted additional experiments using Vicuna-7B to measure uncertainty and performed RAG experiments on LLaMA-2-7B-Chat-HF. The results are presented in the table below. As observed, although the improvements are not as significant as those achieved when using LLaMA-2-7B-Chat-HF itself for measuring the uncertainty (which is expected), the performance still surpasses other baselines.

(2) More importantly, we would like to point out that the statement "it is apparent that when the uncertainty score model does not match the LLM, the performance gain from the proposed method decreases" is not entirely rigorous. As demonstrated, even when comparing two baselines (e.g., Contriever and GRAGON-PLUS), the performance differences vary across different LLMs. Specifically, on LLaMA-2-7B-Chat-HF, GRAGON-PLUS outperforms Contriever by an average of 2.36%; on LLaMA-2-13B-Chat-HF, this advantage decreases to 0.78%; and on Vicuna-7B, it further drops to 0.39%！ This indicates that the improvements achieved by retrieval models inherently differ across various LLMs.

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.

[1] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021.

[2] Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. arXiv preprint arXiv:2302.05698, 2023.

Question 4: Not all training and test sets feature distributional shifts in their contexts. For example, both HotpotQA and 2WikiMQA use Wikipedia as their source of context.

Response 4: Thank you very much for your reminder! We have reselected the training and test datasets. In the new experiments, the training set includes MultiFieldQA, Musique, QMSum, TREC, and TriviaQA, while the test set consists of NarrativeQA, Qasper, SAMSum, and 2WikiMultihopQA. The detailed experimental results are presented below.

All experiments were conducted on LLaMA-2-7B-Chat-HF. As observed, our method continues to achieve the best performance on the new datasets. This demonstrates the robustness of our method, as it does not rely on the specific selection of datasets.

Question 5: In line 451, how do you conclude that "RSA is an effective metric for assessing distribution shifts," given that you only have two data points (GSM8K and 2WikiMQA)?

Response 5: Thank you very much for your reminder. In fact, Figure 3 in the original paper includes three results, representing the original training set, GSM8K, and 2WikiMQA. Additionally, we have obtained results on other test sets as well, but we opted to include only one in the figure for aesthetic reasons. However, we did not consider that this might raise concerns for readers. Once again, thank you for pointing this out. Here, we will present the RSA results on two additional datasets in the anonymous link below: https://ufile.io/uvmt2qsi

Question 6: The "10" mentioned in line 310 should be k

Response 6: Thank you for your suggestion. We have made the corresponding changes in the paper.

Weakness:

Weakness 1: Some content is difficult to understand. For example, in line 245, "The probability distribution within this window is stable," which does not seem to imply a stable distribution. Additionally, in Section 4.1, the phrase "number of chunks" is unclear, as the previous mention of the same term refers to the total number of chunks in the training set.

Response to weakness 1:

(1) Here, I believe directly presenting a specific example might be more helpful for understanding. Below is an anonymous code link: https://ufile.io/56d9jxyh

(2) Thank you very much for your reminder. The "number of chunks" here refers to the quantity of chunks retrieved during the retrieval process. Based on your suggestion, we have revised line 420 in the original text from "Ablation Study on Chunk Length and The Number of Chunks" to "Ablation Study on the Length and Number of Retrieved Chunks."

Weakness 2: The definition of the negative samples is questionable. It appears that the negative samples are among the top M most similar chunks to the anchor. If this is the method for selecting hard negative examples, the hyperparameters in the paper would play a crucial role.

Response to weakness 2: Here, we followed the settings from previous works, such as EPR[1] and CEIL[2], without making any modifications to these hyperparameters.

Dear Reviewer Q3d1,

Thank you very much for your comment, we will address your concerns below.

Question 1: The chunks are divided based on a specific word length (300 in the paper). Below are questions regarding this design: (1) It is possible for 300 words to correspond to more than 512 tokens for BERT. What would be your solution in this scenario? (2) Do you consider punctuation as separators for words as well? (3) If a chunk is cut mid-sentence, concatenating two "incomplete" sentences (chunks) may lead to a "sudden transfer" between them. This could affect uncertainty calculation. What are your thoughts on this?

Response 1: (1) and (2): Thank you very much for your reminder. Our explanation in the paper was not sufficiently clear. We actually used 300 letters rather than words. The specific code is shown below.

text_splitter = CharacterTextSplitter(
    separator="",
    chunk_size=300,
    chunk_overlap=0,
    length_function=len,
    is_separator_regex=False,
)

(3): In the original test set, for few-shot tasks such as TREC, TriviaQA, and SAMSum, the nature of these tasks requires full sentences to be presented in the prompt. Therefore, during retrieval for these three tasks, we used complete sentences. And it can be observed that our method also achieves excellent performance when using complete sentences.

What's more, one of the primary objectives of our method is to address the fragmentation between chunks. To amplify the impact of this challenge, we allowed "mid-sentence" cases to exist and did not apply more sophisticated processing to the chunks. If only the output uncertainty is used for estimation (i.e., the PreciseChunking method shown in Table 1 of our paper), such "sudden transfer" can affect the results. However, after calibrating the uncertainty using SNR, we observed significant performance improvements. Furthermore, our experiments demonstrate that even under the influence of such "sudden transfer," our method can still outperform previous approaches with significantly lower training costs.

Question 2: What is the role of N as defined in line 285?

Response 2: Thank you very much for your reminder! The use of N here is unnecessary and was a typo. In fact, this step involves using BM25 to coarsely filter and select M semantically relevant chunks for each chunk, aiming to reduce the computational cost of subsequent measurements with the LLM. Thank you again! We have corrected this typo in the paper.

Question 3: How did you select the training and test sets from the LongBench suite?

Response 3: Thank you for the insightful question. In fact, we carefully considered the diversity of both the training and test datasets when making our selections. Specifically:

2WikiMultihopQA and Musique are tasks that require answering questions based on multiple provided documents.

TREC is a classification task that involves categorizing questions into 50 predefined categories.

TriviaQA is a single-document question-answering task with several few-shot examples provided.

SAMSum is a dialogue summarization task, also including few-shot examples.

HotpotQA involves answering questions based on multiple given documents.

MultiFieldQA focuses on answering questions in English based on a single document, with documents spanning diverse fields.

Qasper is centered around questions posed on a single research paper, where the questions are generated by NLP readers and answered by NLP practitioners.

NarrativeQA involves answering questions based on stories or scripts, requiring an understanding of key elements such as characters, plots, and themes.

QMSum is a summarization task that requires summarizing meeting transcripts based on a user query.

To further address your concerns about the sensitivity of dataset selection, we have added experiments in our response to Question 4, where we trained and tested the model using newly selected training and test datasets.

Dear Reviewer Q3d1,

In our most recent response, we addressed the concerns you raised previously. However, it is possible that for various reasons, our response may not have reached you. If you find that our previous answers have resolved your concerns, would you kindly consider raising your score? We greatly appreciate your support.

Thank you very much.

Best regards,

The Authors