SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction

Thanks for appreciating the clear formulation and good performance of the proposed SePer metric for evaluating retrieval utility. We believe your concerns can be addressed through clarification and added experiments as follows:

1. "The assumption that retrieval utility is uniformly distributed across different reasoning steps may not hold true universally"

We totally understand your concern, which is reasonable in a general view. However, actually this assumption can hold for current multi-step reasoning datasets, including those in our papers. The reason lies in the construction process of these synthetic datasets. For example, the 2wikimultihopQA dataset constructs a multi-hop reasoning question by chaining two single-hop questions together. In this construction setting, it is reasonable to assume that each hop's retrieval contributes evenly to final success. Besides, to the best of our knowledge, this is the only feasible way to obtain ground-truth middle-step retrieval utility annotations currently. We'd be happy to test for future datasets with more fine-grained annotations.

We believe that the proposed SePer metric works in general multi-hop reasoning tasks by manually examining real cases. We added several qualitative cases in the Appendix A.2.1 to prove that qualitatively for your reference.

2. "Extensive Monte Carlo sampling and clustering in SePer calculation may be costly"

The main cost of calculating SePer did come from sampling the generation 10 times. The clustering process (distance calculation) is actually ignorable compared to the sampling time since it is based on deBERTa. However, since the sampling choice is made following previous works like [1] in the uncertainty estimation area, it is inevitable if we want to achieve human-level and fine-grained evaluation. If more efficient and accurate uncertainty estimation algorithm emerges, they can be plugged in seamlessly as well. We also added a Table in the appendix to show the time cost to compute SePer compared to other LLM-as-a-judge methods marked in purple color. Since SePer is accurate, free from API, and explainable in computation, we believe there still exists many usage scenarios for SePer in spite of the extra cost brought by sampling 10 times.

3. [Major concern] "whether SePer is more about answer quality than information gain"

Thanks for bringing up this issue. We acknowledge that our writing has some ambiguity, especially between SePer and the reduction of SePer, which leads to some misunderstandings and confusion. We distinguished the usage of these two terms by marking SePer reduction as Δ SePer to ensure consistency in the paper and add illustrative content. Here is the explanation FYI.

• SePer (semantic perplexity) is a measure of the generator's confidence in the ground-truth answer, thus indeed more of a measure of answer quality as mentioned by the reviewer.
• SePer reduction, computed from the difference of SePer w/ and w/o retrieval, measures the information gain acquired from retrieval, thus measuring "retrieval utility", as illustrated in the title and Figure 2.

Thus, for the first concern, "Comparing SePer with the baselines" in Table 1 actually aims at proving SePer is a more fine-grained and accurate measure of LLM's knowledge on the correct answer, thus being a solid stone for calculating knowledge gain. Table 2 is the one that proves SePer reduction is a good measure of knowledge gain. For the second concern, "the Answers with high certainty but poor information (e.g., "I don't know")" won't have a high SePer score since we use the confidence assigned to the ground-truth answer (but not the confidence in the generated answer) for calculating SePer, as illustrated in Figure 2.

4. How much does it cost to run SePer during evaluation？

We added the cost in time and money in Appendix A.2.2, marked in purple color.

Finally, thank you for your review and valuable comments. We appreciate your feedback and welcome any additional questions or suggestions.

[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation." The Eleventh International Conference on Learning Representations.

We would like to thank the reviewer for bringing up the ASK theory. We added some thoughts and discussions about the difference between our work and ASK theory after we went through related works suggested by the reviewer. Through this public discussion, we hope to facilitate the understanding of our paper, as well as encourage new insights along the lines of the interesting work in the 1980's.

We find resonance with ASK theory in the assumption that "knowledge gain (bridging knowledge gap) is the key to the success of information retrieval". However, we actually work in different directions, as illustrated in the following aspects:

• Goals: ASK aims at improving the design of IR methods, while our proposed SePer aims at improving the evaluation of retrieval utility in recent RAG pipelines.
• Subjects: In ASK, a real human is the information recipient, while in our evaluation framework, LLM is the information recipient. This results in different methods we can leverage for computation. For example, it's still infeasible to quantitatively estimate belief shifts in human brain, but it's possible to estimate LLM's internal belief in a computable way as described in this paper, thanks to recent research about LLMs and uncertainty estimation.
• Motivations: ASK is motivated by the fact that users often cannot express their real information needs during searching. Thus, it's important for the search engine to identify the lack of knowledge (anomalous state of knowledge) in a specific user, which is underrepresented but critical to satisfying the real information needs of the user. To this end, ASK proposed a new perspective of retrieving documents that can bridge the knowledge gap rather than that can match the expressed knowledge as done by traditional methods such as BM25. While we appreciate the importance and keenness of this perspective, we'd like to kindly state that our motivation stems primarily from a pragmatic need to address the limitation of evaluation in recent RAG pipelines.
• Scenarios: ASK is especially useful in cases when the user cannot fully express their needs. In our setting, our methods do not focus on the gap between "know" and "tell".
• Methods: ASK focuses on picturing users' state of knowledge and detecting the anomalous state using methods like interactive querying, and then improving retrieval accuracy accordingly. We focus on estimating LLM's belief through a Bayesian inference framework via sampling and clustering, and then use it to evaluate retrieval utility.

We welcome and appreciate further discussions on related topics.

It is interesting to link our work to redefining relevance in IR. We believe that this connection may occur due to the reviewer's familiarity with the anomalous state of knowledge (ASK) theory, which proposed to improve IR by mining users' knowledge gap. However, we must clarify here that despite sharing some cognitive similarities, the goals, motivations, and scenarios are all shifted from this line of work. In short, although the reviewer put it in the Strengths part that "Determining the relevance of an item for RAG is exciting", we state that our work does not focus on redefining "relevance" to improve IR, but to propose a computable metric to evaluate the "retrieval utility" in RAG. The in-depth discussion of nuances between them is put at the end of a rebuttal for anyone interested in this topic.

For other concrete questions, we address them as follows:

1. Metric validation.

   We believe there is some misunderstanding regarding this point. We totally agree that a new proposed metric should be tested upon its reliability and validity as mentioned by reference [1] from the reviewer and we believe we had implicitly followed the scientific convention of proposing a new metric in the following part of this paper:

   1. Reliability: The reliability of a metric measures its sensitivity against random measurement errors, such as consistency across repeated measures, and choice of datasets in a benchmark [1].

      1. For the randomness due to repeated sampling, we conducted experiments in Figure 3 in the original paper. In Figure 3, we repeatedly sampled the n generations 10 times at each fixed n and calculated the variance of correlation for repeated measurements. The shadow area in Figure 3 quantifies the error range of our SePer metric. Shortly speaking, there are two conclusions to take away from the experiments in Figure 3:

         1. The Coefficient of Variation of correlation is within 1% using our default hyperparameters as proven by experiments from four major datasets, indicating high reliability and stability of our SePer metric in aligning with human preferences. (1% generally means high precision measurements in common convention )
         2. As n (the number of generations) increases, the variance generally decreases even smaller as shown by the length of the error bar in Figure 3. Until our default setting of n=10, the error range is small enough to qualify SePer as a reliable metric, striking a good balance between accuracy and efficiency.

         To highlight this point as suggested by the reviewer, we further add a Table in Appendix A1.1.2 to illustrate the implementation details of the test of robustness. We also add statistics of standard deviation and coefficient of computing SePer reduction to prove its stability in computation in A1.1.2.

      2. For randomness of datasets, we conducted experiments on major mainstream RAG datasets, covering aspects of simple tasks (NQ, TriviaQA, etc.) and reasoning-based tasks (Hotpot QA, etc.). The results of the proposed SePer consistently align well with human preferences. To add further evidence, we add benchmarks of applying SePer to retriever evaluation on different datasets in Appendix, where we can see from the produced ranks that the proposed SePer is robust and consistent across different datasets.

   2. Validity: the validity of a metric measures whether it reasonably reflects the intended construction purpose and aligns well with human preferences[1]. To prove SePer's validity, we combine theoretical along with experimental analysis. Originated from cognitive perspectives [4][5], the intuition is widely-accepted. Grounded in the Bayesian inference frameworks, we construct the metric with stepwise derivation as proved in section 3. As for experiments, we proved in Table 1 and Table 2 that the proposed metric aligns better with human preferences across different datasets. The P-values in hypothesis tests are all less than 0.01, indicating a significant correlation relationship with human judgments. We added more detailed captions to explain these experiments.

   In all, we believe that our experiments satisfy the required justification of metric design. We follow the writing style of related works such as bertscore [2] and factscore [3] in this field and have at least the amount of workloads on par with these works.

   However, we do agree with the reviewer that the justification of metric validation is critical in a metric design paper and should be highlighted more clearly. To this end, we reorganized the writing of the corresponding part (section 4) based on the taxonomy of [1] mentioned by the reviewer and added more details in the appendix.

2. Related works.

   Thanks. we agree that the premise in the anomalous state of knowledge(ASK) theory shares some similar ideas with our intuition of measuring the success of retrieval by knowledge gain (the "anomaly" in ASK). We cited these works in the section 1 and 3 of our paper.

3. Writing.

   Thanks for this suggestion. We acknowledge that some parts of writing in the original paper are too brief and make it prone to misunderstanding. We added a leading paragraph in section 5 for the background introduction and added citations in section 3 as suggested. We also re-organize section 4 to emphasize the justification of metric design.

Finally, thank you for your review and valuable comments. We appreciate your feedback and welcome any additional questions or suggestions.

[1] Ziang Xiao, Susu Zhang, Vivian Lai, and Q. Vera Liao. Evaluating evaluation metrics: a framework for analyzing NLG evaluation metrics using measurement theory. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 conference on empirical methods in natural language processing, 10967--10982, Singapore, December 2023. , Association for Computational Linguistics.

[2] Zhang T, Kishore V, Wu F, et al. BERTScore: Evaluating Text Generation with BERT[C]//International Conference on Learning Representations.

[3] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.

[4] Cooper, William S. "On selecting a measure of retrieval effectiveness." Journal of the American Society for Information Science 24.2 (1973): 87-100.

[5] Nicholas Belkin. Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, page 133-143, 11 1980.

There is indeed some misunderstanding due to the gap between different subareas, mainly regarding the datasets.

Here are the detailed explanations:

Reference [1], while named as "multiple datasets", is implicitly based on the dimension of "attributes" or "aspects", each "aspect" is represented by a dataset, such as a language (multilingual setting) or a domain (multi-domain setting). In these cases, correction by the number of datasets indeed improves the soundness of replicative analysis, since the hypothesis is based on a composition of all aspects. For example, in the task of multilingual POS tagging, the p-value should be calibrated if one wants to solidly claim that "method A is better than method B in pos tagging of all languages", since the erroneous discovery on any dataset(language) would hurt the integrity of the global hypothesis.

As shown by the experiments in [1], many traditional NLP tasks fall under the scope of "correction of multiple datasets because:

1. the hypothesis is claimed in a multi-aspect way such as multilingual and multi-domain, as illustrated above.
2. each test dataset is relatively small(from 280 to 2327 in [1], for example), thus p-values suffer from large variations.

On the contrary, our "multiple datasets" do not represent different aspects, but rather a cumulation of data samples. This is usually the case in IR and QA area, since it's hard to divide these samples by aspects.

To wrap up the difference between the two mentions of "multiple datasets",

• The more hypotheses, the more prone to type-1 error, thus need calibration by methods mentioned in [1]
• The more data samples, the less prone to type-1 error, reflected in the computation of $t$-value by normalizing by $n$.

To make things more clear, we add a table here of the exact $p$-value of each dataset. The significance claim of $p$<0.01 still holds even if we do multiple corrections since the p-value is too small due to the large number of samples.

Note: the 0 $p$-value does not mean exactly equal to 0, but below the numerical precision between the difference of computed $p$ and 0.

Nevertheless, thanks for bringing up the paper [1], which facilitates our thinking regarding the evaluation of NLP tasks.

References:

[1] Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets. Transactions of the Association for Computational Linguistics, 5:471–486.

We appreciate the reviewer's valuable suggestions. We thank the reviewer for recognizing our work's clear motivation and presentation, as well as the necessity and innovation of SePer for evaluating retrieval in RAG. We are glad to find that the effectiveness, efficiency, and generalizability across RAG scenarios were highlighted. We believe your concerns can be addressed through clarification and additional details in our responses below.

W1: " How different entailment inference methods and different thresholds impact SePer measure "

Thanks for this valuable suggestion. We are adding more ablation experiments in the appendix about SePer's robustness to hyperparameters, and this part will be marked in yellow as we push the revision in 1-2 days. For now, we can say that $SePer_{soft}$ does not bear the problem of choosing a threshold since it directly incorporates the entailment score in weighted sum computation.

W2: "...conduct more concrete semantic analysis of cases to deepen the understanding of SePer".

Thanks for this great suggestion. We do agree that concrete and qualitative case studies can facilitate the understanding of the characteristics of SePer such as its fine-grained evaluation ability. We add several case studies in appendix A.2.1, covering both simple/reasoning-type of questions and known/unknown questions to LLM.

W3: "quite a few typos in the paper"

Thanks a lot for your careful review. We have modified these typos and proofread our paper again to avoid typos.

Finally, thank you again for your review and valuable comments. We appreciate your feedback and welcome any further questions or suggestions.

We are grateful to the reviewer for recognizing the novelty and validity of the metrics we proposed. We appreciate your valuable suggestions and believe that your concerns can be addressed through clarification and additional details in our responses below.

W1&W2. "Some unclear presentation hampers the understanding"

Thanks a lot for this suggestion. We do realize that there are some unclear writings that may confuse the readers. We have added more detailed captions to each figure and table. We also have added an explanation about the concept of SePer reduction highlighted in red and distinguish its use from SePer by marking it as Δ SePer. As for explaining the role of reranking and prompt compression, we added a leading paragraph in section 5 and a detailed tutorial about the current components in RAG in Appendix A.4, which will help wider readers to understand section 5 about how to use SePer to measure the utility of different components in RAG.

W3&Q1. "How could SePer be used concretely in RAG?"

Thanks a lot for this suggestion. We ran experiments and added a table to benchmark current RAG workflows in Appendix A.3.2 and the procedure of applying SePer in RAG. We also agree that SePer can be used in tasks such as providing more fine-grained annotation in RAG datasets and will continue exploring more use cases in future research.

W4. "Similarity to Farquhar et al. (2024) ... has not been explicitly stated"

We actually cited the semantic entropy work from Farquhar et al. (2024) in section 1 and section 3.3, one for the ICLR version and one for the Nature version. We also add some explanations of how our work builds upon and differs from this milestone work at the end of section 3.3, marked in orange color.

Q2: "if SePer correlates well with the the quality of final answer generation?"

Yes, SePer correlates well with the answer generation quality. It measures the generator's belief on reference answer by design. Actually, Table 1 demonstrates whether the compute mechanism in SePer can better reflect answer quality, which is proven yes. Table 2 is about SePer reduction (not SePer), where we compute the correlation of SePer reduction to the retrieval utility and prove it is the best measure of retrieval utility. We apologize again for the confusing part in writing about SePer and SePer reduction. We examined thoroughly and modified relevant parts to ensure the consistent usage of the two terms and mark SePer reduction as Δ SePer. We also added an illustrative paragraph highlighted in red to help the reader clearly understand these two proposed concepts.

Finally, thank you for your review and valuable comments. We appreciate your feedback and welcome any additional questions or suggestions.