DiverseAgentEntropy: Quantifying Black-Box LLM Uncertainty through Diverse Perspectives and Multi-Agent Interaction

We sincerely thank reviewer aMZi for the constructive suggestions. Here we clarify the questions you posed.

Computational cost

We acknowledge that the cost of our method is relatively higher compared to self-consistency-based methods. However, we emphasize the following points:

1. Superior performance: our method outperforms simple self-consistency-based approaches. In high-stakes applications where correctness is prioritized over cost, our calibrated uncertainty score provides users with a reliable measure of how much they can trust the model's output. Additionally, the chosen answers after applying the abstention policy are more accurate.

2. Utility for finetuning: the intermediate results generated by our method, including varied questions and the self-reflection interaction processes, can be further leveraged to create synthetic data for finetuning or training LLMs.

3. Potential for optimization: future work can explore ways to maintain the same level of performance while reducing costs. This could involve using fewer but higher-quality questions from diverse perspectives and minimizing the number of interaction rounds.

We have added a detailed analysis to present the cost/number of inference calls for each method in Appendix Table 6 in the updated version.

Generalization to other types of language tasks

We believe our method has the potential to generalize to a wide range of QA tasks, including those involving RAG and complex QA scenarios. Its ability to generate questions from diverse perspectives and leverage multi-agent interactions makes it well-suited for handling complex reasoning and integrating diverse information sources effectively. For RAG, by generating questions from different perspectives, we prompt the retriever to fetch diverse documents, enriching the range of potential information sources. Additionally, the multi-agent interaction facilitates self-reflection, which is particularly beneficial for QA scenarios involving conflicting retrieved documents. For complex QA, in order to effectively use our pipeline, we propose future work to incorporate an additional meta judge to monitor agents' overall understanding of the query. This mechanism ensures that the model does not resort to taking shortcuts, such as prematurely considering the query invalid.

We sincerely thank reviewer YVnz for the constructive suggestion. Here we clarify the questions you posed.

Computational cost

We acknowledge that the cost of our method is relatively higher compared to self-consistency-based methods. However, we emphasize the following points:

1. Superior performance: our method outperforms simple self-consistency-based approaches. In high-stakes applications where correctness is prioritized over cost, our calibrated uncertainty score can provide users with a reliable measure of how much they can trust the model's output. Additionally, the chosen answers after applying the abstention policy are more accurate.

2. Utility for finetuning: the intermediate results generated by our method, including varied questions and the self-reflection interaction processes, can be further leveraged to create synthetic data for finetuning or training LLMs.

3. Potential for optimization: future work can explore ways to maintain the same level of performance while reducing costs. This could involve using fewer but higher-quality questions from diverse perspectives and minimizing the number of interaction rounds.

We have added a detailed analysis to present the cost/number of inference calls for each method in Appendix Table 6 in the updated version.

Questions about the implementation of the pipeline

We ensure that the original query is preserved in the context of the generated varied questions, but not necessarily the original answer. After generating these varied questions, we immediately prompt the model to self-check whether each generated question strictly requires knowledge of the original query to answer. We adopt the following prompt for extracting the answer to the original query from the response. An answer to the original query will also be extracted after each 1-1 interaction.

System: You will extract the answer to the given question using ONLY the information provided in the "Response" section. You will identify the answer directly without using any additional knowledge or explanation. If the response includes a negation to the question, use those as the answer.

User: Response: <response>
Assistant: <extracted answer>

We show the prompt for 1-1 interaction below.

System: You are an AI assistant that helps people answer questions. Ensure your responses are concise and strictly relevant to the queries presented, avoiding any unrelated content to the question. Do not change your answer unless you think you are absolutely wrong.
<previous interaction conversations…>
User: “When I asked you in another API call that” + selection_agent_question + “You mentioned that’’+ selection_agent_answer_to_original_query + “Which is your actual answer to” + original_query? 
Assistant:  <generate a new answer to the original query>

Note that we have added both prompts to the paper in line 297 and the appendix prompts.

Application to RAG

Our method can be applied to tasks like RAG. By generating questions from different perspectives, we can prompt the retriever to fetch diverse documents, enriching the range of potential information sources. Additionally, the multi-agent interaction facilitates self-reflection, which is particularly beneficial for QA scenarios involving conflicting retrieved documents.

We sincerely thank reviewer D3Hu for the valuable comments. Here we answer all the questions and hope they can address your concerns.

Novelty

We would like to highlight two major novelties for our method:

1. Our work introduces diverse perspective questions by injecting additional context from different perspectives into the original query. For example, in Figure 1, "What is the most common symptom experienced by women with the leading cause of cancer deaths in the U.S.?" adds context about the cancer diagnosis perspective to the original query "What type of cancer kills the most women in the U.S.? ". This approach differs from works like Semantic Uncertainty (ICLR 2023), which focuses on the original query, and SAC3 (EMNLP 2023), which only paraphrases queries into semantically equivalent forms without adding explicit new context. In fact, both tested models in our paper will still consistently give the wrong answer to the semantically equivalent form of the original query, e.g., “Which organ does the cancer that kills the most US women affect?” . These diverse perspective questions further enable unique agent backgrounds for same-model multi-agent interactions (lines 220-224) and facilitate a novel and unique analysis of the retrievability of a model, showing that models fail to consistently retrieve the correct answer to the same query under diverse perspective questions (lines 416-454).

2. Unlike the debate framework in Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024), which involves agents from different models debating and persuading each other to improve downstream performance, our approach focuses on same-model agent interaction. By exposing the same model to different contextual hints, we aim to analyze an individual model’s uncertainty more effectively. In this setting, we allow the model to state "I don’t know" rather than forcing it to generate an answer, emphasizing a fundamentally different goal and interaction paradigm. We also propose a novel weighted algorithm combined with classical entropy for uncertainty estimation during agent interactions (lines 255-269). To the best of our knowledge, we are the first to develop a method for measuring a model’s uncertainty after agent interaction.

Evaluation metrics

We address both uncertainty estimation and hallucination detection in our work. In Table 1, we compare our approach with rigorous uncertainty estimation methods to evaluate the calibration of our method ( We adopt three new popular and recent uncertainty estimation baselines from Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (TMLR 2024), see lines 196-202). In Table 2, we benchmark our approach against hallucination detection and direct answering methods to assess its ability to identify and mitigate hallucinations effectively.

We evaluate AUROC in Table 1, as it is the primary metric used in uncertainty estimation papers for assessing calibration and ranking ability in a threshold-independent manner. However, AUROC is less informative for final question-answering scenarios, where users expect a model for either a single correct answer or an explicit acknowledgment of uncertainty, such as stating, "I don’t know."

To address this, we employ the evaluation metrics shown in Table 2, which focus on the final question answering with the optimal threshold that a user would select for each method. Our results demonstrate that our method achieves the highest accuracy on questions where the model does not abstain from answering. Additionally, it achieves the highest correctness and informativeness across the entire dataset, metrics that are also used in TruthfulQA (ACL 2022). This indicates that for users seeking an optimal answer for each question, our proposed method is the best choice.

Furthermore, as presented in Figure 3, we evaluate cov@acc by sampling all possible coverage rates and plotting their corresponding accuracy. The results confirm that, among coverage rates where all methods are applicable, our method consistently achieves the highest accuracy.

We agree with the reviewer that we should also present the cost/number of inference calls for each method. We have added a detailed analysis in Appendix Table 6 in the updated version.

We sincerely thank reviewer KJxh for the valuable comments. Here we answer all the questions and hope they can address your concerns.

Novelty

We would like to highlight two major novelties for our method:

1. Our work introduces diverse perspective questions by injecting additional context from different perspectives into the original query. For example, in Figure 1, "What is the most common symptom experienced by women with the leading cause of cancer deaths in the U.S.?" adds context about the cancer diagnosis perspective to the original query "What type of cancer kills the most women in the U.S.? ". This approach differs from works like SelfCheckGPT (EMNLP 2023), which focus on the original query, and Knowing What LLMs Do Not Know (NAACL 2024) or SAC3 (EMNLP 2023), which only paraphrase queries into semantically equivalent forms without adding explicit new context. In fact, both tested models in our paper will still consistently give the wrong answer to the semantically equivalent form of the original query, e.g., “Which organ does the cancer that kills the most US women affect?” . These diverse perspective questions further enable unique agent backgrounds for same-model multi-agent interactions (lines 220-224) and facilitate a novel and unique analysis of the retrievability of a model, showing that models fail to consistently retrieve the correct answer to the same query under diverse perspective questions (lines 416-454).

2. None of the mentioned methods introduce multi-agent interaction, which is crucial in our approach to enable the model to be exposed to diverse perspectives and self-reflect. We propose a weighted algorithm combined with classical entropy for uncertainty estimation during agent interactions(lines 255-269). To the best of our knowledge, we are the first to develop a method for measuring a model’s uncertainty after agent interaction.

Baselines

We would like to emphasize that our primary goal is to compare our method with entropy-based uncertainty estimation methods, as these are the most commonly used approaches in the literature for uncertainty estimation in NLG. ( We already highlighted this in Section 3.1 and have revised Section 3.2 to clarify this focus). Specifically, we compare against Self-Consistency(SE), also known as SemanticEntropy, which is a widely recognized and effective uncertainty estimation method (ICLR 2023, Nature 2024). Additionally, we adopt three new and popular black-box uncertainty estimation baselines from Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (TMLR 2024). We have added a description of these methods in lines 196–202 and present their results in Table 1.

We acknowledge that Knowing What LLMs Do Not Know (NAACL 2024) is a very related paper. However, it is proposed as a nonfactuality detection method, not an uncertainty estimation method. We will thus use it as a hallucination detection method for comparison in Table 2. Since the codebase of this paper lacks sufficient details for proper implementation of the Atypicality component and XGBoost fitting, we emailed the authors for clarification and plan to include the full baseline in the final version. Nonetheless, SeQ in Table 2 is a simple version of Knowing What LLMs Do Not Know (NAACL 2024) without the Atypicality component.

Tables 1&2 and Figure 3 show that our method consistently performs better than any baselines in terms of uncertainty score calibration (Table 1) and hallucination detection (Table 2 and Figure 3). We highlight that ab-R is not our primary metric, as it only reflects how often the model abstains and closely correlates with accuracy (acc). With similar ab-R values, the method with higher accuracy (acc) is preferable.

Related works

We agree with the reviewer that related work should be enriched and we have added more related papers including the ones that the reviewer has mentioned in the related work section.