Thank you for your thoughtful response. We address each comment below, and provide additional details regarding added baselines in our general response above.

[W1] Writing clarity

Thank you for pointing out these points of confusion in your questions below. We address these points below and revise them in our updated manuscript.

[W2] Unclear Preference Data Construction

We do not collect new human annotations in this work. Instead, we use different annotator intents and their respective expected answers from NaturalQuestions dataset. We depict how these annotations are used for preference data construction in Figure 2. We use the annotations from NaturalQuestions to simulate the clarifying answers from each annotator. We then simulate each annotator’s preference by comparing the LLM’s predicted output against the respective annotator’s expected answer.

[W3-4] Additional Baselines and Related Works

In our related work, we discuss STaR-GATE and add it as one of our additional baselines in our general response above. There, we also add further discussion on these other cited works, and will include these in our revisions. Additionally, our baselines using the Starling Reward model to rank clarifying questions is used to demonstrate the efficacy applying standard RLHF methods in our clarifying questions setting.

To compare against the original performance of these LLMs, in our General Response above, we add a comparison against using CoT prompting to evaluate the original performance of the base LLM. We find that CoT performance of the base LLM (post instruction-tuning) underperforms when compared to trained methods.

[Q1] Clarification on $a_i$ in lines 120 and 129

In both lines, $a_i$ is the $i$-th user’s clarifying question to the LLM’s proposed clarifying question. During evaluation, we simulate $a_i$ by conditioning on the input query $x_i$, clarifying question $r_{init}$, and the $i$-th user’s expected answer $y_i$.

In line 120, there was a typo where we are $M(x)$ instead of $r_{init}$. The correct equation is represented in line 129 and in Figure 1. Thank you for pointing this out! We fix this in our revisions.

[Q2] Are the input query ($x$) and the user intent ($x_i$) supposed to be different forms?

The $x$ refers to the potentially ambiguous input query passed to the LLM. In Figure 1, we use $x_i$ illustrate $i$-th user’s disambiguated intent in the form of a re-written input query. $x_i$ is only used to help illustrate our example in Figure 1, and is not directly used in our methods or settings. Thanks for pointing out this point of confusion, we will remove the unnecessary notation to avoid confusion in our revisions.

[Q3-5] Additional Baselines.

Thank you for your suggestions. We address this in our response to [W3-4] above, where we discuss additional baselines provided in our general response.

Thank you for your thoughtful review! Due to space constraints, we address comments here and will answer questions in the following response.

[W1] Unclear Relationship Between Human Annotators and the User Simulator

We do not collect human annotations ourselves for this work. Human annotations for different users' answers to each query $(x, \{y_1, \dots, y_n \})$ are sourced from the AmbigQA and NaturalQuestions datasets. We use these annotations to simulate the clarifying answers $(a_1, \dots, a_n)$ from each human annotator to generated clarifying question $(q)$. This is done by training a user simulator model to that conditions on the input query, clarifying question, and human-annotated answer to simulate the annotator's clarifying answer $(x, y_i, q) \rightarrow a_i$.

[W2] Additional Baselines

Thank you for your suggestions. In our common response, we include additional baselines for points of comparison, and include additional discussion on methods from the other works you listed. While some methods are not immediately adaptable to our setting (e.g., they are designed to only work for retrieval-based QA systems), we include further discussion.

[W3] Need for More Insightful Analysis (Error Accumulation)

This is a fascinating point! While such an analysis is quite interesting, in practice we find that it is not always obvious when the system is asking an “incorrect clarifying question”. Consider the example from our LLama2 method below where the query is unambiguous and clarifying the question seems unnecessary.

• Question: “When did gods not dead 3 come out?”
• Gold Answer: “March 30, 2018”
• Clarifying Question: “Which “God’s Not Dead 3” are you referring to? The first, second, or third installment?”
• Clarifying Answer: “The third installment.”
• Predicted Answer with Clarification: “March 30, 2018”
• Direct Answer without Clarification: “April 2, 2018”

Here, we find that seemingly extraneous clarifying questions can often improve model performance. Another way to determine whether a clarifying question is irrecoverably bad during evaluation is by using GPT-4. Recall that during evaluation, use GPT-4 as our clarifying-answer simulator and allow it to refrain from providing a clarifying answer if it judges that none exists. Using this as an indicator of an irrecoverably bad clarifying question, we can compute the percent of ambiguous questions where each system produces clearly erroneous clarifying questions. For example, we find that our 20% of our LLAMA2-based Clarify SFT system’s clarifying questions contain errors. We also find that with additional DPO training, this number is reduced to only 13%. In our revisions, we will include these additional figures and discussions for additional error analysis.

Thank you for your thoughtful comments, and thorough notes on how we can improve our writing. We will incorporate these suggested edits and upload them in our revisions. Below, we address each comment. Due to space constraints, we address questions in our following comment.

[W1, W2] Clarity of Methods Description in Sections 2-4

We provide these revisions in our updated draft. Below, we clarify a few of these points.

• Line 196: Our method uses expected answers from multiple human annotators to simulate each annotator’s interactions with the LLM and their preferences over different responses. Thus, we discuss other methods of using multiple human annotators in RLHF training here. These methods do not simulate interactions from these multiple human judgments.
• Line 208: We clarify that these answer sets are sourced from multiple human annotators. We also provide a reference to our dataset descriptions (Section 4: Experiments) where the annotation setups are described in detail.

[W2] Grammatical issues

Thank you for identifying these errors! We have revised each in our updated draft.

[W3] Clarifying Questions versus Multi-Answer responses

We agree the task of determining when Clarifying Questions versus Multi-Answer responses are appropriate is an important line of inquiry. There have been some prior efforts [1] demonstrating that users prefer systems to ask Clarifying Questions responses over Multi-Answer responses about twice as frequently; however, there has been limited work analyzing when this is the case or building systems to determine which response is more appropriate. Ultimately, we agree that systems should be able to predict either response type based on what is most appropriate for the specific setting, user, and query. While this is an important line of inquiry for future work, here we focus solely on the task of generating clarifying questions.

[1] Asking Clarification Questions to Handle Ambiguity in Open-Domain QA

• Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwanhee Lee, Joonsuk Park, Sang-Woo Lee, Kyomin Jung
• EMNLP 2023 Findings

[W4] No experiment involving training a model directly on multi-turn QA dialogues. Instead, the authors only experiment with combining the weights of two separate models.

As we note in Section 5, model merging is an efficient alternative to training an additional model from scratch on the full multi-turn dialogues, and we expect only a minor gain in performance with training another model rather than merging (Wortsman et al., 2022). We also note recent works that propose methods that could yield even greater performance gains than training from scratch over the full dialogues. We chose to use parameter merging in our experiments for the following two reasons: We expect such improvements from training from scratch or using mixture-of-experts models to be orthogonal to the methods proposed in this work for assigning preferences over clarifying questions. The results from parameter merging are sufficient for demonstrating that performance gains over the Direct-Answer SFT baseline are not simply due to the increased parameter count from using separate models for each conversation turn, as merging to a single joint model still outperforms these methods.

[W5] Evaluation Metric in Section 5.1

In Section 5.1, our Direct-Answer Accuracy metric directly evaluates whether systems are able to ask clarifying questions when they are necessary and directly answer user queries when they are not. This metric differs slightly from simply evaluating binary classification accuracy of determining whether or not the query is ambiguous, as clarifying questions often helps systems recover correct answers to unambiguous questions as well (as demonstrated in the our main results in Table 1). To account for this, Direct-Answer Accuracy also rewards systems for asking clarifying questions for unambiguous queries if it would otherwise get the question wrong. We include examples of such instances Table 12.

Thank you for your thoughtful comments. We address comments below and questions in the following commment.

[W1&2] Generalizability to multi-turn setting + Scope of experiments

We agree that extending our evaluation and methods to arbitrary conversation lengths is an exciting avenue for future work. We select this QA setting for two reasons: (1) QA is one of the most prevalent request types in real user-LLM interactions [1] and (Zhao et al. 2024) and (2) the scope of possible clarifying questions is unbounded (prior works, as we note in L107-110 in Section 2 and L498-510 have focused on grounded settings where possible ambiguities are identified within the provided context or fixed by the task.). This setting, however, often does not necessitate more than two conversation turns to resolve the ambiguity in the request (see response to Q3 below for an analysis of different ambiguity types). To the best of our knowledge, we are the first Future work may explore extending our settings requiring more conversation turns.

[1] LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

• Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang
• Arxiv 2023

[W3] Limited Baselines

Thank you for your suggestions! In our general response, we add such baselines including the PPDPP method from “Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents” you suggested and include it as an additional baseline. Overall, the new baseline achieves slightly lower performance than our proposed method. See the general response above for further discussion.

Regarding your comments on the low Direct-Answer Accuracy performance of clarify-or-answer systems, we discuss its weakness in Section 5.1. In our General Response above, we additionally include the binary classification accuracy comparing each system’s clarify vs. direct-answer prediction against the human-annotated ambiguous vs. unambiguous labels from AmbigQA. Note that PPDPP is the only method that directly utilizes human-annotated labels during training, and archives the best accuracy using this metric. Furthermore, note that this performance does not directly correlate to performance on our end-task F1 metric, while our original Direct-Answer Accuracy metric does. Our Direct-Answer Accuracy metric is slightly different than ambiguity detection, as it directly measures whether systems are predicting to ask clarifying questions where they will improve the model’s response, also considering the cases where (1) asking a clarifying question for an unambiguous question improves the model’s response and (2) asking a clarifying question for an ambiguous question does not improve the model’s response. As we note in our introduction and results sections, our main results in Table 1 demonstrate that clarifying questions frequently improve model performance on ambiguous questions, which points toward discrepancy in what humans and models often identify as ambiguous.

Thank you for your thoughtful remarks! We address your comments individually below.

[W1] Additional Baselines: Thank you for your suggestions! Please see the general response where we add and discuss several additional baselines.

[W2] Section 5.1 (When to Clarify) Accuracy Metric: In Section 5.1, our metric for Direct Answer accuracy (DA Acc) is intended to measure exactly this: how often are clarifying questions asked when they help LLMs recover more correct answers to the question. We chose to evaluate this accuracy metric rather than accuracy on identifying ambiguous/unambiguous questions because, as our main results in Table 1 demonstrate, clarifying questions not only help LLMs recover correct answers to ambiguous questions, but also for unambiguous questions as well. As such, this accuracy metric has the additional benefit of also rewarding models for asking questions in response to unambiguous questions if it helps the model recover the correct answer. We include examples of such questions in Table 12.

[Q1] Would it be helpful & possible to include some more informative comparisons and evaluation metrics (see "Weaknesses" above)?

See our discussions of weakness above where we address adding such additional comparisons.

[Q2] Human Annotation Details and Reliability All annotations (both identifying ambiguous questions, and identifying the set of answers from different interpretations) were collected by the authors of the original AmbigQA work. We state that the annotations are sourced from the original datasets in Section 4, and clarify this in earlier mentions.

[Q3] Table 2 doesn't say what metric it uses - presumably F1? Yes, we use Answer F1. We added this to the caption of Table 2.

[Q4] Typos We have fixed the typos in the updated manuscript. Thank you for catching them.

Additional Related Works: Reviewers suggested several other methods. Below, we include discussions for several of these methods, and discuss their similarities.

• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training, ArXiv 2024 Suggested by vGfa. While this method’s training approach similarly simulates conversation and to rank responses for DPO training, their proposed method relies on access to gold conversation trajectories during RLHF training and evaluation. Such trajectories are not available in the NaturalQuestions dataset.

• STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents ACL Findings 2024, Asking Clarification Questions to Handle Ambiguity in Open-Domain QA EMNLP Findings 2023 Suggested by vGfa and kypX. These two methods are designed for the RAG setting where each question is also supplemented with a set of retrieved documents paired with their retrieval/relevance scores. In this work, we focus on the closed-book setting.

• Self-Evaluation Guided Beam Search for Reasoning NeurIPS 2023. Suggested by kypX. This work presents a decoding method to achieve better accuracy and calibration with Chain-of-Thought prompting. While it can allow for models to consider different intermediate / chain-of-thought-style questions, it doesn’t enable interaction.

• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations SIGIR 2019 Suggested by kypX. This early work explores the task of asking clarifying questions in similar open-domain clarifying question settings. The methods in this work, however, rely on selecting from a fixed pool of existing clarifying questions, severely limiting their applicability.