Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training

Thank you so much for taking the time to review our paper, and for acknowledging the novelty of our work as well as the clarity of our paper! We appreciate your helpful feedback which has greatly helped to strengthen our paper, and we have done our best to respond to your questions below.

“The experiments focus on…”

”a particular skill which larger language model already posses.”

R: Thank you for raising this point! While large LLMs have excelled at many difficult tasks, we would like to clarify that unfortunately, even larger LLMs do not currently possess the ability by default to proactively lead conversations by asking clarification questions. This type of skill is important to learn in adaptation to downstream tasks, where interactivity may be more desirable (e.g. specialized applications such as building a dialogue system for customer support; coding assistants), and we observe ACT is more effective for downstream task adaptation than generic strategies such as SFT or in-context learning.

In our Action F1 results in Table 1 of our paper (we note that there is high inter-annotator agreement on PACIFIC, indicating ambiguity here is mostly objective), and we have now run additional benchmarking experiments with zero-shot dialogue context as input at inference time and longer adapter tuning on Gemini to further evaluate this ability:

Prompting Gemini zero-shot with specific instructions for disambiguation only achieves an Action F1 of 58.1. Gemini Pro achieves an Action F1 of 81.4 using 10 full in-context conversation examples at inference time. Implicit ambiguity recognition performance can be improved further with Gemini using 250 samples for SFT, reaching 88.0 F1 without any additional inference-time cost.

We have additionally run experiments with 10-shot in-context learning on a smaller LLM, Zephyr 7B, finding that the model only achieves 55.4 Action F1. We find that with 250 conversations for tuning, Zephyr tuned with ACT reaches as high as 89.6 Action F1, outperforming Gemini tuned with SFT.

Overall, with Zephyr, we consistently observe the performance ranking as ACT > SFT > few-shot ICL. With Gemini, there is not a commercial interface that allows us to run ACT, but we see the potential of SFT is higher than few-shot ICL. Overall, we find that specialized tuning approaches may have a higher ceiling for effective LLM customization than generic prompting approaches as they are better aligned with downstream tasks. This has the potential to greatly reduce inference time latency/complexity (e.g., reducing the need for many-shot prompting).

”a specific sub-domain of dialogue systems (SQL-based)”

R: We thank you for your valuable suggestion on making our paper more impactful on various domains! First and foremost, we do agree with you that a method is more useful when it can be applied to various domains. Following this reason, our work has been evaluated on a variety of domains covering finance, news articles, childrens stories, etc., and feature complex reasoning subtasks over both structured tables and natural language documents. Our work is not restricted to SQL-specific subdomains. Please refer to Table 1, which demonstrates our experimental results on PACIFIC which mixes structured tables with natural language, and Table 2, our Abg-CoQA results which involve reasoning about natural language documents.

Secondly, we would also like to mention that text-to-SQL is an important task which has received much attention in recent years both in academia (e.g. [1,2,3,4,5]) and industry (e.g. [6,7,8,9]). Our SQL-based conversational question answering results are only found in Table 3.

We have also run limited experiments on the planning task of another general goal-oriented dialogue corpus [10] to demonstrate that our approach can generalize to interactive settings with non-conversational actions like tool use, featuring a non-binary action space. The results below highlight that our method achieves a higher average reward even on this action optimization task that does not contain explicit clarification labels in the data.

”I found the sections on "conversations in the wild" difficult to follow and disconnected from the rest of the text. … what specific problem you are trying to address?”

”L425-427: ACT does better than frontier LLMs when it has 250 examples to learn from. I assume you don't need human labeling.”

R: To clarify, ACT is a data-efficient tuning framework, and our main experiments use ground-truth action labels, which is commonly the case with academic datasets. Each dataset has dialogue turns which are labeled with whether they are ambiguous, which is the basis of “optimal”/“sub-optimal” actions. Our in-the-wild experiments refer to scenarios where these labels are not directly available. In Sec5.4 we have discussed this point, and Tab4 demonstrates our methods are effective in scenarios without label supervision.

”if I understand correctly, the success of the method does depend on prior model alignment)”

R: The method does not rely on prior alignment, but intuitively, prior alignment provides a stronger initialization.

“why not try with more examples than 250?”

R: For fixed tasks where you may expect your training data to cover the target distribution, it is possible to achieve strong performance with standard SFT. [1] finds most of the performance gains come from SFT and there is limited improvement even from on-policy DPO, and [2] finds SFT on high quality data can outperform RLAIF. [3] finds that SFT with hindsight relabeling can outperform many offline RL techniques. The exploration of on-policy tuning strategies like ACT shine especially in the setting where there is insufficient training data to cover the target task. Given these previous findings, we decided to explore customized tuning approaches for the low-data region.

[1] LIONs: An Empirically Optimized Approach to Align Language Models, EMNLP 2024

[2] A Critical Evaluation of AI Feedback for Aligning Large Language Models, NeurIPS 2024

[3] RvS: What is Essential for Offline RL via Supervised Learning? At ICLR 2022

”Algorithm 2 you say (line 14-16) that the trajectory is assigned to the winning or losing labels--but in the text it is just the sampled y_j that is assigned to these”

R: Thank you for pointing out this typo. As in Algorithm 2, y_{lj} is set to the incorrect trajectory. The trajectory contains the responses from the original sampled response y_j leading to the incorrect outcome g_j’. We have fixed this in the revised manuscript (L291-294).

”Line 304: How do you guarantee that the responses from Gemini Ultra model M should be rejected? Won't it often come up with a good response?”

R: In Alg1, L3 says that we infer a ground truth pragmatic action (a_i), L4 states that we determine a conjugate rejected action (a_i’), and in L6 we prompt an LLM conditioned on the rejected action (a_i’).

”Line 354: Why not stick with F1, even though AmbigSQL is balanced?”

R: In our updated experiments (which include an Iterative Reasoning Preference Optimization baseline), we have added Macro F1.

”Line 448: It seems odd to me ACT outperforms Zephyr SFT on overall trajectory outcome but not action recognition. Does the A model show bias to asking questions?”

R: The Action Classifier does not bias towards clarifying questions. It classifies an utterance to compare it against a ground-truth label rather than predicting one. In Tab 1-3, ACT does outperform Zephyr SFT on action recognition in nearly every condition. Even when SFT outperforms ACT in terms of action recognition (in Table 2), ACT outperforms SFT in multi-turn performance. This indicates that ACT is able to better teach models how to reason about its own clarifying questions.

In Appendix A.1 of our revised paper, we have now noted that action recognition performance can be impacted by label noise (Abg-CoQA in particular has relatively low inter-annotator agreement on ambiguity compared to PACIFIC). Trajectory outcome performance is less dependent on labels and is improved by online sampling and multi-turn optimization. Our results indicate ACT is consistently able to improve upon baselines on multi-turn goal completion.

We thank you again for your very thoughtful review and helpful feedback!

Thank you for your careful review and helpful feedback! We appreciate you acknowledging the soundness of our work with extensive experiments, the impact of our demonstration of the limitations of off-policy learning, and the novelty of our text-to-SQL dataset. We are grateful for your helpful suggestions which has greatly helped to strengthen our paper, and we have responded to each of them below.

”The paper is…”

”probably over-stuffed with results”

”generally very wordy making it difficult to read. It is saved by the clear algorithms and figures, but the text should be shortened considerably. This would also allow for more discussion of the AmbigSQL data set, which is a significant contribution”

”You could also omit most or all of the DPO-Dist results from the main text--it never gets close to the best results”

”L 527: this paragraph can just be a single sentence--just tell us prior alignment was helpful and how much, don't need to discuss more.”

R: We appreciate your very helpful suggestions! We have condensed the writing in the paper, and we have reorganized our results, as follows:

1. Our supplementary baselines (e.g. DPO-Dist, prompting variants) have been moved to our Additional Experiments Section (Appendix Tables A7-9).

2. We have shortened our discussion of the ablations.

3. We have shortened the description of Implicit Ambiguity Recognition in Sec 4.2.

4. We have shortened the description of Agent task performance in Sec 4.2

5. We have merged our initial Qualitative Analysis section with the extended section in the Appendix (see Appendix Section I).

6. We have added a discussion on user simulators and action classifiers to the main text.

7. We have added more details on AmbigSQL in Sec 4.1.3 of the revised manuscript.

We believe your suggestions have greatly helped to improve the presentation of our paper.

”The AmbigSQL data might be too easy to learn, as some models achieve near perfect performance on it.”

R: It is true that with sufficient fine-tuning data, models are able to achieve near perfect performance on the implicit ambiguity recognition task on AmbigSQL. This is somewhat expected due to the three systematic perturbation patterns we detail in Appendix B.1. However, we believe that the value in AmbigSQL is the complexity of the multi-turn text-to-SQL task. Table A11 demonstrates that it is not possible to achieve high performance on the task without asking clarifying questions, and despite the promise of using tuning to learn the question-asking pattern in the data, Table 3 shows that Gemini struggles to learn question-asking using only limited in-context examples. Smaller models also struggle greatly on the SQL generation task.

”One thing you could do after shortening the rest of the paper is to consider the use of user simulators more, and find parallels between earlier work and yours. I think the novelty of the paper lies mostly in the exact implementation of data generation for preference optimization, summarized in Algorithm 1 and Figure 3, and it would be good to focus on that specifically.”

”I would like to see details of the user simulator U and action classifier A in the main text.”

”there wasn't much discussion of the limitations of user simulations.”

R: We appreciate your suggestions on simulation. We have clarified the description of our approach and added the following discussion of user simulators to Section 3.1:

“Works such as Deng et al. (2023c); Yu et al. (2023) directly prompt LLMs for goal-oriented tasks conditioned on dialogue context and task objectives. Our implementation ofUis inspired by this setup. We first prompt an LLM to summarize the user’s information-seeking goal. Then, we form another prompt using this summary along with the current dialogue context to simulate a user response. Prompting with this goal summary allows for more flexibility than directly providing the user simulator with the ground truth information objective. We provide details on our implementation of U in Appendix H.”

We have also added a discussion on Action Classifiers in the same section.

“Figure 3 was critical to my understanding of your approach. It would help the text to label figure 3 with the notation used in the text”

R: Thanks for your helpful suggestion! Our revised paper contains an updated Figure 3 with the notation (temporarily written in red) in Section 3.1. Additionally, we have updated the iterator notation used in Algorithm 2 and Section 3.2.2 for clarity and consistency with the Figure. We hope that our changes help to create a balance between including a detailed explanation without overwhelming readers with too many details.

“this paper doesn't discuss DPO variants built for tasks that require multi-step trajectories. … I would suggest the authors check into DPOs for conversational tasks or reasoning tasks that require COT. “

R: Thank you very much for your suggestion. At the time of submission, we cited several recent works examining on-policy variants of DPO, and following your feedback and we have now added new citations to the following concurrent works:

[1] Iterative Reasoning Preference Optimization (NeurIPS 2024)

[2] Parrot: Enhancing multi-turn instruction following for large language models (ACL 2024)

[3] Self-Play Preference Optimization for Language Model Alignment (arXiv May 2024)

[4] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024)

Our related works section now includes this additional paragraph in Section 2.2: “Many of our contemporaries also question the limits of fully offline preference learning and have examined “online” variants of them (Guo et al., 2024; Xu et al., 2023; 2024b). Yuan et al. (2024) proposes iterative DPO, and Chen et al. (2024) proposes a variant where ground-truth responses are considered winning, and responses sampled from the previous iteration of a policy model are considered losing. Pang et al. (2024) applies a variant of iterative DPO to optimize externalized reasoning chains. Our work differs from these in that we are proposing a novel approach to customize LLMs for specific conversational settings. While other works look at customizing preference data for DPO for conversations in general (e.g. Sun et al. (2024), the focus is still on single-turn response optimization. ACT considers multi-turn trajectories for preference optimization, and the first paper to consider contrastive learning on the basis of conversational actions.”

If you have any suggestions for other more related conversational DPO works to discuss in our paper we would be glad to reference them!

We really appreciate your helpful feedback and we hope that we have addressed each of your concerns.

Thank you for taking the time to carefully read our paper and provide helpful feedback. We appreciate your acknowledgement of the quality of our writing, as well as the extensiveness of our experiments across multiple domains. Below, we have responded to each of your helpful suggestions, which we believe has greatly helped to increase the quality of our paper.

“The biggest weakness of the paper should be the choice of weak baselines … One paper is https://arxiv.org/pdf/2404.19733. This paper has many identical settings: online sampling and a heuristic to filter trajectories. … the authors have similar ablation comparisons in Table 6. I would like to see some clear discussions”

R: Thank you for your helpful feedback! You are right that there are some other concurrent works which have introduced on-policy learning into offline DPO-type methods recently, including IRPO (which was recently accepted at NeurIPS). ACT fundamentally differs from existing on-policy DPO variants (e.g., IRPO) in two aspects.

First, ACT is designed for multi-turn conversation modeling specifically. While ACT includes online sampling combined with heuristic-based automatic evaluation, we believe our paper is one of the first works to consider multi-turn trajectories for preference optimization, and the first paper to consider contrastive learning on the basis of conversational actions. As you pointed out, we have ablated many of the components of ACT in Table 6. The ablation condition where we take away multi-turn simulation can be considered a special case of Online or Iterative DPO in which we have a special reward signal based on conversation actions. Our ablations found that perturbing the action signal greatly hurts performance across all metrics, and that removing multi-turn simulation greatly hurts clarification reasoning ability.

Second, recent works such as IRPO have considered improving reasoning by self-improving explicit reasoning by optimizing reasoning chains. Our problem setup (Sec4) aims to internalize the reasoning process through direct preference optimization. The current formulation of ACT bypasses explicit reasoning, attempting to encourage the model to generate action-optimal responses directly. Explicit reasoning optimization approaches like IRPO follow the recently popular literature on optimizing strategies which leverage inference-time computation. ACT is compatible with such reasoning approaches, but this was not the focus of our work.

We have included the above discussion in Section 2.2. Following your suggestion, we have also run new experiments comparing ACT against IRPO on PACIFIC and AmbigSQL, as IRPO has been demonstrated to achieve strong improvements in numerical reasoning tasks like GSM-8K. As a reward signal, we use the downstream task metrics (e.g. DROP F1) for each of our tasks.

We find the following results on PACIFIC:

We also obtain the following results on AmbigSQL:

We find that ACT outperforms IRPO tuning holistically, even though IRPO optimizes the use of extra inference-time compute. These findings are consistent with our Ablation studies in Table 5, which demonstrate that action-based optimization and multi-turn trajectories are crucial to conversational modeling. We have updated these results in the text in Tab. 1-3, and we have strengthened this discussion in Section 5.

Thank you for taking the time to review our paper, and for acknowledging the advantages of our multi-turn approach, our strong experimental results on a breadth of real-world tasks, and the clarity of our paper! We have addressed the two concerns you mentioned below.

”experiments on limited applications of dialogue tasks”

R: We would like to mention that dialogue tasks are a broad and useful domain which have received a lot of attention in the last ten years. We would also like to clarify that we do not only focus on a single sub-domain of dialogue systems. Table 1 demonstrates our experimental results on PACIFIC, which involves reasoning about structured tables as well as natural language paragraphs describing various financial information. Table 2 demonstrates our experimental results on Abg-CoQA, which only reasons about natural language documents. Table 3 provides results on AmbigSQL, which is our ambiguous text-to-SQL task. Overall, our proposed method is task agnostic, and it is designed to improve downstream LLMs in task-specific ways for multi-turn conversational modeling.

Although our focus is on ambiguous conversations, we also ran limited experiments on another planning task in a general goal-oriented interaction corpus [1] to demonstrate that our approach can also generalize to settings with non-conversational actions like tool use and a non-binary action space. The results below highlight that our method achieves a higher average reward even on this action optimization task that does not contain explicit clarification labels in the data.

[1] Decision-oriented dialogue for human-ai collaboration, TACL 2024 https://arxiv.org/abs/2305.20076

”there are also many variants for DPO based methods, however, the authors have not compared them as baselines.”

In our paper, we had originally considered a baseline in which we distill information from different LLMs using DPO, and we have provided many different experimental comparisons in the Additional Experiments in Appendix A.

Following Reviewer g3nF’s suggestion, we have now additionally added comparisons to Iterative Reasoning Preference Optimization, a concurrent work which was recently accepted at NeurIPS 2024 and has received much attention. We find that our work greatly outperforms IRPO by up to 21% in terms of implicit action recognition and 12% in terms of trajectory-level F1. We have pasted those results here as well:

PACIFIC:

AmbigSQL:

We see that ACT outperforms IRPO tuning holistically, even though IRPO optimizes the use of extra inference-time computation. These findings are consistent with our Ablation studies in Table 5, which demonstrate that action-based optimization and multi-turn trajectories are crucial to conversational modeling.

Thank you again for your helpful feedback, and we hope that we have addressed each of your concerns!