Parrot: Enhancing Multi-Turn Chat Models by Learning to Ask Questions

We sincerely thank the reviewer for the valuable comments. We hope our rebuttal will address the concerns of the reviewer.

Q1: Contributions.

Although intuitive, previous works have not provided quantitative evidence to support or verify that chat models' ability to follow multi-turn instructions needs high-quality multi-turn data. Existing evaluation benchmarks like Alpaca Eval only cover single-turn instructions, and MT-Bench is limited to two turns. To the best of our knowledge, we are the first to quantitatively assess the multi-turn capabilities of current chat models and to reveal the gaps between them and GPT-4, ChatGPT. We think this should be considered a contribution.

Q2: Values of asking model.

The evolution of asking models and chat models should progress in parallel, akin to the interplay between the spear and the shield. An asking model that can provide diverse, in-depth, and extensive questions is indeed critical for effectively evaluating chat models, as it challenges them to prove their ability to handle a wide range of topics and ensures that issues of safety are thoroughly tested. A strong asking model can drive the progress of chat models by expanding their capability boundaries and revealing potential areas for improvement. Therefore, it is imperative for the academic community to explore building more powerful asking models that can drive the progression of chat models towards higher levels of reliability, safety, and versatility.

In this paper, we propose training an asking model and exploring its application in enhancing the multi-turn capabilities of chat models. Our experimental results reveal that an asking model is capable of generating human-like questions, which can help improve a chat model's multi-turn abilities. Furthermore, we believe that training better asking models will have even more applications in the future.

Q3: Using ShareGPT.

We would like to clarify that the purpose of extending dialogues from ShareGPT is not merely to increase their length, but to enhance their depth and topic diversity. Dialogues in real life do not have predetermined endpoints. In real-life communications, it is common for users to not obtain the information or answers they need in just one round of conversation. They may delve deeper into a topic or transition to related new topics.

Our method, therefore, employs an asking model specifically trained to generate contextually appropriate and topically relevant questions, thereby ensuring a natural progression of the conversation. When provided with short dialogues from ShareGPT, our asking model is capable of introducing new questions that deepen the existing topic or smoothly transition to a new related one. This is in contrast to the approach of simply concatenating short dialogue sessions, which often results in incoherence and logical inconsistencies [R1]. Our data collection method, therefore, not only increases the length of the dialogues but also improves their depth, and the naturalness of topic transitions.

Furthermore, the quality of the data collected through our method is tailored to meet the complex requirements of users in multi-turn dialogue scenarios. The resulting data is not just longer; it is richer and more representative of genuine human conversation patterns, enabling the trained model to better understand and respond to users' needs. To vividly demonstrate the effectiveness of our asking model, we have included additional examples from our dataset in the appendix (Section G).

[R1] Re3Dial: Retrieve, Reorganize and Rescale Conversations for Long-Turn Open-Domain Dialogue Pre-training, EMNLP 2023.

We sincerely thank the reviewer for the valuable comments. We hope our rebuttal will address the concerns of the reviewer.

Q1: Novelty.

Our work is different from previous works that train models with ChatGPT-generated data. We clarify our novelty and significance to the community in the following aspects:

1. we analyze the issues of existing multi-turn datasets such as non-human-like instructions, less detailed responses, short sessions, or limited topic transitions.
2. to overcome these issues, we propose to train an asking model that can simulate user questions and then collect a new multi-turn dataset. The experimental results demonstrate that our dataset surpasses existing datasets in terms of topic diversity, number of turns, and human-likeness.
3. we introduce the new MT-Bench++ benchmark, designed to quantitatively evaluate a model's multi-turn performance. By utilizing our Parrot dataset with more dialogue turns and higher quality, we have enhanced the multi-turn capability of a chat model based on LLaMA-2.

Q2: Values of our work.

As academic researchers, our primary focus is know-how, e.g., how to mitigate the gap between open source models and the state-of-the-art API models (e.g., GPT-4 and ChatGPT) in multi-turn capability. For this, we indicate that existing multi-turn datasets have some issues and propose to train an asking model to help collecting a multi-turn dataset with superior quality. Our model trained with our dataset show significant improvement on multi-turn capability compared with models trained on ShareGPT, Baize or UltraChat. Our dataset is fully available for faciliting future academic research, akin to UltraChat and Baize.

Moreover, we believe our proposed method of training a human-like asking model can be applied to generate multi-turn data if using an alternative model that permit commercial usage when necessary. For companies, even in cases where manual annotation is involved in collecting multi-turn data, our asking model can be utilized to enrich topic diversity and prolong the duration of conversation. This ensures that annotators refrain from prematurely ending dialogues or encountering limited topic transitions.

Q3: Revealing how to collect high-quality dataset.

We have delved deeper into the issues present in current multi-turn datasets, such as non-human instructions, less detailed responses, short sessions, or limited topic transitions. Our approach effectively overcomes these issues by employing a specially trained 'asking model' that can promote richer dialogues. This model is designed to simulate user asking, which in turn prompts more human-like questions, longer sessions and broader topics. Through the improved performance of the chat model, manual evaluation and qualitative examples, it is proved that our method can collect higher-quality multi-turn data than existing methods.

We sincerely thank the reviewer for the valuable comments. We are encouraged by the recognition given to our work. We hope our rebuttal will address the concerns of the reviewer.

Q1: Details of human evaluation.

We utilize the first-turn questions from MT-Bench because it widely covers eight domains, then we employ Parrot-Ask and ChatGPT iterative self-chatting methods (e.g., UltraChat) to collect dialogues of ten-turns, each method generating 800 questions in total. Three annotators are asked to evaluate the quality of these generated questions following these criteria:

• Repetitiveness: Questions should not be repetitive, ensuring each is unique and has some contributions to the conversation.
• Conciseness: Questions should be free of unnecessary verbosity, maintaining clarity and brevity.
• Politeness Patterns: Questions should avoid excessive politeness or expressions of gratitude that are not typically used in natural human questioning.
• Relevance: Questions should be directly related to the previous context or responses, showing a clear understanding of the conversation flow.

The result indicates that 81.8% of the questions generated by our Parrot method are human-like and high-quality, while only 36.8% for the iterative self-chatting method. The Kappa among annotators is 0.72, which reaches a high significant level.

For ChatGPT self-chatting method (e.g., Baize), we observed that 82.5% of the dialogues would end in a single turn, and fail to generate new turns of dialogues. Therefore, we did not include this method for human evaluation. We will add these important details in the final version of our paper.

Q2: Proposed prompts.

We use the following prompts for the Parrot-Ask model: "The following is a conversation between a user and an AI assistant. User statements start with [USER] and AI assistant statements start with [ASSISTANT]. You need to tell me what the user is likely to ask in each round of the conversation." We have updated an appendix to include this important information (Section B). Please refer to the supplementary material.

Q3: Follow-up questions in MT-Bench++.

We decide the follow-up questions in MT-Bench++ according to the following standards:

1. The question should be challenging and require AI to perform complex reasoning or rely on wide knowledge to answer.
2. The question should be related to the previous text. Try to reference or imply the content of the previous dialogue in the question.
3. The questions in a session should include appropriate topic transitions.

We have updated an appendix to include this important information and provided more examples of MT-Bench++ (Section A), where we have underscored specific features of the questions such as references, topic transitions, and knowledge requirements. These features are highlighted in blue and accompanied by annotations for clarity. For further details, please refer to the supplementary material.

Q4: Missing supplementary materials.

We apologize for missing the supplementary materials. We have updated an appendix, which introduces more details of MT-Bench++ and evaluations, and more qualitative examples. Please refer to the supplementary material.

We sincerely thank the reviewer for the valuable comments. We are encouraged by the recognition given to our work. We hope our rebuttal will address the concerns of the reviewer.

Q1: Novelty.

We clarify our novelty and significance to the community in the following aspects:

1. we analyze the issues of existing multi-turn datasets such as non-human-like instructions, less detailed responses, short sessions, or limited topic transitions.
2. to overcome these issues, we propose to train an asking model that can simulate user questions and then collect a new multi-turn dataset. The experimental results demonstrate that our dataset surpasses existing datasets in terms of topic diversity, number of turns, and human-likeness.
3. we introduce the new MT-Bench++ benchmark, designed to quantitatively evaluate a model's multi-turn performance. By utilizing our Parrot dataset with more dialogue turns and higher quality, we have enhanced the multi-turn capability of a chat model based on LLaMA-2.

Q2: Long-context models.

We fully recognize that long-context models require long conversations for alignment. However, due to limitations of ChatGPT, such as the 16K token support, as well as the high costs associated with collecting very long dialogues, as an initial research exploration, we have constructed dialogues of up to ten turns, most of which are around 4K in length. We have demonstrated the effectiveness of our methodology for long conversations through sufficient experiments. Given the emergence of more powerful models like GPT-4-Turbo-128K, we believe our methods will be applicable for the alignment of long-context models, and we plan to explore this in the future.

Q3: Out-of-domain performance.

MT-Bench has questions across 8 domains. In Table R1, we compare our proposed Parrot-Chat model with OpenAI GPTs and several open-source models across these domains. Although there remains a significant gap compared to GPT-4 overall and across individual domains, our Parrot-Chat-13B model outperforms GPT-3.5-Turbo in the roleplay and humanities domains. Our model also performs the best among all open-source models in 6 out of 8 domains and the second best in the other two domains. In addition, compared to GPT models, big gaps occur in math, coding, and extraction domains. To further enhance performance, we plan to enrich our dataset with more examples in these domains.

Table R1. Evaluation results of our proposed and baseline chat models in different domains on MT-Bench. We show the average scores in Overall column. Among those open source models, we make the best performance in bold and the second one underlined.

Q4: Qualitative examples.

We have added an appendix containing qualitive examples, including the model generated responses (Section E) and examples of our dataset (Section G). Please refer to the supplementary material.

Q5: Hallucinations in data.

Our dataset occasionally exhibits instances of "hallucinations," where ChatGPT generates responses that contain information not aligned with reality. This phenomenon is somehow unavoidable for now, even in existing datasets such as ShareGPT and UltraChat.

More importantly, we have observed that even when ChatGPT generates hallucinatory responses, our Parrot-Ask model can identify mistakes and guide ChatGPT to produce correct responses in subsequent interactions. For example, when ChatGPT initially incorrectly stated that automating the Cisco AnyConnect Secure Mobile Client via a script was not possible, our Parrot-Ask model pointed out that it could be accomplished using C#. ChatGPT then revised its initial response and provided relevant examples, successfully correcting a hallucination. We have shown more such examples like this in Appendix F.

We find this discovery intriguing, as it suggests that our asking model may have the potential to correct model hallucinations through multi-turn interactions. This is a topic we intend to explore in further detail in the future.