VoxDialogue: Can Spoken Dialogue Systems Understand Information Beyond Words?

Thank you for your recognition of our work and for acknowledging the value of the 12 acoustic factors in the field. Allow me to clarify some of your questions:

Q1: Original Metrics to Evaluate Dialogue Systems

A1: To better evaluate the ability of spoken dialogue systems to focus on each attribute, we have made certain improvements to the GPT-based metric. In previous studies[1,2], GPT-based evaluation metrics were often vague and lacked clear standards, resulting in assessments that did not adequately focus on specific acoustic information. Inspired by the way human MOS (Mean Opinion Score) evaluations require descriptive explanations for different score levels, our work introduced detailed scoring guidelines for GPT (as mentioned in the second paragraph of Section 4.2), enabling it to provide more systematic and attribute-specific evaluations.

Q2: Human Evaluation

A2: To ensure consistent evaluation standards across different researchers, we opted for the improved GPT-based evaluation method rather than human evaluation. Since human evaluators may carry personal preferences, human evaluation often carries a greater risk of inconsistent scoring, which can undermine the reliability of assessments. This decision was made to address potential reproducibility concerns.

To meet your expectations, we are currently conducting human evaluations of the experimental results and will present them before the discussion concludes. If you have any suggestions for further improving our benchmark evaluation system while ensuring standard reproducibility, we would greatly appreciate your insights and look forward to discussing them further.

Q3: Discussion on Gendered Titles and Potential Offense

A3: Thank you for your in-depth consideration of our dataset. Using gendered titles can indeed raise concerns about potential offense. However, it is important to note that spoken dialogue is a one-to-many problem, and our provided answers serve only as reference responses, not definitive ones. Beyond titles like Mr./Ms., the dataset includes many other gender-related information in the responses. As long as the other content aligns with the context, the performance metrics of the model should not be significantly impacted.

It is also worth noting that other widely used datasets, such as SpokenWoz[3], adopt similar approaches by including gendered titles like Mr./Ms. This type of bias has been present in many existing datasets and models, reflecting broader challenges in the field. However, we believe it is essential to separate the discussion of potential bias from the primary goal of evaluating acoustic attributes.

To date, there has been no systematic discussion of potential biases in spoken dialogue datasets, and many biases may remain subtle and difficult to detect. Therefore, it is currently impossible to guarantee that our dataset is entirely free from bias. However, we are committed to systematically reconstructing the dataset to ensure fairness once a comprehensive review of bias phenomena becomes available.

Additionally, as an evaluation dataset, our model does not participate in the training process, which minimizes the risk of propagating biases to new spoken dialogue models. While we recognize that this does not entirely eliminate the issue, it ensures that the potential impact is manageable at present. We remain open to further feedback and discussions on how to enhance fairness in our dataset and beyond.

Q4: How many languages are included?

A4: Currently, the dataset includes only two languages (Chinese and English), as these are the two primary languages used in most spoken dialogue models. Due to the significant variability in how different models perform with other languages, adding additional languages would require designing separate evaluation metrics for each language to effectively and fairly measure model performance on a per-language basis. This might be better suited as a separate study specifically aimed at evaluating models’ capabilities in handling multiple languages.

Q5: Other Issues

A5: We have added LeBenchmark in Table 1.

Thank you once again for your insightful discussion of our work. Your feedback has been invaluable, and we hope this response adequately addresses your concerns. If you have any further questions or suggestions, we would be delighted to engage in continued dialogue.

[1] Yang Q, Xu J, Liu W, et al. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension[J]. ACL2024

[2] Ao J, Wang Y, Tian X, et al. SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words[J]. NeurIPS2024

[3] Si S, Ma W, Gao H, et al. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents[J]. NeurIPS2023

Thank you for your recognition of our work and for acknowledging the importance of our contributions to understanding paralinguistic information. Allow me to clarify some of your questions:

Q1: Human Preference Evaluation

A1: Thank you for your suggestion. We have manually filtered the synthesized data to ensure it aligns with human conversational standards as much as possible. However, collecting real spoken dialogue data is particularly challenging, especially when constructing and simulating scenarios that emphasize various paralinguistic information, as highlighted in this paper. This scarcity of real dialogue data has made it difficult for us to find sufficient real dialogue data for conducting human preference comparison on all attributes.

Based on your advice, we conducted some evaluations on emotional data. Through an A/B test with DailyTalk[1] data, our data received 47% human approval, demonstrating that the authenticity of our data is reasonably assured to some extent.

Q2: Fidelity Processing

A2: Thank you for your suggestion. In the latest version, we have revised the description of the construction process for simulating poor audio fidelity:

For fidelity, according to the Nyquist-Shannon sampling theorem, the sampling rate must be at least twice the highest frequency of the signal to ensure lossless reconstruction. To capture frequencies up to 4 kHz, the minimum sampling rate should be 8 kHz. Therefore, we downsampled the speech to 4 kHz (to simulate the loss of speech signal and represent ‘poor’ audio quality, resulting in the loss of some speech information) and then resampled it back to 16 kHz to simulate poor audio fidelity.

Thank you for your valuable suggestions on our work. We have indeed put a lot of thought into constructing processes for each attribute to ensure the data reflects the corresponding acoustic properties. If you have any further writing suggestions or find any ambiguous points in our explanations, we look forward to further discussion.

Q3: Spoken Dialogue is a One-to-Many Problem—How to Evaluate Reasonably?

A3: Yes, spoken dialogue is a highly open-ended process where multiple responses can be considered correct. To make our evaluation system more rigorous, we referred to evaluation approaches from previous works [2,3] and chose to evaluate only the final response in multi-turn dialogue data. Under this constraint, the response must align with the preceding context, thereby limiting the openness of possible answers and enabling more effective evaluation.

Q4: How can the dataset be extended to include more languages?

A4: As long as corresponding TTS models are available, language-specific evaluation benchmarks can be generated. However, due to the significant variability in how different models perform with other languages, adding additional languages would require designing separate evaluation metrics for each language to effectively and fairly measure model performance on a per-language basis. This approach would ensure that each language is evaluated comprehensively and accurately.

Thank you once again for your insightful discussion of our work. Your feedback has been invaluable, and we hope this response addresses your concerns. Should you have any further questions, we are always open to continued dialogue.

[1] Li Y, Su H, Shen X, et al. Dailydialog: A manually labelled multi-turn dialogue dataset[J]. ACL 2017

[2] Lin G T, Chiang C H, Lee H. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations[J]. ACL2024

[3] Ao J, Wang Y, Tian X, et al. SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words[J]. NeurIPS2024

Q4: Why are audio and music considered attributes? (weakness 6)

A4: We refer to audio and music as attributes because they are forms of acoustic information that contribute to spoken dialogue beyond words. Although not properties of the spoken language itself, they can influence spoken dialogue as environment information. We apologize if this terminology was misleading, and we will revise it to describe these as environmental attributes. We welcome your suggestions on further improvements.

Q5: What does ‘meeting the desired expectations’ mean? (weakness 6)

A5: As shown in Table 3, each speech instance in the spoken dialogue dataset has corresponding attribute labels. For attributes such as volume, fidelity, audio events, and music, it is not possible to directly use TTS models to generate audio that perfectly matches the labels. Therefore, additional post-processing is required to meet these expectations. For example, for samples labeled as “Low Fidelity,” we must transform normal fidelity audio into low fidelity through specific processing methods.

Additionally, we have clarified the processing of music in the latest version as follows:

“For music, we randomly spliced it before the first speech segment or set it to play in a loop as background sound.”

Q6: Why only evaluate the final response? (weakness 7)

A6: Understanding context is crucial for spoken dialogue systems. A short context sequence may simplify the task into a basic QA exercise, which does not require substantial contextual understanding. We consistently use the final response for evaluation to highlight the model’s capability to comprehend and utilize context effectively.

Additionally, spoken dialogue is a highly open-ended process where multiple responses can be considered correct. To make our evaluation system more rigorous, we referred to evaluation approaches from previous works [1,2] and chose to evaluate only the final response in multi-turn dialogue data. Under this constraint, the response must align with the preceding context, thereby limiting the openness of possible answers and enabling more effective evaluation.

Q7: ASR-based vs. spoken dialogue models (weakness 8)

A7: In our experiments, it is important to note that spoken dialogue models did not outperform ASR-based systems across all metrics. ASR-based systems demonstrated robust context understanding capabilities, surpassing direct spoken dialogue models in certain attributes. Although ASR-based models may have limitations in understanding acoustic information, comparing them provides a valuable performance reference, representing the upper bound performance without the integration of acoustic information.

Q8: Writing Issues (weakness 3,4)

A8: We have made the following improvements based on your suggestions:

• We have revised the related works section to emphasize the importance of acoustic information in spoken dialogue models, especially at the end of Section 2.1.

• Additionally, we have included more detailed descriptions of synthesized dialogue scripts and updated the explanation for the processing of music.

• Lastly, we have removed the periods from the section titles in 2.1 and 2.2 to ensure consistency in formatting.

Thank you for your attention and suggestions; they have enhanced the quality of our paper. We hope the updated version meets your expectations. If you have any additional feedback or questions, please feel free to reach out.

[1] Ao J, Wang Y, Tian X, et al. SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words[J]. NuerIPS2024

[2] Lin G T, Chiang C H, Lee H. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations[J]. ACL2024

[3] Chu Y, Xu J, Zhou X, et al. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models[J]. arXiv2023

Thank you for your recognition of the significance of the 12 key acoustic information aspects of our work and its overall value. Allow me to clarify your questions and address any concerns:

Q1: What is ‘most appropriate’ spoken dialogue synthesis pipeline for each attribute? (weakness 1, 2, 6)

A1: We noted your comment: “If another work also considered paralinguistic information for their data synthesis process, why is your approach more ‘appropriate’ than theirs?” We believe there may have been some misunderstanding. By ‘most appropriate,’ we mean that different attributes require distinct synthesis methods tailored specifically for their needs. This does not imply that our method is inherently more suitable than those proposed in other works.

That said, we have indeed introduced new pipelines for certain acoustic attributes that have not been previously explored. These novel methods help address existing gaps in the field and offer fresh perspectives on the synthesis and control of various acoustic features in spoken dialogue. In the case of age, for instance, there is currently no TTS model capable of directly controlling vocal age. To address this, we use reference voices of different ages and apply zero-shot TTS models to achieve age-specific control. As detailed in Section 3.2, Stages 2 and 4 of our work, we outline the synthesis requirements for different attributes and propose five distinct synthesis methods using various TTS models to meet these needs effectively.

Previous works [1,2] have made similar attempts but with notable limitations. For instance, SD-eval [1] used zero-shot TTS to generate adult voice data but did not achieve synthesis for children or elderly voices. StyleTalk [2] focused on emotional spoken dialogue synthesis but left many other attributes unexplored. To the best of our knowledge, our work is the first to comprehensively explore 12 different acoustic attributes in spoken dialogue, including several attributes (e.g., language, volume, speed, fidelity, stress, and non-verbal expressions) that have not been systematically studied in spoken dialogue tasks. As noted by other reviewers:

It is creative to use GenAI to create spoken data for target dimensions related to Speech.

We hope that our exploration of these diverse attributes will enhance the ability of spoken dialogue models to comprehensively understand and generate spoken dialogue across multiple dimensions.

Q2: Difference between audio-language models and spoken dialogue models (weakness 3)

A2: As described in Qwen-Audio [3]:

“Qwen-Audio, a fundamental multi-task audio-language model that supports various tasks, languages, and audio types, serving as a universal audio understanding model.”

“Building upon Qwen-Audio, we develop Qwen-Audio-Chat through instruction fine-tuning, enabling multi-turn dialogues and supporting diverse audio-oriented scenarios.”

We consider an audio-language model to be a foundational model primarily focused on audio understanding, whereas spoken dialogue models build upon audio-language models and are specifically designed for multi-turn dialogue interactions. We apologize for any confusion this may have caused. Due to the frequent overlap between these models as different versions within the same project, we have included clarifications in the latest version to clearly distinguish these model types and their respective purposes.

Q3: Explanation of examples (weakness 5)

A3: When using ChatGPT-4o to generate spoken dialogue data, it was observed that the model, drawing on common market research, made the following recommendations based on shopping habits across different genders and ages:

• Men typically prioritize features such as suitability for business travel and long battery life when purchasing smartphones.
• Women often pay more attention to the aesthetics, tactile feel, and camera performance of smartphones.
• Young people tend to prefer foods with stronger flavors, such as fruit juices and beef burgers.
• Elderly individuals are more inclined to choose foods that are easy to digest and nutritious.

These recommendations are based solely on general trends from market research and statistical data, aimed at generating suggestions without bias. We believe that detailed user information could further rationalize these recommendations; however, this goes beyond the primary focus of our work, which centers on age, gender, and other acoustic information. We have conducted manual checks to ensure the data does not contain strong group discrimination.

Thank you for your high evaluation of our work and for recognizing the value of the 12 acoustic attributes in spoken dialogue systems, as well as the significance of our proposed methods for constructing spoken dialogue data tailored to these attributes. Allow me to share the updates we made based on your review:

• In Table 5, we have added the specific values for each metric in Figure 2 to provide clearer references.
• In Table 3, avg represents the average number of dialogue turns, and dur indicates the total duration of the audio files (in hours). We have clarified these definitions in the table caption for better understanding.
• Following your suggestion, we included the distribution of dialogue durations in Figure 1c, visually illustrating the length distribution of dialogue.

Thank you again for your valuable feedback and for recognizing the contributions of our work. Your suggestions have helped us refine and improve our paper further. We hope these updates address your concerns. If you have any additional questions or suggestions, please feel free to reach out to us.

Thank you for your positive score and thoughtful insights on our work, recognizing the value of our dataset. Allow me to address the questions you raised and discuss our considerations.

Q1: Discussion on Task-Oriented Dialogue

A1: We fully agree with your perspective on the significance of task-oriented dialogue. Initially, we had ever considered constructing our dataset based on task-oriented dialogues (such as MultiWoz [1] and SpokenWoz [2]). However, after careful deliberation, we decided to put this in the next stage of our research. The current spoken dialogue tasks can be categorized into daily dialogues (e.g., Daily Dialogue [3], MELD [4], focusing on emotion and content comprehension) and task-oriented dialogues (e.g., MultiWoz, SpokenWoz, emphasizing task objectives and dialogue state tracking). However, we realized that these task-oriented dialogues often prioritize content information (the main consultation content) and contextual memory (user-specific information), rather than the acoustic features emphasized in this paper. As a result, we opted for daily dialogue scenarios, which are more sensitive to acoustic nuances.

However, your insightful example has underscored the importance of acoustic information in real-world scenarios. Following your advice, we will explore whether spoken dialogue models can more effectively handle task-oriented dialogues while leveraging acoustic features in our future work. Thank you for your valuable suggestion.

Q2: FunAudioLLM vs. Spoken Dialogue Models

A2: FunAudioLLM processes speech by converting it into text using ASR, which is then passed into a large language model (LLM) for inference. During training, LLMs are likely exposed to extensive textual dialogue data, enhancing their ability to infer attributes such as age, gender, timbre, and language based on context on dialogue tasks. In contrast, as noted by Tang et al. [5], task overfitting remains a significant challenge in multi-modal LLM training. Current audio-language models primarily focus on audio comprehension and often lack training data formatted for spoken dialogues, which hampers their performance in dialogue-specific tasks.

This discrepancy explains why spoken dialogue models currently underperform compared to FunAudioLLM in dialogue tasks. A viable solution would be to leverage synthetic data to strengthen the dialogue capabilities of these models. Our proposed spoken dialogue synthesis pipeline is well-positioned to support such efforts by providing diverse and attribute-specific synthetic dialogue data.

Thank you once again for your insightful discussion of our work. Your feedback has been invaluable, and we hope this response addresses your concerns. Should you have any further questions, we are always open to continued dialogue.

[1] Si S, Ma W, Gao H, et al. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents[J]. NeurIPS 2023

[2] Budzianowski P, Wen T H, Tseng B H, et al. Multiwoz--a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling[J]. ACL 2018

[3] Li Y, Su H, Shen X, et al. Dailydialog: A manually labelled multi-turn dialogue dataset[J]. ACL 2017

[4] Poria S, Hazarika D, Majumder N, et al. Meld: A multimodal multi-party dataset for emotion recognition in conversations[J]. ACL 2019

[5] Tang C, Yu W, Sun G, et al. Salmonn: Towards generic hearing abilities for large language models. ICLR 2024