Thanks for the valuable comments and acknowledging that our work offers noteworthy observations about current spoken dialogue systems. We address your concerns below.

1. Generalization to non-task-oriented spoken dialogue datasets

We evaluate our turn-taking prediction model on Fisher. Please check the general response.

2. Difference with prior turn taking evaluation metrics

What is the main difference between the proposed evaluation protocol and the previous approach by Ekstedt and Skantze (2022)? Is it impractical to apply the metrics from prior turn-taking evaluation methods to audio FMs?

Thank you for this thoughtful question, and we apologize if this distinction was unclear in the draft. The key difference between our evaluation protocol and prior approaches [1], lies in the evaluation focus and application context.

[1] focuses on assessing turn-taking model's ability to predict turn-taking events that will happen in the near future. This involves assessing how well turn-taking model forecasts when specific turn-taking events will happen in human-human conversations based on contextual cues.

In contrast, our protocol evaluates AI dialogue systems' ability to perform turn-taking events in spontaneous, interactive conversations with human users. This involves assessing how well the AI system actively decides to take the conversation floor, yield it’s turn, backchannel or interrupt the user during live human-AI interactions, reflecting its capability to engage in natural dialogue with the end user. This shift required adapting existing metrics and creating new ones tailored to human-AI interaction.

For example, while [1]'s SHIFT vs. HOLD (S/H) metric evaluates how well a turn-taking model predicts whether the current speaker will hold a turn or there will be a turn shift, we adapted it to evaluate an audio FM’s decision to speak or allow the user to continue during pauses. Pseudo ground-truth labels are generated using a judge turn-taking model, and the agreement between the AI's decisions and these pseudo labels is used to assess the quality of the AI system's turn-taking decisions. Additionally, we introduced novel metrics, such as evaluating how well AI systems manage user interruptions—an essential aspect of realistic human-AI interactions.

In summary, our protocol represents the first effort to assess audio FMs' ability to perform turn-taking events in human-AI conversations, offering valuable insights for future research. In response to your feedback, we have added this discussion in Sec. 4.3 and A.1.1 to clarify this novel contribution.

3. Scalability and Applicability of Approach

Turn-taking prediction models used in evaluation protocol require training, which limits scalability and applicability.

Please check the general response.

References

[1] Ekstedt, Erik, and Gabriel Skantze. Voice activity projection: Self-supervised learning of turn-taking events. Interspeech 2022 (https://arxiv.org/abs/2205.09812 )

Thank you for your reply and hard work. I think 1) Generalization to non-task-oriented spoken dialogue datasets is resolved with your additional experiments on Fisher and 3) Scalability and applicability is somewhat explained while it remained a limitation However, I still have a major concern about 2) Difference with prior turn-taking evaluation metrics: If current turn-taking metrics evaluate human-human conversation and your proposed metric evaluates human-AI interaction, isn't it possible to apply the current turn-taking metrics to human-AI interaction? In particular, could you explain what specific aspects of human-AI turn-taking cannot be captured by adapting existing metrics? Could you elaborate on this further?

Thank you for your insightful comments. We address your concerns below.

1. Thresholds for proposed metrics

The thresholds in Sec. 4.4-4.8 seem arbitrary. Is there a specific reason for choosing these values?

We apologize if this was not sufficiently clear. As mentioned in Sec 4.3 (lines 307-309) on page 6, we carefully tuned the thresholds for all proposed metrics using an in-domain validation set to maximize the agreement between the judge model's labels and human judgments. We present the agreement between judge labels and human judgments on the in-domain validation set in Table 6 in the Appendix. We hope this addresses your concern.

2. Reliability of judge model

There are concerns about the reliability of the judge model. Since all results are based on comparisons with this model, is there concrete evidence supporting its credibility?

We agree that establishing the reliability of the judge model is essential, given that our results depend on using the judge label. To address this, we outline our validation approach in Sec. 4.3 (lines 302-312) on page 6, inspired by prior works [1, 2] showing high consistency between LLM predictions and human relevance judgments.

In our study, we assess judge model consistency with human judgments by analyzing instances in a human-human conversation dataset corresponding to each metric. For example, Metric A considers scenarios where a listener decides whether to speak up during a speaker’s pause. Detailed explanations for all metrics are in sections 4.4-4.8 (e.g., lines 322-323 for Metric A, lines 374-375 for Metric B, lines 399-401 for Metric C, lines 414-415 for Metric D, and lines 448-450 for Metric E). As shown in Figure 3 on page 7, our judge labels have good agreement with human judgments on both in-domain (blue) and out-of-domain (green) test sets for most metrics, supporting the model's credibility. We hope this clarifies our approach.

3. Big Confidence Intervals for Proposed Metrics

The confidence in some evaluations (Fig. 3(b), (e)) appears high, but no explanation is provided.

Thank you for pointing out this important observation. We acknowledge the reviewer’s concern regarding the high confidence intervals for Metric (B) in Fig. 3(b) and Metric (E) in Fig. 3(e). We hypothesize that this result from a small sample size, as shown in Table 2, where occurrences of the AI system backchanneling (0.01% for both Moshi and Cascaded systems) and user interruptions (0.2% for Moshi and 0.1% for the Cascaded system) are indeed very rare in human-AI conversations. We updated the paper (lines 387-389 in Sec. 4.5 and lines 458-460 in Sec. 4.8) to clearly address and discuss this.

4. Biased participants

The participants introduced in Sec 3 seem biased, consisting of the authors and related individuals.

Thank you for raising this important concern. We acknowledge the potential for bias in our user study, as participants were primarily the authors and their research colleagues. Similar “lab experiments” are common in prior work [3], enabling focused testing of specific functionalities (eg. prompting participants to interrupt the AI system) and facilitating deeper insights through discussions about their subjective experiences. However, we agree that this controlled setup may introduce bias. We hope our evaluation protocol inspires future studies to conduct larger-scale evaluations with more diverse participants. We explicitly discuss and acknowledge this limitation in Sec. 6 and A.2 in the paper.

References

[1] AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (https://arxiv.org/abs/2402.07729)

[2] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (https://arxiv.org/pdf/2306.05685)

[3] Survey on evaluation methods for dialogue systems [https://link.springer.com/article/10.1007/s10462-020-09866-x]

Threshold and Reliability

L303-304

we take inspiration from prior works (Yang et al., 2024; Zheng et al.,2024) that have experimented with using an LLM as a judge.

When considering the concept of an LLM as a judge, it is customary to employ a robust model such as GPT-4. Typically, these models demonstrate agreement rates exceeding 80% when compared with human evaluations [1]. Furthermore, such models are validated not only against subjective measures like agreement but also through objective benchmarks [2]. These high standards highlight the importance of evaluating how well the proposed approach aligns with them.

Although the proposed model shows high agreement with human judgments within the 'dataset' used for human-human comparisons, this does not necessarily establish the model as 'objectively' strong. Compared to the numerous benchmarks used to evaluate LLMs, the evidence provided in Sections 4.4–4.8 (e.g., lines 322–323 for Metric A, lines 374–375 for Metric B, lines 399–401 for Metric C, lines 414–415 for Metric D, and lines 448–450 for Metric E) may not be sufficient to conclude that it is an excellent judge. That said, I want to stress that this critique does not imply the evaluation methods themselves are flawed.

we carefully tuned the thresholds ... to maximize the agreement between the judge model's labels and human judgments.

Therefore, a good judge model should minimize the need for extensive hyperparameter tuning to achieve consistent results. Alternatively, it might be beneficial to first demonstrate that the model's performance is robust and consistent across different hyperparameter settings.

[1] Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena https://arxiv.org/abs/2306.05685

[2] OpenAI, https://arxiv.org/abs/2303.08774

Finally, I have adjusted the contribution score to reflect the strengths of this work, particularly its thoughtful exploration of evaluation techniques and efforts to align human and model judgments. While there are areas for improvement, the study provides valuable insights that advance discussions in this domain.

Thanks for the insightful suggestions and acknowledging that our work offers valuable references for future development of voice dialogue systems. We address your concerns below.

1. Evaluated few audio FMs

This study only tested a few open-source and closed-source audio FM models.

Check general response

2. There is a lack of comprehensive performance evaluation and summary.

We apologize if our comprehensive evaluation was not clearly emphasized in the draft. Our work provides a thorough assessment of spoken dialogue systems by reviewing prior literature to identify key turn-taking abilities for effective human-AI interaction and developing specific metrics (Sections 4.4-4.8) to evaluate these skills. Our analysis highlights significant limitations in current AI dialogue systems and outlines key research directions for improvement, as noted by reviewers (@R Ddi6, @R BKtK, @R mo78). If you have specific suggestions for additional evaluations, we will try to incorporate them into the paper.

Thank you for your valuable comments and acknowledging the originality of our work. Weakness 3 and 4 are similar to weakness 1 and 2 and we address remaining concerns below.

1. Results on Fisher:

Please check general response.

2. Application to other languages:

“How can your evaluation protocol be adapted or applied in scenarios where supervised datasets are not available for other languages, such as Chinese?”

Please check general response on scalability and applicability of our approach.

3. Backchannel detection is heuristic

The filler word set for backchannel detection is heuristic. It may ignore some backchannel case not in filler word set.

Thank you for your insightful observation. We acknowledge that our approach for identifying backchannels relies on heuristics, using common one- and two-word phrases as indicators of backchannels. While this may miss some backchannels, it aligns with standard practices in prior turn-taking models [1, 2], which also use similar heuristic methods. Based on your comment, we explicitly discussed this limitation in Sec. 6 in the main paper.

References

[1] TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog (https://aclanthology.org/2020.findings-emnlp.268/)

[2] Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion (https://arxiv.org/abs/2401.14717)

Thanks for the valuable comments and suggestions. We address your concerns below.

1. GPT-4o was not evaluated.

Please check the general response on evaluating more audio FMs.

2. Agreement between the judge labels and human judgments

My primary concern is the relatively low agreement between the majority of judge labels and human judgments, with most falling below 80%. This raises questions about the strength of the claim that the proposed metrics maintain high consistency with human decisions.

We acknowledge the reviewer’s concerns. We would like to emphasize that turn-taking prediction is indeed a challenging task, as turn-taking behavior can vary widely even among different users. Despite this complexity, our turn-taking model achieves performance comparable to those reported in prior studies, which we believe demonstrates its robustness and practical utility. Our model also maintains moderate alignment with human decisions even on OOD spoken conversation datasets—an indicator of its generalizability.

That said, we would like to stress that the primary contribution of this work lies in our novel evaluation protocol, which is designed to be adaptable and can integrate any turn-taking prediction model. We agree that enhancing the model’s accuracy would further improve the protocol's reliability, and explicitly discuss this in Sec. 6 (Limitations) in the main text.

3. Low consistency with human judgments for Metric (E)

In Metric (E), the judge labels show low consistency with human relevance judgments, indicating that this metric may have limited reliability in assessing the model's ability to handle user interruptions effectively.

This is indeed a limitation of our work and we have acknowledged this explicitly in section 4.8. Moving forward, we consider this an area for further improvement, where enhancing the judge model could lead to better alignment with human judgments.

3. Scalability and Applicability of Approach

Reviewers (@R WGPU, @R mo78) mentioned that the evaluation protocol needs a supervised dataset to train the judge model, limiting its scalability and applicability. We agree that this is a limitation of our approach and updated the draft to explicitly acknowledge it. While our current approach indeed relies on a supervised dataset to train a turn-taking model as the judge, this model can be trained on any spoken conversation dataset containing speaker turns, transcripts, and timestamps, which are often available even for non-English languages. As noted in Appendix A.1, prior work [2] successfully trained turn-taking models on Chinese (Mandarin) and Japanese using publicly available datasets [3, 4]. Interestingly, multilingual models trained on English, Chinese, and Japanese perform comparably to monolingual models despite the diverse turn-taking behaviors of these languages.

To address scenarios without supervised datasets, we propose a low-cost solution: collecting a small spoken dataset for the target language and generating annotations through human efforts or using tools like PyAnnote (speaker diarization) and Whisper (ASR). We can then train multilingual turn-taking models that leverage high-resource language data to improve performance on low-resource languages. This approach is far more cost-effective than collecting human relevance judgments for every turn-taking event. In response to your feedback, we added a discussion in the paper (Limitations in Sec. 6, A.7) on adapting our evaluation to non-English and low-resource languages.

References:

[2] Multilingual Turn-taking Prediction Using Voice Activity Projection (https://aclanthology.org/2024.lrec-main.1036.pdf )

[3] HKUST Mandarin Telephone Speech (https://catalog.ldc.upenn.edu/LDC2005S15 )

[4] Japanese Travel Agency Task Dialogues (https://aclanthology.org/2022.lrec-1.619/)