A2. Thank you for raising this important question. Firstly, it's crucial to note that intent recognition primarily stems from natural language processing, with significant foundational research and advancements in this area [12, 13, 14], including high performance in some goal-oriented dialogue systems. As a result, the text modality often plays a central role in deciphering complex human intentions, as evidenced in [1]. Nevertheless, in real-world settings, the integration of non-verbal modalities, such as video and audio, along with conversational context and scene information, is crucial to accurately infer intentions. Recognizing the limitations of existing datasets, which are often small-scale, single-turn, and limited to closed-world classification, our MIntRec2.0 dataset aims to facilitate research into the effectiveness of non-verbal modalities in multi-turn conversations and in contexts that more closely resemble real-world situations, including out-of-scope utterances.

In the process of multimodal intent annotation, our annotators utilized textual, video, and audio information, combined with contextual data within each dialogue, to analyze the underlying intention of each utterance. To investigate the relative importance of different modalities in understanding conversational intent, we gathered insights on key aspects including text modality (spoken language), video modality (facial expressions and body language), audio modality (tone), and background (context information and conversational scenes). We asked 9 annotators involved in both annotation and human evaluation to rank these six aspects based on their importance in understanding human intentions, drawing from their experience. The aggregated ranking results are as follows:

These results suggest that spoken language is the most critical factor, followed by facial expressions, tone, context information, body language, and conversational scenes. While spoken language is predominant, non-verbal cues like facial expressions and tone are valuable in perceiving emotions or attitudes, especially in the Express Emotions or Attitudes category. Context information provides essential background knowledge about the speakers, aiding in a more profound understanding of intentions. Body language, though complex and implicit, is also insightful, particularly in the Achieve Goals category. Conversational scenes, while less critical in open-ended dialogues, still contribute to understanding intent in specific contexts. We plan to include these insights and discussions in the Appendix of the revised paper to provide a more comprehensive understanding of the multimodal intent recognition process.

References:

[1] MIntRec: A New Dataset for Multimodal Intent Recognition. Zhang et al. ACM MM 2022.
[2] Switchboard: Telephone speech corpus for research and development. Godfrey et al. ICASSP. 1992.
[3] MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis. Hazarika et al. ACM MM 2022.
[4] Integrating Multimodal Information in Large Pretrained Transformers. Rahman et al. ACL 2020.
[5] Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis. AAAI 2021.
[6] Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Han et al. EMNLP 2021.
[7] MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos. Zadeh et al. arXiv. 2016.
[8] Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Zadeh et al. ACL 2018.
[9] Intention,–Plans,–and–Practical–Reason. Michael E Bratman. Mind. 1988.
[10] Intelligent agents: Theory and practice. Michael Wooldridge and Nicholas R Jennings. The knowledge engineering review. 1995.
[11] Intention, emotion, and action: A neural theory based on semantic pointers. Schröder et al. Cognitive science. 2014.
[12] Snips Voice Platform: An Embedded Spoken Language Understanding System for Private-by-design Voice interfaces. Coucke et al. arXiv. 2018.
[13] An evaluation dataset for intent classification and out-of-scope prediction. Larson et al. EMNLP-IJCNLP 2019.
[14] Efficient Intent Detection with Dual Sentence Encoders. Casanueva et al. arXiv. 2020.

A2. Thank you for your question. I'll address the significance of the 1-4% improvements from both qualitative and quantitative perspectives:

Firstly, from a qualitative standpoint, in the realm of existing multimodal language analysis tasks, state-of-the-art multimodal fusion methods typically exhibit modest improvements, generally ranging between 1-3 points, over previous baselines, such as in multimodal sentiment analysis [3, 4, 5, 6]. These incremental advancements are considered significant as they demonstrate the effectiveness and robustness of the proposed methods in real-world multimodal language datasets [7, 8]. The complexity of multimodal language analysis, necessitating nuanced modeling of cross-modal interactions, inherently makes significant performance enhancements challenging.

Secondly, regarding quantitative results, we conducted statistical t-tests to compare the performance of multimodal fusion methods (MAG-BERT and MulT) against the text-only baseline (TEXT). The results demonstrate that the multimodal fusion methods significantly outperform the text baseline, with p-values < 0.05 (†) and < 0.1 (*), as detailed in the following tables. The p-values are provided in parentheses:

These results indicate that the multimodal fusion methods significantly outperform the text baseline in 4 out of 6 metrics for in-scope classification (F1, Recall, Accuracy, Weighted F1) and in 3 out of 4 metrics for combined in-scope and out-of-scope classification (In-scope F1, Accuracy, F1). While other metrics such as Precision, Weighted Precision, and Out-of-scope F1 show more variability, they also demonstrate substantial average improvements across five experimental runs. These findings validate the effectiveness of our methods and highlight the significance of the observed improvements across most metrics.

A3. Thank you for your insightful question. In our experiments comparing ChatGPT and humans, the set of 10 dialogues totaling 227 utterances remained consistent across all experiments to ensure a fair comparison between ChatGPT and human participants. The performance metrics for each method were aggregated across all intent classes. To delve deeper into the performance on specific intent classes, we conducted additional experiments for ChatGPT-10, Humans-10, and Humans-100. We calculated the F1-score for each intent class and have presented these results as follows:

Second, to further illustrate the impact of multimodal information in intent recognition, we selected a specific dialogue from our dataset for a detailed case study. This allowed us to compare the predictive accuracy of both text and MAG-BERT models for each utterance, considering speaker identity and ground truth. The results are as follows:

The data reveals that MAG-BERT generally achieves high accuracy in most in-scope classes, except for some with complex semantics like Taunt, Oppose, and Confirm. Furthermore, in many correctly predicted in-scope classes, MAG-BERT demonstrates a high confidence level, often exceeding a 0.5 probability. In contrast, the TEXT model showed more errors than MAG-BERT in two utterances and tended to exhibit higher confidence in certain utterances (e.g., index 4, 5, 8, 18). This comparison highlights the effectiveness of incorporating non-verbal modalities over relying solely on textual information. However, it's important to note that MAG-BERT struggles with out-of-scope (OOS) utterances, often making errors in these categories. This observation suggests that while existing multimodal fusion methods have capabilities in recognizing known intents, their performance in detecting out-of-scope utterances is limited, pointing to a significant area for future research and development.

Finally, we conducted an analysis of the composition of out-of-scope utterances. Upon examining a random sample of 100 OOS utterances, we identified several utterances that potentially belong to new intent classes with limited examples, such as help, drive person away, guess, and wish. We also noted that a significant portion of the sample (nearly 60%) consists of utterances that can be categorized as statement-opinion, where the speaker expresses personal views on a subject or individual. Another notable segment (nearly 20%) falls under statement-non-opinion, encompassing subjective statements. These findings correlate with the 42 dialogue act classes defined in the SwDA dataset [2] and align with our discussion in the main paper's Section 3, particularly in the subsection on out-of-scope utterances. Our study indicates the need for continuous exploration in this field to effectively identify and categorize a broader range of human intents, especially those that fall outside the currently defined scope.

Thanks for your suggestions. We will address your concerns point by point.

Reply to Weaknesses:

A1. Thank you for your insightful question. To address the impact of incorporating multimodal information in intent recognition, we first carried out additional experiments to examine the fine-grained performance of each class using both text-only and multimodal fusion baselines. We analyzed 30 specific intent classes and one out-of-scope category, employing the F1-score as the evaluation metric, akin to the approach in [1]. The average performances from 5 experimental runs are presented below:

a. Single-turn Conversations:

Our results show significant enhancements in understanding 26 intent classes when integrating nonverbal modalities. Notably, 14 classes saw improvements of over 3 points, including Acknowledge, Agree, Apologise, Arrange, Ask for help, Care, Criticize, Flaunt, Invite, Joke, Plan, Refuse, Warn, and OOS. These classes represent a mix of common and challenging scenarios, as well as out-of-scope instances, all requiring high-level cognitive inference and semantic understanding. Remarkably, we observed substantial improvements of over 10 points in challenging classes like Flaunt ($\uparrow$ 10.29), Invite ($\uparrow$ 19.51), and Warn ($\uparrow$ 11.92). This underscores the importance of nonverbal modalities in recognizing human intentions. While there are also classes that show lesser improvements or rely more on text modality, the performance with non-verbal modalities is competitive, demonstrating the substantial benefit they bring to intent recognition in multimodal scenarios.

Moreover, we extended our research to multi-turn conversations, maintaining the same experimental settings as in single-turn conversations.

b. Multi-turn Conversations:

In these settings, MAG-BERT outperformed text-only modality in 18 classes. Specifically, it achieved improvements of over 3 points in four classes, including Acknowledge, Arrange, Ask for help, and Ask for opinions, and 1-2% improvements in 8 classes, such as Agree, Comfort, Doubt, Explain, Greet, Inform, Praise, and Warn. These classes encompass a significant portion of common interaction intents. However, the gains from nonverbal modalities in multi-turn conversations were not as pronounced as in single-turn conversations, indicating existing methodological limitations in handling out-of-scope utterances and fully utilizing context information. Addressing these challenges is crucial for future research and highlights both the importance and complexity of the MIntRec2.0 dataset.

The results illustrate that humans significantly outperform ChatGPT. With the same foundational knowledge of 10 dialogues encompassing 227 utterances, Humans-10 exhibited superior performance across nearly all intent classes and the out-of-scope category, outperforming ChatGPT-10 by over 10 points in most cases. Notably, 15 intent classes and one out-of-scope category saw improvements of over 30 points. Classes like Care and Prevent achieved improvements of over 50 points, and Ask for help, Criticize, Doubt, and Oppose saw over 40 points improvement. These findings highlight a significant gap between ChatGPT and human capabilities, underscoring humans' adeptness at using limited prior knowledge from multimodal contexts, such as body language and facial expressions, to infer and synthesize complex intents at a cognitive level—a skill where current machine learning methods, including large language models, fall short.

Additionally, with more extensive prior knowledge of 100 dialogues comprising 997 utterances, Humans-100 performed even better compared to Humans-10 and achieved state-of-the-art performance across all classes. This included markedly improved performance of over 10 points in 16 intent classes and the out-of-scope category. This demonstrates the remarkable potential of humans to leverage multimodal knowledge and their ability to learn effectively with only a marginally larger dataset (7% of all training data). This proficiency even surpasses current fully supervised multimodal fusion methods, as shown in Table 4. The detailed intent performance comparison between ChatGPT and humans further validates the challenges presented by the MIntRec2.0 dataset, indicating that there is still considerable progress to be made in AI for the complex task of multimodal intent recognition. We plan to include these results in the Appendix of the revised paper.

Reply to Questions:

A1. Thank you for your question. In the process of data collection and dialogue segmentation from the TV series, we indeed encountered laugh tracks and other audience signals following the utterances. To ensure data integrity and focus on the complexity of the intents, we meticulously annotated only the speaking parts, excluding these external audience cues. This approach helps maintain data quality by providing relatively pure multimodal signals for analysis.

Regarding the TV series as a data source, they are derived from real-world, open-ended, multi-turn conversational scenarios with a diverse range of scenes and topics (as detailed in our response to Reviewer SFq3-Reply to Weaknesses-A1). This makes them substantially similar to multi-turn conversations in real-world contexts, offering a rich resource for gathering human intents from everyday interactions.

However, a primary distinction lies in the scope of intent categories defined in our study. The 30 intent categories we identified are based on existing hierarchical multimodal intent taxonomies [1], encompassing 16 classes in Express Emotions or Attitudes and 14 classes in Achieve Goals. These categories, which are prevalent in daily life, were determined after extensive analysis and summarization of dialogue videos. They theoretically align with philosophical bases of intent, encompassing both the traditional artificial intelligence definitions of intent (as plans or goals of an agent, coupled with corresponding feedback actions [9, 10]) and the emotional evaluation factors influenced by the brain [11]. However, we recognize that some real-world intent classes may not be fully covered by our defined categories. To address this, we included an additional out-of-scope (OOS) category for utterances that do not fit into any known classes. This category facilitates the exploration of new potential intents and, with further real-world data, can aid in broadening the scope of intent classification beyond a closed-world framework.

In summary, our intent detection system is designed to be adaptable to real-world scenarios, and the chosen data sources provide a solid foundation for future research into human intent recognition.

We are greatly appreciate for your recognition of our contributions in this work.

Thanks again for your positive evaluations!

a. Single-turn Conversations:

Our results show significant enhancements in understanding 26 intent classes when integrating nonverbal modalities. Notably, 14 classes saw improvements of over 3 points, including Acknowledge, Agree, Apologise, Arrange, Ask for help, Care, Criticize, Flaunt, Invite, Joke, Plan, Refuse, Warn, and OOS. These classes represent a mix of common and challenging scenarios, as well as out-of-scope instances, all requiring high-level cognitive inference and semantic understanding. Remarkably, we observed substantial improvements of over 10 points in challenging classes like Flaunt ($\uparrow$ 10.29), Invite ($\uparrow$ 19.51), and Warn ($\uparrow$ 11.92). This underscores the importance of nonverbal modalities in recognizing human intentions. While there are also classes that show lesser improvements or rely more on text modality, the performance with non-verbal modalities is competitive, demonstrating the substantial benefit they bring to intent recognition in multimodal scenarios.

Moreover, we extended our research to multi-turn conversations, maintaining the same experimental settings as in single-turn conversations.

b. Multi-turn Conversations:

In these settings, MAG-BERT outperformed text-only modality in 18 classes. Specifically, it achieved improvements of over 3 points in four classes, including Acknowledge, Arrange, Ask for help, and Ask for opinions, and 1-2% improvements in 8 classes, such as Agree, Comfort, Doubt, Explain, Greet, Inform, Praise, and Warn. These classes encompass a significant portion of common interaction intents. However, the gains from nonverbal modalities in multi-turn conversations were not as pronounced as in single-turn conversations, indicating existing methodological limitations in handling out-of-scope utterances and fully utilizing context information. Addressing these challenges is crucial for future research and highlights both the importance and complexity of the MIntRec2.0 dataset. We would like to include these experimental results and discussions in the appendix of the revised paper.

A3. We appreciate your attention to detail and will ensure that all the identified errors are corrected in the revised version of the paper.

Thanks for your suggestions. We will address your concerns point by point.

Reply to Weaknesses:
A1. Thank you for your valuable feedback regarding the diversity of scenes and topics in our dataset. The first multimodal intent recognition dataset, MIntRec [1], is based on the TV series Superstore, featuring a rich array of characters and a multitude of storylines set in various scenes like shopping malls, warehouses, and offices. This dataset also presents hierarchical intent taxonomies with two broad categories, Express Emotions or Attitudes and Achieve Goals, which are grounded in philosophical theories [2], and delineates 20 fine-grained classes. Nonetheless, its data scale is somewhat limited.

To address this and provide a more comprehensive resource for multimodal intent research, we introduced MIntRec2.0. This enhanced version incorporates two additional popular TV series, Friends and The Big Bang Theory. With 34 main characters and more than 10 primary types of conversational scenes and 15 distinct topics, MIntRec2.0 significantly enriches the dataset in terms of content. It covers a wide range of common intents encountered in daily life and expands the existing 20 classes to 30. Specifically, these conversational scenes and topics are diverse and include:

a. Scenes and Settings:
Superstore provides a unique retail environment with scenes in the store, cash registers, break room, parking lot, managerial offices, and warehouse, reflecting workplace dynamics and customer interactions.
Friends showcases diverse social settings like the Central Perk, apartments, travel locations, and various city spots, emphasizing personal and relational interactions.
The Big Bang Theory offers academic and living spaces, including apartments and the university, highlighting intellectual and social engagements.

Each of these series brings a unique set of environments and interaction dynamics, ranging from personal and intimate to professional and public. The diversity in character backgrounds, professions, and social settings across these shows ensures a wide-ranging exploration of human interactions and conversational intents.

Moreover, these series are culturally iconic and have significantly influenced societal communication patterns, making them highly relevant for studying contemporary conversational trends. Their popularity also ensures that the dataset is relatable and accessible for a broad range of researchers and applications.

b. Topics and Themes:
Superstore touches on workplace relations, management challenges, customer service scenarios, labor issues, and social issues like immigration and corporate dynamics.
Friends explores friendship, romantic relationships, career challenges, and urban living, offering insights into a variety of emotional and relational topics.
The Big Bang Theory delves into scientific discourse, geek culture, technological advancements, scientific research, social awkwardness, and the balance between intellectual pursuits and everyday life.

The combination of these series presents an extensive range of human experiences and topics, from the mundane to the complex. This diversity enriches our dataset, making it an invaluable tool for studying and understanding the nuances of multimodal intent recognition in varied conversational contexts.

Furthermore, these series, with their wide cultural impact, provide a relatable and realistic reflection of contemporary social dynamics, essential for developing robust and applicable AI models in the field of human-computer interaction.

In summary, the chosen TV series offer a balanced mix of scenes and topics, providing a comprehensive resource that captures the complexity of human interactions and conversational intents. We are confident that our dataset's scope and diversity significantly contribute to the advancement of multimodal intent recognition research.

A2. Thank you for your insightful feedback. In our experiments, incorporating nonverbal modalities with multimodal fusion methods resulted in noticeable performance improvements. Specifically, we observed improvements of 1-4% in in-scope classification and 1-8% in both in-scope and out-of-scope utterance identification in single-turn conversations. To further demonstrate the effectiveness of nonverbal modalities, we conducted experiments across 30 specific intent classes and one out-of-scope category, using the F1-score as the evaluation metric, similar to [1]. The average performance over 5 experimental runs is presented below:

Reply to Questions:

A1. Thank you for your insightful feedback.

Firstly, intent recognition has its roots in natural language processing, where numerous foundational works have focused on understanding the semantics of utterances, particularly within goal-oriented dialogue systems [3, 4, 5]. Consequently, it's unsurprising that text modality often serves as the primary means for interpreting multimodal language. This predominance of text is further evidenced in other well-known multimodal tasks like sentiment analysis and emotion recognition [6, 7, 8, 9, 10]. However, in real-world scenarios, the integration of non-verbal modalities becomes essential. Our world is inherently multimodal, and non-verbal cues significantly enhance the comprehension of true intentions, especially in open-ended dialogues. Recognizing the need for a high-quality, large-scale multimodal intent dataset, we developed MIntRec2.0 to bridge this gap and facilitate further research.

Secondly, the incorporation of non-verbal modalities indeed yields notable improvements. As indicated in Table 4 of our main paper, multimodal fusion methods such as MAG-BERT and MulT demonstrate score enhancements of 1-4% across various metrics in both in-scope classification and out-of-scope detection. While these modalities might not always yield substantial improvements in certain aspects like handling out-of-scope utterances or contextual information, this is largely due to the absence of fusion methods specifically designed for these purposes. However, as detailed in Reply to Weaknesses-A2, MAG-BERT notably enhances performance across many intent classes, highlighting the significant potential of non-verbal modalities.

Finally, the MIntRec2.0 dataset presents the first large-scale resource in multimodal intent research, offering a comprehensive framework for benchmarking with advanced multimodal fusion methods. Our extensive evaluations reveal that while non-verbal modalities contribute to recognizing complex cognitive-level intents, they face challenges in effectively leveraging context and out-of-scope utterances. This limits their robustness and generalizability in this demanding task. Comparing the performances of humans and ChatGPT, we observe that humans excel at using limited multimodal language as prior knowledge for synthesizing and inferring real intentions, significantly outperforming even sophisticated large language models. This distinction underscores the importance and challenges posed by this dataset, marking it as a valuable contribution to the research community.

A2. In addressing the complexity of understanding a speaker's intentions during interactions, two main aspects emerge: non-verbal information and background information. The first aspect involves effectively leveraging non-verbal modalities, such as video and audio, to capture cross-modal interactions. The second aspect encompasses various types of side information, including conversational context, out-of-scope utterances, and other objective factors (e.g., cultural and social background, scenes, and topics). These elements serve as implicit prior knowledge, aiding in the understanding of human intentions. A detailed introduction of these perspectives is as follows:

• Non-Verbal Cues: Non-verbal modalities such as video and audio are also significant for understanding human intentions in multimodal scenarios. Tones, gestures, facial expressions, body language, and eye contact are crucial in providing additional information about the intention of a speaker. These cues can add to, contradict, or enhance the verbal information, offering deeper insights into the speaker's real intentions.
• Background Information:
  • Conversational Context: This refers to the specific flow of information within the dialogue and the relevance of previous utterances to the current one. Understanding the conversational history is crucial for interpreting the speaker's current intentions. The context may include the topics being discussed, the progress of the dialogue interactions, and the shared knowledge and experiences between the speakers, which are significant in helping understand intentions.
  • Out-of-scope Utterances: The occurrence of statements or new intentions that deviate from the known intentions of the speaker can help discover new intents. On the one hand, understanding out-of-scope utterances helps avoid misclassifying them as in-scope intents. On the other hand, detected out-of-scope utterances (e.g., unrelated or new topics) can be used to understand broader communicative goals, facilitate human communication, and identify potential communication interests.

• Background Information:
  • Objective Factors: (1) Cultural and Social Background: The cultural norms, social backgrounds, and linguistic styles of the speakers influence how intentions are expressed and perceived. This includes factors like accent, dialect, and the social context of the conversation. (2) Scenes and Topics: The physical setting of the conversation and the subject matter can have a significant impact on a speaker's intent. Different environments and topics can elicit varied communicative intents and styles.

In developing the MIntRec2.0 dataset, we have diligently incorporated these key factors to facilitate a more accurate understanding of human intentions. Firstly, it contains 1,245 dialogues, each with a conversational context. Secondly, of the total 15,040 utterances, each contains both verbal and nonverbal (video and audio) information. Thirdly, we consider out-of-scope utterances that typically occur in natural interactions, annotating not only 30 in-scope intent categories but also one out-of-scope category, resulting in 9,304 in-scope and 5,736 out-of-scope utterances. Finally, we annotate the speaker identity for each utterance to help researchers distinguish the different objective factors of various speakers. Therefore, we believe MIntRec2.0 is a foundational and significant contribution to research in human intention understanding.

A3. The difference between human-computer and human-human interaction datasets is substantial, primarily due to the inherent differences in the nature of communication and interaction in each setting. Here are the primary three distinctions:

• Interaction Dynamics: In human-computer interactions, the dynamics are typically unidirectional or asymmetrical. Users often lead the conversational directions and give commands to generate dialogue utterances (e.g., in goal-oriented dialogue systems [3, 4]). In contrast, human-human interactions are more dynamic and bidirectional (e.g., in open-ended dialogue systems [11]), with both parties actively contributing, responding, and adapting to the conversation flow.
• Complexity of Communication: Human-computer interactions are generally more structured and predictable, with a limited range of intents that follow specific orders or needs and relatively simple responses. Human-human interactions are far more complex, involving a wider range of intents, subtleties, emotions, and unpredictability.
• Non-Verbal Cues: Non-verbal cues are often limited or absent in human-computer interactions, as seen in many task-oriented datasets in NLP [3, 4, 5]. Even with advanced multimodal fusion algorithms, interpreting non-verbal cues from humans remains challenging for computers. In human-human interactions, non-verbal cues play a crucial role in understanding intent, with nuances in body language, facial expressions, and tone carrying significant information.

Therefore, constructing a human-human interaction dataset for multimodal intent recognition is more appropriate, as it satisfies interaction dynamics, supports complex communication, and incorporates non-verbal cues. MIntRec2.0 makes a pioneering contribution in this area and aims to facilitate related research and application.

References:
[1] MIntRec: A New Dataset for Multimodal Intent Recognition. Zhang et al. ACM MM 2022.
[2] Intention, emotion, and action: A neural theory based on semantic pointers. Schröder et al. Cognitive science. 2014.
[3] Snips Voice Platform: An Embedded Spoken Language Understanding System for Private-by-design Voice interfaces. Coucke et al. arXiv. 2018.
[4] An evaluation dataset for intent classification and out-of-scope prediction. Larson et al. EMNLP-IJCNLP 2019.
[5] Efficient Intent Detection with Dual Sentence Encoders. Casanueva et al. arXiv. 2020.
[6] Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis. AAAI 2021.
[7] Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Han et al. EMNLP 2021.
[8] MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos. Zadeh et al. arXiv. 2016.
[9] Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Zadeh et al. ACL 2018.
[10] MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. Poria et al. ACL 2019.
[11] IEMOCAP: interactive emotional dyadic motion capture database. Busso et al. Language resources and evaluation. 2008.

a. Single-turn Conversations:

Our results show significant enhancements in understanding 26 intent classes when integrating nonverbal modalities. Notably, 14 classes saw improvements of over 3 points, including Acknowledge, Agree, Apologise, Arrange, Ask for help, Care, Criticize, Flaunt, Invite, Joke, Plan, Refuse, Warn, and OOS. These classes represent a mix of common and challenging scenarios, as well as out-of-scope instances, all requiring high-level cognitive inference and semantic understanding. Remarkably, we observed substantial improvements of over 10 points in challenging classes like Flaunt ($\uparrow$ 10.29), Invite ($\uparrow$ 19.51), and Warn ($\uparrow$ 11.92). This underscores the importance of nonverbal modalities in recognizing human intentions. While there are also classes that show lesser improvements or rely more on text modality, the performance with non-verbal modalities is competitive, demonstrating the substantial benefit they bring to intent recognition in multimodal scenarios.

Moreover, we extended our research to multi-turn conversations, maintaining the same experimental settings as in single-turn conversations.

b. Multi-turn Conversations:

In these settings, MAG-BERT outperformed text-only modality in 18 classes. Specifically, it achieved improvements of over 3 points in four classes, including Acknowledge, Arrange, Ask for help, and Ask for opinions, and 1-2% improvements in 8 classes, such as Agree, Comfort, Doubt, Explain, Greet, Inform, Praise, and Warn. These classes encompass a significant portion of common interaction intents. However, the gains from nonverbal modalities in multi-turn conversations were not as pronounced as in single-turn conversations, indicating existing methodological limitations in handling out-of-scope utterances and fully utilizing context information. Addressing these challenges is crucial for future research and highlights both the importance and complexity of the MIntRec2.0 dataset. We would like to include these experimental results and discussions in the appendix of the revised paper.

A4. Thank you for your insightful question. To address the impact of incorporating multimodal information in intent recognition, we first carried out additional experiments to examine the fine-grained performance of each class using both text-only and multimodal fusion baselines (as detailed in responses to Reply to Weaknesses-A2&A3).

A5. Thank you for raising this important question. Firstly, it's crucial to note that intent recognition primarily stems from natural language processing, with significant foundational research and advancements in this area [4, 5, 6], including high performance in some goal-oriented dialogue systems. As a result, the text modality often plays a central role in deciphering complex human intentions, as evidenced in [1]. Nevertheless, in real-world settings, the integration of non-verbal modalities, such as video and audio, along with conversational context and scene information, is crucial to accurately infer intentions. Recognizing the limitations of existing datasets, which are often small-scale, single-turn, and limited to closed-world classification, our MIntRec2.0 dataset aims to facilitate research into the effectiveness of non-verbal modalities in multi-turn conversations and in contexts that more closely resemble real-world situations, including out-of-scope utterances.

In the process of multimodal intent annotation, our annotators utilized textual, video, and audio information, combined with contextual data within each dialogue, to analyze the underlying intention of each utterance. To investigate the relative importance of different modalities in understanding conversational intent, we gathered insights on key aspects including text modality (spoken language), video modality (facial expressions and body language), audio modality (tone), and background (context information and conversational scenes). We asked 9 annotators involved in both annotation and human evaluation to rank these six aspects based on their importance in understanding human intentions, drawing from their experience. The aggregated ranking results are as follows:

These results suggest that spoken language is the most critical factor, followed by facial expressions, tone, context information, body language, and conversational scenes. While spoken language is predominant, non-verbal cues like facial expressions and tone are valuable in perceiving emotions or attitudes, especially in the Express Emotions or Attitudes category. Context information provides essential background knowledge about the speakers, aiding in a more profound understanding of intentions. Body language, though complex and implicit, is also insightful, particularly in the Achieve Goals category. Conversational scenes, while less critical in open-ended dialogues, still contribute to understanding intent in specific contexts. We plan to include these insights and discussions in the Appendix of the revised paper to provide a more comprehensive understanding of the multimodal intent recognition process.

References:

[1] MIntRec: A New Dataset for Multimodal Intent Recognition. Zhang et al. ACM MM 2022.
[2] Intention, emotion, and action: A neural theory based on semantic pointers. Schröder et al. Cognitive science. 2014.
[3] Switchboard: Telephone speech corpus for research and development. Godfrey et al. ICASSP. 1992.
[4] Snips Voice Platform: An Embedded Spoken Language Understanding System for Private-by-design Voice interfaces. Coucke et al. arXiv. 2018.
[5] An evaluation dataset for intent classification and out-of-scope prediction. Larson et al. EMNLP-IJCNLP 2019.
[6] Efficient Intent Detection with Dual Sentence Encoders. Casanueva et al. arXiv. 2020.
[7] Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis. AAAI 2021.
[8] Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Han et al. EMNLP 2021.
[9] MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos. Zadeh et al. arXiv. 2016.
[10] Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Zadeh et al. ACL 2018.
[11] MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. Poria et al. ACL 2019.
[12] IEMOCAP: interactive emotional dyadic motion capture database. Busso et al. Language resources and evaluation. 2008.
[13] Intention,–Plans,–and–Practical–Reason. Michael E Bratman. Mind. 1988.
[14] Intelligent agents: Theory and practice. Michael Wooldridge and Nicholas R Jennings. The knowledge engineering review. 1995.

Regarding ambiguous or contradictory multimodal signals, we prioritize the clarity of video content, ensuring it contains clear textual utterances and discernible facial or body language. Contradictory signals across different modalities highlight the complexity of understanding human intent, necessitating nuanced analysis and inference of multimodal synergy. We preserve these multifaceted signals in their natural form to maintain the dataset's real-world applicability and generalizability. Hence, MIntRec2.0 represents a significant step forward in researching human intention understanding, especially in tackling the inherent challenges posed by ambiguous and complex multimodal interactions.

A3. The difference between human-computer and human-human interaction datasets is substantial, primarily due to the inherent differences in the nature of communication and interaction in each setting. Here are the primary three distinctions:

• Interaction Dynamics: In human-computer interactions, the dynamics are typically unidirectional or asymmetrical. Users often lead the conversational directions and give commands to generate dialogue utterances (e.g., in goal-oriented dialogue systems [4, 5]). In contrast, human-human interactions are more dynamic and bidirectional (e.g., in open-ended dialogue systems [12]), with both parties actively contributing, responding, and adapting to the conversation flow.
• Complexity of Communication: Human-computer interactions are generally more structured and predictable, with a limited range of intents that follow specific orders or needs and relatively simple responses. Human-human interactions are far more complex, involving a wider range of intents, subtleties, emotions, and unpredictability.
• Non-Verbal Cues: Non-verbal cues are often limited or absent in human-computer interactions, as seen in many task-oriented datasets in NLP [4, 5, 6]. Even with advanced multimodal fusion algorithms, interpreting non-verbal cues from humans remains challenging for computers. In human-human interactions, non-verbal cues play a crucial role in understanding intent, with nuances in body language, facial expressions, and tone carrying significant information.

Therefore, constructing a human-human interaction dataset for multimodal intent recognition is more appropriate, as it satisfies interaction dynamics, supports complex communication, and incorporates non-verbal cues. MIntRec2.0 makes a pioneering contribution in this area and aims to facilitate related research and application.

A4. Thank you for your question. In the process of data collection and dialogue segmentation from the TV series, we indeed encountered laugh tracks and other audience signals following the utterances. To ensure data integrity and focus on the complexity of the intents, we meticulously annotated only the speaking parts, excluding these external audience cues. This approach helps maintain data quality by providing relatively pure multimodal signals for analysis.

Regarding the TV series as a data source, they are derived from real-world, open-ended, multi-turn conversational scenarios with a diverse range of scenes and topics (as detailed in our response to Reviewer SFq3-Reply to Weaknesses-A1). This makes them substantially similar to multi-turn conversations in real-world contexts, offering a rich resource for gathering human intents from everyday interactions. However, a primary distinction lies in the scope of intent categories defined in our study. The 30 intent categories we identified are based on existing hierarchical multimodal intent taxonomies [1], encompassing 16 classes in Express Emotions or Attitudes and 14 classes in Achieve Goals. These categories, which are prevalent in daily life, were determined after extensive analysis and summarization of dialogue videos. They theoretically align with philosophical bases of intent, encompassing both the traditional artificial intelligence definitions of intent (as plans or goals of an agent, coupled with corresponding feedback actions [13, 14]) and the emotional evaluation factors influenced by the brain [2]. However, we recognize that some real-world intent classes may not be fully covered by our defined categories. To address this, we included an additional out-of-scope (OOS) category for utterances that do not fit into any known classes. This category facilitates the exploration of new potential intents and, with further real-world data, can aid in broadening the scope of intent classification beyond a closed-world framework.

In summary, our intent detection system is designed to be adaptable to real-world scenarios, and the chosen data sources provide a solid foundation for future research into human intent recognition.

Secondly, the incorporation of non-verbal modalities indeed yields notable improvements. As indicated in Table 4 of our main paper, multimodal fusion methods such as MAG-BERT and MulT demonstrate score enhancements of 1-4% across various metrics in both in-scope classification and out-of-scope detection. While these modalities might not always yield substantial improvements in certain aspects like handling out-of-scope utterances or contextual information, this is largely due to the absence of multimodal fusion methods specifically designed for these purposes. However, as detailed in Reply to Weaknesses-A2&A3, MAG-BERT notably enhances performance across many intent classes, highlighting the significant potential of non-verbal modalities.

Finally, the MIntRec2.0 dataset presents the first large-scale resource in multimodal intent research, offering a comprehensive framework for benchmarking with advanced multimodal fusion methods. Our extensive evaluations reveal that while non-verbal modalities contribute to recognizing complex cognitive-level intents, they face challenges in effectively leveraging context and out-of-scope utterances. This limits their robustness and generalizability in this demanding task. Comparing the performances of humans and ChatGPT, we observe that humans excel at using limited multimodal language as prior knowledge for synthesizing and inferring real intentions, significantly outperforming even sophisticated large language models. This distinction underscores the importance and challenges posed by this dataset, marking it as a valuable contribution to the research community.

A2. In addressing the complexity of understanding a speaker's intentions during interactions, two main aspects emerge: non-verbal information and background information. The first aspect involves effectively leveraging non-verbal modalities, such as video and audio, to capture cross-modal interactions. The second aspect encompasses various types of side information, including conversational context, out-of-scope utterances, and other objective factors (e.g., cultural and social background, scenes, and topics). These elements serve as implicit prior knowledge, aiding in the understanding of human intentions. A detailed introduction of these perspectives is as follows:

• Non-Verbal Cues: Non-verbal modalities such as video and audio are also significant for understanding human intentions in multimodal scenarios. Tones, gestures, facial expressions, body language, and eye contact are crucial in providing additional information about the intention of a speaker. These cues can add to, contradict, or enhance the verbal information, offering deeper insights into the speaker's real intentions.
• Background Information
  • Conversational Context: This refers to the specific flow of information within the dialogue and the relevance of previous utterances to the current one. Understanding the conversational history is crucial for interpreting the speaker's current intentions. The context may include the topics being discussed, the progress of the dialogue interactions, and the shared knowledge and experiences between the speakers, which are significant in helping understand intentions.
  • Out-of-scope Utterances: The occurrence of statements or new intentions that deviate from the known intentions of the speaker can help discover new intents. On the one hand, understanding out-of-scope utterances helps avoid misclassifying them as in-scope intents. On the other hand, detected out-of-scope utterances (e.g., unrelated or new topics) can be used to understand broader communicative goals, facilitate human communication, and identify potential communication interests.
  • Objective Factors: (1) Cultural and Social Background: The cultural norms, social backgrounds, and linguistic styles of the speakers influence how intentions are expressed and perceived. This includes factors like accent, dialect, and the social context of the conversation. (2) Scenes and Topics: The physical setting of the conversation and the subject matter can have a significant impact on a speaker's intent. Different environments and topics can elicit varied communicative intents and styles.

A5. Thank you for your insightful question. In our experiments comparing ChatGPT and humans, the set of 10 dialogues totaling 227 utterances remained consistent across all experiments to ensure a fair comparison between ChatGPT and human participants. The performance metrics for each method were aggregated across all intent classes. To delve deeper into the performance on specific intent classes, we conducted additional experiments using the same datasets for ChatGPT-10, Humans-10, and Humans-100 as outlined in Table 5. We calculated the F1-score for each intent class and have presented these results as follows:

The results illustrate that humans significantly outperform ChatGPT. With the same foundational knowledge of 10 dialogues encompassing 227 utterances, Humans-10 exhibited superior performance across nearly all intent classes and the out-of-scope category, outperforming ChatGPT-10 by over 10 points in most cases. Notably, 15 intent classes and one out-of-scope category saw improvements of over 30 points. Classes like Care and Prevent achieved improvements of over 50 points, and Ask for help, Criticize, Doubt, and Oppose saw over 40 points improvement. These findings highlight a significant gap between ChatGPT and human capabilities, underscoring humans' adeptness at using limited prior knowledge from multimodal contexts, such as body language and facial expressions, to infer and synthesize complex intents at a cognitive level—a skill where current machine learning methods, including large language models, fall short.

Additionally, with more extensive prior knowledge of 100 dialogues comprising 997 utterances, Humans-100 performed even better compared to Humans-10 and achieved state-of-the-art performance across all classes. This included markedly improved performance of over 10 points in 16 intent classes and the out-of-scope category. This demonstrates the remarkable potential of humans to leverage multimodal knowledge and their ability to learn effectively with only a marginally larger dataset (7% of all training data). This proficiency even surpasses current fully supervised multimodal fusion methods, as shown in Tables 4 and 5. The detailed intent performance comparison between ChatGPT and humans further validates the challenges presented by the MIntRec2.0 dataset, indicating that there is still considerable progress to be made in AI for the complex task of multimodal intent recognition. We plan to include these results in the Appendix of the revised paper.

Reply to Questions:

A1. Firstly, intent recognition has its roots in natural language processing, where numerous foundational works have focused on understanding the semantics of utterances, particularly within goal-oriented dialogue systems [4, 5, 6]. Consequently, it's unsurprising that text modality often serves as the primary means for interpreting multimodal language. This predominance of text is further evidenced in other well-known multimodal tasks like sentiment analysis and emotion recognition [7, 8, 9, 10, 11]. However, in real-world scenarios, the integration of non-verbal modalities becomes essential. Our world is inherently multimodal, and non-verbal cues significantly enhance the comprehension of true intentions, especially in open-ended dialogues. Recognizing the need for a high-quality, large-scale multimodal intent dataset, we developed MIntRec2.0 to bridge this gap and facilitate further research.

Second, to further illustrate the impact of multimodal information in intent recognition, we selected a specific dialogue from our dataset for a detailed case study. This allowed us to compare the predictive accuracy of both text and MAG-BERT models for each utterance, considering speaker identity and ground truth. The results are as follows:

The data reveals that MAG-BERT generally achieves high accuracy in most in-scope classes, except for some with complex semantics like Taunt, Oppose, and Confirm. Furthermore, in many correctly predicted in-scope classes, MAG-BERT demonstrates a high confidence level, often exceeding a 0.5 probability. In contrast, the TEXT model showed more errors than MAG-BERT in two utterances and tended to exhibit higher confidence in certain utterances (e.g., index 4, 5, 8, 18). This comparison highlights the effectiveness of incorporating non-verbal modalities over relying solely on textual information. However, it's important to note that MAG-BERT struggles with out-of-scope (OOS) utterances, often making errors in these categories. This observation suggests that while existing multimodal fusion methods have capabilities in recognizing known intents, their performance in detecting out-of-scope utterances is limited, pointing to a significant area for future research and development.

Finally, we conducted an analysis of the composition of out-of-scope utterances. Upon examining a random sample of 100 OOS utterances, we identified several utterances that potentially belong to new intent classes with limited examples, such as help, drive person away, guess, and wish. We also noted that a significant portion of the sample (nearly 60%) consists of utterances that can be categorized as statement-opinion, where the speaker expresses personal views on a subject or individual. Another notable segment (nearly 20%) falls under statement-non-opinion, encompassing subjective statements. These findings correlate with the 42 dialogue act classes defined in the SwDA dataset [3] and align with our discussion in the main paper's Section 3, particularly in the subsection on out-of-scope utterances. Our study indicates the need for continuous exploration in this field to effectively identify and categorize a broader range of human intents, especially those that fall outside the currently defined scope.

Thanks for your suggestions. We will address your concerns point by point.

Reply to Weaknesses:

A1. Thank you for your valuable feedback regarding the diversity of scenes and topics in our dataset. The first multimodal intent recognition dataset, MIntRec [1], is based on the TV series Superstore, featuring a rich array of characters and a multitude of storylines set in various scenes like shopping malls, warehouses, and offices. This dataset also presents hierarchical intent taxonomies with two broad categories, Express Emotions or Attitudes and Achieve Goals, which are grounded in philosophical theories [2], and delineates 20 fine-grained classes. Nonetheless, its data scale is somewhat limited.

To address this and provide a more comprehensive resource for multimodal intent research, we introduced MIntRec2.0. This enhanced version incorporates two additional popular TV series, Friends and The Big Bang Theory. With 34 main characters and more than 10 primary types of conversational scenes and 15 distinct topics, MIntRec2.0 significantly enriches the dataset in terms of content. It covers a wide range of common intents encountered in daily life and expands the existing 20 classes to 30. Specifically, these conversational scenes and topics are diverse and include:

a. Scenes and Settings:
Superstore provides a unique retail environment with scenes in the store, cash registers, break room, parking lot, managerial offices, and warehouse, reflecting workplace dynamics and customer interactions.
Friends showcases diverse social settings like the Central Perk, apartments, travel locations, and various city spots, emphasizing personal and relational interactions.
The Big Bang Theory offers academic and living spaces, including apartments and the university, highlighting intellectual and social engagements.

Each of these series brings a unique set of environments and interaction dynamics, ranging from personal and intimate to professional and public. The diversity in character backgrounds, professions, and social settings across these shows ensures a wide-ranging exploration of human interactions and conversational intents.

Moreover, these series are culturally iconic and have significantly influenced societal communication patterns, making them highly relevant for studying contemporary conversational trends. Their popularity also ensures that the dataset is relatable and accessible for a broad range of researchers and applications.

b. Topics and Themes:
Superstore touches on workplace relations, management challenges, customer service scenarios, labor issues, and social issues like immigration and corporate dynamics.
Friends explores friendship, romantic relationships, career challenges, and urban living, offering insights into a variety of emotional and relational topics.
The Big Bang Theory delves into scientific discourse, geek culture, technological advancements, scientific research, social awkwardness, and the balance between intellectual pursuits and everyday life.

The combination of these series presents an extensive range of human experiences and topics, from the mundane to the complex. This diversity enriches our dataset, making it an invaluable tool for studying and understanding the nuances of multimodal intent recognition in varied conversational contexts.

Furthermore, these series, with their wide cultural impact, provide a relatable and realistic reflection of contemporary social dynamics, essential for developing robust and applicable AI models in the field of human-computer interaction.

In summary, the chosen TV series offer a balanced mix of scenes and topics, providing a comprehensive resource that captures the complexity of human interactions and conversational intents. We are confident that our dataset's scope and diversity significantly contribute to the advancement of multimodal intent recognition research.

A2 & A3. Thank you for your insightful feedback. In our experiments, incorporating nonverbal modalities with multimodal fusion methods resulted in noticeable performance improvements. Specifically, we observed improvements of 1-4% in in-scope classification and 1-8% in both in-scope and out-of-scope utterance identification in single-turn conversations. To further demonstrate the effectiveness of nonverbal modalities, we conducted experiments across 30 specific intent classes and one out-of-scope category, using the F1-score as the evaluation metric, similar to [1]. The average performance over 5 experimental runs is presented below:

2. New Results in Revision

1. We fine-grained intent performance results in single-turn and multi-turn conversations, illustrating the effectiveness of non-verbal modalities (page 31, Appedix U.1, Un9C, SFq3, ju3D).
2. We conduct a case study to demonstrate the effectiveness of non-verbal modalities, especially in out-of-scope data (page 30, Appendix T, Un9C, ju3D).
3. We fine-grained intent performance of ChatGPT-10, Humans-10, Humans-100, highlighting the significant gap between humans and AI methods (page 31, Appendix U.2, Un9C, ju3D).
4. We add voting results on the significance of modalities during multimodal intent annotation (page 22, Appendix L, Un9C, ju3D).

3. Revisions to the Text

1. We introduce scenes and topics in the dataset to demonstrate data diversity (page 18, Appendix E, Un9C, SFq3).
2. We correct minor typos throughout the text (pages 2, 5, SFq3, Un9C).
3. We discuss the differences between human-computer and human-human interaction datasets (page 16, Appendix C, Un9C, SFq3).
4. We clarify the composition of out-of-scope data (page 5, Un9C, ju3D).
5. We clarify there are no audience signals and the consistency of prior knowledge in the Humans-10 and ChatGPT-10 evaluations (pages 4, 8, Un9C, ju3D).

Each reviewer's concerns have been individually addressed. We humbly request careful review and feedback on our responses and revised work and sincerely appreciate your support during this review process.