Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark

Thanks for recognizing the novelty of our ideas and for your positive feedback on our extensive experiments and writing. While there are some minor issues with the expression, we have made every effort to clarify them. We believe this work is an important extension of the traditional MER task and will contribute to the development of this field.

Q1: The dataset scale is not mentioned or compared in Sec 2 or Tab 1.

A1: Our benchmark dataset contains 1,121 emotional labels and 332 samples, all of which are used to establish a zero-shot learning benchmark for the OV-MER task. The main contribution of this work is the definition of a new task and the development of foundational research, including the datasets, metrics, and solutions. In addition to this dataset, we are also working on a series of follow-up efforts to expand the dataset.

Q2: There is no quality evaluation for the generated labels (e.g., comparison with manual annotations), and their validity has not been assessed (e.g., by using the dataset for pre-training or other downstream tasks). Although there are comparisons with humans regarding clue length, label count, and word cloud, the label accuracy is never evaluated or discussed. This raises concerns about the reliability and overall usefulness of the dataset for the study.

A2: As shown in Figure 2, our ground truth generation process has gone through multiple rounds of strict manual checks by professional annotators. Therefore, the ground-truth labels are accurate and reliable.

In Appendix F, we provide a deeper comparison between the one-hot labels in the MER2023 dataset and the OV labels in the OV-MERD dataset. We observe that OV labels cover a wider variety of emotions, some of which (such as shy, nervous, and grateful) are rarely discussed in previous datasets. Meanwhile, we notice that most samples have about 2 to 4 labels, much more than the traditional task where each sample is assigned only one emotion. Therefore, our dataset provides richer and more comprehensive labels for each sample.

Q3: The setup in Table 2 and 3 presents a data leakage issue. The ground truth label is generated using SALMONN and GPT-4V, but both of these methods are also being evaluated.

A3: SALMONN and GPT-4V were used only to generate initial descriptions. Subsequently, we conducted manual checks independent of SALMONN and GPT-4V, and the final descriptions differed significantly from the initial ones. Therefore, there is no issue of data leakage. Tables 2 and 3 list various MLLMs, and the primary reason for including SALMONN and GPT-4V in these tables is to provide a comprehensive performance comparison.

Q4: Sec 2.1, pre-annotation, what is the average duration of the videos? I'm concerned that using only three frames as input may miss a significant amount of visual info, especially since the paper later notes that there are errors, duplicates, and missing info related to visual clues.

A4: Thanks for your comments. In Appendix H, we analyze the duration distribution of the OV-MERD dataset. We observe that the majority of the samples have a duration between 1 and 4 seconds.

Regarding the sampling strategy, we did not provide a detailed explanation due to space limitations. Specifically, we categorize visual clues into two types: (1) visual clues with relatively long durations; and (2) visual clues with fast movements, such as eye movements, head movements, and micro-expressions.

For the first type, since the duration of most videos is between 1 and 4 seconds (see Appendix H) and the video content is usually continuous with only minor differences between adjacent frames (see Appendix C), uniformly sampling three frames are sufficient to capture this information; For the second type, we observe that current MLLMs (including the GPT-4V used in this paper) struggle to capture these fast movements. Increasing the number of sampled frames does not address this issue. Previous research has also shown that GPT-4V cannot recognize micro-expressions [1]. To capture these fast movements, we employ multiple professional annotators to manually add this information.

[1] Lu, Hao, Xuesong Niu, Jiyao Wang, Yin Wang, Qingyong Hu, Jiaqi Tang, Yuting Zhang et al. "Gpt as psychologist? preliminary evaluations for gpt-4v on visual affective computing." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 322-331. 2024.

Q5: Sec 2.1, manual check, could you please clarify why ALLM struggles to capture emotion-related acoustic features? From my understanding, paralinguistic information in the audio modality contains meaningful emotion-related features, which should be useful in this context.

A5: The paralinguistic information in the audio contains meaningful emotion-related features. However, current ALLMs struggle to capture these emotion-related paralinguistic features. We believe the main reason is that existing ALLMs primarily focus on datasets related to tasks like ASR or audio event detection, with less emphasis on paralinguistic information, which results in relatively poor performance in paralinguistic understanding.

Q6: Table 1, CMU-MOSEI has discrete emotion labels.

A6: Thanks for your comments. We correct this issue in the revised PDF. More dataset comparison is provided in Appendix G.

Q7: In Sec 3.1, the GPT-based grouping is described as costly, but if the entire process only costs $1, it doesn't seem to be a significant expense. Could you clarify the reason?

A7: Thanks for pointing it out. This paper reports zero-shot performance, only focusing on the inference process. The cost of evaluating our OV-MERD dataset is about $1 per evaluation, which may not seem high.

However, for future work aimed at training frameworks to better address the OV-MER task, this cost will become prohibitive. For example, if we plan to train a model for 100 epochs, the evaluation cost will rise to $100.

If we intend to test N different parameter combinations and M different frameworks, the evaluation cost will further increase to $100 * M * N. Moreover, we plan to expand the OV-MERD dataset in the future. This cost will further increase. Therefore, this paper explores alternatives to GPT-based metrics. We place this discussion in the appendix.

Q8: Fig 4 (a), typo in video branch: ALLM $\rightarrow$ VLLM.

A8: Thanks for your careful reading. We correct it in our revised PDF.

Q9: Sec 4.1, CLUE-A, could you explain why text is used to generate acoustic clues when the audio already contains the necessary content information? It seems redundant.

A9: Thanks for your comment. In Section 5, we discussed this issue and highlighted the impact of different CLUE-MLLM generation strategies: 1) S0 does not use text and inputs the audio into MLLM; 2) S1 inputs both text and audio into MLLM simultaneously; 3) S2 first uses MLLM to extract descriptions and then combines with text using another LLM. In general, S2 achieves the best performance. These results suggest that the ALLM cannot fully leverage the text, and using an additional LLM to emphasize the text can further improve performance. Based on these observations, we adopt this strategy for generating CLUE-Audio.

Q10: It is not clear what fusion method you use in Tab 3.

A10: Thanks for your careful reading. To fuse different modalities, we input prelabeled acoustic and visual clues into LLM and leverage its reasoning capabilities for multimodal integration (see Figure 4). This approach is consistent with the fusion method used during our dataset construction process (see Section 2). We add this to our revised paper.

Thanks for your reply. I can see that you have made some revisions in the rebuttal revision based on the replies. For example, in the section Main Results on CLUE-M/A/T/V. You mentioned that the results on CLUE-Multi/A/V/T reflect the performance upper bound. Could you tell me briefly what changes have been made?

We greatly appreciate your positive comment on the task and dataset we proposed, as well as your encouraging evaluation of our experiment. While there are some minor issues with the expression, we have made every effort to clarify them. We believe this work is an important extension of the traditional MER task and will contribute to the development of this field.

Q1: The baseline results rely on CLUE-Multi/A/V/T descriptions that include human checks. In real-world scenarios, the human intervention is impractical and the actual performance may be lower.

A1: You may misunderstand the purpose of these baselines. The results on CLUE-Multi/A/V/T only reflect the performance upper bound, which is used to evaluate whether current MLLMs can address the OV-MER task.

Q2: The models in the paper are trained using CLUE-Multi/A/V/T data that includes manual checks, which ensures high-quality and detailed emotion descriptions. However, in real-world applications, it is likely that only LLMs-generated CLUEs without any human oversight will be available. Would this discrepancy lead to significant performance degradation in models trained using CLUE-Multi/A/V/T data?

A2: It seems there have been some misunderstandings. The main goal of this paper is to provide a valuable dataset with rich emotional labels to further advance the development of MER.

Q3: The dataset’s cartoonish style could hinder the model’s ability to generalize to real-world data, potentially limiting its effectiveness in practical applications. Would the cartoonish style of the dataset cause a significant domain shift problem? Specifically, how would a model pre-trained on this dataset perform when tested or fine-tuned on other datasets like DFEW and IEMOCAP? Do you consider any adaptation strategies to mitigate potential domain shifts?

A3: Our dataset is not in a cartoon style. The original data comes from MER2023, which is a dataset containing real-person videos collected from movies and TV series. The cartoon-style videos mentioned in the paper were generated by using DemoAI to remove personal information, addressing copyright concerns. For the original data, please download the MER2023 dataset.

Q4: There is a lack of information on the total number of samples and the train/validation/test split. What is the sample size of the dataset, and how are the train/validation/test sets divided?

A4: This paper provides a zero-shot learning benchmark that does not require the training process. Our benchmark dataset contains 1,121 emotional labels and 332 samples, all of which are used to test the performance of various MLLMs. The main contribution of this work is the definition of a new task and the development of foundational research, including the datasets, metrics, and solutions. In addition to this dataset, we are also working on a series of follow-up efforts to expand the dataset.

Q5: The paper mentioned, "This is because, in MER, text is emotionally ambiguous; without integrating other modalities, we cannot accurately understand the emotions conveyed through text." However, as far as I know, there are datasets like MOSI and MOSEI where the text/language modality contributes more than other modalities, as evidenced by relevant experimental results [1-4]. Is this explanation of yours appropriate, or is it specific to this particular dataset?

A5: Thanks for your valuable comment. The contribution of different modalities varies across different datasets. As proved in previous research [1], the contribution of the text/language modality in MER2023 is smaller than that of other modalities (our OV-MERD is derived from MER2023). However, in the CMU-MOSI and CMU-MOSEI datasets, the contribution of the text/language modality is greater than that of the other modalities. The original text was not accurate, and we will correct our expression in the revised PDF.

[1] Lian, Zheng, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. "Merbench: A unified evaluation benchmark for multimodal emotion recognition." arXiv preprint arXiv:2401.03429 (2024).

Hi, sorry for the late reply. I found this paper interesting, and you have addressed most of my concerns. Therefore, I have increased my score to 6. However, I have also noticed some questions raised by other reviewers and the AC ikpq, which require further discussion and consideration. I reserve the possibility of further adjustment of scores in the follow-up.

In addition, I would like to thank the AC for his/her active participation in the discussion, which I believe is beneficial to the ICLR.

We sincerely thank you for your positive feedback on our motivations, problem formulation, experiments, and reproducibility. Meanwhile, your valuable comments help us improve this paper. We believe this work is an important extension of the traditional MER task and will contribute to the development of this field.

Q1: While it's true that nuanced emotional labels can't always be conveyed by a single adjective, like "surprised," as it may stem from either positive or negative experiences, the VAD (valence, arousal, and dominance) framework can help mitigate this issue. This is why emotion recognition tasks are generally categorized as either discrete label-based or dimensional-based. By using VAD scores, we can better distinguish whether "surprise" is elicited by positive or negative emotions. Discrepancies do exist within each subject's understanding and annotators's agreement. However, it is not clear whether the issue was addressed by the proposed approach. In Appendix Section G, the authors state, 'Specifically, in the first round, we randomly select four annotators to check the clues and labels; in the second round, we merge the clues and labels reviewed by the first four annotators and ask another four annotators to perform a second round of checks.' How exactly are they aligned, and what has been removed/kept?

A1: In Section 1, we discuss the motivation behind OV-MER. Although dimensional models can be used to distinguish different emotions, they are abstract and difficult for the general public to understand. For instance, the valence-arousal-dominance (VAD) model is one of the most widely used dimensional models. Given a VAD score (V=3.3, A=2.9, D=4.7), it is challenging for people to accurately interpret the corresponding emotion. According to previous research [1], this VAD score represents sad. Furthermore, the VAD model attempts to simplify emotions into three linear dimensions. However, emotions are inherently complex and often involve combinations of multiple emotional states, which cannot be fully captured by these three dimensions. Therefore, this paper uses discrete models to describe emotions. To address the limitations of traditional one-hot MER, we propose the OV-MER task, aiming to better describe subtle emotional nuances in an open-vocabulary manner.

Additionally, in contrast to dimensional models, discrete models align more closely with how humans intuitively understand emotions, making it easier to achieve consensus across different annotators.

[1] Verma, Gyanendra K., and Uma Shanker Tiwary. "Affect representation and recognition in 3D continuous valence–arousal–dominance space." Multimedia Tools and Applications 76 (2017): 2159-2183.

Q2: Section 2, which focuses on the construction of the 'OV-MERD dataset', is one of the most significant contributions of this work. However, the authors first discuss feature extractions and annotations, leaving readers unclear about the original data source until Section 2.3. This structure is not well-designed; as a reader, it became frustrating to navigate from the first page to Figure 2 and then through Sections 2.1 and 2.2 without understanding the origins (where and what) of the original data until Section 2.3 on page 4. This makes the paper difficult to follow.

A2: The reason we use this structure is that in Section 2, we primarily emphasize the human-LLM collaborative annotation process, as this is one of the key contributions of our paper, while the data source is just the practical part and can be replaced with other datasets. Therefore, we first discuss the annotation process and then use this process to create the OV-MERD dataset. To address your concerns, we will add an additional introduction about the dataset in Section 2.

Q3: In Section 2.3, the phrase "evenly select samples with further annotation" is vague. Does this refer to the original dataset (MER2023), which (I can only assume) has an even number of samples under each category? Or is it discussing the newly constructed dataset? I interpret it as referring to the original dataset based on Section E in the Appendix, but this should be stated more clearly.

A3: The original text is: "This dataset is an extension of MER2023, from which we evenly select samples with further annotations.". This sentence does not mean that MER2023 has an equal number of samples for each category but discusses the newly constructed dataset. We will clarify this sentence in the revised PDF.

Thanks for your reply. In the revised paper, we further clarify our work and highlight all changes in red.

Q1: Both dimensional and discrete approaches have their respective advantages and disadvantages in the emotion recognition task. For VAD, using a Likert scale to quantify the scores allows participants to rate the intensity of emotions on a defined scale, which can be helpful. On the other hand, discrete labels are more intuitive for the general public but often fail to quantify the exact degree of emotions experienced. It is unnecessary for the authors to argue the advantages of one approach over the other, as both serve different purposes and contexts. Additionally, my question regarding the annotators mentioned in Appendix Section G was not addressed.

A1: Yes, both discrete models and dimensional models have their advantages and drawbacks, and our OV-MER captures the strengths of both emotional models. Compared with the one-hot discrete models, OV-MER uses more dense discrete labels to capture more nuanced emotions, mimicking the continuous attributes of the dimensional model. Compared with the dimensional model, OV-MER uses discrete labels, making it closer to the way humans understand emotions. Therefore, our OV-MER provides a bridge connecting discrete and dimensional models.

To illustrate which labels are removed or kept, we provide two examples in Appendix J, each with labels from multiple annotators. To ensure annotation quality, we hire professional annotators who are experts in affective computing and familiar with the definition of emotions. Some of these annotators are members of our team who specialize in affective computing. In Figure 18, as the character doesn't know which doctor to see, most annotators provide labels such as confused or puzzled. Based on his tone and expression, some annotators further provide labels like anxious and serious. In Figure 19, most annotators notice his disapprove based on the textual content. Combining other modalities, some annotators further note his blame and accuse of what others are planning to do. From these examples, we can observe that these annotators provided relatively reliable labels. However, some annotators may focus only on the most relevant labels and overlook some details. To ensure the comprehensiveness of the annotation results, we merge the labels checked by four annotators. For example, in Figure 18, the final merged labels are troubled, focused, puzzled, anxious, worried, confused, and serious. In the next round of checks, we invite another four annotators for a second round of checks. Through this process, we can ensure that each retained label is confirmed by at least one annotator in each round, thereby ensuring the comprehensiveness and accuracy of the annotation results.

Q2: It is certain that the original data (MER2023) is not a contribution for this work, but the constructed new dataset is. Therefore, it is necessary to introduce it first with enough background information. If the annotation process is the primary contribution, and can be applied to other datasets as authors stated, then OV-MER should be applied to more datasets in this work, particularly those with a limited number of emotion labels, as listed in Table 1, instead of focusing solely on MER2023, even just for comparison purpose.

A2: Thanks for your comment. In the revised paper, we add more background information about the dataset.

Q3: Please clarify whether the current PDF available for review includes all the revisions made in response to the feedback.

A3: Thanks for your comments. In the revised paper, we clarify this statement in Section 2.3.

Q4: If changing the order of the annotations does not affect the evaluation results, this should be demonstrated through experiments or at least discussed in the paper.

A4: In the revised paper, we add more discussion on the label order. This paper proposes three set-based metrics.

$\text{Precision}_{\text{s}}$ indicates the number of correctly predicted labels;

$\text{Recall}_{\text{s}}$ indicates whether the prediction covers all ground truth;

and $\text{F}_{\text{s}}$ is the harmonic mean of two metrics, which is used for the final ranking. It should be noted that if we change the label order in $\text{Y}$ and $\hat{\text{Y}}$, it will not result in any change in performance. Therefore, the order of the emotions has no effect on the final result.

Q5: To confirm, it is the textual content generated by a LLM? That is ok; I just wanted to ensure this with a clear answer.

A5: Yes. We use ALLM and VLLM for pre-annotation and LLM for clue merging. All outputs are in text format.

We appreciate your recognition of the "vital contribution to the field" made by this paper. We also thank you for highlighting that our approach can "make the model's emotional comprehension more transferable and adaptable." While there are some minor issues with the expression, we have made every effort to clarify them. We believe this work is an important extension of the traditional MER task and will contribute to the development of this field.

Q1: Dependency of this approach to a certain versions of large language model definitely has its own limitations. From known weaknesses of this models to potential future risks of such approach. Furthermore, attempts to control this is quite expensive and risky. The bigger concern that comes to my mind is the effect of mirrors reflection in learning. How can we be sure that from this circle of learning, a limited and biased understanding is not created for machine learning? A broader concern is the issue of model feedback loops—how can we be confident that machine learning isn't developing a narrow, biased interpretation of emotions from these circular, LLM-driven annotations? This feedback cycle may compromise the depth of emotional understanding in machine learning.

A1: You raise a very interesting point. In fact, we considered this issue when designing the annotation process and conducted some experimental validation.

(1) Annotation Process Design: We employed multiple professional annotators and conducted several rounds of checking to address machine bias. Specifically, LLM-driven annotations are used as initial results, and all final decisions are made by multiple annotators (see Figure 2). To ensure annotation quality, we selected annotators with expertise in affective computing and familiarity with different emotion definitions. The involvement of multiple professional annotators ensures that the machine does not lead to narrow or biased interpretations. Additionally, as shown in Figure 17 (in Appendix J), our annotation platform includes an "Other Emotion" part. If the LLM-driven annotations fail to perfectly capture the character’s emotional state, annotators can manually add other labels. This process is also conducted by multiple professional annotators, ensuring that the emotional understanding is not constrained by machine learning biases.

(2) Experimental Validation: We also considered this issue in our experiments by comparing the human-only annotation strategy with the human-LLM collaborative strategy (see Figure 5). From these results, the human-LLM collaborative strategy covers a broader range of emotions and provides more diverse labels for each sample. These results confirm that this feedback loop does not undermine the depth of emotional understanding. On the contrary, it helps uncover more complex and subtle emotional nuances.

Thank you for your detailed responses. After reviewing the Area Chair's (AC) comments and re-evaluating the feedback from other reviewers, I have a few additional questions and concerns:

1. With updates to LLMs, would the label space change? One concern is the cost of labeling datasets using your approach, especially given the significant reliance on expert annotators, as you indicated. Do experts need to control the label space each time a new dataset or new version of LLM is used? How feasible is it in practice?

2. How can we ensure that existing machine learning solutions can adapt to this new baseline? From your earlier response, I understand that adapting to this setup is "almost infeasible." If this is the case, how practical is it to implement this shift, particularly when the benchmark is based on a limited set of datasets (e.g., a single dataset)? My concern is whether the broader research community will accept this if it limits baseline comparisons or the range of datasets.

3. How can methods be fairly compared in this setup to confidently conclude that your approach improves learning of emotional models? Table 7 is presented as evidence of effectiveness, but I don’t see a clear or fair comparison—lower performance is reported, but relative to what? Furthermore, is there any discussion or analysis showing that models trained with your dataset are demonstrably more emotionally intelligent compared to those trained on traditional datasets? While the variation in label count or the use of a broader label range is mentioned in response to a similar comment by Reviewer GTfB (Q2), I don’t believe this necessarily indicates greater informativeness or improved learning.

4. How does training work in this setup? How is the model provided with a loss function, and how is the loss landscape defined and optimized in this new framework? You reported a cost of $1 per epoch, so am I correct in assuming that GPT acts as a loss function in this setup? I find this approach challenging both in terms of its cost and its potential reception in the literature, as well as concerns about its trustworthiness.

5. I find question 3 from the AC debatable. While I agree that the approach does not necessarily introduce new emotions, as suggested, the discussion around Fig. 5 indicates that it enables the use of a broader set of emotion labels rather than expanding the label space itself. However, I do not see a clear analysis demonstrating how these additional labels are more informative or how they necessarily introduce new information. From your response to reviewer tZuA Q1, I understand that your method differs from VAD systems by being more aligned with the "general public's" understanding of emotions. However, this does not necessarily address how it contributes to improved machine learning performance, which ties back to question 3.

Thanks for your valuable comments. Below are some clarifications:

Q1: With updates to LLMs, would the label space change? One concern is the cost of labeling datasets using your approach, especially given the significant reliance on expert annotators, as you indicated. Do experts need to control the label space each time a new dataset or new version of LLM is used? How feasible is it in practice?

A1: Thanks for your insightful comments. We would like to clarify that the discussion regarding LLMs is beyond the scope of this paper. Technically, during the annotation process, we used GPT, as it is one of the most powerful LLMs. Meanwhile, we propose to use the human-LLM collaborative strategy in emotion annotation, which is a step from zero to one. The detailed impact of different LLMs will be discussed in our future work.

In our annotation process, experts are not required to control the label space. Annotators are free to use any emotional label they deem appropriate, with LLMs serving only as a reference. Specifically, we employed eight annotators and conducted two rounds of checks to ensure that each label was confirmed by at least one annotator in each round, thereby ensuring the accuracy of the annotations.

Q2: How can we ensure that existing machine learning solutions can adapt to this new baseline? From your earlier response, I understand that adapting to this setup is "almost infeasible." If this is the case, how practical is it to implement this shift, particularly when the benchmark is based on a limited set of datasets (e.g., a single dataset)? My concern is whether the broader research community will accept this if it limits baseline comparisons or the range of datasets.

A2: Thanks for your comments. Regarding the baseline, this paper shifts from traditional classification models to LLM-driven approaches, aligning with current research trends. As more LLMs and MLLMs emerge, the baseline for OV-MER will become increasingly rich. We believe that this shift is practical and positive for the future of emotion recognition research.

Meanwhile, introducing new concepts inevitably will face many challenges. We do not intend to replace the mainstream work focused on basic emotion recognition. Our goal is to provide the community with a different perspective, highlighting that some subtle emotions cannot be perfectly described by basic emotions. In other words, we believe that emotion recognition should not be limited to basic emotions but should expand to include more nuanced emotions. Our work is only laying the foundation for OV-MER, and the further development of this task will require the collective effort of many researchers. Currently, some researchers have already started using our dataset in their works.

Q3: How can methods be fairly compared in this setup to confidently conclude that your approach improves learning of emotional models? Table 7 is presented as evidence of effectiveness, but I don’t see a clear or fair comparison—lower performance is reported, but relative to what? Furthermore, is there any discussion or analysis showing that models trained with your dataset are demonstrably more emotionally intelligent compared to those trained on traditional datasets? While the variation in label count or the use of a broader label range is mentioned in response to a similar comment by Reviewer GTfB (Q2), I don’t believe this necessarily indicates greater informativeness or improved learning.

A3: Thanks for your comments. First, the main purpose of Table 7 is to demonstrate that basic emotions cannot fully capture the complex emotional states of characters. Therefore, we need to use more nuanced labels beyond basic emotions and provide richer labels for each sample. Here, we provide an example. In predicting the emotional labels of the instance in Figure 1(b), existing methods can only predict labels within the basic emotion categories (e.g., "surprise"), whereas our method can output more nuanced and diverse emotions (e.g., "surprise, nervous, dissatisfied"). By identifying these more subtle emotional labels, we can enhance the model's emotional intelligence.

To address your second question, we conducted a user study to verify whether our dataset provides greater informativeness than basic emotions. Specifically, we hired 4 participants and randomly selected 10 samples from our dataset. For each sample, we provided both the basic emotion label and the OV-MER label. The following instruction was given: Which label provides greater informativeness? The label with more information was marked as 1, and the other label as 0. The results showed that 97.5% of the results believed that our OV-MER provided more information. This user study also helps address the concerns raised by AC. The user study results will be submitted to the supplementary material.

We sincerely appreciate your recognition of the importance of this paper and the acknowledgment of our contributions. In this paper, we introduce a new task called OV-MER, for which we further construct a dataset, define evaluation metrics, and propose solutions. While there are some minor issues with the expression, we have made every effort to clarify them. We believe this work is an important extension of the traditional MER task and will contribute to the development of this field.

Q1: The color scheme in Figure 5 does not align with the main visual style of the paper. It is recommended to redraw this figure for consistency.

A1: Thanks for your suggestion. We update Figure 5 in the revised PDF.

Q2: In Section 6, please provide additional elaboration on the recommendations for framework design, as this could be valuable for future researchers.

A2: In this paper, we prove that how to integrate subtitle information and fuse multimodal inputs plays a crucial role in the final performance. We will work on these aspects in the framework design. Meanwhile, we plan to follow the current mainstream approaches in MLLMs, including incorporating more emotion-related instruction datasets and utilizing more powerful LLMs. We add this discussion in our revised paper.

Q3: Traditional facial emotion recognition methods often utilize AU units and assign a single label per face. However, this paper’s OV-MER assigns multiple labels to a video. Could you clarify why videos in multimodal scenarios are suited to convey multiple emotions?

A3: Video emotion is more complex than facial emotion. This is because, in videos, we need to capture subtle changes in the temporal dimension and integrate multimodal clues. Take Figure 1(b) as an example. In the temporal dimension, we need to infer a person's nervousness based on his stuttering; in the multimodal dimension, we need to combine information from different modalities to gain a more comprehensive understanding of emotion. Due to the complexity of video emotion, using a single label is limiting, and more discrete labels are required to better describe video emotion. We add this discussion to Appendix A.

Q4: Table 1 includes only a limited selection of emotion datasets. Please consider expanding it to include additional datasets, such as MSP-Podcast.

A4: Thanks for your suggestion. In Appendix G, we compare with more emotion datasets, including MSP-Podcast, JL-Corpus, EmoDB, EMOVO, MESD, eNTERFACE, etc,

Q5: In Table 2, please further clarify the distinctions between the labels extracted from CLUE-Multi and the final ground truth.

A5: We highlight this point in Figure 2(b). In our preliminary experiments, we observed that the labels extracted from different language versions of CLUE-Multi show some discrepancies. To eliminate language influence and achieve consensus labels, we merge these labels and conduct two-round manual checks. The output after manual checks is considered the final ground truth.

Thanks for your interest in our work. We are honored to have the opportunity to answer your questions.

Q1: How does the proposed framework/paradigm, namely “open-vocabulary,” differ from established concepts such as “multi-label learning” and “partial label learning”? Both of these areas have been explored extensively, including in affective computing and other domains of human-machine systems/computing.

A1: Thanks for your meaningful comments. We would like to clarify the differences between them from the following aspects:

1. Different Task Definition. Open-vocabulary learning (OVL) differs from multi-label learning (MLL) and partial-label learning (PLL) primarily in its focus on generalizing to unseen or novel labels, whereas MLL and PLL operate within a fixed label space. In MLL, multiple labels are assigned to each sample [1], while in PLL, a set of candidate labels are assigned for each sample, with only one correct label [2]. In OVL, we aim to recognize labels outside of the predefined label space [3]. As such, the key difference lies in the ability of OVL to generalize to new labels dynamically, which extends the static capabilities of MLL and PLL.

2. Emergent Affective States in Real Worlds. Human affective states are diverse and context-dependent, often extending beyond predefined taxonomies (e.g., beyond Ekman’s basic emotions). As psychologist Plutchik pointed out, humans can express approximately 34,000 different emotions [4]. Open-vocabulary methods allow models to understand and predict emotional states beyond these predefined taxonomies. Meanwhile, existing MLL and PLL methods have limited emotion label scope, and failed to adapt to open-world tasks.

3. Relationship Between OVL and MLL&PLL. Theoretically, MLL or PLL can be converted to OVL by spanning the label space of MLL or PLL to a complete set that includes all the emotion labels in the language (e.g., around 34,000 different emotions). However, this is not feasible in practice to construct such kind of dataset for MLL and PLL, i.e., annotators need to label each sample with 34,000 different emotions, and it's hard to collect sufficient samples for every emotion category, let alone multiple labels). This is why the largest emotion category number in existing datasets is only 23 (EmotioNet), while in our work, using OVL, we can easily extend the number of emotion categories to 248.

4. Evaluation Settings. In our OVL benchmark, we adopt a zero-shot setting where the sample labels are selected as the most reliable emotional descriptors from an unconstrained vocabulary. Consequently, the label spaces in the training and test sets are different, meaning the test set may include novel emotion labels that were unseen during training. In contrast, traditional MLL and PLL approaches require the label spaces of the training and test sets to be strictly consistent, making them unsuitable for scenarios where the label space dynamically evolves (i.e., the dimensions of label vectors can be mismatched, impossible to calculate loss or similarity score). This is why we proposed a novel set-based evaluating metric to measure the performance of emotion recognition. This fundamental difference highlights our work as not only a novel OVL benchmark for emotion recognition but also a unique testbed for evaluating open-vocabulary learning approaches, which are critical for addressing the challenges of real-world, open-ended tasks.

5. Feasibility. Previously, OV-MER was difficult to solve due to the near-infinite emotion label space. However, with the development of LLMs, we see the potential to propose a feasible OV-MER framework. The main reason is that LLMs contain an extensive vocabulary, enabling them to output highly diverse emotional labels. Ideally, since LLMs are trained on large-scale data, the upper limit on the number of emotions is the ability of LLMs, instead of the predefined label spaces. Specifically, we leverage LLMs in our annotation process and solutions. (1) Our annotation process relies on LLMs to provide reference labels for annotators, thereby improving the comprehensiveness of the annotation results; (2) Compared to traditional methods (such as MLL and PLL), our solution also leverages LLMs to output a richer set of emotional labels. Meanwhile, it's almost infeasible to apply existing MLL and PLL baselines to OVL tasks, as they are all designed for fixed label space.

In summary, allowing the model to predict a richer set of emotions (such as OV-MER), rather than limiting the label space to a few basic emotions (such as PLL and MLL), offers a better solution to the complexity of human emotional expression. We believe this work is the future of MER. It will advance emotion recognition from basic emotions to more subtle emotional expressions, contributing to the development of emotional AI.