OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition

Q1: This research aims to overcome the limitations of previous studies that rely on predefined taxonomies to capture more complex, subtle emotions. However, the grouping and emotion recognition performance results are not so different from those of the existing studies. The methods and evaluation criteria are sound and reasonable but limited to identifying L1-level groups, which is not very different from the present classification framework.

A1: It seems the reviewer may have some misunderstandings about our work. The L1-level group is just one of several grouping techniques used in this paper. Specifically, Table 2 presents results using GPT-based grouping, Table 3 provides results using L1-level grouping, and Table 12 shows results using L2-level grouping.

In this paper, our prediction approach follows an open-vocabulary setting, which allows for the recognition of unseen emotions. This is fundamentally different from traditional MER, which relies on fixed taxonomies. However, this flexibility introduces new evaluation challenges. Since there is no predefined label space, the model may predict closely related but differently expressed emotions (e.g., joyful and happy). To address this issue, we grouped similar emotions before computing evaluation metrics. Thus, the grouping operation is not intended to restrict emotion recognition to L1-level groups. Instead, to align with our open-vocabulary setup, we designed grouping techniques to facilitate performance evaluation. Of course, other evaluation metrics that do not rely on grouping techniques could also be used.

Meanwhile, due to the open-vocabulary setting of our OV-MER, we need to design not only new evaluation metrics but also novel dataset construction methods and baselines. These innovations are also key contributions of our work. Therefore, OV-MER is fundamentally different from traditional MER in terms of the task, dataset, evaluation metrics, and solution approach.

Q2: As the authors claim, human emotion includes many different elements. Therefore, emotion recognition should be a "multi-label" problem. I strongly recommend that the proposed framework be extended in this direction.

A2: Thanks for your comment. We would like to clarify that OV-MER has already attempted to address MER in a multi-label manner. As shown in Figure 2 and Appendix E, a single instance can be associated with multiple emotion labels simultaneously. Furthermore, we argue that our proposed OV-MER task encompasses the concept of a multi-label problem but extends beyond it by incorporating flexible and expandable emotion expressions. Consequently, conventional evaluation protocols (e.g., L2 loss) cannot be directly applied to the OV-MER task.

Q3: I wonder how much agreement there is among annotators (Kappa value) in the annotation framework.

A3: Unlike the traditional single-label-based annotation method with a fixed label space, OV-MER employs a multi-label-based annotation method without a fixed label space. Therefore, we cannot directly compute the Kappa value between different annotators. Therefore, we draw inspiration from Section M and utilize the Jaccard similarity coefficient to measure the inter-annotator agreement.

Specifically, assume there are $N$ samples and $K$ annotators. For each pair of annotators $A_m$ and $A_n$, their annotation results for each sample $x_i$ are denoted as $Y_m^i$ and $Y_n^i$, respectively. Here, $Y_m^i$ and $Y_n^i$ contain a set of emotion labels. We calculate the agreement score between annotators $A_m$ and $A_n$ as:

In our annotation process, we hired 8 annotators and conducted two rounds of checks, with no overlap among annotators in each round (see Appendix K). For the first round, the inter-annotator agreement is shown as follows:

For the second round, the inter-annotator agreement is shown as follows:

We observe that through multi-round checks, the inter-annotator agreement gradually increases. These results demonstrate the necessity of multi-round checks, which help enhance label reliability.

We sincerely appreciate your positive feedback and recognition of our contributions to advancing MER research. Your comments on our work are truly valuable to us.

Q1: In Table 2, some baseline models include both 7B and 13B versions. Please specify which version is used in the leaderboard.

A1: Thank you for your suggestion. In this paper, we use the 7B model by default, and we will clarify this in the revised manuscript.

Q2: Please explain the reason for reporting both English and Chinese results in Table 2.

A2: In Figure 2, we observe that there are certain differences in the labels extracted from different languages. To study the impact of language differences, we report results for both English and Chinese descriptions in Table 2.

Q3: Please discuss the correlation between English and Chinese results.

A3: We utilize the results from Table 2 to compute Pearson correlation coefficients (PCC) between the English and Chinese results for each metric. As illustrated in the following table, all metrics demonstrate strong cross-linguistic correlations.

Q4: Please explain how the OV-MERD dataset is annotated. Traditionally, due to the fixed label space, MER datasets usually employ multiple annotators and use majority voting to determine the final label. In OV-MER, please explain the method for determining the final label.

A4: We appreciate your comment regarding our annotation process. As detailed in Appendix K, our labeling procedure involved eight annotators familiar with emotion definitions and utilized a two-round verification pipeline. In the first round, four randomly selected annotators performed independent annotations. In the second round, the labels reviewed by the first group of annotators were merged, and the remaining four annotators conducted a second round of checks. This approach ensures that each preserved label receives confirmation from at least one annotator per round, thereby guaranteeing both the comprehensiveness and accuracy of the annotation results.

Q5: Please discuss the reasons behind why the GPT-based scores are highly correlated with the EW-based metrics, but less correlated with the matching-based scores?

A5: GPT-based and EW-based metrics are two grouping techniques used in our work, focusing on calculating emotion label similarity. In contrast, matching-based scores emphasize word-level matching between two descriptions, including non-emotional words. Consequently, matching-based metrics exhibit lower similarity compared to GPT-based and EW-based metrics.

Q6: In Table 12, it seems that M*-L2 is always more correlated with the GPT-based metrics than M*-L1. Please explain this further.

A6: M*-L1 emphasizes coarse-grained clustering information, whereas M*-L2 emphasizes fine-grained clustering information. The higher correlation between M*-L2 and GPT-based metrics suggests that GPT-based metrics primarily rely on fine-grained emotion clustering during the metric calculation.

Thank you for your positive feedback on our work. We greatly appreciate your recognition that OV-MER represents a significant advancement in MER systems, enhancing their generalizability and enabling more human-like emotional understanding in AI applications.

Q1: While human reviewers refine dataset labels, there is no separate human assessment of final model outputs. A user study evaluating how well OV-MER aligns with human perception would be beneficial.

A1: Thanks for your insightful comment. To assess how well OV-MER aligns with human perception, we conducted a user study. Specifically, we hired 9 annotators and randomly selected 20 samples from our dataset. Each annotator was presented with (sample, OV-MERD label) pairs and asked to judge their alignment with human perception using a binary (Yes/No) response format. To ensure annotation quality, we also included inspection data consisting of (sample, incorrect label) pairs. We observe that 96% of the annotations confirm the alignment between OV-MERD labels and human perception. Considering the potential annotator errors, this result demonstrates that our OV-MERD labels align well with human perception.

Q2: Ablation studies on the impact of different modalities (text, audio, video) are limited. While results indicate multimodal inputs improve performance, further analysis on their individual contributions would be valuable.

A2: Thanks for your suggestion. In Table 2, we observe that CLUE-Video outperforms CLUE-Text, consistent with the nature of our OV-MERD dataset. To be specific, OV-MERD is derived from MER2023, where the textual modality contributes less than the visual modality in emotion recognition [1]. Meanwhile, CLUE-Audio achieves superior performance over both CLUE-Text and CLUE-Video, suggesting that although textual expressions may be ambiguous for emotion recognition, combining them with audio cues can effectively resolve these ambiguities, leading to better performance.

[1] Lian, Zheng, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. "Merbench: A unified evaluation benchmark for multimodal emotion recognition." arXiv preprint arXiv:2401.03429 (2024).

Q3: The dataset is primarily sourced from movies and TV series, which may not fully represent spontaneous, real-life emotional expressions. A discussion on potential domain adaptation strategies would strengthen the work.

A3: OV-MERD is derived from MER2023, which is sourced from high-rated movies and TV shows. The high ratings serve as an implicit validation of the actors' performances, ensuring spontaneous and realistic emotional expressions. Currently, this type of dataset is the mainstream in the MER research community, as it provides a cost-effective means to expand dataset scale. In the future, we plan to apply for additional funding to collect data featuring spontaneous, real-life emotional expressions by recruiting participants. Furthermore, we will employ domain adaptation techniques (e.g., Domain-Adversarial Neural Networks, DANN) to address potential domain gaps between different data sources. These will be incorporated into the future work section of the revised manuscript.

Q1: CLUE-Multi achieves higher scores than GPT-4V. This performance gap suggests potential methodological issues and undermines the credibility of the results.

A1: We believe the reviewer may have misunderstood the results in Table 2. As explained in Section 4.1 and illustrated in Figure 2, CLUE-Multi is the ground truth derived from manually verified visual and acoustic clues. In contrast, as shown in Figure 3, GPT-4V belongs to the CLUE-MLLM category, where it does not utilize manually verified clues. Therefore, CLUE-Multi is the upper-bound performance, while GPT-4V is a baseline. To enhance the clarity, we will revise the manuscript as follows: (1) The caption for the second part will be updated from "CLUE-MLLM" to "CLUE-MLLM (Baselines)". (2) The caption for the third part will be revised from "CLUE-M/A/T/V" to "CLUE-M/A/T/V (Upper-Bound Performance)".

Q2: Comparison with conventional MER methods.

A2: Traditional MER methods are not applied due to fundamental differences in our experimental setup. Specifically, conventional methods require identical label spaces $\mathcal{Y}$ for both training and testing sets. They cannot predict unseen emotions, i.e., $y \notin \mathcal{Y}$. However, we use an open-vocabulary annotation manner for each sample, which inherently cannot guarantee alignment between training and testing label spaces. Since MLLM-based methods give us more freedom in emotion prediction, our work primarily leverages MLLM-based solutions.

If we force the traditional MER approach, these models can only predict labels that belong to their training label space. To address your concerns, we train on the IEMOCAP (or MELD) dataset and test on OV-MERD, following the zero-shot experimental setup for a fair comparison. For the model architecture, we evaluate both the attention model and MulT.

Q3: Manual checks were performed only for ALLM/VLLM outputs, not for texts generated by LLMs.

A3: During our pipeline design, we observed noticeable errors and hallucinations in the outputs from ALLM/VLLM. In contrast, when it comes to the text merging task, given that GPT-series models exhibit impressive performance in reading comprehension [1] (close to human performance) and considering that multi-clue merging is a fundamental function in reading comprehension, we directly adopt the merging results from GPT-series models. This decision balances dataset reliability and construction cost. The user study in our response to Reviewer 2UWh further validates the high quality of LLM-based MER outputs.

[1] Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.

Q4: Training, validation, and testing are absent. This omission raises concerns regarding potential overfitting due to insufficient data.

A4: We believe the reviewer may have misunderstood some aspects of our work. This paper focuses on establishing a zero-shot benchmark for OV-MER. Consequently, all data is used for inference, meaning every sample serves as part of the testing set. Since our method does not involve any training process, the overfitting problem does not exist at all. In this paper, our primary contributions are to propose a new task (OV-MER) and lay the groundwork for OV-MER. In our follow-up work (as discussed in Section B. Limitations), we plan to develop more effective frameworks to better address OV-MER.

Q5: Meaning of the subscript "s".

A5: The subscript ``s'' indicates that these metrics are set-based, distinguishing them from the traditional single-label metrics.

Q6: Lack theoretical parts. More suitable for the NeurIPS Datasets and Benchmarks track.

A6: This paper goes beyond merely proposing a dataset or establishing benchmarks within existing paradigms. Instead, it represents a significant task-wise innovation in the MER field by introducing a novel paradigm that enables more accurate emotion modeling. Additionally, this paper employs a psychological theory-based method to address the challenging evaluation problem of OV-MER. Given these substantial contributions to the MER research community, we firmly believe this work meets ICML's standards.