AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models

Q1: It would be beneficial to also explore and discuss the impact of different audio and video encoders.

A1: (1) Impact of Audio encoders. The choice of audio encoder does not significantly impact performance. This confirms that AffectGPT's remarkable performance is primarily attributed to our proposed high-quality, large-scale dataset and effective framework rather than acoustic encoders. Meanwhile, we note that ImageBind achieves slightly worse performance compared to other audio encoders. One possible explanation for this is that other audio encoders are widely applied in audio content understanding tasks (such as ASR), where audio content plays a crucial role in emotion recognition.

(2) Impact of Video Encoder. The choice of video encoder has a limited impact on performance. Interestingly, CLIP_VIT slightly outperforms EVA_CLIP and DINOv2, consistent with the findings in MERBench [1], a unified benchmark for traditional categorical frameworks. These results suggest that conclusions drawn from traditional categorical frameworks, such as encoder selection, may also be applicable to MLLM-based descriptive frameworks.

[1] Lian, Zheng, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. "Merbench: A unified evaluation benchmark for multimodal emotion recognition." arXiv preprint arXiv:2401.03429 (2024).

Q2: Please further discuss the influence of rank selection in LoRA.

A2: In the following table, we count the increase in trainable parameters when using LoRA for the LLM branch. The first row represents the model without the LoRA module. Experimental results demonstrate that fine-tuning the LLM with LoRA improves performance compared to models without LoRA. However, increasing the rank for LoRA-based models does not yield significant performance gains and instead increases computational costs.

Q3: How are videos sampled for the video branch?

A3: We uniformly sample 8 frames per video by default.

Q4: The impact of varying the number of sampled frames on the model's performance.

A4: In the following table, we compare two types of inputs: (1) face-only and (2) face-text combinations, and evaluate model performance across different sampling frame counts, ranging from 2 to 64. We observe that using too few frames (e.g., fewer than 2) results in a noticeable decline in performance, indicating that insufficient frames lead to information loss. However, further increasing the number of sampling frames (e.g., more than 8) does not yield significant performance improvements. This can be attributed to the fact that MER tasks typically use short-duration videos with relatively stable facial expressions. Therefore, we default to sampling 8 frames per video in this paper.

Q5: The dataset construction pipeline could benefit from further explanation.

A5: We adopt a model-led, human-assisted annotation strategy. This approach leverages human priors to guide description generation and sample filtering, ultimately enabling automatic annotation for unlabeled data.

(1) Description Generation. During the description generation process, we first conduct preliminary experiments. In this phase, a small subset of samples is selected, and annotators are asked to assign fine-grained emotional labels to each sample. Based on the insights gained from these preliminary experiments, we perform a model selection process to ensure the quality of the automatically generated descriptions.

(2) Sample Filtering. During sample filtering, we employ a two-stage filtering technique. In the first stage, we remove samples with mismatched audio and video, as well as those with abnormal description lengths. In the second stage, we use model-based crowdsourcing to generate relatively reliable emotion labels. If the labels derived from the descriptions differ significantly from those obtained through model-based crowdsourcing, the descriptions are considered to be of low quality and are then removed.

In summary, we integrate human priors into both description generation and sample filtering to ensure the high quality of the generated descriptions in MER-Caption+.

Q6: The details of the filtering technique.

A6: For general instruction datasets, we utilize the prompt provided in Appendix E to extract emotion labels from each instruction-answer pair. Samples that result in empty emotion outputs are removed.

Q1: Limited discussion on computational costs of pre-fusion operations. The pre-fusion mechanism lacks theoretical analysis of modality interaction dynamics. The structural innovation of AffectGPT is relatively insufficient, and it does not fully explain why the "pre-fusion" operation can serve as a solution for cross-modal interaction.

A1: Regarding computational efficiency, the pre-fusion operation only involves Q-Former or attention operations, which are significantly less computationally intensive than LLMs. Theoretically, the Q-Former facilitates cross-modal interaction by distilling knowledge from multimodal content into query tokens, while the attention mechanism achieves this through dynamically predicted attention weights based on multimodal inputs.

The model architecture innovation represents one aspect of our work, and our contributions extend significantly beyond this. Specifically, we introduce the largest-scale emotion description dataset to date, constructed using an efficient model-led, human-assisted approach; we establish a comprehensive benchmark encompassing three key tasks: fine-grained emotion recognition, basic emotion classification, and sentiment analysis; and we develop specialized evaluation metrics designed for assessing free-form text outputs.

Q2: Regarding the ablation study of the model, since the video and audio features have already been fused, why are these features still fed separately into the LLMs? Have experiments been conducted using only the fused features?

A2: Thanks for your valuable comment. Following your suggestion, we conducted additional experiments to investigate this aspect. Our results demonstrate that incorporating raw audio and video features alongside the fused features yields modest performance improvements compared to using fused features alone.

Q3: What is the purpose of designing the evaluation metric $F_s$, or in other words, what specific aspect does this metric reflect?

A3: In this paper, $Precision_s$ indicates the number of correctly predicted labels, and $Recall_s$ indicates whether the prediction covers all ground truth. $F_s$ is a harmonic mean of two metrics $Precision_s$ and $Recall_s$.

Q4: Is it possible that MER-Caption+ also contains some instances where the data does not match the descriptions?

A4: Thanks for your valuable comment. Yes, there may be inaccurate descriptions in MER-Caption+ because we used an automatic annotation strategy without manual checks. However, the experimental results in Table 3 demonstrate that MER-Caption+ achieves significantly better performance than the manually annotated MAFW dataset. The main reason is that humans tend to focus on major clues, which can easily lead to incomplete descriptions. These results confirm that, despite the lack of manual checks in MER-Caption+, we can still ensure the quality of the labels. In the future, we will investigate other post-filtering techniques to further improve MER-Caption+'s annotation quality.

We sincerely appreciate your recognition of our fair comparative experiments, innovative dataset, effective key components, and comprehensive evaluation benchmark.

Q1: In Appendix F, experiments have shown that using a combination of SALMONN and mPLUG-Owl results in better performance. However, the authors still chose the combination of SLAMONN and Chat-UniVi. An explanation for this choice is needed.

A1: Thanks for your careful review. In this paper, we do not use the combined results for model selection but instead rely on the performance of individual models. For example, for VLLM, Chat-UniVi outperforms mPLUG-Owl, and for ALLM, SALMONN outperforms SECap. Therefore, we use the combination of Chat-UniVi and SALMONN for description generation. The combination experiments are primarily designed to demonstrate that integrating multimodal cues can lead to better performance. Your suggestion of using the combined results for model selection is insightful, and we will conduct more experiments in this direction.

Q2: The origin of the raw data used in the dataset construction is not provided by the authors.

A2: The raw data comes from the unlabeled portions of MER2024, which is used with permission from the dataset owners. In this work, we annotate each unlabeled sample with emotion descriptions. The complete annotation procedure is illustrated in Figure 2.

Q3: A large amount of data was filtered out during the dataset construction process, which limits the application scenarios of the dataset.

A3: We would like to highlight that dataset quality is equally important as quantity. It is verified by the experimental results in Table 4: Given the same testing set (with diverse sources/scenarios), the model trained on the filtered datasets performs better than the one trained on the noisy dataset.

Q4: The evaluation metric used in the benchmark for basic emotion recognition tasks may have the risk of being inaccurate. The current evaluation metric only calculates the number of correctly matched emotion words in the output of the multimodal emotion recognition model. However, if the model outputs entirely incorrect results, the impact of these errors is not accounted for.

A4: Thanks for your comment. In fact, we have taken this into account during the metric design process. Basic emotion recognition tasks provide the majority-voted labels $y$, which are generally reliable. However, emotion descriptions produce free-form outputs $\mathbf{\hat{Y}}$ that may contain multiple labels, including fine-grained ones beyond basic emotions. Therefore, we use the HIT rate as the metric, ensuring that the basic label $y$ should be at least in $\mathbf{\hat{Y}}$.

Meanwhile, we attempt to design metrics for evaluating potentially incorrect labels in $\mathbf{\hat{Y}}$. However, the labels in $\mathbf{\hat{Y}}$ that differ from the basic label $y$ are not necessarily incorrect - they may represent some fine-grained emotions not covered by basic categories. But since basic emotion recognition tasks lack fine-grained reference labels, we have not yet established appropriate evaluation metrics for this purpose. This remains an important research direction for our future work.

Q5: In Table 3, the performance of the LLaVA dataset decreased after applying the filtering strategy. The authors did not explain what the filtering strategy was. If it is the strategy mentioned in the paper, it suggests that this filtering strategy does not work well for the LLaVA dataset. The authors should analyze the reason.

A5: Thanks for your comment. Regarding general instruction datasets, we use a filtering process to retain only emotion-related instruction-answer pairs. Specifically, we use the prompt in Appendix E and extract emotion labels from each instruction-answer pair. Samples yielding empty emotion outputs are removed. As shown in Table 3, this filtering approach proves less effective for the LLaVA dataset. We hypothesize that the detailed descriptions in non-emotion subsets may also provide valuable cues for inferring emotional states in some scenarios.

Q6: Although the paper proposes a new multimodal emotion recognition model with significant performance improvement, the model's innovation compared to previous work is limited, as it mainly adds a pre-fusion mechanism between multiple modalities.

A6: The model architecture novelty is only a part of our work, and our contributions extend significantly beyond this. Besides the model architecture, we also use a model-led, human-assisted strategy to minimize human effort while constructing the largest multimodal emotion dataset to date. Also, we present a comprehensive benchmark covering fine-grained emotion recognition, basic emotion classification, and sentiment analysis, with metrics designed for free-form text outputs.

We sincerely appreciate your recognition of our work's significance and your acknowledgment of our contributions—including the novel dataset, model architecture, and comprehensive benchmark for descriptive emotion understanding. These innovations enable richer, more flexible emotion representation that extends beyond conventional predefined-label paradigms.

Q1: Lack of Baseline Comparisons on MER-Caption+: To better isolate the impact of the new dataset vs. the new model design, it would be helpful to compare baseline models trained on MER-Caption+ against AffectGPT trained on MER-Caption+. This would clarify whether the performance boost comes primarily from the dataset or the model architecture.

A1: In our experimental design, we systematically validate the effectiveness of both the new dataset and model using the control-of-variables method. Specifically, in Table 3, we demonstrate the impact of our MER-Caption+ dataset by maintaining identical model architecture while varying only the training data. In Table 4, we verify the necessity of AffectGPT by keeping the training data constant while modifying only the model structure. These carefully designed ablation studies can already verify the effectiveness of the new dataset and the new model.

Q2: Dataset Coverage and Diversity: The paper does not specify the scenarios included in the dataset (e.g., daily conversations, news reports, or movie reviews). Additionally, it is unclear whether the dataset primarily focuses on first-person speaking or also includes multi-person videos, which could affect its generalizability.

A2: Thanks for your suggestion, and we will incorporate the following additional information in our revised manuscript. In this paper, we intentionally focus on single-person videos, as this allows us to eliminate interference from other speakers and reduce task difficulty. Multi-person MER belongs to another research topic and will be discussed in our future work.

Q3: Impact of Frame Signals on Performance: The results suggest that using face signals alone outperforms frame signals, raising the question of whether frame signals introduce more noise. It would be valuable to explore whether this issue can be mitigated through data cleaning or improved visual signal alignment with other modalities.

A3: Thanks for your comment. We agree that using frame signals may introduce additional noise, resulting in performance degradation. Your suggestions for further verification are insightful, and we plan to explore these aspects through additional experiments in future work.