Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models

Dear Reviewer W74m:

We are deeply grateful for your positive feedback on our work and the insightful suggestions. We have carefully reviewed each point and provided detailed responses accordingly.

Q1: Performance under different prompts (Other Strengths And Weaknesses)

To explore the performance under different prompts, we designed a prompt augmentation semantically similar to Focus-on-Emotion: Identify the predominant emotion conveyed, and conducted corresponding experiments. As presented in the table, the performance of the new prompt is noticeably inferior to that of Focus-on-Emotion. We hypothesize that more direct prompts tend to yield better results. We plan to further explore the impact of various prompt designs on performance in future work.

Table: Ablation experiments on different prompts

Q2: More analysis on removing visual information (Other Strengths And Weaknesses)

Compared to general tasks, visual redundancy in emotion-related tasks is more likely to interfere with the inference bias. Therefore, dropping a small amount of the least relevant visual information can help improve the model’s performance. In Figure 3 of the original paper, we conduct a sensitivity analysis under different drop rates and observe that discarding a small amount of redundant information leads to performance gains. However, excessive dropping results in the loss of some informative content, causing a decline in performance to some extent. Additionally, Figure 4 of the paper visualizes the specific content being dropped, showing that at lower drop rates, the removed tokens are mostly irrelevant, which allows the model to focus more on salient information and enhances its reasoning capability.

Q3: The confidence-based prompt selection process in the first stage (Other Strengths And Weaknesses)

Sorry for the confusion. To encourage the model to focus on relatively simpler problems at each stage, we introduce prompt enhancement for fine-grained inference based on the outcomes of coarse-grained inference. To mitigate the risk of error propagation from the first stage adversely impacting the second-stage inference, we assess the confidence of the model using the variance of the output logits, rather than relying solely on predicted labels. Specifically, when the variance among logits is high (indicating strong confidence), we apply direct positive/negative prompt augmentation. Conversely, when the variance is low (suggesting uncertainty), we incorporate ambiguity-related descriptions to characterize the model decision process in the first stage, which are then used to guide the second stage. The selection of the variance threshold is discussed in the experimental section, where the effectiveness of the proposed strategy is demonstrated.

Q4: Inappropriate expressions (Other Strengths And Weaknesses)

We will revise the inappropriate expressions in the final version. Thanks for your suggestions.

Dear Reviewer 6rbK:

Thank you very much for your valuable comments and constructive feedback. Below, we carefully respond to each of your concerns point-by-point, providing detailed explanations and supplementary evidence to further clarify our approach and demonstrate its effectiveness.

Q1: Attention map computation (Other Strengths And Weaknesses)

Sorry for any inconvenience caused. Within the transformer architecture of MLLM, each block generates an attention map that reflects the model focus at that layer. Notably, attention maps from deeper layers tend to encode more abstract, high-level semantic representations, aligning more closely with the model learned conceptual understanding of the input. Therefore, we use the attention map from the final transformer block as the basis for token dropping in the second stage. Additionally, to enhance the accuracy of emotional focus, we retain only the attention maps between visual tokens and the Focus-on-Emotion prompt.

Q2: More comprehensive ablation experiment (Other Strengths And Weaknesses)

Since we perform a classification task using an MLLM, the metric Accuracy serves as an appropriate indicator of performance. We conducted further ablation studies on the EmoSet dataset. As shown in the table, the effectiveness of each module is further demonstrated. More details will be provided in the final version.

Table: Ablation experiments on additional datasets

Q3: Discussion on inference efficiency (Other Strengths And Weaknesses)

Although the two-stage inference process introduces a certain increase in inference time, the overall inference latency for emotion recognition tasks remains low, due to the relatively lightweight nature of the task, making this additional overhead negligible in practical use. Additionally, we effectively reduced part of the inference time by discarding tokens. Overall, the slight extra time cost results in improved inference performance.

Dear Reviewer jWJJ:

Thank you again for your thoughtful and constructive suggestions. In the following responses, we address each of your points thoroughly, providing additional explanations and supporting evidence to strengthen the clarity of our methods.

Q1: Discussion on generalizability of SEPM (Other Strengths And Weaknesses)

Thank you for highlighting this issue. It should be clarified that 1) the two-stage inference process in our method is specifically designed to address the progressive categorization from coarse-grained to fine-grained levels inherent in emotion recognition tasks, and 2) the visual enhancement component of our method is motivated by the interference caused by redundant visual information in emotion-related tasks. Consequently, both modules are specifically tailored for emotion recognition and are not directly transferable to other tasks. Nonetheless, we have explored applying the Focus-on-X prompt to other domains and conducted experiments on a medical dataset (VQA-RAD [1]). As shown in the table, models across various architectures all exhibited improved performance, indicating that Focus-on-X can effectively serve as a generalized prompt enhancement.

Table: Generalizability experiment on the VQA-RAD Dataset (Medical Domain)

Q2: Experiment on higher drop rate (Other Strengths And Weaknesses)

We conducted ablation experiments with higher drop rates (beta), as shown in the table below. As the drop rate increases, the performance of the model declines to some extent but still maintains a certain inference capability. We believe discarding the least relevant redundant information helps the model better focus on emotion-related content, thereby improving performance. However, as the drop rate continues to increase, the model begins discarding some moderately important information, leading to a reduction in inference accuracy. Nonetheless, since the most critical emotional information is preserved, the model's performance does not completely collapse.

Table: Ablation experiment on drop rate

Q3: Lack of clarity in Equation 6 (Other Strengths And Weaknesses)

Sorry for the confusion. We have revised the relevant part of Equation 6:

Then we extract the FoE prompt set as emotion-related text, leveraging the query-dimension of textual logits and the key-dimension of the visual modality to construct the drop matrix $P \in \mathbb{R}^{L_t \times L_v}$, where $L_t$ and $L_v$ are the lengths of the text tokens (FoE) and visual tokens, respectively. The drop matrix $P$ is defined as:

$$P[i, j] = \mathcal{A}(i, j)$$

$i \in$ { $x \mid \mathbb{I}[x] \in \mathbb{T}$ }, $\quad j \in$ { $y \mid \mathbb{I}[y] \in \mathbb{V}$ },

where $\mathbb{I}$ denotes the set of all input tokens, including both textual and visual modalities, $\mathbb{T}$ denotes FoE prompt token set and $\mathbb{V}$ is visual tokens set, i.e.

$$(\mathbb{T} \cup \mathbb{V}) \subseteq \mathbb{I}$$

We will update the above content in the final version.

Q4: Problem on Figure 6 (Other Strengths And Weaknesses)

Thank you for tips. We will update in the final version.

[1] A dataset of clinically generated visual questions and answers about radiology images, Scientific data, 2018

Dear Reviewer wn8T:

We sincerely appreciate your time and effort in reviewing our paper. We have provided further clarification on each of the issues you raised. We hope the detailed responses below fully address your concerns, and we would be grateful if you would consider updating your score.

Q1: Different with SPARSEVLM & innovations (Relation To Broader Scientific Literature & Other Strengths And Weaknesses & Other Comments Or Suggestions)

Thank you for raising this valuable question. Compared to general tasks, emotion recognition is often susceptible to interference from redundant visual information (noise), leading to inference biases. While SparseVLM accelerates inference for general tasks by retaining only the most crucial information in the visual modality, our method, in contrast, designs a specialized Focus-on-Emotion prompt, effectively identifying emotion-related visual redundancy through prompt-guided relevance. By discarding such redundant information, we can significantly enhance the accuracy of the model in emotion inference. Additionally, in emotion recognition tasks with a large number of categories, semantic similarities between emotions often lead to confusion. Therefore, we introduce Confidence-Guided Coarse-to-Fine Inference, enabling the model to handle simpler subtasks incrementally, thereby improving overall performance.

Q2: Reason for not comparing with other emotion-related methods (Other Strengths And Weaknesses)

To the best of our knowledge, we are the first to explore training-free optimization in general emotion recognition tasks. Therefore, we do not identify suitable emotion-related methods for a fair and meaningful comparison. Although the SoV [1] method mentioned in the paper is also training-free, it is specifically designed for facial emotion images and requires additional manual bounding box annotations. Moreover, since its code has not been released, we do not have the capacity to adapt it to our model architecture and more general emotion recognition tasks in a short period of time. We will try to include a comparison with SoV in the final version, and will continue to follow similar work in the future. Thank you for your valuable suggestion.

Q3: More comparative experiment (Other Comments Or Suggestions)

In the absence of suitable emotion-related baselines for direct comparison, we introduced a comparative experiment with PDrop [2] based on its official code, which aims to eliminate visual redundancy to optimize the model. The experimental results in the table show that PDrop yields a slight improvement over the zero-shot setting by mitigating visual redundancy, supporting the hypothesis that emotional visual redundancy may interfere with the judgment of the model. Moreover, the results highlight the superior effectiveness of our proposed method.

Table: Comparison with PDrop on various emotion datasets.

Thank you for your valuable advice. We are deeply grateful and appreciate your guidance.

[1] Visual Prompting in LLMs for Enhancing Emotion Recognition, EMNLP, 2024

[2] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction, CVPR, 2025