Response to Reviewer Wv59

Response to W1

Thank you for the question. We follow standard practice in MSA and formulate the task as regression because all three benchmark datasets (SIMS, MOSI, MOSEI) provide continuous sentiment scores (SIMS: [-1, 1], MOSI/MOSEI: [-3, 3]). As mentioned in Appendix B, these fine-grained labels support regression-based training. To compute classification metrics (Acc-2, F1), we adopted a threshold-based way which is consistent with previous studies [1, 2]. For example, scores >0 are treated as positive, ≤0 as non-positive (for binary classification). This method is widely used in the MSA studies.

Response to W2

Thank you for the insightful comment. As shown in Table 1 below, although it is expected that the student may underperform the teacher (which uses MLLM prompts as input), the student achieves competitive results with much fewer parameters and significantly lower inference cost. For example, our student model achieves competitive performance (F1: 81.85±1.41 on SIMS) with only 0.82M parameters, 8.61 GFLOPs, and 6.39s test-time inference. These results demonstrate the potential of our framework that make general knowledge of MLLMs to guide the training of task-specific MSA models, offering new insight into applying general MLLMs to improve MSA. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs are relatively higher. This is because we do not compress the input sequences' length as the prior methods [1, 2]. With further optimization (e.g., sequence dimension reduction), the computational cost of MMSLF can be further reduced.

Table 1. Comparison of Efficiency (Evaluate on SIMS dataset). Bold results indicates the optimal result. The experiments were conducted on a Computer with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40 GPU.

Response to W3

Thank you for the observation. The teacher model in our framework serves as a medium to transfer the general knowledge of MLLMs into a smaller student model. While the teacher leverages prompts from a powerful MLLM (e.g., GPT-4o-mini), some performance gap is expected due to information loss during knowledge transfer, which is common in teacher-student frameworks. The purpose of introducing the teacher is not to surpass the MLLM, but to bridge its capabilities into a more efficient, task-specific model. Ultimately, our student model achieves trade-off performance, validating the effectiveness of our idea.

Response to Q1

It is directly obtained from the datasets. We will emphasize these points in the revised dataset introduction. Thank you for your question.

Response to Q2

Thank you for the question. We model the task as regression because all three benchmark datasets (SIMS, MOSI, MOSEI) provide continuous sentiment annotations (SIMS: [-1, 1], MOSI/MOSEI: [-3, 3]), and regression enables better supervision for fine-grained sentiment intensity. As mentioned in Appendix B, these fine-grained labels support regression-based training. To compute classification metrics (Acc-2, F1), we adopted a threshold-based way which is consistent with previous studies [1, 2]. For example, scores >0 are treated as positive, ≤0 as non-positive (for binary classification). This method is widely used in the MSA studies.

Response to Q3

Thank you for the question. In this work, we use preprocessed modality sequence provided by each dataset, which are commonly adopted in prior MSA methods. These feature representations vary across datasets in both dimension and format, making direct cross-dataset evaluation infeasible. We follow this setting to ensure fair comparison with existing baselines.

Response to Q4

As shown in Table 2 below, we evaluated our method using different prompt templates to assess robustness. We selected GPT-4o-mini and Gemini-2.0-Flash because these larger MLLMs typically exhibit stable outputs across prompt variations, which is essential for reliable knowledge distillation. Our experimental results show minimal performance variance across different prompt formulations within the same MLLM, confirming that advanced large-scale MLLMs generate consistent outputs regardless of reasonable prompt variations. This stability validates our framework's robustness to prompt engineering choices. However, the more significant factor is the selection of an appropriate teacher mLLM with strong domain-specific capabilities. The performance difference between GPT-4o-mini and Gemini-2.0-Flash may stem from their varying abilities to generate accurate and informative prompts for sentiment-centered multimodal tasks, rather than sensitivity to prompt variations. Therefore, we believe this might be due to that Gemini‑2.0- Flash was trained on comparatively less fine‑grained affective data, limiting its ability to craft sentiment‑focused multimodal descriptions. We will include this discussion in the revised manuscript.

Table 2. Comparison of Prompt Sensitivity and Robustness. Bold results indicates the optimal result.

Reference

[1] MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In ACM MM 2020.

[2] Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP 2023.

Response to Reviewer ctoX

Response to W1

Thank you for your valuable comments. (1) As shown in Table 1, in the revised manuscript, we will add the following which quantitatively compares advanced MLLMs and our MMSLF in terms of parameter count, GFLOPs, and inference time. Obviously, our student model achieves competitive performance (F1: 81.85±1.41 on SIMS) with only 0.82M parameters, 8.6 GFLOPs, and 6.39s test-time inference. In contrast, Gemini-2.0-Flash and GPT-4o-mini require larger parameters and >22min inference time. These results demonstrate that our method offers a trade-off between performance and computational cost. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs are relatively higher. This is because we do not compress the input sequences' length as the prior methods [1, 2]. With further optimization (e.g., sequence dimension reduction), the computational cost of MMSLF can be further reduced.  (2) We can see that the Student’s performance is lower than that of the Teacher, GPT‑4o‑mini, and Gemini‑2 Flash because it omits MLLM‑generated prompts and inevitably loses some information during distillation. However, this design itself confirms the feasibility of using general MLLMs to improve a much smaller model’s performance. (3) Although there remains a gap between the Student and GPT‑4o‑mini or Gemini‑2.0-Flash, the student still surpasses other large models such as GPT‑4V. In summary, we believe our motivation is sound and the corresponding exploration meaningful, and we will further clarify this in the revised manuscript. In addition, due to the limitation on the number of rebuttal words. Overall, these results demonstrate that our method offers a trade-off between performance and computational cost.

Table 1. Comparison of Efficiency (Evaluate on SIMS dataset). Bold results indicates the optimal result. The experiments were conducted on a Computer with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40 GPU.

Response to W2

Thank you for pointing this out. As shown in Table 2 below, we have added an experiment: on the SIMS dataset we conducted two‑tailed t-tests between the Student model and the best task‑specific baseline. The resulting p-values are MAE = 0.0029, Corr = 0.00022, Acc‑2 = 0.095, and F1 = 0.066. Under the conventional 0.05 threshold, the Student’s improvements on MAE and Corr are statistically significant, while the gains on ACC‑2 and F1 show the same positive trend. Because of time and rebuttal‑length constraints, we will include a more detailed comparison on all datasets in the revised manuscript.

Table 2. Two‑tailed t-tests between the Student model and the ALMT on SIMS dataset.

Response to W3

Thank you for raising this point. (1) In Section 4.8, we mentioned that "the prompts may contain more misleading information, which may result in poorer performance when using prompts generated by Gemini-2.0-Flash for model training as discussed in Section 4.7. (2) We have provided one case (you can also refer to Case 1 in Appendix Figure 6), and the descriptions generated by Gemini sometimes contain errors. (3) As shown in Table 3 below, we use different templates to generate prompts and then use them to train the model. The results show that minimal performance variance across different prompt formulations within the same MLLM, confirming that advanced large-scale MLLMs generate consistent outputs regardless of reasonable prompt variations. Therefore, we believe this might be due to the fact that Gemini‑2.0- Flash was trained on comparatively less fine‑grained affective data, limiting its ability to craft sentiment‑focused multimodal descriptions. We will include this discussion in the revised manuscript.

Case 1 in Appendix Figure 6:

Sentiment Label: Positive

Audio Prompts of Gemini-2.0-Flash: The speaker has a conversational tone that comes off as conversational.

Table 3. Comparison of Prompt Sensitivity and Robustness. Bold results indicates the optimal result.

Response to W4

Thank you for pointing this out. In the revised manuscript we will improve the table caption. For example, stating the brief information and explanations of any symbols or abbreviations.

Response to Q1

Please see the Response to W3. We will include a more detailed comparison in the revised manuscript.

Response to L1

Thank you for this question. As mentioned in Response to W1 and the corresponding table, although Gemini‑2 .0-Flash delivers the advanced performance, it relies on an external 7 B‑parameter API whose end‑to‑end inference exceeds 22 min and is bounded by network latency. By contrast, with the guidance of the teacher during the training stage, our MMSLF‑Student achieve competitive performance with fewer parameters, fewer GFLOPs, and fewer inference time. Similar phenomena can also be observed in GPT-4o-mini. These results illustrate that our framework offers a new insight into real-world applications of MSA.

Response to L2

Thanks for your suggestions! We will specifically mention this in the revised paper. Actually, all task‑specific models, as well as Gemini‑2.0‑Flash and Video‑LLaMA, take vision, audio, and language as inputs. GPT‑4o‑mini and GPT‑4V take vision and text as inputs because they do not support audio analysis.

Response to L3

As shown in Response to W2, we have added the significance test to compare with the suboptimal model ALMT. We will include a more detailed comparison in the revised manuscript.

Reference

[1] MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In ACM MM 2020.

[2] Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP 2023.

Response to Reviewer RVt6

Response to Cons 1

Thank you for the suggestion. It should be noticed that five runs results are usually lower than searched best-run results. We have realized that including more recent methods can strengthen our claims. As shown in the Table 1 below, we compare methods without open-sourced code (i.e., TCAN, DeepMLF) using reported best-run results. For those with open-sourced code (i.e., MAG-BERT, DMD, DLF), since the limited time of rebuttal, we will reproduce five-run averages of these methods under the same setting on all datasets in the revision. Obviously, the results show our MMSLF achieves competitive/better performance in many metrics when compared with task-specific models on MOSI, MOSEI and SIMS datasets. For example, both teacher and student outperform the work DLF on MOSI, while DLF performs better on MOSEI. Compared to DeepMLF (task-specific MLLM), the student underperforms on several metrics. We believe this is primarily due to the fact that part of the knowledge transfer from the MLLM and the teacher may be lost due to the student’s limited capacity. Despite this, our method achieves competitive performance, demonstrating a favorable trade-off. We will also discuss the recent/contemporaneous works (i.e., DLF and TCAN) in the revised version in section relevant work.

In addition, regarding MMML, since it employs stronger encoders for text and audio feature extraction, we are working toward a fair comparison under a consistent setting. Specifically, we adopt RoBERTa for text encoding and HuBERT for audio encoding, and we will report the average performance over three runs. These experiments are currently in progress, and we will update the results on OpenReview once they are available.

Table 1. Performance comparison. Bold results indicates the optimal result.

Response to Cons 2

Thank you for the suggestion. We provide a detailed comparison in Table 2. Notably, our student model achieves competitive performance (F1: 81.85±1.41 on SIMS) with only 0.82M parameters, 8.61 GFLOPs, and 6.39s test-time inference. In contrast, Gemini-2.0-Flash and GPT-4o-mini require larger parameters and >22min inference time. These results demonstrate that our method offers a trade-off between performance and computational cost. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs are relatively higher. This is because we do not compress the input sequences' length as the prior methods [1, 2]. With further optimization (e.g., sequence dimension reduction), the computational cost of MMSLF can be further reduced.

Table 2. Comparison of Efficiency (Evaluate on SIMS dataset). Bold results indicates the optimal result. The experiments were conducted on a Computer with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40 GPU.

Response to Cons 3

Thank you for the suggestion. We include a visualization in Appendix Section D.6 and Appendix Figure 4 that illustrates the alignment learned by the student model under the guidance of MLLMs. The results show that prompts from MLLMs can improve the student’s multimodal alignment. As images and hyperlinks are not supported in the rebuttal, we will include more qualitative cases in the revised version to further highlight the unique contributions of our method.

Response to Q1

Please see the Response to Cons 1.

Response to Q2

Thank you for the question. Considering both cost and performance, we obtain all prompts once before training the teacher model. We also experimented with generating three prompts per sample and randomly sampling one during training on SIMS dataset, but found that this way lead to higher cost without significant performance improvement. As shown in Table 3 below, the "obtain once prompt" setting achieves comparable or even better results. Moreover, we observe that the “sampling from three prompts” strategy introduces noticeably higher variance for MMSLF-Teacher, especially in MAE and Corr (e.g., 2.71 vs. 0.50 for MAE std), suggesting that sampling different prompts may introduce inconsistent guidance and lead to unstable training. However, for MMSLF-Student, the variance across runs is relatively small in both settings (e.g., F1 std: 1.41 vs. 0.66), indicating that the student model is less sensitive to prompt sampling. This is likely because the student learns from the teacher’s distilled representations and attention patterns, rather than directly using the prompts.

Table 3. Effect of prompt sampling methods on performance. Bold results indicates the optimal result.

Reference

[1] MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In ACM MM 2020.

[2] Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP 2023.

Thank you for your feedback and for raising your score! And sorry for the late of the results of the MMML comparison. We have been working to address all your concerns.

Consistent with MMML, we replaced the original text encoder and audio encoder with RoBERTa and HuBERT. Then, we ran the code three times on largest MOSEI datasets and report the results in the table below. We can see that the student outperforms MMML on most metrics. We also observed an interesting phenomenon: the student slightly outperforms the teacher on MAE and on some of the Acc-2 metrics. For example, the teacher achieves an MAE of 0.521 while the student achieves an MAE of 0.508. We believe that this occurs because we retained direct supervision from the ground truth labels rather than fully aligning with the teacher during student training, so some of the teacher’s subtle biases were not transferred.

We sincerely appreciate your suggestions and have agree your opinion that the findings and the experimental support are important. What we want to emphasize is also these points. In the revised manuscript, we will highlight these points and include MMML comparisons across the datasets.

Table 3. Comparison with MMML on MOSEI dataset. Bold results indicates the optimal result.

Thank you for your response and for your engagement in the discussion. The MMML paper mentions that different encoders (i.e., Data2Vec and HuBERT) were used for different datasets to obtain optimized extraction of features, thus achieving a solid foundation for the subsequent fusion process. And according to your suggestion, we also conducted experiments using RoBERTa and Data2Vec as the text and audio encoders, respectively. The results are shown in the Table below. Although MMML achieves higher performance on Acc2-Has0 (86.32%) and F1-Has0 (86.23%), our method outperforms on more metrics. For example, the teacher achieves better results on Corr (0.792±0.15). The student achieves better results on Acc2-Non0 (87.09±0.25), F1-Non0 (87.18±0.24) and MAE (0.513±1.27). Interestingly, similar to the case when using HuBERT as the encoder, the student model slightly surpasses the teacher on some metrics. We believe that ground-truth supervision was retained alongside the teacher signal in student training allowing the itself to avoid some potential noise or biases learned by the teacher.

After referring to your suggestions and having a discussion among the authors, we think a cautious description would be more appropriate when MMML is added for comparison. In the revised version, we will revise the use of the term “SOTA” and opt for a more accurate description of our results. For example, explicitly state on which metrics the model achieves the better performance, such as Acc2-Non0, F1-Non0, MAE, and Corr.

Finally, we consider the comparison with MMML to also serve as an discussion that shows the impact of different encoder choices on our model performance. We will include further discussion in the Appendix.

We sincerely appreciate your feedback and hope that our response can address your concerns.

Table 5. Comparison with MMML on the MOSEI dataset. RoBERTa and Data2Vec are used as the text and audio feature extraction, respectively. Bold values denote the best results for each metric.

Response to Reviewer EAd9

Response to W1

We sincerely thank your feedback. Since the submission was only 9 pages long, some details may not be included. In the revised paper, we will: (1) Explain the information and format contained in the prompts. (2) Emphasize the innovation of the framework. (3) Clarify our ideas more clearly, which refer to using general knowledge from large-scale language models to guide the training of MSA models for specific tasks. Please let us know if you have any more specific suggestions. Thank you very much!

Response to W2

Thank you for bringing this important recent work to our attention. (1) We missed this concurrent work (arXiv:2504.11082), which was released just one month before the NeurIPS submission deadline. We will add comparisons in our revision. (2) As shown in Table 1 in Response to Reviewer RVt6, we include the methods for comparison. We compare methods without open-sourced code (i.e., TCAN, DeepMLF) using reported best-run results. For those with open-sourced code (i.e., MAG-BERT, DMD, DLF, MMML), since the limited time of rebuttal, we will reproduce five-run averages under the same setting on all datasets in the revision. Obviously, the results show our MMSLF achieves competitive/better performance in many metrics compared with task-specific models. For example, both teacher and student outperform the work DLF on MOSI, while DLF performs better on MOSEI. Compared to DeepMLF (MLLM), the student underperforms on several metrics. We believe this is due to the fact that part of the knowledge transfer from the MLLM and the teacher may be lost due to the student’s limited capacity. Despite this, our method achieves competitive performance, demonstrating a favorable trade-off. We will discuss these works in the revised version.

Response to W3

We sincerely thank you for your sharing and suggestions! In our revision, we will: (1) Expand the literature review to discuss novel conceptual contributions and emerging ideas in the field. (2) Include deeper discussion of the new insights we discovered, incorporating the key points raised in our responses to all three reviewers.

Response to W4

We sincerely thank you for your recognition of our teacher model design. Regarding the student architecture, we would like to clarify: (1) While the pipeline is relatively straightforward, this design choice was made based on empirical findings that this structure can achieve distillation efficiency for our specific task. The simplicity of the architecture provides a clearer demonstration of our method's effectiveness by minimizing confounding factors from complex architectural innovations. (2) As shown in Table 1 below, we conducted preliminary experiments attempting to directly integrate MLLM prompts into the existing ALMT model. Obviously, ALMT-Teacher achieved the best performance, but ALMT-Student performed worse than MMSLF-Student. We believe that architectural simplicity is not merely a limitation but rather an informed design choice that balances effectiveness with training stability. (3) As mentioned in lines 56-57 in the paper, we believe the contribution lies in our whole framework that explores using the general knowledge of MLLMs to guide the training of task-specific MSA models, offering a new insight into applying general MLLMs to improve MSA.

Table 1. Generality of the proposed framework. Bold results indicates the optimal result.

Response to W5

Thank you for raising this concern. The choice of MLLM significantly impacts performance through the quality of generated guidance signals. As discussed in Section 4.8, our analysis reveals that Gemini-2.0-Flash produces less accurate and informative prompts compared to GPT-4o-mini, reflecting its weaker capabilities in sentiment-aware multimodal understanding. However, (1) We provide our detailed prompt templates in our appendix. (2) As general MLLMs continue to advance rapidly, we believe that this dependency will become less problematic. We will discuss this in our revision.

Response to W6

This phenomenon is caused by the architectural constraints. (1) To effectively learn from MLLM prompts, our teacher model requires sufficient capacity. As shown in Table 8 in the Appendix, we set it to 12 layers. When mLLM prompts are removed, this relatively large teacher model suffers from severe overfitting on the limited MSA training data, leading to poor generalization on test sets. (2) While the student model is parameter-efficient, it lacks the representational capacity to achieve competitive performance without teacher guidance. Therefore, our current design represents a carefully calibrated trade-off between performance and parameters. We will clarify this in the revised version.

Response to W7

While our current implementation has constraints that limit direct plug-and-play application, we believe this represents an important first step toward a broader research direction. (1) Our idea can be adapted to existing MSA models. For example, we tested this on a strong baseline ALMT. Specifically, we concatenate prompts with datasets' text inputs followed by dimensionality reduction. As shown in Table 1 above, although the final performance was also good, we observed slower convergence and suboptimal performance compared to our proposed MMSLF-Student model. (2) Despite these constraints of direct plug-and-play application, our work demonstrates the potential of utilizing MLLMs to improve task-specific model learning. We aim to validate this research direction and offer new insight into applying general MLLMs to improve MSA. We will discuss this in the revised paper.

Response to Q1

We attempted to apply our framework to the recent work (arXiv:2504.11082). However, no open-source implementation was available for this method. Therefore, we present experimental results using ALMT, a strong baseline in the MSA domain. More detailed discussion can be found in our response to W4, W7 and Table 1 above.

Response to Q2

As shown in Table 2 below, we evaluated our method using different prompt templates to assess robustness. We selected GPT-4o-mini and Gemini-2.0-Flash because these larger MLLMs typically exhibit stable outputs across prompt variations, which is essential for reliable knowledge distillation. Our experimental results show minimal performance variance across different prompt formulations within the same MLLM, confirming that advanced large-scale MLLMs generate consistent outputs regardless of reasonable prompt variations. This stability validates our framework's robustness to prompt engineering choices. However, the more significant factor is the selection of an appropriate teacher mLLM with strong domain-specific capabilities. The performance difference between GPT-4o-mini and Gemini-2.0-Flash may stem from their varying abilities to generate accurate and informative prompts for sentiment-centered multimodal tasks, rather than sensitivity to prompt variations.

Table 2. Comparison of Prompt Sensitivity and Robustness. Bold results indicates the optimal result.

Response to Q3

As mentioned in Section Introduction, cost-effectiveness is mainly compared with MLLMs. Because directly applying MLLMs to MSA is costly. To make a more comprehensive comparison, we have also provided the parameter comparison with the task-specific model. As for the work (arXiv:2504.11082) you mentioned, since there is no open-source code available, we are unable to provide a comparison. Due to the limitation on the number of rebuttal words, the detailed results and discussion can be found in Response to Cons2 and Table 1 in Response to Reviewer RVt6.

Response to Q4

We have extended our framework to Emotion Recognition and Sentiment Analysis in subsequent work, with preliminary results demonstrating that appropriately selected MLLMs can effectively guide task-specific model learning across different domains.