Response to Weakness

Thank you very much for your feedback for our work! We do appreciate your recognition of the originality of our work. Indeed, we are the first to introduce this specific framework that leverages MLLMs to help the training of the task-specific small-scale models for better MSA.

About the weakness, our choice of a simple and straightforward model structure was intentional. By keeping the design clear and direct, we aimed to demonstrate the core effectiveness of our idea and framework itself. This can ensure that the improvements we observe are due to the framework rather than any particular trick or complex structural design. We believe that a simple yet effective method can also offer valuable contributions to the community, as has been shown by many solid works [3,4,5]. Our goal is to do a solid work which we hope can inspire further research and practical applications in MSA.

Dear Reviewer wMDu,

We hope you are doing well. Thank you very much for your positive comments on our paper! We are writing to follow up on the response to our submission.

We have submitted a revised version and responded to the reviewers' comments. We would appreciate your kindly seeing our response and appreciate it if you would consider re-evaluating our paper.

If there are any remaining questions or if further clarification would be helpful, please do not hesitate to let us know.

We are looking forward to your response.

Best regards,

The Authors

Response to W1

Thank you for suggesting the inclusion of the CMU-MOSEI dataset for evaluation. Due to the financial cost of using chatGPT API to help in training on such a large dataset, we faced some economic constraints during the initial experiments, which made it challenging to conduct evaluations on MOSEI. In response to your feedback, we have conducted experiments on the MOSEI dataset. As shown in the table below (Table 3 of the paper), the results on the larger dataset (MOSEI) show that PGMF-Teacher/-Student achieves advanced performance on most of the metrics with few parameters. This demonstrates that PGMF has good generalization ability on data sets of different sizes. It is worth noting that Self-MM with the fewest parameters shows well performance on the MOSEI dataset. This also demonstrates that the feasibility of suitable strategies to achieve strong performance with smaller parameters.

Response to W2, W3 and W4

Thanks for your suggestion. We have made some effort to move the relevant work (Section 2) to the main body and ensure clarity and conciseness of the method presentation. The main revisions are: 1) Remove the "Searched best seed" related rows and only report the average results of five runs. 2) The size of Figures 1, 2, and 3 are reduced to make room for the related work. 3) Remove some equations in Sections 3.4 and 3.5 to ensure clarity and conciseness. 4) Explain explain the fusion operation in detail in Section 3.5. The revised content can be found in the PDF of the revision rebuttal.

Response to W5

1. Figure 3 is an attention difference map (not attention map), obtained by subtracting the attention map without MLLM guidance from that of the PGMF-Student. The values represent the difference in attention scores, with positive differences indicating areas where the PGMF-Student pay more attention to.
2. In the original attention map, each row’s attention scores sum to 1 after the softmax operation. However, due to the relatively long sequence length (e.g., 55 frames for video sequences in the SIMS dataset), the attention scores become more distributed across frames, resulting in a sparsity and low attention scores. Despite the low individual values, we can see from the difference map that these subtle changes in scores effectively shift the model’s focus across large regions, showing the prompt’s impact on attention distribution.
3. In long-sequence video data, we think changes between adjacent frames are continuous and gradual, making large attention score changes less possible. However, small changes in attention scores across many regions are sufficient to shift the model’s focus and significantly impact the multimodal alignment.
4. Our pipeline is intentionally designed to be simple, without any complex mechanisms in the alignment and fusion modules. This choice ensures that the improvements in performance are due to the framework itself, rather than relying on intricate structures or tricks. We believe this simplicity also demonstrates the effectiveness of our proposed framework.

Hi, Reviewer PNKv. We have updated the responses, including results on the MOSEI dataset and some revisions to the paper. The details can be found in the PDF of rebuttal revision. If you have any further questions, please do not hesitate to discuss them with us. Thanks for your suggestion.

Dear Reviewer PNKv,

We hope you are doing well. We are writing to follow up on the response to our submission. We appreciate your time and effort in reviewing our work. We know this is a busy time, but we would appreciate your kindly seeing our response. We would appreciate it if you would consider re-evaluating our paper.

If there are any remaining questions or if further clarification would be helpful, please do not hesitate to let us know.

We are looking forward to your response.

Best regards,

The Authors

Response to Q4/W4

In our initial submission, the most recent baseline with best results was ALMT. It was published in December 2023 at EMNLP, which was one of the best results with open-source code available at the time. However, with recent releases from conferences like EMNLP 2024, we now have additional methods for comparison. Specifically, we have included the latest baselines for comparison, including KuDA [6] and FISFN [7]. As shown in the table below, we can see that PGMF-Teacher/Student achieves SOTA performance in all metrics on the SIMS dataset, demonstrating the effectiveness of our idea and framework. On MOSI datasets, the PGMF-Teacher/Student also can achieve good performance, especially in Acc-2 and F1. In addition, it should be noticed that these recent methods are not open-sourced. So we are unable to conduct multiple runs to report the mean and standard deviation for a comprehensive comparison.

Response to Q5/W5

Thank you very much for your feedback on the "EFFECT OF EACH COMPONENT" section! We would like to clarify that this paragraph was not generated by AI. In our submission, we condensed this section to meet length requirements, which may have inadvertently impacted the depth of the analysis. To address your concern, we have added detailed discussion in this section to provide a more critical analysis. In addition, we will reduce less essential content, such as removing single-run results and retaining only the multi-run averages and standard deviations for comparison, as suggested by Reviewer PNKv. The revised paragraph is as follows:

To evaluate the impact of each component within the framework, we conducted experiments by removing specific components. First, when we removed the MLLMs' prompt from the PGMF-Teacher, we observed a significant drop in performance across both datasets. Specifically, on the SIMS dataset, the F1 score decreased from 84.06% to 80.84%, and MAE increased from 0.370 to 0.436. A similar trend was observed on the MOSI dataset, where the F1 score dropped from 85.15% to 79.60%, and MAE increased from 0.734 to 0.914. These phenomenoa show that the MLLMs plays a crucial role in helping the model capture relevant multimodal information more effectively. Scond, we removed the guidance of the PGMF-Teacher during the training of the PGMF-Student. This led to a noticeable decrease in the student model's performance, with the F1 score on SIMS dropping from 81.85% to 78.72%, and on MOSI from 83.68% to 83.00%. The increase in MAE values on both datasets also reflects the PGMF-Student model's reduced ability to align multimodal information without teacher guidance. This result shows that the importance of knowledge distillation, as the PGMF-Teacher's guidance can help the PGMF-Student learn the relationship between each modality effectively.

In addition, we also observed that the guidance from the PGMF-Teacher had a greater impact on the student model’s performance on the SIMS dataset compared to the MOSI dataset. We believe that this difference may be because of the diversity of data in the SIMS dataset. Specifically, the data of SIMS dataset contains complex environments and disturbances such as lighting, head pose and audio background noise. This makes the data difficult for the PGMF-Student to achieve better performance without relying on the guidance of the PGMF-Teacher.

Response to Q6/W6

Thank you for highlighting the importance of comparing parameter counts with other baselines. In Table 7 of the appendix, we included a comparison with ALMT (the best-performing prior method). However, we also have realized that it is important to compare with more methods. Therefore, as shown in Table blow, we have added parameter count comparisons with other relevant methods. In addition, it is worth noting that for the latest methods like CuDA [6] and FISFN [7] mentioned above, we were unable to include parameter counts due to the lack of open-source code.

As we can see from the table below, although PGMF-Student is with the second smallest parameter size only to Self-MM (a simple, direct and effective method), it achieves better performance. This demonstrates the effectiveness of our proposed framework and shows that PGMF achieves a balance of performance and parameters. In addition, we have also achieved significant improvements compared to the Transformer-based methods with few parameters.

Response to Q1/W1

As shown in Figure 2 of the paper, we first use the MLLMs' prompt as a query to extract essential information from other modalities, obtaining a shifted attention map △. This shifted attention map △ is then applied to the original attention map by the dot product (it can be seen as an attention map fusion/transfer), effectively adjusting and optimizing the alignment process with the help of MLLMs. To verify the effectiveness of our idea, we choose the straightforward (dot product) to design the conditional alignment module. By employing a simple pipline, we focused on demonstrating that the MLLMs, even with basic structures/operations, could provide guidance that helps improve the task-specific small-scale model’s representation learning and overall performance. Specifically, the responses to contributions are as follows:

1. As mentioned in General Response, the dot product within the conditional alignment is not the innovation point of the PGMF. Our focus is to validate the effectiveness of the framework through the straightforward design. Instead, the core contribution lies in providing a novel way for MLLMs to directly participate in regulating the alignment process, helping the task-specific small-scale model focus on more relevant cross-modal relationships. This innovation combines MLLMs' prompts with cross-modal alignment, enabling more efficient completion of alignment tasks and improving performance in multimodal sentiment analysis. Although some existing methods [1,2] use MLLMs/LLMs to assist task-specific small-scale model training, they typically rely on MLLMs/LLMs to generate high-quality data. In contrast, our approach differs fundamentally in principle.
2. As shown in Figure 3 of the paper, we demonstrates the effect of the conditional alignment module by showing the difference between vision-language attention maps with and without the MLLMs' help. The difference map clearly indicates that the MLLMs can help the model focus more precisely on key regions in the language and visual modalities, demonstrating the effectiveness of this core idea. Additionally, experiments on the SIMS and MOSI datasets (Table 1 and Table 2 of the paper, and Table 7 of the appendix) show that the prompt-guided alignment module enables both the PGMF-Teacher and PGMF-Student to achieve state-of-the-art performance across most metrics. These results further confirm the empirical contribution of our framework.

Response to Q2/W2

As mentioned in General Response, our primary focus is not on the fusion method itself, but rather on how MLLMs are leveraged to help the learning of task-specific small-scale models. Therefore, our innovation lies in the whole pipeline of the proposed framework PGMF. Therefore, we opted for a simple way in the fusion module, using concatenation and a Transformer encoder, to emphasize the core impact of MLLMs in improving the task-specific small-scale model’s performance. By achieving performance improvements without complex module design**, we believe the model's improvements in** MSA are mainly derived from the framework. We hope this clarification helps to convey the main contribution to our work.

Response to Q3/W3

As mentioned in General Response, our core idea centers on the whole pipline that using MLLMs to help the learning of task-specific small-scale models. We intentionally chose the simplest method to validate our idea, which we believe can better demonstrate the effectiveness of our method. By keeping the implementation straightforward, we can clearly show that the performance gains are due to the prompt-guided alignment rather than any additional complex module designs.

In addition, Reviewer wMDu confirmed that our method has not been seen in other papers, it is original. Reviewer PNKv recognized that our method is with solid contribution and could be utilized for more tasks by the research community. These comments also demonstrate the novelty of our PGMF. In a word, we believe that simple and effective methods are also valuable, simplicity is not an indication of lack of innovation. There are many works [3,4,5] that has simple architecture but make a great contribution to the community.

Thank you for your detailed response and the additional experiments provided. I have carefully reviewed your rebuttal, and I appreciate the effort and thoughtfulness that went into addressing the initial comments.

Your approach to leveraging LLM to train a lightweight model for multimodal sentiment analysis is indeed meaningful and has potential. However, considering the current version of the work, I believe it aligns more with a score of 5 at best. ICLR places significant emphasis on algorithmic innovation, and I feel that aspect could be further strengthened in your submission.

Thank you again for your efforts and engagement.