Response to Reviewer LeNg

Response to W1

LNLN is well-suited for applications where multimodal data is often incomplete, which is a common challenge in many real-world scenarios. For example, in platforms like Twitter, Instagram, and TikTok, users often express sentiments through a combination of text, images, videos, and audio. However, due to limitations such as incomplete captions, low-quality images, or background noise in videos, the data collected can be noisy or incomplete. LNLN's ability to prioritize the language modality while robustly integrating information from other modalities makes it ideal for analyzing such content, where data quality is variable. Moreover, in online learning platforms, students interact with content through text, video lectures, and audio discussions. LNLN can be used to analyze student sentiment and engagement, even when certain modalities, such as audio or video, are of poor quality or partially missing.

However, limitations also exist. The data in real-world scenarios is much more complex. In addition to the presence of missing data, other factors need to be considered, such as diverse cultural contexts, varying user behavior patterns, and the influence of platform-specific features on the data. These factors can introduce additional noise and variability, which may require further model adaptation and tuning to handle effectively.

In addition, from the experiments, it can be seen that the LNLN is able to better processing different levels of missing data noise. However, there are still some challenges and areas for improvement. 1) There are differences in how well the model generalizes across different datasets, which suggests that further work is needed to ensure consistent performance. 2) Tuning the hyperparameters, particularly those related to the loss functions, can be challenging and may require more sophisticated approaches to achieve optimal performance. 3) Current MSA datasets are relatively small, and acquiring and annotating such data is difficult. Therefore, improving model performance with limited data is an important area of study. Given the strong performance of large language models in many domains, we believe that future research should explore MSA based on zero-shot and few-shot learning using large language models as a potential research direction.

Response to Q1

For the balance of hyperparameters, we empirically selected them through experiments as described in Sections 3.6 and Sections 4.5. Specifically, the hyperparameters of the loss function were tuned using a combination of grid search and validation performance. We first identified a range of reasonable values for each hyperparameter and then systematically evaluated their impact on a validation set, aiming to find the combination that optimally balances the trade-offs between different objectives.

Additionally, as shown in Table 1 of the uploaded PDF, we have included detailed experimental results on the MOSI and SIMS datasets, further demonstrating the effectiveness of the loss function and the tuning process.

Response to Reviewer xa9F

Response to Limitations

Regarding the Use of Transformer:

First, our primary focus is on the innovation of the algorithm, not on the innovation of the Transformer. We have utilized Transformer layers in a similar manner to how CNNs and LSTMs are commonly employed in various models. It is only a tool to effectively model sequence data. Extensive MSA-related works [2, 4, 6, 7, 8, 10, 11] has demonstrated the efficacy of Transformers in processing sequential data. Thus, we chose to design the LNLN framework based on the Transformer structure. Second, to address concerns about the algorithm’s reliance on Transformers, we evaluated our model by replacing all Transformer layers with LSTM and MLP layers, which are widely used in earlier MSA works [1, 3, 5]. The Table below shows that while there is a slight performance degradation, the overall performance remains competitive and still achieves SOTA results across many metrics. It also demonstrates that our algorithm's structure is not overly reliant on transformers. In addition, the use of Transformer to implement the reconstructor is also seen in the previous methods [4, 8], which is normal practice in this field. Our focus is on validating the effectiveness of our proposed hypothesis through the LNLN framework.

Regarding the effectiveness and necessity of each loss function:

As shown in Tab. 4 of the paper, we have discussed the effectiveness of each loss. We have additionally conducted our experiments on MOSI and SIMS datasets. In addition, as shown in Tab. 1 of the uploaded PDF, the results show that our losses are effectiveness and necessity.

Response to W1/Q1

Regarding the contributions: Please see the general Response.

Regarding the use of language as the primary information carrier:

While language plays a crucial role in MSA, previous works [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] have not fully emphasized the importance of the language modality in method design. Our proposed LNLN addresses this gap by introducing the DMC module, specifically designed to improve the robustness of language features under varying noise scenarios.

Regarding the differentiation from existing methods:

1. Unlike many existing MSA methods [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] that treat all modalities equally, our method prioritizes the integrity of the dominant modality (language). We propose that by ensuring the completeness and quality of this modality, the overall robustness of the model can be significantly improved.
2. Our method uniquely incorporates a Completeness Check step within the DMC module, which evaluate and adjusts the representation of the dominant modality to maintain its quality. This method is distinct from other methods [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] used in MSA, which do not emphasize the completeness of the dominant modality to this extent.
3. Our model is specifically designed to tackle a wide range of noise levels, reflecting real-world scenarios where data quality can vary. The ability to maintain robust performance across these conditions is a key differentiator of our approach compared to other methods that may only perform well under limited or ideal scenarios.

Response to W2/Q2, W3/Q3 and W4/Q4

As mentioned in the above, our primary goal is not to introduce a novel Transformer architecture but to explore how to achieve robust MSA. The innovation of our LNLN lies in the overall framework, particularly in the design of the DMC module. The collaboration between AHL and DMC ensures that the model can effectively mitigate the impact of noise, thus achieving robust MSA. For more details, please refer to the Response to Limitations above.

Response to W5/Q5 and W7/Q7

Tab. 8 and Tab. 9 of the paper have detailed the performance of each method under the influence of different noise intensities, demonstrating the superiority of our method compared to ALMT. In addition, as shown in Fig. 1 and Fig. 2 of the uploaded PDF, we have compared the confusion matrices of ALMT and LNLN, and plotted their F1 variation curves under different noise intensities. It is clear that ALMT does not perform as well as LNLN in high-noise scenarios, especially when it also tends to favour almost all of its predictions to specific classes. As shown in Fig. 3 of the uploaded PDF, We conducted a Case Study through visualisation, and we can see that our LNLN is able to perceive the sentiment cues accurately for the hard samples and thus make accurate predictions. Due to word limitations of rebuttal, we will provide a more detailed analysis in the revised version.

Response to W6/Q6

We chose to perform our ablation studies on the MOSI dataset because it is widely used in previous works for ablation studies [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. This allowed us to maintain consistency with existing works. Additionally, due to time limitation of rebuttal, we have tried our best to conduct the ablation studies on the SIMS dataset. The SIMS dataset is a Chinese dataset which differs significantly from MOSI. It allows us to demonstrate the generalizability and robustness of our approach across different languages and cultural contexts. As shown in Tab. 4 of the uploaded PDF, the result highlights the effectiveness of our method on this diverse dataset.

Response to Reviewer ugFZ

Response to W1

We believe that our contributions are significant and address the gap in current MSA research. The robustness of MSA models in real-world, noisy environments is an important area of study. Most previous methods [1, 2, 3, 4, 5, 6, 7, 11] are evaluated on standard datasets, which may not accurately represent the challenges posed by real-world data. While some recent works [8, 9, 10] have attempted to explore robust MSA in noisy environments, these efforts often selectively report partial metrics, such as Acc-2 and F1, leading to the performance of various MSA methods in noisy scenarios under-explored. Our work aims to bridge this gap by providing a comprehensive evaluation of advanced MSA methods under noisy conditions, which we believe is a crucial step toward making these models more applicable to real-world scenarios. During this evaluation, we identified several overlooked issues. For example, our evaluation revealed a critical but neglected issue: existing models generally fail under high noise conditions. In such cases, most predictions tend to favor the more numerous categories, making the model appear to achieve SOTA results. However, this behavior is misleading and does not reflect the true performance of the model, leading to an unfair comparison with other approaches. More details can be seen in Fig. 3, Fig. 4 and Fig. 5 in the appendix of the paper, and Fig. 1 of the uploaded PDF.

Response to W2

We would like to clarify and emphasize the following points to address your concern:

1. As shown in Tab. 1 and Tab. 2 of the paper, we believe that our method outperforms existing methods in several key aspects. For example, on the MOSI dataset, our method achieves significant improvements in challenging metrics such as Acc-7 and Acc-5, with relative increases of 9.46% and 2.74%, respectively.
2. The reason for not achieving significant improvements in some cases is that our overall performance results are averaged across different noise levels. As mentioned in the Response to W1 and in Section 4.3 of the paper, some metrics did not show significant improvement. This is because that the predictions of other models were heavily biased towards the more numerous categories under high noise scenarios, resulting those predictions meaningless. For example, in Tab. 8 in the appendix of the paper, when the data missing rate is 0.9, the Acc-7 of TFR-Net, which is specifically designed for data-missing scenarios, is 41.73%. It is slightly higher than that of LNLN at 40.10%. However, as shown in the confusion matrix in Fig. 4 of the appendix, TFR-Net predicted almost all test samples to be in Category 3 (i.e., Neutral). This bias significantly inflated the overall performance of TFR-Net in Tab. 1 of the paper, but these predictions are not meaningful in a high-noise scenario. This issue is one reason why TFR-Net's performance appears higher, but it does not reflect genuine robustness.
3. As mentioned in Response to W1, we aimed to provide a fair and comprehensive evaluation, reporting as many relevant metrics as possible rather than selectively reporting only favorable results. It is unrealistic to expect a model to show significant improvements across all fine-grained metrics simultaneously. However, our method consistently maintains a competitive performance level across all reported metrics, highlighting its robustness.

Response to W3

We would like to clarify that while adversarial learning is employed as a tool within our model, the innovation lies in how we utilize it to address specific challenges in MSA. Similar to how other works employ CNNs and LSTMs as basic layers, we have chosen this pipeline to achieve our specific objectives. The innovation is embodied in the design of the DMC module. This pipeline is carefully designed based on our hypothesis and offers a novel approach to improving the robustness in MSA. Unlike previous approaches [8, 9] that typically treat each modality equally and reconstruct whichever information is missing, our method prioritizes the dominant modality (language) and focuses on improving its completeness in noisy scenarios. Overall, our innovative application of adversarial learning is not just in the use of GANs, but in how it is integrated into our approach to address the specific challenges of robust MSA.

Response to Q1

As to the format of the references themselves, any style is acceptable as long as it is used consistently. You can refer to the PDF of "Formatting Instructions For NeurIPS 2024".

Response to Q2

The hypothesis is first proposed and then experimentally validated.

Response to Q3

The Completeness Check mainly used to evaluate the completeness of the language features and determine to what extent these features are affected by noise or missing data.

1. $H_{cc}$ is a randomly initialized token used as an initial embedding that is concatenated with language features $H_l^1$. Through learning, it is used to predict the completeness of the $H_l^1$. Random initialization ensures that the feature is free from any a priori bias.
2. First, the language representation $H^1_l$ and $H_{cc}$ are fed into the encoder. The encoder processes the input and capture the completeness of the $H^1_l$. Then, the encoder outputs a completeness weight ($w$) that indicates the degree of completeness. This weight is used to balance the $H^1_l$ and the proxy features $H^1_p$ generated by the adversarial learning. Finally, the $w$ is used to combine the $H^1_l$ with the generated proxy features. This ensures that the final dominant modality representation maintains high quality and integrity in the presence of noise.
3. By evaluating and adjusting the dominant modality representation through this process, the Completeness Check helps maintain the robustness and reliability of the model.