Complete multi-modal metric learning for multi-modal sarcasm detection

[Q3] Row 153: What is the size of the target range k and whether it will affect the module.

[A3] We are grateful for the reviewer’s detailed feedback. k is the dimension of the object existence vector generated by YOLO. For instance, an object existence vector could be represented as [car: 1, cat: 0, dog: 1, ...]. Since YOLOv10 is pre-trained on the COCO dataset, the value of k is the same as the number of target categories in the COCO dataset, which is 80. We can use other object detection models, such as Fast RCNN, to annotate the object existence vector for the training set. But if they are also pre-trained on the COCO dataset, the k value will also be equal to 80.

The main purpose of the YOLO-task is to learn a fine-grained fact representation space that is separated from the sentiment representation space. Therefore, the k value will directly affect the training effect of the YOLO-task on the fact projection layer and may even be misleading. We conducted a simple experiment by randomly selecting k target categories to prove it. The experimental results are shown in the following Table 1.

Table 1: Results on MSD Dataset with different k

This table shows the model performances with different k on the MSD dataset.

The experimental results confirm the conjecture that the k value will directly affect the training effect of the YOLO-task on the fact projection layer. Therefore, we recommend choosing an object detection model that is pre-trained on a larger dataset.

We will make the definition of k more explicit in the revised manuscript.

[Q5] Supplemental baseline: Comparisons of relevant sarcasm detection work are missing, and it is recommended to add, e.g., G2SAM[4], DynRT-Net[5], DMSD-CL[2].

[A5] We greatly appreciate the reviewer’s suggestion. We have supplemented the experimental results of G2SAM [1], Multi-view CLIP [2], and DMSD-CL [3] on the MSD dataset. The supplemented content is shown in the following Table 4.

Table 4: Supplemented Results on MSD Dataset

This table shows the model performances on the MSD dataset.

Results with ⋆ denote the significance tests of our CMML-Net over the baseline models at p-value < 0.01. The best results are highlighted in boldface, while the second-best results are underlined.

We list the experimental results of these existing methods which use RoBERTa as the text encoder on the MSD dataset in the following Table 5.

Table 5: Supplemented Results on MSD Dataset with RoBERTa encoder

This table shows the model performances on the MSD dataset.

Results with ⋆ denote the significance tests of our CMML-Net over the baseline models at p-value < 0.01. The best results are highlighted in boldface, while the second-best results are underlined.

By supplementing these baselines and demonstrating consistently state-of-the-art (SOTA) results (Table 4, Table 5), our experimental analysis substantiates the superiority of CMML-Net in multimodal sarcasm detection tasks, both in terms of its methodological innovation and model performance.

We will add these additional experiments in the revised manuscript. Our paper will help readers better understand the existing work in the field of multimodal sarcasm detection.

[1] Wei Y, Yuan S, Zhou H, et al. G^ 2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(8): 9151-9159.

[2] Qin L, Huang S, Chen Q, et al. MMSD2. 0: towards a reliable multi-modal sarcasm detection system[J]. arXiv preprint arXiv:2307.07135, 2023.

[3] Jia M, Xie C, Jing L. Debiasing Multimodal Sarcasm Detection with Contrastive Learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(16): 18354-18362.

[4] Qiao Y, Jing L, Song X, et al. Mutual-enhanced incongruity learning network for multi-modal sarcasm detection[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(8): 9507-9515.

[5] Tian Y, Xu N, Zhang R, et al. Dynamic routing transformer network for multimodal sarcasm detection[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023: 2468-2480.

[Q4] Inadequate experimentation: It is not enough to adopt only one dataset, more datasets including MMSD2.0 [1], DMSD [2], RedEval [3] verification model need to be adopted.

[A4] We want to thank the reviewer’s careful review and valuable comments on our work. We conducted further experiments on MMSD2.0 and DMSD and will report the main results of the experiments below (Table 2, Table 3).

Table 2: Results on MMSD2.0 Dataset

This table shows the model performances on the MMSD2.0 dataset.

Results with ⋆ denote the significance tests of our CMML-Net over the baseline models except Multi-view CLIP (Full Finetuned) at p-value < 0.01. The best results are highlighted in boldface, while the second-best results are underlined.

Table 3: Results on DMSD Dataset

This table shows the model performances on the DMSD dataset.

Results with ⋆ denote the significance tests of our CMML-Net over the baseline models at p-value < 0.05. The best results are highlighted in boldface, while the second-best results are underlined.

Due to time constraints, we use the frozen version of CLIP as the encoder. CMML-Net still performs better than the fully fine-tuned Multi-view CLIP on MMSD2.0 (Table 2). Our CMML-Net achieves state-of-the-art (SOTA) results on DMSD (Table 3).

According to the results of the experiment on MMSD2.0 and DMSD (Table 2, Table 3), CMML-Net consistently demonstrates its effectiveness in multimodal sarcasm detection. By explicitly and efficiently capturing complete multi-modal incongruities in both fact and sentiment perspectives, our approach achieves competitive performance without requiring extensive fine-tuning or large-scale architectures. This highlights CMML-Net's robustness and strong potential for broader applications in multimodal tasks.

We will add these additional experiments in the revised manuscript to further demonstrate the generalizability of our model.

[Q1] Metrics Learning Related Work: This paper is inspired by metrics learning, but lacks work on metrics learning.

[A1] Thank the reviewer for the reviewer’s thorough and insightful comments, which have significantly improved the quality of our work. We will add a summary of the relevant work on "metric learning" in the revised manuscript. The general content of the supplement is shown below.

Metric learning

Metric learning is a technique for measuring the similarity between objects based on distance metrics. Traditional methods usually transform the original feature space into a representation where the distance can capture meaningful relationships. These methods usually include Mahalanobis distance [6-8] and rely on linear transformations (e.g., symmetric positive definite matrices) to project the data into Euclidean space. They are limited by their reliance on predefined distance functions.

Deep metric learning extends this concept by leveraging nonlinear transformations through deep neural networks. This approach creates a flexible embedding space. It enables the model to minimize the distance between similar samples and maximize the separation between different samples [2, 3]. Deep metric learning has been successful in various applications, including image text retrieval, text classification, face recognition, and multimodal data representation [1]. Existing works widely adopt architectures such as Siamese networks and loss functions (e.g., triplet loss [4] and contrastive loss [5]) to effectively capture the relationship between pairs or groups of samples.

We introduce the concepts of deep metric learning to calculate incongruity at the representation level rather than at the sample level. Our method iteratively calculates complete multi-modal incongruity to capture the subtle relationships between representations for multi-modal sarcasm detection. This approach generalizes deep metric learning concepts to capture more complex relationships, offering a broader and more adaptable solution for multimodal learning tasks.

[1] Suárez J L, García S, Herrera F. A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges[J]. Neurocomputing, 2021, 425: 300-322.

[2] Peng L, Jian S, Li D, et al. Mrml: Multimodal rumor detection by deep metric learning[C]//ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.

[3] Wang H, Tang P, Kong H, et al. DHCF: Dual disentangled-view hierarchical contrastive learning for fake news detection on social media[J]. Information Sciences, 2023, 645: 119323.

[4] Hoffer E, Ailon N. Deep metric learning using triplet network[C]//Similarity-based pattern recognition: third international workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3. Springer International Publishing, 2015: 84-92.

[5] Hadsell R, Chopra S, LeCun Y. Dimensionality reduction by learning an invariant mapping[C]//2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06). IEEE, 2006, 2: 1735-1742.

[6] Globerson A, Roweis S. Metric learning by collapsing classes[J]. Advances in neural information processing systems, 2005, 18.

[7] Wang F, Sun J. Survey on distance metric learning and dimensionality reduction in data mining[J]. Data mining and knowledge discovery, 2015, 29(2): 534-564.

[8] Weinberger K Q, Saul L K. Distance metric learning for large margin nearest neighbor classification[J]. Journal of machine learning research, 2009, 10(2).

[Q2] Typographical errors: There are errors in some of the corner marks in the text, e.g. line 140. Some punctuation errors, such as line 148. Some sentences are redundant, such as lines 165 to 167.

[A2] We sincerely thank the reviewer for carefully pointing out typographical errors, punctuation issues, and redundant sentences in the manuscript. These observations are indeed valuable for improving the clarity and presentation of our work.

We will thoroughly review the manuscript to correct the identified issues and perform comprehensive proofreading to ensure overall linguistic accuracy.

Thanks to the author for the reply, I will revise my score and hope that the author will add the missing details in a future edition.

[Q4] This paper highlights the incongruity from two perspective: fact and sentiment. The sentiment aspect is easy to understand. However, there lacks more discussion on why the fact aspect is important for multimodal sarcasm detection in introduction section.

[A4] We apologize for not intuitively clarifying the importance of fact incongruity in the Introduction. According to the reviewer's insightful comments, we provided several specific examples of sarcasm on fact incongruity, sentiment incongruity, and intra-modal incongruity in CMML-Net Anonymous GitHub. Additionally, we have included test results of CMML-Net and different models on these examples for further clarity.

The test results clearly show that fact incongruity is useful for sarcasm detection. Facts describe the existence of objects or events, which are hidden in semantics. Fact incongruity refers to the incongruity between factual semantic information in multi-modal data. For example, there is a strong factual semantic contrast between a dress and a bedsheet in (a). Our proposed CMML-Net utilizes a fact-sentiment multi-task representation learning module to produce refined fact and sentiment text-image representation pairs. It then explicitly constructs the fact-sentiment dual-stream network to capture incongruity in the fact and sentiment perspectives.

In the revised manuscript, we will include a clearer and more intuitive explanation of the role of fact incongruity in the Introduction. We present test results here again (Table 1) to deal with the connection being inaccessible.

Table 1: Test Results on several examples.

This table shows the test results of CMML-Net and different models on several examples including fact incongruity, sentiment incongruity, intra-modal text incongruity, and intra-modal image incongruity. [CMML-Net Anonymous GitHub]

[Q2] The method of this paper exhibits limited novelty. In my opinion, introducing the Yolo task, the fact stream aims to find the image-based incongruity. However, the image-based incongruity have been discussed in existing works. And they proposed many effectiveness approach to solve this problem.[1,2,3,4]

[A2] According to the reviewer’s concerns, we provide a more detailed explanation. Specifically, the fact stream leverages the proposed complete multi-modal metric learning to jointly and iteratively capture image-based incongruities, text-based incongruities, and inter-modal incongruities, rather than being limited to detecting only image-based incongruities. We introduce this process in detail in Section 3.2.1.

Then, we emphasize that the main purpose of introducing YOLO-task is to learn the fact representation space separated from the sentiment stream. We conduct experiments to replace the backbone of YOLO-task. The experimental results are shown in the following Table 2.

In addition, existing methods use YOLO to detect objects in images and then extract features from the detected objects. They rely heavily on models such as YOLO. However, our YOLO-task only guides the fact projection layer in paying attention to more fine-grained fact semantic information during the training process. It is removed during reasoning. Our main result in the paper shows that our method has a competitive performance advantage.

Table 2: Results on MSD Dataset with different backbones

This table shows the model performances with different backbones on the MSD dataset.

The experimental results suggest that the performance of the framework is primarily influenced by the guiding role of the YOLO-task, while the complexity of the chosen object detection backbone plays a less significant role.

We will present the purpose of the YOLO task and its difference from other work more clearly in the revised manuscript.

[Q3] There is unclear motivation of why this paper introduces deep metric learning. And what’s it’s advantage compared to traditional deep learning in existing works?

[A3] We appreciate the reviewer for raising concerns. We propose a deep metric learning method, which is called complete multi-modal metric learning. It can jointly and iteratively calculate inter-modal and intra-modal incongruity in a unified metric space (e.g., fact and sentiment metric space). The underlying motivation is to build an efficient model instead of constructing multiple independent architectures in fact and sentiment streams to capture inter-modal and intra-modal incongruity. Deep metric learning has the characteristics of containing fewer parameters and having a consistent computational structure in the same space. Therefore, we introduce deep metric learning into multimodal sarcasm detection.

In addition, existing works determine the degree of incongruity between representations before training and then learn the overall effective representation during training. Deep metric learning measures the incongruity between representations through explicit calculation and learnable distance. It provides a way to capture complete incongruity more intuitively and effectively than traditional deep learning.

We will strengthen the motivation for introducing deep metric learning and its advantages over traditional deep learning in the revised manuscript.

[Q1] Lack of insights in the proposed approach. Motivation of the proposed module in the overall framework is unclear in this paper.

[A1] We would like to reiterate the novelty of our approach. Incongruity is the key clue for multimodal sarcasm detection. However, existing methods neglected inter-modal or intra-modal incongruities in fact and sentiment perspectives, leading to incomplete sarcasm information and biased performance. We illustrate this intuitively with several examples and the test results of CMML-Net and different models on several examples in CMML-Net Anonymous GitHub.

In multimodal sarcasm detection, fact incongruity reflects contradictions in factual information across modalities, while sentiment incongruity highlights mismatches in emotional tone. We design a multi-task representation learning module to explicitly construct a fact-sentiment dual-stream network. The fact stream (FISN) and the sentiment stream (SISN) capture fact incongruity and sentiment incongruity respectively in different representation spaces.

We propose a complete multi-modal metric learning method to jointly and iteratively calculate inter-modal and intra-modal incongruities in fact and sentiment metric space. This method avoids the construction of multiple independent architectures for intra-modal and inter-modal incongruities extraction in either the FISN or SISN. Our proposed method is efficient for explicitly capturing complete multi-modal incongruities.

Therefore, our motivation is sufficient and insightful. The test results in CMML-Net Anonymous GitHub indicate that our model can adapt well to inter-modal or intra-modal incongruity in the fact or sentiment perspective. In addition, our model shows extremely superior performance in the main experiment. This also demonstrates the novelty of our method. We present test results here (Table 1) to deal with the connection being inaccessible.

Table 1: Test Results on several examples.

This table shows the test results of CMML-Net and different models on several examples including fact incongruity, sentiment incongruity, intra-modal text incongruity, and intra-modal image incongruity. [CMML-Net Anonymous GitHub]

Dear Reviewer TiNN,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. Since only two days remaining for discussion, we would like to confirm whether our responses have effectively addressed your concern. If you require further clarification, please do not hesitate to contact us. We are more than willing to continue our communication with you. If you are willing to improve the score, we would be grateful.

Thank you for your hard work and support.

Best regards,

The authors of Paper 13962.

Dear Authors,

Thank you for your detailed response.

I appreciate the effort you put into clarifying the motivation behind your paper. However, I still find it difficult to distinguish your work from prior research in the field. Moreover, considering the feedback from the other reviewers, I will keep my score.

Best regards, Reviewer TiNN

[Q6] The Main Result lacks significance analysis.

[A6] We thank the reviewer for identifying this oversight. We randomly sampled several random number seeds to run the model and conduct significance tests. In the Main Result of MSD, the significance test indicates that all metrics of CMML-Net outperform all baseline models selected at p-value < 0.01. We will add this additional experiment in the revised manuscript to further demonstrate the robustness of our model.

[Q7] In Section 4.6, the authors analyze the YOLO-task representation, but is it possible to replace it with other models to achieve multimodal sarcasm detection results based on different backbones?

[A7] We are grateful for the reviewer’s detailed feedback. The main purpose of the YOLO-task is to train the fact projection layer to learn a fine-grained fact representation space separated from the emotion representation space. YOLO does not participate in reasoning. Therefore, we can use other object detection models, such as Fast RCNN, to annotate the object existence vector for the training set. For instance, an object existence vector could be represented as [car: 1, cat: 0, dog: 1, ...].

Due to time constraints, we conducted simple experiments using different parameter versions of YOLOv10. The experimental results are shown in the following Table 4. They reveal that the performance improvement is more due to the separation of the representation space and the fact-sentiment dual-stream network.

Table 4: Results on MSD Dataset with different backbones

This table shows the model performances with different backbones on the MSD dataset.

The experimental results suggest that the performance of the framework is primarily influenced by the guiding role of the YOLO-task, while the complexity of the chosen object detection backbone plays a less significant role. We will add this additional experiment in the revised manuscript to further show the role of YOLO-task.

[Q5] Only one dataset is considered in the experiments, such as MMSD2.0 and MSTI, which are not included, and the generalizability of the method cannot be demonstrated.

[A5] Thanks for the reviewer’s careful review and valuable comments. We conducted further experiments on MMSD2.0 and DMSD and will report the main results of the experiments below (Table 2, Table 3).

Table 2: Results on MMSD2.0 Dataset

This table shows the model performances on the MMSD2.0 dataset.

Results with ⋆ denote the significance tests of our CMML-Net over the baseline models except Multi-view CLIP (Full Finetuned) at p-value < 0.01. The best results are highlighted in boldface, while the second-best results are underlined.

Table 3: Results on DMSD Dataset

This table shows the model performances on the DMSD dataset.

Results with ⋆ denote the significance tests of our CMML-Net over the baseline models at p-value < 0.05. The best results are highlighted in boldface, while the second-best results are underlined.

Due to time constraints, we use the frozen version of CLIP as the encoder. CMML-Net still performs better than the fully fine-tuned Multi-view CLIP on MMSD2.0 (Table 2). Our CMML-Net achieves state-of-the-art (SOTA) results on DMSD (Table 3).

According to the results of the experiment on MMSD2.0 and DMSD (Table 2, Table 3), CMML-Net consistently demonstrates its effectiveness in multimodal sarcasm detection. By explicitly and efficiently capturing complete multi-modal incongruities in both fact and sentiment perspectives, our approach achieves competitive performance without requiring extensive fine-tuning or large-scale architectures. This highlights CMML-Net's robustness and strong potential for broader applications in multimodal tasks.

We will add these additional experiments in the revised manuscript to further demonstrate the generalizability of our model.

[Q3] The authors mention two innovative points in the Introduction section, but these two points essentially seem to be the same.

[A3] We apologize for any misunderstanding we may have caused the reviewer. We clarify the differences between the two contributions below.

The first contribution focuses on the proposed deep metric learning method, which is called complete multi-modal metric learning. It can jointly and iteratively calculate inter-modal and intra-modal incongruity in fact and sentiment metric space. This deep metric learning method is the key to capturing complete multi-modal incongruity in an efficient model architecture. It avoids the construction of multiple independent architectures in fact or sentiment sub-networks to extract intra-modal and inter-modal incongruity.

The second contribution focuses on the overall structure of CMML-Net. We design multi-task representation learning to explicitly construct a fact-sentiment dual-stream network. The fact-sentiment dual-stream network we constructed effectively captures incongruity from both fact and sentiment perspectives, facilitating complete incongruity extraction.

We will further strengthen the presentation of the innovative points in the revised manuscript to make it clearer to readers.

[Q4] The authors explain that "Fact incongruity means sarcasm occurs when the literal meaning and the observed facts unexpectedly contrast." Additionally, in the method design, both the FISN and SISN modules take the combined results of image and text as input. This indicates that the work primarily focuses on addressing inter-modal incongruities. However, the authors describe the FISN and SISN as aiming to capture both intra-modal and inter-modal incongruities (Sections 3.2.1 and 3.2.2). Where is the intra-modal incongruity reflected?

[A4] We appreciate the reviewer’s insightful question regarding the role of intra-modal incongruity in our design. We clarify that the proposed complete multi-modal metric learning method can jointly and iteratively calculate inter-modal and intra-modal incongruity in FISN and SISN.

Specifically, both FISN and SISN receive text and image representations projected into a unified fact representation space $\mathcal{S}^F$ or sentiment representation space $\mathcal{S}^S$. In this unified space $\mathcal{S}^F$, the complete multi-modal metric learning method iteratively calculates fact incongruity (Eq 3.) between any two representations including intra-modal pairs (e.g., text-text $(\mathbf{r}_T^F, \mathbf{r}_T^F)$ or image-image $(\mathbf{r}_I^F, \mathbf{r}_I^F)$ ) or inter-modal pair (e.g., text-image $(\mathbf{r}_T^F, \mathbf{r}_I^F)$ ):

$\mathcal{I}_{u,v}^F = \| \mathbf{r}_u^F - \mathbf{r}_v^F \|$ (Eq 3)

Here, $u \in set( T, I )$ and $v \in set( T, I )$, and $\mathcal{I}_{u,v}^F$ measures the fact incongruity including intra-modal incongruity (e.g., text-text or image-image) and inter-modal incongruity (e.g., text-image) in the unified space. Our method ensures that intra-modal incongruity is fully captured alongside inter-modal incongruity. We will more clearly demonstrate the extraction of intra-modal incongruity in the revised manuscript.

[Q2] The authors mention that previous work neglected the importance of intra-modal incongruity in sarcasm detection, leading to incomplete incongruities and biased performance, but do not provide reasons why intra-modal incongruity is useful for sarcasm. It is suggested to use examples in the Introduction section to intuitively demonstrate this.

[A2] We apologize for not intuitively clarifying the importance of intra-modal incongruity in the Introduction. According to the reviewer's insightful comments, we provided several specific examples of sarcasm on fact incongruity, sentiment incongruity, and intra-modal incongruity in CMML-Net Anonymous GitHub. Additionally, we have included test results of CMML-Net and different models on these examples for further clarity.

The test results clearly show that intra-modal incongruity is useful for sarcasm detection. Incongruity comes from within a modality when the relationship between modalities is congruous, such as the emphasis relationship in (c) and the complement relationship in (d). Our proposed complete multi-modal metric learning method jointly and iteratively calculates the inter-modal and intra-modal incongruity in fact and sentiment metric space.

In the revised manuscript, we will include a clearer and more intuitive explanation of the role of intra-modal incongruity in the Introduction. We present test results here (Table 1) to deal with the connection being inaccessible.

Table 1: Test Results on several examples.

This table shows the test results of CMML-Net and different models on several examples including fact incongruity, sentiment incongruity, intra-modal text incongruity, and intra-modal image incongruity. [CMML-Net Anonymous GitHub]

[Q1] The summary of related work is incomplete.

[A1] Thanks for the reviewer’s thorough and insightful comments, which have significantly improved the quality of our work. We will add a summary of the relevant work on "metric learning" in the revised manuscript. The general content of the supplement is shown below.

Metric learning

Metric learning is a technique for measuring the similarity between objects based on distance metrics. Traditional methods usually transform the original feature space into a representation where the distance can capture meaningful relationships. These methods usually include Mahalanobis distance [6-8] and rely on linear transformations (e.g., symmetric positive definite matrices) to project the data into Euclidean space. They are limited by their reliance on predefined distance functions.

Deep metric learning extends this concept by leveraging nonlinear transformations through deep neural networks. This approach creates a flexible embedding space. It enables the model to minimize the distance between similar samples and maximize the separation between different samples [2, 3]. Deep metric learning has been successful in various applications, including image text retrieval, text classification, face recognition, and multimodal data representation [1]. Existing works widely adopt architectures such as Siamese networks and loss functions (e.g., triplet loss [4] and contrastive loss [5]) to effectively capture the relationship between pairs or groups of samples.

We introduce the concepts of deep metric learning to calculate incongruity at the representation level rather than at the sample level. Our method iteratively calculates complete multi-modal incongruity to capture the subtle relationships between representations for multi-modal sarcasm detection. This approach generalizes deep metric learning concepts to capture more complex relationships, offering a broader and more adaptable solution for multimodal learning tasks.

[1] Suárez J L, García S, Herrera F. A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges[J]. Neurocomputing, 2021, 425: 300-322.

[2] Peng L, Jian S, Li D, et al. Mrml: Multimodal rumor detection by deep metric learning[C]//ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.

[3] Wang H, Tang P, Kong H, et al. DHCF: Dual disentangled-view hierarchical contrastive learning for fake news detection on social media[J]. Information Sciences, 2023, 645: 119323.

[4] Hoffer E, Ailon N. Deep metric learning using triplet network[C]//Similarity-based pattern recognition: third international workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3. Springer International Publishing, 2015: 84-92.

[5] Hadsell R, Chopra S, LeCun Y. Dimensionality reduction by learning an invariant mapping[C]//2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06). IEEE, 2006, 2: 1735-1742.

[6] Globerson A, Roweis S. Metric learning by collapsing classes[J]. Advances in neural information processing systems, 2005, 18.

[7] Wang F, Sun J. Survey on distance metric learning and dimensionality reduction in data mining[J]. Data mining and knowledge discovery, 2015, 29(2): 534-564.

[8] Weinberger K Q, Saul L K. Distance metric learning for large margin nearest neighbor classification[J]. Journal of machine learning research, 2009, 10(2).

Dear Reviewer Trxx,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. Since only two days remaining for discussion, we would like to confirm whether our responses have effectively addressed your concern. If you require further clarification, please do not hesitate to contact us. We are more than willing to continue our communication with you. If you are willing to improve the score, we would be grateful.

Thank you for your hard work and support.

Best regards,

The authors of Paper 13962.

[Q3] The framework is built largely on existing approaches, enhancing its reproducibility. However, this reliance on established methodologies limits its originality, as there is a lack of significant methodological contribution. This may affect its impact within the research community, which typically values innovation in addition to reproducibility.

[A3] We acknowledge that we stand on the shoulders of giants, but we believe that this work is novel and contributes to the field of multimodal satirical detection.

Incongruity is the key clue for multimodal sarcasm detection. However, existing methods neglected inter-modal or intra-modal incongruities in fact and sentiment perspectives, leading to incomplete sarcasm information and biased performance. In CMML-Net Anonymous GitHub, we present several examples to show the importance of these incongruities and show the results of CMML-Net and different models on these examples.

In multimodal sarcasm detection, fact incongruity reflects contradictions in factual information across modalities, and sentiment incongruity highlights mismatches in emotional tone. We design a multi-task representation learning module to explicitly construct a fact-sentiment dual-stream network. The fact stream FISN and the sentiment stream SISN respectively capture fact incongruity and sentiment incongruity in different representation spaces.

Moreover, we propose a complete multi-modal metric learning method to jointly and iteratively calculate inter-modal and intramodal incongruities in a unified metric space (e.g., fact and sentiment metric space). This method avoids the construction of multiple independent architectures for extracting intra-modal and inter-modal incongruities in FISN or SISN. Our proposed method performs well in explicitly capturing complete multi-modal incongruities.

In experiments, the test results in CMML-Net Anonymous GitHub indicate that our model can adapt well to inter-modal or intra-modal incongruity in the fact or sentiment perspective. Furthermore, our model shows extremely superior performance in the main results. This also demonstrates the novelty of our method. Therefore, our work is novel and contributes to multimodal sarcasm detection. We present test results here (Table 1) to deal with the connection being inaccessible.

Table 1: Test Results on several examples.

This table shows the test results of CMML-Net and different models on several examples including fact incongruity, sentiment incongruity, intra-modal text incongruity, and intra-modal image incongruity. [CMML-Net Anonymous GitHub]

[Q1] The paper lacks clarity due to some undefined symbols and terms. For instance, the architecture is based on "units" in section 3.2 that are repeatedly referenced as learnable, but their exact nature is not defined. It is unclear whether these units are neuron clusters, specific network layers, or memory mechanisms designed to integrate multiple modalities. Additionally, other symbols, such as the capital "S" and "F," are not explicitly defined. While "S" seems to represent sentiment-related information, the meaning of "F" is ambiguous and should be clarified. Could you please clarify those terms where they are firstly introduced?

[A1] We acknowledge that some terms and symbols are not fully defined, and we apologize for any confusion caused. We will address the specific issues in the revised paper:

The CMML Unit in Section 3.2 is the designed basic component for complete multi-modal metric learning in the fact incongruity sub-network(FISN) and sentiment incongruity sub-network(SISN). It calculates the incongruity of all text-text, text-image, and image-image representation pairs in a unified metric space, and updates them via dynamic separation and non-linear adjustment to obtain more discriminative incongruity representations. FISN and SISN will gradually obtain complete multi-modal incongruities by stacking CMML Units.

The superscript capital letters "F" and "S" stand for Fact and Sentiment, respectively. They distinguish the different learning processes for fact and sentiment representations in the multi-task learning module and the calculations in the fact and sentiment streams of the dual-stream network.

We thank the reviewer for the careful review work, and we will revise the manuscript to ensure that all terms and symbols are clearly defined and unambiguous.

[Q2] The paper’s focus on multi-modal sarcasm detection is narrow, limiting its potential impact and relevance for the broader machine learning community. Could you please give some potential broader implications or applications of their work beyond sarcasm detection.

[A2] Thanks for the reviewer’s thorough and insightful comments. This study focuses on multimodal sarcasm detection, but we believe that the proposed model and method have broader application potential.

Our proposed complete multi-modal metric learning method can jointly and explicitly calculate inter-modal and intra-modal incongruity. It is applicable to multimodal tasks such as fake news detection and sentiment transition analysis It can jointly and efficiently reveal the key incongruous features in these tasks from both inter-modal and intra-modal aspects, thereby improving the performance of the model. This study can provide an important reference for researchers in these similar tasks and promote the further development of multimodal learning methods.

We will add a discussion of potential broader implications and applications of CMML-Net in the revised manuscript.

Dear Reviewer BJ3o,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. Since only two days remaining for discussion, we would like to confirm whether our responses have effectively addressed your concern. If you require further clarification, please do not hesitate to contact us. We are more than willing to continue our communication with you. If you are willing to improve the score, we would be grateful.

Thank you for your hard work and support.

Best regards,

The authors of Paper 13962.