M3CoL: Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification

The motivation of the manuscript is not strong. The process of aligning Positive couplets and Negative couplets in pairwise manner do not necessarily ignore the shared relational information exist between samples. There are lines of contrastive learning work (e.g. [1]) which align representations of sample within the same class together. Why does the mixup can better improve the performance compared to these approaches?

• We appreciate the reviewer’s feedback but respectfully disagree that the motivation of our manuscript is weak. Our key contribution addresses a critical limitation of traditional contrastive learning: its reliance on strict one-to-one correspondences across modalities, an assumption often violated in real-world data. As discussed in Line [49-76] and illustrated in Figure 1 (Left panel), traditional methods align paired modalities as positives and treat non-corresponding ones as negatives, failing to capture shared relations across different samples. For example, shared elements like "tomato sauce" and "basil" can relate across separate samples, which such methods overlook. In contrast, M3CoL (Figure 1, Right panel) generates mixed samples through convex combinations of data points, allowing the model to learn shared multimodal relationships beyond pairwise associations. This is crucial for modeling complex real-world relationships, such as imperfect bijections, which enhance generalization. While related works (e.g., Zhang et al., 2022) cluster semantically similar samples within class boundaries, they rely on rigid label hierarchies. M3CoL explicitly captures shared structures that may not align strictly with predefined classes, leveraging mixup-based contrastive loss to map representations in a latent space respecting multimodal semantics. This is especially valuable in datasets like ROSMAP/BRCA (shared biological pathways) and Food-101 (overlapping food attributes). Empirically, M3CoL demonstrates state-of-the-art performance on N24News, ROSMAP, and BRCA and comparable results on Food-101, validating its ability to model shared relations and generalize across domains. We believe this directly addresses the limitations of traditional approaches, offering a robust alternative for multimodal learning.

The rationale of using MixUp technique is not well stated. Is there any reason behind the choice of MixUp as a way to combine samples? Additional ablation studies can be provided to strengthen the choice empirically.

• We respectfully disagree with the reviewer’s concern regarding the rationale for Mixup, as it is clearly addressed in lines 77-86. Mixup and its variants (e.g., Zhang et al., 2017; Yun et al., 2019) are well-established for enhancing feature spaces, improving robustness, and mitigating overfitting, particularly in low-sample settings. Our method leverages Mixup not only as a data augmentation tool but as a mechanism to generate mixed samples that capture shared relationships across modalities, addressing limitations of traditional contrastive methods. Building on recent advances (e.g., Shen et al., 2022; Kim et al., 2020), M3CoL adapts Mixup to multimodal contexts, providing a principled approach for modeling complex shared relationships.

Beside the idea of MixUp contrastive learning strategy, the rationale of applying unimodal downstream loss is also short of explanation. While it does show improvement via Ablation study, why is it the case that it can indeed help the overall system?

• We respectfully disagree with the concern that the rationale for applying the unimodal downstream loss is insufficiently explained. This is explicitly discussed in lines 114-117 of the manuscript. The unimodal prediction modules provide additional supervision during training through classifiers 1 and 2. This strategy enables deeper integration of modalities by allowing the system to leverage complementary strengths of different modalities. For instance, when one modality is weak or noisy, the other can compensate, enhancing overall robustness and performance. The ablation study further validates this design choice, demonstrating its effectiveness in improving the system's capability to learn robust representations.

The reasons for the training sample selection strategy are not explained, and some experimental results are incomplete.

• As stated in the paper (Line 159), the mixing indices j, k are drawn arbitrarily, without replacement, from [1, N], for both the modalities where N denotes the total number of samples present in the batch. j and k are kept different to ensure diversity in mixed samples. We respectfully ask the reviewer to specify any perceived incompleteness in results, as we believe all key experiments have been included.

The ACC for the Body section in N24 News has not been provided.

• Thank you for pointing this out. We did not report the ACC for the Body section in N24News due to availability issues, as major baselines do not provide corresponding numbers for comparison. This ensures a fair and consistent evaluation across all reported metrics.

Samples from modality 1 (x_i^1,x_j^1) and modality 2 (x_i^2,x_k^2), along with their respective mixed data, are fed into encoders to generate embeddings. How were samples j and k selected, and why can’t they both be j?

• Thank you for your observation regarding the selection of mixing indices j and k for modalities 1 and 2, respectively. As mentioned in the paper (Line 159), the indices j and k are drawn randomly, without replacement, from the set [1, N], where N denotes the total number of samples in the batch. The indices are kept distinct to ensure diversity in the mixed samples.

Dear Reviewer MUir, We hope we have addressed all the concerns you pointed out. Please let us know if you need any further clarifications; we will be happy to address them.

The paper does not deeply address how M3CoL scales with very large datasets, which could be a limitation given the increasing size of real-world datasets.

• We appreciate this concern. Our experiments demonstrate M3CoL's effective handling of moderately large datasets (Food-101: 60K samples, N24 News: 48K samples). The model's architecture enables batch processing and scales linearly with sample size, while memory requirements depend primarily on batch size rather than total dataset size. These characteristics make M3CoL practically applicable for large-scale deployments. Additionally, the consistent performance across datasets of varying sizes (from hundreds to tens of thousands of samples) indicates robust scalability.

There's a potential risk of overfitting with mixup, especially in early training stages. More analysis on balancing generalization and overfitting would be valuable.

• We respectfully disagree with the concern about overfitting during early training stages. As clearly demonstrated in Figure 6(a) and 6(b), our test accuracy curves show stable and consistent improvement throughout training, with no signs of performance degradation that would indicate overfitting. In fact, the learning curves demonstrate that both mixup and M3CoL variants maintain steady improvement or stable performance even after 40,000 steps, suggesting effective regularisation rather than overfitting. The smooth, monotonically increasing nature of these curves provides empirical evidence that our approach successfully balances model generalisation and learning capacity.

M3CoL's effectiveness relies heavily on the quality of mixed samples. Discussion on how data quality variations across modalities might affect performance is lacking.

• We acknowledge this valuable point about modality-specific data quality variations. While our current work focuses on establishing M3CoL's core methodology and demonstrating its effectiveness on standard benchmark datasets, analysing the impact of varying data quality across modalities represents an important future direction. We plan to explore this through systematic experiments with controlled quality variations in our future work.

Originality: Incorporating Mixup in contrastive learning is not new [1-3], even in a multimodal setting ([4-6], see Questions.) The reviewer would truly appreciate the authors’ further discussions on [4-6].

• We appreciate the reviewer highlighting these relevant works. While mixup in contrastive learning has indeed been explored, our work differs in several key aspects: Previous works [1-3] focus on single-modality scenarios, while M3CoL introduces a novel multi-modal contrastive framework that specifically addresses cross-modal interactions. Regarding multimodal works: VLMixer [4] focuses on vision-language pre-training using CutMix, while M3CoL proposes a more general framework for any number of modalities PowMix [5] is specific to sentiment analysis, whereas M3CoL is domain-agnostic [6] focuses on mixing synthetic/natural images, while M3CoL addresses general multimodal data mixing Our key novelty lies in combining multimodal supervision with mixup-based contrastive learning in a unified framework that can handle arbitrary numbers of modalities, which hasn't been explored in previous works.

Significance: datasets, especially the non-medical datasets, are relatively small. The effectiveness of the method is yet to be seen from larger, real-world datasets. Since this method is relatively straightforward, larger-scale experiments will improve the significance of the submission.

• Thank you for the valuable feedback. Our experiments demonstrate M3CoL's robust scalability, with consistent performance across datasets ranging from hundreds to tens of thousands of samples (e.g., Food-101: 60K, N24 News: 48K). The model's architecture scales linearly with dataset size, with memory requirements tied to batch size, making it practical for larger datasets. While our medical datasets are smaller, they include more modalities, demonstrating M3CoL's ability to handle diverse multimodal scenarios. Testing on larger real-world datasets is a promising future direction to further validate its effectiveness.

Could the authors kindly discuss the following related work ([6] being concurrent):

• As discussed in our previous response, we have highlighted the key differences between M3CoL and works [1-6]. We will rewrite our related work section to include comprehensive discussions of all these works to better contextualise our contributions.