Thank you for your positive feedback. We have addressed the comments on Weaknesses (W) as outlined below.

W1: Relationship between the proposed method and [1]
Thank you for highlighting the relevance of [1]. While both [1] and our work aim to learn representations robust to missing views using information-theoretic objectives, they differ in the following key aspects:

1. Use of labels: [1] directly incorporates label information during training to isolate task-relevant information in the learned representation. In contrast, our method does not utilize labels or task-specific information, as we assume no prior access to such additional data. This makes our approach more general and applicable to scenarios where task-relevant labels are unavailable.

2. Treatment of view-specific information: [1] discards view-specific information, assuming it is irrelevant to the task at hand. On the other hand, our method seeks to capture all factors of variation across views, as we do not assume prior knowledge of what information will be necessary for unknown downstream tasks. This ensures our representations retain both shared and view-specific information, making them broadly applicable to a wide range of potential tasks.

W2: Calibration verification
We would like to clarify that our evaluation of subset-view representations inherently assesses their calibration. Specifically, we evaluate each subset-view representation by using it as input to a downstream task model (e.g., classifier or regressor) that has been trained solely on complete-view representations. For a subset-view representation to generalize well in these downstream tasks, it must be well-calibrated toward the complete-view representation. Therefore, the strong performance of our method in handling subset views, as demonstrated in our experimental results, implicitly validates the calibration of these representations.

If you have any suggestions for additional calibration-specific metrics or experiments, we would be happy to consider them.

Reference
[1] Tian et al., Variational Distillation for Multi-View Learning, TPAMI 2023.

Dear Reviewer 48cg,
Thank you once again for your encouraging and thoughtful feedback.

We would like to kindly remind you that we have clarified:

1. The relationship between our work and [1].
2. The calibration verification.

In addition to these clarifications, we have also addressed issues raised by other reviewers, which we hope you will have an opportunity to consider as well.

Our revised manuscript includes new results, such as the analysis of what is being learned and ignored in the representation (Section 4.1) and the experiment on ImageNet-100 (Section E.6). These updates, along with our responses to other reviewers, are highlighted in the updated document for your reference.

We look forward to your response and would be happy to address any further comments or questions you may have.

Best regards,
The Authors

We appreciate your constructive review. We have addressed the comments on Weaknesses (W) as outlined below.

W1: Choice of Mutual Information (MI) estimator
We adopted the InfoNCE objective as the MI lower-bound estimator for two primary reasons:

• Low variance and numerical stability: InfoNCE is widely recognized for its low variance in MI estimation. Other MI lower-bound estimators [1,2,3] often suffer from the high variance of the exponential of their critic function outputs (e.g., $e^{f(z, v_{1:N})}$ in our context). In contrast, InfoNCE applies a softmax operation to the critic's output, ensuring that the value inside the log in Equation (7) is bounded between 0 and 1. This leads to reduced variance and improved numerical stability.

• Fair comparisons with CMC: CMC also utilizes the InfoNCE objective to lower bound and maximize their MI objective between every pair of single-view representations. By adopting the same MI estimator, we ensure that our comparison to CMC isolates the effect of optimizing different MI objectives rather than the influence of different MI estimator choices.

W1: Choice of distance metric
We chose the (squared) Mahalanobis distance as the critic function for our InfoNCE objective because it naturally aligns with our use of multivariate Gaussian representations. Specifically, the Mahalanobis distance measures how far the subset-view representations are from the complete-view representations, accounting for the variance of the complete-view representation. This ensures that each dimension is scaled appropriately based on its variance.

While cosine similarity [4,5,6,7] and L2 distance [8,9] are popular choices for critic functions, they are not suitable for our representations. Cosine similarity is not applicable because our representations are not L2-normalized, and L2 distance is less desirable as it assumes equal scaling across all dimensions, which is not the case for our multivariate Gaussian representations.

We will revise Section 3.3 to elaborate on the rationale behind selecting the InfoNCE objective and Mahalanobis distance as the critic function.

W2: Single-view performance
Firstly, while our method shows lower performance on single views in the Synthetic and MOSEI datasets, it tends to outperform baseline methods with single views in the remaining datasets, as demonstrated in Tables 1, 2, and Figure 4, as well as in the additional experiment on ImageNet-100.

The reason our method may be less effective with single views lies in its design. Our approach focuses on jointly learning representations for all combinations of views, rather than optimizing specifically for single views or particular subsets of views. This enables our method to perform robustly across any combination of available views, but this may also incur a small performance loss in single views. In contrast, most baseline methods prioritize learning representations for specific subsets, such as single views (e.g., CMC, GMC(s), and MVTCAE) or complete views (e.g., GMC(s) and MVTCAE), which limits their ability to effectively aggregate information from multiple available views, but may perform well in single views.

===== Updated equation index =====
We kindly notify you that Equation (7) referenced in our response to W1 (Choice of MI estimator) is now Equation (4) in the updated manuscript.

W3: Validation on more diverse datasets
We conducted additional experiments on the ImageNet-100 dataset, which consists of 100 subsampled classes selected by CMC from the full ImageNet dataset. Following the experimental setup in CMC, we transformed RGB images into two views in different color spaces (L and ab).

To accommodate our limited computational resources, we used pretrained ResNet34 [4] to extract 512-dimensional deep features from each view. We then trained each representation learning method with MLP encoders to learn 128-dimensional representations from these features. For evaluation, we followed the same linear classification protocol as in Sections 4.2 and 4.3, training a linear classifier to predict image labels from the frozen representations.

The table below summarizes the results. The results demonstrate that all methods, including ours, benefit from the addition of views. Notably, our method achieves superior performance compared to all baseline methods for both single-view inputs (L, ab) and two-view inputs (Lab). These findings indicate that our method can effectively learn representations from realistic datasets, providing further validation of its generalizability and effectiveness.

We appreciate your suggestion and will include this additional experiment in our revised manuscript.

W4: Use of advanced pretrained backbones
We would like to clarify that we employed pretrained deep features for feature extraction in our experiments. For instance, on the MUSTARD dataset (Section 4.2), we used BERT for the Text modality and ResNet for the Video modality.

Additionally, with the exception of the synthetic dataset we generated, all the datasets used in our experiments are standard multi-view benchmarks that have been widely utilized in prior studies [11,12,13, 14]. Given our focus on evaluating the representation learning methods themselves rather than the impact of backbone architectures, we prioritized following standardized preprocessing to enable a direct comparison with existing methods.

We leave exploring more advanced backbones for feature extraction in future work.

References
[1] Nowozin et al., f-gan: Training generative neural samplers using variational divergence minimization, NeurIPS 2016.
[2] Belghazi et al., Mutual information neural estimation, ICML 2018.
[3] Poole et al., On variational bounds of mutual information, ICML 2019.
[4] Tian et al., Contrastive multiview coding, ECCV 2020.
[5] Poklukar et al., Geometric multimodal contrastive representation learning, ICML 2022.
[6] Redford et al., Learning transferable visual models from natural language supervision, ICML 2021.
[7] Cherti et al., Reproducible scaling laws for contrastive language-image learning, CVPR 2023.
[8] Wang et al., Rethinking minimal sufficient representation in contrastive learning, CVPR 2022.
[9] Grill et al., Bootstrap your own latent: A new approach to self-supervised Learning, NeurIPS 2020.
[10] He et al., Deep Residual Learning for Image Recognition, CVPR 2016.
[11] Liang et al., Multibench: Multiscale benchmarks for multimodal representation learning, NeurIPS 2021.
[12] Liang et al., Factorized contrastive learning: going beyond multi-view redundancy, NeurIPS2023.
[13] Hwang et al., Multi-view representation learning via total correlation objective, NeurIPS 2021.
[14] Li et al., Large-scale multi-view spectral clustering via bipartite graph, AAAI 2015.

Dear Reviewer hoNY,
Thank you once again for your thoughtful and constructive feedback.

We have worked diligently to address your concerns, specifically:

1. Providing a justification for our choice of MI estimator and distance metric.
2. Discussing the observed single-view performance.
3. Conducting additional experiments on ImageNet-100.
4. Clarifying the usage of deep pretrained backbone encoders.

In addition, we have addressed issues raised by other reviewers, which we hope you may find relevant as well.

Our revised manuscript includes new results, such as the analysis of what is being learned and ignored in the representation (Section 4.1) and the experiment on ImageNet-100 (Section E.6). These updates, along with our detailed responses to the reviews, are highlighted in the document for your reference.

We would greatly appreciate your further feedback and would be happy to address any additional comments or questions you may have.

Best regards,
The Authors

W3: Lower performance with combined views
Thank you for highlighting the case of reduced performance when combining views, which is indeed an important observation. While this issue is specific to the Text modality on the MOSI dataset in our method, it can occur when the informativeness of modalities is highly unbalanced for the downstream task. Specifically, the Text modality (T) is inherently more informative for sentiment classification, as it often contains keywords that make sentiment inference trivial. This explains why, in Table 1, the performance of the Text modality alone is consistently higher than that of any other single-view modality across all methods. In such cases, combining representations from multiple views via a (weighted) average may slightly degrade the representation from the most informative view. This occurs because each view’s representation contributes to all dimensions of the combined representation to some extent, potentially diluting the signal from the dominant view. Importantly, this phenomenon is not unique to our method. For example:

• CMC achieves its highest performance using the Text modality alone on MOSI and MOSEI.
• GMC achieves its highest performance using the Text modality alone on MUSTARD.
• Similarly, MoPoE and MVTCAE fail to improve upon the performance of the Text modality when adding Video or Audio as additional views on MUSTARD.

We appreciate the opportunity to address this observation and will include a discussion of this phenomenon in the revised paper to provide further insights.

References
[1] Sutter et al., Generalized multimodal elbo, ICLR 2021.
[2] Cochran et al., A sampling investigation of the efficiency of weighting inversely as the estimated variance, Biometrics 1953.
[3] Cochran et al., The combination of estimates from different experiments, Biometrics 1954.
[4] Hwang et al., Information-theoretic state space model for multi-view reinforcement learning, ICML 2023.
[5] Hinton et al., Training products of experts by minimizing contrastive divergence, Neural Computation 2002.

Thank you for your insightful review. We have addressed the comments on Weaknesses (W) as outlined below.

W1: Clarification on main claims
We have outlined our contributions more explicitly as follows:

1. Theoretical contribution: Proposition 1 in Section 3.2 formally demonstrates that the single MI term between complete views and the joint representation encoded by any WMoE joint encoders (e.g., MoE, PoE, MoPoE) provides a lower bound of the weighted average of various MI terms. This enables learning representations for various subsets of views efficiently.

2. Algorithmic contribution: In Section 3.3, we derived a tractable lower bound for our single MI objective, enabling the MoPoE joint encoder to learn and calibrate exponentially many subset-view representations with a computational cost that scales linearly with the number of views. Importantly, this represents a significant improvement over previous work [1], which trained the same encoder with a computational cost that increased exponentially with the number of views.

3. Empirical contribution: Through comprehensive evaluations on seven MVRL benchmark datasets, including newly added results on ImageNet-100 (as presented in our response to reviewer hoNY), spanning 2 to 8 views, we demonstrated that our method achieves robust performance across diverse input view combinations, consistently surpassing strong baseline methods.

We will revise the introduction section to explicitly highlight these contributions to ensure clarity.

W1: Situating our work within related work
Thank you for pointing this out. To address this concern, we would like to note that our method is a Contrastive MVRL approach that builds upon and optimizes the encoders originally introduced by Multi-View VAEs. As such, it is closely related to prior work in both Contrastive MVRL and Multi-View VAEs. We will revise Section 2 to explicitly highlight these connections.

If you have any additional suggestions for better situating our work within the context of related research, we would greatly appreciate your input.

W2: On the optimality of IVW fusion
Inverse-Variance Weighted (IVW) average [2, 3, 5] is a classical (late) fusion method widely used in statistical sensor fusion. When observations are generated from the underlying state with independent Gaussian random noise, the IVW average serves as the Maximum Likelihood Estimation (MLE) of the state. To assess its optimality within the context of MVRL, let $\mu$ represent the ground-truth state that encapsulates all factors of variation for a given instance. Assuming that the mean of each single-view representation $\mu_i$ from the i-th view is an observation of $\mu$ with Gaussian noise $\sigma^2_i$ s.t. $p(\mu_i | \mu) = N(\mu, \sigma^2_i \text{I})$, the IVW average $\mu_{1:N} = \frac{\sum_{i=1}^{N} \mu_i / \sigma^2_i}{\sum_{i=1}^{N} 1 / \sigma^2_i}$ is the MLE of $\mu$. This assumption aligns naturally with our method, as all 𝜇ᵢ are derived from the same instance but exhibit different precisions based on the characteristics of each view. These precisions are captured by the learned variance $\sigma^2_i$ of each single-view representation. Consequently, IVW allows each single-view representation to contribute to the complete-view representation in proportion to its precision $\frac{1}{\sigma^2_i}$, making it an optimal choice for MVRL.

Beyond its statistical foundation, IVW offers computational scalability, with costs that scale linearly with the number of input views. This aligns with our goal of retaining scalability in MVRL. While alternatives like cross-view attention-based fusion could be considered, they incur a quadratic computational cost relative to the number of input views, making them less practical for scenarios involving many views. Moreover, such approaches have been empirically shown to perform less effectively when handling subset views, as observed in [4].

If there are other fusion approaches you believe could serve as valuable comparisons, we would greatly appreciate your suggestions to explore them further.

Additionally, regarding the statement about the effectiveness "when applied with other metrics," we kindly request clarification on the specific metrics you are referring to so that we can address this point more effectively.

I thank the authors for the rebuttal, which partially resolve my concerns.

1. I think the contributions made by the paper are kind of clear now. To me, the theoretical and algorithmic contributions revolve around the MI term which acts as a lower bound of the objective. However, I think the derivation of this term is highly inspired from (Shi et al., 2019) and (Sutter et al., 2021).

2. This is now resolved from authors answers and the additional experiments on ImageNet-100 dataset that they have presented in the reply of reviewer hoNY.

3. The performance of information fusion methods on a given dataset seems to depend on the specific method-dataset combination. This highlights the importance of discussing how to select the appropriate method for a particular dataset. Additionally, it is crucial to consider how the informativeness of a specific modality is defined and what happens to that informativeness when the modality is combined with others. These are significant questions that need to be addressed in the context of any information fusion method. I think some analysis of what is being learnt and what is being ignored in the context of modality fusion should be done, which is kind of missing in the present version.

Dear Reviewer Vxpo,
Thank you once again for your thoughtful follow-up review.

As the author-reviewer discussion period is set to conclude in a few days, we would greatly appreciate your feedback on whether our response and the revised manuscript have effectively addressed your concerns. If there are any remaining comments or questions, we would be more than happy to address them.

Thank you for your valuable time and input.

Best regards,
The Authors

1. Their paper states in section 3.1 that the parameter count and computational cost of the encoder increase relatively less with increasing views compared to other models. If it is a deep learning model, can the network weight parameter quantity or iteration time be provided for comparison, which will further prove its viewpoint. In addition, in the ablation experiment of computational cost, in the case of relatively few views, the difference in computational cost among multiple models is not significant, and even the computation time of CwA is longer.

2. The explanation of the relevant work in the paper is relatively simple, appearing to simply cite a large number of references rather than introducing methods, and it is not clear how the multi-view VAE methods are related to the method proposed in this paper. In addition, the explanation of the role of the encoder and the significance of the operations performed in this paper is not clear enough.

3. This paper proposes that the model can handle multi-view extensions, but I feel that this has not been fully validated from the experimental part. The experiment is divided into three parts: synthetic dataset, multimodal dataset, and conventional multi-view dataset. I am not sure if the synthetic dataset has practical significance, and how to discuss the effectiveness of the simulation experiment? The multimodal dataset used by the author lacks detailed explanations, and only four comparison methods were selected, lacking more state-of-the-art methods for comparative experiments.

We appreciate your valuable feedback. We have addressed the comments on Weaknesses (W) and Questions (Q) as outlined below.

W1: Parameter count

We have summarized the number of parameters used by each method across all datasets in the table above. It is important to note that all the methods compared in this work are late-fusion algorithms that use per-view backbone encoders with the same number of parameters for preprocessing input views. Consequently, the total number of model parameters for all methods scales linearly with the number of views, as each additional view introduces a fixed number of parameters. For example, doubling or tripling the number of input views simply doubles or triples the parameter count for all methods. However, the specific design choices in processing features encoded by backbone encoders and additional network components (e.g., decoders or extra backbone encoders) result in differences in the overall parameter count, which we elaborate on below:

CMC encodes features from the backbone encoders into deterministic single-view representations using one linear layer per view. In comparison, CwA uses two linear layers to encode the features into the mean and variance of single-view representations. This design results in a slightly higher parameter count for CwA in Caltech-101 with small backbone encoders, while the parameter counts remain nearly identical for the rest of the bigger datasets.

GMCs encodes backbone features into single-view and complete-view representations using two linear layers shared across all views. However, the linear layer for complete-view representation processes a stack of all backbone features, leading to a relatively larger parameter count in datasets like Synthetic and Caltech-101. GMC introduces an additional backbone encoder per view on top of GMCs, leading to up to twice the number of parameters compared to CMC, CwA, and GMCs.

While sharing the same per-view encoder architecture as CwA, MoPoE-VAE and MVTCAE incorporate decoders, which can at most double their overall parameter count compared to other methods. Further details on the architectures of encoders and decoders for each dataset can be found in Tables 4, 5, 6, and 7 in Section D of the supplementary material.

W1: Iteration time
Regarding iteration time, we would like to highlight that computation time becomes increasingly critical as the number of views grows, which is the primary focus of our work. Figure 3 in the main text reports the running time for 10 training epochs, showing that our method is relatively slower when handling a small number of views (2–4) compared to contrastive learning methods. This is because computing the Mahalanobis distance between representations in our method is more computationally intensive than the inner product operations used by CMC and GMC(s). However, when scaling to 8 or 16 views, our method outperforms all other approaches, including GMCs, GMC, and MVTCAE, even though these methods also have computation costs that scale linearly with the number of views.

W2: Shortened explanation of related work
Due to space limitations, we had to keep the discussion of early-fusion approaches in Section 2 brief. However, we provided an in-depth explanation of late-fusion approaches, including Contrastive MVRL methods and Multi-View VAEs, as our method is a Contrastive MVRL approach that builds on and optimizes the encoders introduced by Multi-View VAEs. This makes our method closely related to these works. For a more detailed explanation of the relationships between our method and these late-fusion approaches, we direct the reviewer to Section B in the supplementary material.

W2: Unclear explanation of the encoder and the operations
Regarding the explanation of the encoder and the operations performed in this paper, we kindly ask the reviewer to specify the unclear parts (e.g., by referencing line numbers), which would allow us to provide a more precise and helpful clarification.

W3: The purpose and the significance of the synthetic dataset
The purpose of using the synthetic dataset was two-fold: (1) to ensure the presence of both a shared factor observable across all views and view-specific factors unique to each view, and (2) to evaluate whether each MVRL method can effectively capture both types of information. The synthetic dataset is generated with 9 disentangled factors: 1 shared factor and 8 view-specific factors across 8 views. Regression on these factors as a downstream task provides a controlled way to evaluate whether a given MVRL method captures all important information in its representations. While the synthetic dataset may lack direct practical applicability, we believe that its ability to offer a controlled setup for verifying key capabilities (e.g., aggregating complementary information across views) makes it valuable for tasks with real-world analogs, such as autonomous driving, medical diagnostics, or multi-sensor systems.

In contrast, tasks from other datasets tend to focus more on shared factors. For example, in the four datasets from MultiBench, sentiment is often inferable from any single modality (video, audio, or text), with text being more advantageous when it contains critical keywords. Similarly, in Caltech-101, the identity of each object is inferable from any of its six visual features.

W3: Explanation on multimodal datasets
Thank you for pointing this out. We will revise Section D.2 in the supplementary material to provide the following information about the datasets:

The four multimodal datasets (MOSI, MUSTARD, FUNNY, and MOSEI) were collected to explore human affective states through diverse expressions, including spoken language, facial expressions, gestures, and speech tone. These datasets represent human expressions as multimodal time-series data comprising text, video, and audio modalities, enabling the prediction of sentiment (MOSI), emotion (MOSEI), humor (FUNNY), and sarcasm (MUSTARD). These modalities provide complementary information, highlighting the importance of understanding their complex relationships. Further details on each dataset are available in [1].

W3: Comparison to state-of-the-art methods
We would like to ask you to provide references to any late-fusion approaches we may have missed so that we can include them for comparison.

Q1: Single-view performance and hyperparameter influence
Firstly, while our method shows lower performance on single views in the Synthetic and MOSEI datasets, it tends to outperform baseline methods with single views in the remaining datasets, as demonstrated in Tables 1, 2, and Figure 4, as well as in the additional experiment on ImageNet-100 (in the response to reviewer hoNY).

The reason our method may be less effective with single views lies in its design. Our approach focuses on jointly learning representations for all combinations of views, rather than optimizing specifically for single views or particular subsets of views. This enables our method to perform robustly across any combination of available views, but this may also incur a small performance loss in single views. In contrast, most baseline methods prioritize learning representations for specific subsets, such as single views (e.g., CMC, GMC(s), and MVTCAE) or complete views (e.g., GMC(s) and MVTCAE), which limits their ability to effectively aggregate information from multiple available views, but may perform well in single views.

Regarding hyperparameters, our method demonstrates robust performance across a wide range of input views, as evidenced by the hyperparameter sensitivity analysis provided in Section E.1. This suggests that performance is not significantly impacted by the choice of hyperparameters.

Q2: Regarding reference value
Since our evaluation scenarios are not identical to the settings used in the original papers for the baseline methods, we conducted all evaluations using our own computational resources. While we do not have reference values for direct comparison, we have provided anonymized code and detailed explanations on evaluation protocol (Section 4) and implementation (Section D in the supplementary material) to ensure the correctness of our evaluations and support reproducibility.

Q3: Regarding subsequent task
Could you kindly clarify what is meant by "subsequent tasks"?

Reference
[1] Liang et al., Multibench: Multiscale benchmarks for multimodal representation learning, NeurIPS 2021.

We sincerely appreciate your follow-up review. Below, we address each aspect of the remaining concerns.

1: 7 MVRL benchmark datasets
In addition to the six datasets that were originally in the paper, we conducted additional experiments on ImageNet-100, as requested by Reviewer hoNY. The results from the new dataset are reported in Section E.6 of the revised manuscript and also discussed in our response to Reviewer hoNY.

2: Inconsistent performances
The main purpose of the synthetic dataset experiments is to complement the standard benchmark datasets (i.e. MultiBench, Caltech-101, ImageNet) commonly used in the MVRL literature. Since these datasets are focused on measuring the task-specific performance, it is not clear that the baseline methods are successfully capturing view-specific factors even when they are performing well on the downstream task. The complexity of the data (image, text, video, etc.) also hinders an in-depth analysis. This is an on-going problem with all the evaluations appearing in the prior literature on MVRL, as they don’t show the whole picture. In order to raise awareness about the issue for the community, we have proposed the synthetic dataset that allows us to directly measure whether the method is successfully capturing shared as well as view-specific factors, shown in figure 2 (a) and (b) in the revision.

As shown in figure 2 (b), all the baseline methods fail to encode view-specific factors. This is because our method is designed to successfully aggregate both shared and view-specific factors, while other methods tend to discard view-specific factors. This distinction is discussed in detail in the results paragraph of Section 4.1 in the revision.

The apparent ‘inconsistent’ performance in MultiBench and Caltech-101 is because, as we stated earlier, it only measures the downstream task performance. Throwing away view-specific factors, such as the video-specific information that is irrelevant to the downstream task (such as the gender of the speaker for the sentiment classification) will not affect the performance, and as such, all the baseline methods tend to do so as observable from the synthetic dataset experiments. The takeaway from these experiments is that, CwA doesn’t fall into this common trap, and in addition, the shared factors found by CwA are generally better than those found by other baselines, albeit it doesn’t completely dominate over all datasets. It is important to note that it is extremely challenging for a single method to achieve superior performance across all input-view combinations and benchmark datasets, as evidenced by the evaluation results of other baseline methods [1, 2, 3].

Lastly, as emphasized in prior works [3, 4, 5], learning complete representations that capture shared as well as view-specific factors of variation is crucial for MVRL because we cannot assume prior knowledge about downstream tasks or which information will be most relevant. This highlights the importance of our synthetic dataset, which allows us to rigorously assess whether a method captures all factors of variation. We are unable to provide reference values, as the synthetic dataset experiment is a novel contribution of our work. The results presented in our paper serve as the reference values.

References
[1] Tian et al., Contrastive multiview coding, ECCV 2020.
[2] Sutter et al., Generalized multimodal elbo, ICLR 2021.
[3] Hwang et al., Multi-view representation learning via total correlation objective, NeurIPS 2021.
[4] Lee et al., Image representation using 2d gabor wavelets, IEEE TPAMI 1996.
[5] Zhang et al., CPM-Nets: cross partial multi-view networks, NeurIPS 2019.\

3: Performance of GMC with single text view
Regarding the performance of GMC with the single text view, its use of two backbone encoders per view provides a significant advantage, particularly given the high dimensionality of the text view in the MUSTARD dataset (50x300 dimensions). This is supported by the results of GMCs, which employs a single backbone encoder per view (indicated by "s" for single backbone) for fair comparison, which consistently underperforms compared to our method across all input view combinations.

3: Lower performance with additional views on MOSI
Regarding the lower performance with additional views on the MOSI dataset, we reiterate our response to Reviewer Vxpo:

This phenomenon is specific to the Text view on the MOSI dataset. More generally, this can occur when the informativeness of views is highly unbalanced for the downstream task. For example, the Text view (T) is inherently more informative for the downstream task (i.e. sentiment classification), as it often contains keywords that make it trivial to infer the sentiment. This explains why, in Table 1, the performance of the Text view alone is consistently higher than that of any other single views across all methods. In such cases, combining representations from multiple views via a (weighted) average may slightly degrade the representation from the most informative view. This occurs because each view’s representation contributes to all dimensions of the combined representation to some extent, potentially diluting the signal from the dominant view. More importantly, this phenomenon is not unique to our method. For example:

• CMC achieves its highest performance using the Text view alone on MOSI and MOSEI.
• GMC achieves its highest performance using the Text view alone on MUSTARD.
• Similarly, MoPoE and MVTCAE fail to improve upon the performance of the Text view when adding Video or Audio as additional views on MUSTARD.

We included our discussion on this issue in Section 4.2 of our revised manuscript.

3: Analysis of influencing factors
Please understand that analyzing performance on in-the-wild datasets, such as MOSI, MUSTARD, and others, is inherently challenging due to the lack of controlled variables. This is one of the primary motivations for including our synthetic dataset, where all influential factors can be explicitly designed and evaluated consistently across all methods.

4: CwA+recon
Thank you for raising this question. When input views are well feature-engineered to represent only the essential characteristics of an instance, autoencoder-based approaches can be advantageous, as the reconstruction objective encourages the representation to memorize the views. For this reason, we evaluated CwA+recon on Caltech-101, which consists of six handcrafted visual feature views, to explore whether our method could benefit from the reconstruction objective in such scenarios.

However, in-the-wild datasets in practice consist of high-dimensional data with sparse and redundant information. For example, the video, audio, and text views in the four datasets from MultiBench typically exhibit sequential structures, making them high-dimensional and containing overlapping information across adjacent timesteps. In such cases, forcing the representation to memorize every detail through autoencoding may not be beneficial and can hinder the discovery of meaningful global context information. This is reflected in our results on MultiBench (Tables 1 and 2), where autoencoder-based methods such as MoPoE-VAE and MVTCAE generally underperform compared to contrastive learning (CL) methods, especially on MUSTARD and MOSEI. In these scenarios, incorporating an autoencoder objective into our method is unlikely to provide additional benefits.

A similar trend is evident in our new results on ImageNet-100, where CL-based methods remain more effective than Multi-View VAEs. For these reasons, we opted not to evaluate CwA+recon on datasets with high-dimensional, in-the-wild views.

Dear Reviewer zRh5,
Thank you again for your thoughtful follow-up review.

With the author-reviewer discussion period coming to a close in a few days, we would sincerely appreciate your feedback on whether our response and the revised manuscript have sufficiently addressed your concerns.

Additionally, we would like to highlight some new results included in the revised manuscript, such as the analysis of what is being learned and ignored in the representation (Section 4.1) and the experiment on ImageNet-100 (Section E.6), which we hope you find relevant.

If there are any remaining questions or additional comments, we would be happy to address them promptly.

Thank you for your valuable time and input.

Best regards,
The Authors

Dear Reviewers,

In response to the suggestion from Reviewer Vxpo, we have incorporated an analysis of what is learned and ignored in the representation using the synthetic dataset. This analysis is detailed in the revised Section 4.1.

Additionally, we updated Figure 5 to clarify the data generation process for the synthetic dataset and elaborated on the importance of this dataset in Section D.1.

All updates in this second revision are highlighted in magenta.

If you have any remaining concerns, we would be more than happy to address them.

Thank you for your constructive feedback.