Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs

We deeply appreciate your thoughtful and constructive feedback, which has been invaluable in refining and improving our manuscript. Below, we provide detailed responses to your insightful comments:

Q1. Clarification on "Underlying Semantic Information"

Thank you for your careful review of both the main text and the appendix. We appreciate your insightful observation and acknowledge that the term "underlying semantic information" was indeed too vague and lacked precision. Specifically, we intended to convey that the encoded information of the latent variables in matrix $Z_0$ remains invariant under specific permutations. To address this, we have revised the relevant text (lines 150–154, highlighted in blue) to provide better clarity and context. The updated text reads as follows:

"The core idea is that variables with the same superscript $l$ encode similar information about the $l$-th view, regardless of whether they are directly encoded or transformed. Thus, even if the columns of $Z_0$ are reordered, as illustrated in the transition from $Z_0$ to $Z_1$ in Figure 1, each column continues to represent the same view, and the information encoded by elements at corresponding positions remains invariant."

This revision clarifies that the latent variables consistently represent view-specific information even after reordering, enhancing the clarity of our approach.

Q2. Sensitivity to Regularization Parameters and Cyclic Permutation Setting

We appreciate your inquiry regarding the sensitivity of our method to regularization parameters and cyclic permutation settings.

As detailed in Appendix B.4, we analyzed the sensitivity of the KL divergence regularization parameters for $z$ and $\omega$, selecting 5 and 2.5, respectively. To further investigate, we performed a finer-grained analysis and visualized the results as a 3D surface plot (Figure 13). The relatively flat surface indicates overall robustness. Importantly, the KL term for $z$ benefits from a larger value, as it is critical for establishing inter-view correspondences, while the KL term for $\omega$ serves as a smaller auxiliary regularizer to enforce consistency.

Regarding the potential sensitivity to cyclic permutations, we understand the reviewer’s concern about the randomness introduced by different cyclic permutations (e.g., 1234 cyclically permuted to 2341 or 3421). Actually, we pre-compute all possible cyclic permutations for the observed view indices and uniformly sample from them during training. This ensures that, over sufficient training iterations, all permutations are equally represented. Furthermore, the goal of cyclic permutations is consistent—to encourage all distributions within the same set ($\mathcal{S}_i$ in our paper) to become similar. Our ablation study (Table 5) includes a comparison between cyclic and random permutations, indeed validating the crucial role of cyclic permutations in establishing a strong similarity measure. Finally, all experiments are run with multiple random seeds, reporting mean and standard deviation, demonstrating robustness to stochastic effects from permutation sampling.

In conclusion, the cyclic permutations ensure consistency, and the regularization parameters are robust under proper tuning, as validated by our analysis. We sincerely thank you again for your thoughtful review.

Dear Reviewer yAd5,

Thank you sincerely for your efforts in reviewing our paper. We have carefully addressed your concerns with corresponding responses and results. We would appreciate your feedback to confirm if our clarifications resolve your concerns. Please feel free to let us know if any part remains unclear.

Best regards,

Authors

We sincerely appreciate your thorough and constructive feedback, which has been instrumental in improving the clarity and depth of our work. Below, we address your insightful comments in detail:

Q1. Comparison of Invariant Feature Learning: Other Works vs. Our Novelty

Thank you for the opportunity to highlight the novelty of our work and its distinctions from prior methods.

The essence of multi-view data lies in capturing both shared and unique information across diverse modalities or perspectives. Many methods have been proposed to leverage this inherent property, but prior works have not fully explored the information potential of multi-view data within the VAE framework, particularly under incomplete scenarios. For instance:

• DIMVC [1]: Enforces strict consistency between views to retain only shared information, reducing reliance on missing views. While this improves clustering performance, it has not been validated for generative tasks.
• Multi-VAE [3]: Models view-common information using discrete distributions to achieve strict disentanglement, minimizing the influence of view-specific details on clustering. However, it requires concatenating representations from all views, making it unsuitable for incomplete scenarios.
• APLN [4]: Addresses uncertainty and conflicts in multi-view classification caused by imputation, focusing on resolving conflicting opinions. This method differs fundamentally from ours, relying on label supervision and a multi-stage iterative strategy tailored to classification tasks.
• CMVAE [2]: Assumes invariant linear relationships between views in the latent space. This strict assumption may limit the preservation of view-specific details, as seen in blurred PolyMNIST backgrounds. Its GMM-based latent space, designed to capture class information, requires predefined modes and is less effective at learning inter-view relationships.

We would like to emphasize that, unlike previous works, our method fully leverages the invariant relationships between views through two key innovations:

1. Embedding Invariant Relationships into the Latent Space:
   We establish explicit correspondences between views to flexibly capture their invariant relationships. By leveraging set permutations and partitions, we directly embed these relationships into the variational inference framework, ensuring that the latent space inherently reflects this invariance.
2. Latent Space Regularization with Inter-View Relationships:
   We introduce a prior based on inter-view relationships to self-supervise latent variable learning, enhancing efficiency and creating a synergistic interplay between inter-view correspondences and the latent space. This interplay makes our method more flexible than [2]. Furthermore, our method does not require strict alignment between views. Instead, we adopt a soft alignment approach, where views only need to be consistent after transformation. This flexibility enables the latent variables to preserve both view-common and view-specific information while facilitating the imputation of missing views through correspondences. Our framework effectively addresses information imbalance and bias caused by incomplete fusion, a critical challenge that has not been well-addressed within the variational framework for incomplete multi-view learning. Finally, our representation has been thoroughly validated through clustering and generation tasks, demonstrating its robustness and versatility.

We hope this clarifies the unique contributions of our work and its advancements over existing methods.

Q2. Publication of Code

We understand the importance of reproducibility and have taken several steps to ensure transparency. The code has been included in the supplementary material and is also available in an anonymous GitHub repository: https://anonymous.4open.science/r/MVP-F154/ . This repository includes expanded technical details, such as implementations for cyclic and batch permutations, to facilitate reproduction.

To further ensure accessibility, we will release complete implementations for all publicly available datasets used in our paper. We are also committed to refining and optimizing the code for enhanced usability.

We hope this demonstrates our dedication to transparency and reproducibility.

We are deeply grateful for your thoughtful and constructive feedback, which has greatly contributed to enhancing the clarity and rigor of our work. Below, we provide detailed responses to each of your comments:

Q1. Computational Complexity of MVP

To address your concern regarding the computational complexity, we have conducted a detailed comparison of our method and the baseline methods. The table below highlights the computational cost of encoding, transformation, and decoding across different approaches:

Here, $V$ denotes the number of views, and $K$ is the number of latent variable samples per view for optimizing the IWAE bound. While our method introduces additional computational cost during the transformation step due to explicit channel construction in the latent space, these operations are performed using a lightweight MLP in a low-dimensional space, keeping the overall complexity manageable.

We hope this analysis clarifies the trade-offs of our approach and provides a clear understanding of its computational efficiency.

Q2. Latent Dimension Selection for Different Datasets

Thank you for your question regarding how we select the first $k$ dimensions of $z$ to encode shared information across different datasets.

First, to prevent any potential misunderstanding, we would like to clarify that our approach is distinct from feature or latent variable disentanglement methods [1], which aim to capture shared information strictly for interpretability. In our method, $k$ is used to derive an average representation $\omega$ from the latent variables $z$ of different views, providing additional regularization while allowing flexibility by choosing $k \leq d$.

Second, we conducted ablation studies, detailed in Appendix B.4, to illustrate the flexibility of our approach in selecting $k$ based on task requirements. Below is a summary of our findings:

• Clustering Tasks: For clustering, strong inter-view consistency is crucial to effectively capture shared information and reveal underlying categories. Thus, we set $k = d$, using all latent dimensions. Our ablation results show that the optimal choice of $k$ depends on dataset characteristics (e.g., view dimensions, number of classes, Table 7). Smaller datasets benefit from careful selection of latent dimensions, while larger datasets exhibit robust performance across different values of $d$, reducing the need for fine-tuning. This Parameter Sensitivity Analysis aligns with established practices [2, 3].
• Generation Tasks: For generation tasks, where retaining unique features from each view is important, we tested different ratios of $k/d$ (25%, 50%, 75%, 100%) and found robust results for both PolyMNIST and MVShapeNet, given their larger size (Table 8). Therefore, we chose different ratios based on dataset characteristics: 50% for PolyMNIST, which has higher view variability, and 75% for MVShapeNet, where views are more consistent.

We hope this explanation clarifies our rationale and demonstrates the adaptability of our method.

[1] Chen R T Q, Li X, Grosse R B, et al. Isolating sources of disentanglement in variational autoencoders[J]. Advances in neural information processing systems, 2018, 31. [2] Zhang C, Cui Y, Han Z, et al. Deep partial multi-view learning[J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 44(5): 2402-2415. [3] Cai H, Huang W, Yang S, et al. Realize Generative Yet Complete Latent Representation for Incomplete Multi-View Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

We sincerely appreciate your valuable feedback, which has greatly contributed to improving the clarity and robustness of our work. Below, we address each of your insightful comments in detail:

Q1. Incomplete Data Processing Clarification

Thank you for pointing out the importance of providing more clarity on the incomplete data processing steps. We have moved the description of the incomplete data processing to the beginning of the experimental section to improve readability and comprehension. Additionally, we have added a direct reference to Appendix C1 for further details. The revised content can be found in lines 318-322, with changes highlighted in blue for your convenience.

Q2. Computational Complexity of MVP

Your suggestion to provide a computational complexity comparison was incredibly helpful in clarifying the trade-offs of our approach. We have included a detailed comparison between our method and several baselines, as shown below:

Here, $V$ denotes the number of views, and $K$ is the number of samples of latent variables per view for optimizing the IWAE bound. It is worth noting that our method introduces additional computational cost during the transformation step because it explicitly constructs channels between views in the latent space. However, these transformations are performed in the low-dimensional latent space using a lightweight MLP, which keeps the added complexity manageable.

We hope this comparison addresses your concern, and we thank you for encouraging us to include this analysis.

Q3. Strategy of implementing Cyclic Permutations

Your observation regarding the computational efficiency of cyclic permutations prompted us to elaborate further on the optimizations implemented in our method. As detailed in Appendix C.2, we adopted the following strategies to mitigate overhead:

1. Pre-computation of Cyclic Permutation Indices
   • Using Sattolo's Algorithm, all cyclic permutations of $V$ indices can be efficiently generated with a time complexity of $O(V)$ for each permutation, resulting in $(V−1)!$ permutations in total. These permutations are precomputed and stored in a "fingerprint" file, along with pre-defined masks for incomplete multi-view datasets. By treating missing views as fixed points, we ensure that the permuted index sets retain consistent lengths, simplifying downstream operations.
2. Batch Processing with Pre-stored Indices
   • During training, the precomputed indices are directly applied via indexing operations, enabling efficient cyclic permutations of latent variables (e.g., rearranging $Z_0$​ into $Z_1$​). This transformation is performed in batches using $L \times L$ arrays, ensuring seamless integration with the training pipeline.

By trading a small amount of storage space for computational efficiency, our approach eliminates repetitive computations, ensuring scalability even with multiple views. We hope this explanation provides sufficient clarity on how we achieve computational efficiency.

Q4. KL Divergence in ELBO Computation

We appreciate your inquiry about the KL divergence terms in the ELBO computation. We used an analytical form for the KL divergence, as the terms involving $z$ and $\omega$ can be fully decoupled, as shown in Appendix A.4. Specifically, both the posterior and prior distributions in each KL term are multivariate Gaussians of the same dimensionality, which allows us to compute the KL divergence directly without any approximation.

Thank you once again for your thoughtful and constructive comments, which have been instrumental in refining our manuscript. We are confident that the revisions based on your feedback have significantly improved the quality of our work.