COPER: Correlation-based Permutations for Multi-View Clustering

We thank the reviewer for their thoughtful and constructive feedback and for acknowledging the strengths of our work, particularly the effectiveness of our end-to-end multi-view clustering framework and the validation of our theoretical contributions through extensive experiments. We address the reviewer's specific comments and suggestions in detail below.

W1 Motivation

We thank the reviewer for their valuable feedback. To clarify, our motivation stems from the limitations of existing multi-view clustering (MVC) methods, particularly those that adopt a two-stage process where representation learning is followed by clustering. This two-stage process can be suboptimal, as the representations learned are not directly optimized for clustering, often resulting in subpar performance. While we acknowledge that a few end-to-end methods have been proposed, they are often constrained by limited adaptability to various data types or require specific assumptions that restrict their generalizability. Our work proposes an end-to-end approach that is distinct in leveraging self-supervised learning to improve canonical correlation analysis (CCA)-based clustering. Specifically, we incorporate a novel self-supervision scheme into an end-to-end clustering procedure, which can enhance any CCA-based method with better representations tailored for clustering. Unlike existing end-to-end approaches, our method introduces within-cluster permutations and multi-view pseudo-labeling, which allow for improved cluster separation and representation alignment, thereby addressing the inherent limitations of prior approaches. This motivation is grounded in the understanding that direct optimization of the representations for clustering tasks can lead to better generalization and more effective model performance across different data types, as evidenced by our extensive empirical results.

W2 Additional baselines

Thank you for suggesting this end-to-end approach. In response, we introduced an evaluation of OPMC [1], L0-CCA [2], and MVCAN [3] in the revised version (see Tables 2 and 6). OPMC performs well against other baselines and outperforms COPER in 2 out of 10 datasets regarding accuracy (ACC). However, COPER still surpasses OPMC, L0-CCA, and MVCAN in most metrics and across most datasets.

W3 Parameter Study

Thank you for your suggestion. We provide a sensitivity analysis for $\lambda$ value in Appendix E.3 and extend it with an additional analysis of the batch size hyperparameter. We train our model on the MSRCv1 dataset with batch sizes [128, 256, 512, 1024] five times for each batch size, each time with a different random initialization seed. Then, we measure the metrics ACC, ARI, and NMI and put the results on the box-plot chart (E.2 Figure 6).

References:

[1] Liu J, et al. One-pass Multi-view Clustering for Large-scale Data, ICCV 2021.

[2] Lindenbaum, O., Salhov, M., Averbuch, A., & Kluger, Y. (2021). L0-sparse canonical correlation analysis. In International Conference on Learning Representations.

[3] Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios (CVPR 2024)

We thank the reviewer for thorough and constructive feedback on our submission. We appreciate the acknowledgment of the strengths of our approach, particularly the simplicity and effectiveness of the end-to-end framework, the theoretical proofs and visualizations, and the innovative integration of multi-view pseudo-labeling techniques. Below, we provide detailed responses to the reviewer's comments.

Q1 Uniform Model Architecture + Weaknesses

We appreciate the feedback on uniform model architecture. To address this concern, we have conducted an experiment to demonstrate that the performance of our model is not too sensitive to batch size choices. Specifically, we trained our model with varying batch sizes in [128, 256,512,1024], with 5 random initializations for each batch size. Then, we compute the metrics ACC, ARI, and NMI for each setup and provide a box-plot chart in Appendix E.3. This experiment is an addition to the sensitivity analysis we provided for hyperparameter $\lambda$ in E.3.

Variations in model architecture are influenced by the complexity of the dataset, such as feature dimensions, and align with established practices in multi-view clustering. Our benchmark methods, which have different configurations tailored to specific datasets, include CVCL [3], ICMVC [5], RMCNC [4], and MVCAN [2]. Furthermore, although our gradual training strategy is straightforward, it can be automated using heuristic criteria once the loss converges to a predefined threshold. This approach reduces the need for manual adjustments. Other benchmark methods, such as CVCL [3] and RMCNC [4], also incorporate various gradual steps as part of their frameworks.

Finally, in Appendix E2, we demonstrate how the Silhouette score, an unsupervised metric, could be used to tune model hyperparameters.

Q2 Compare with Latest Self-Supervised Approaches

We appreciate the Reviewer’s suggestion to compare our approach with the latest self-supervised multi-view clustering methods and have included a comparison to MVCAN [1] and L0-DCCA [7], OPMC [6]. As shown in Tables 2 and 6, COPER consistently outperforms MVCAN [1] and L0-DCCA [7], OPMC [6] across most datasets at ACC, ARI, and NMI, demonstrating its robustness and adaptability across diverse clustering scenarios. Although MVCAN achieves competitive performance in specific cases, such as Caltech101-20 for NMI, its results are more variable and generally lower than COPER, particularly on complex or noisy datasets like RBGD and VOC.

We tried to use the DeepMVC [2] Git repository but faced considerable challenges in reproducing the environment. We were unable to establish a working environment, and the package dependencies led to complex compatibility issues that led to initialization errors. We will work toward solving these issues by reaching out to the owners of this package via Github.

References:

[1] Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios (CVPR 2024)

[2] On the Effects of Self-Supervision and Contrastive Alignment in Deep Multi-View Clustering" (CVPR 2023)

[3] Deep Multiview Clustering by Contrasting Cluster Assignments" is accepted by ICCV 2023

[4] Sun, Y., Qin, Y., Li, Y., Peng, D., Peng, X., & Hu, P. (2024). Robust multi-view clustering with noisy correspondence. IEEE Transactions on Knowledge and Data Engineering.

[5] Chao, G., Jiang, Y., & Chu, D. (2024, March). Incomplete contrastive multi-view clustering with high-confidence guiding. In Proceedings of the AAAI Conference on Artificial Intelligence

[6] Liu J, et al. One-pass Multi-view Clustering for Large-scale Data, ICCV 2021.

[7] Lindenbaum, O., Salhov, M., Averbuch, A., & Kluger, Y. (2021). L0-sparse canonical correlation analysis. In International Conference on Learning Representations.

We thank the reviewer for their thoughtful and detailed feedback. We are grateful for the acknowledgment of the strengths of our work, including the clarity of our motivation, the rigor of our theoretical analysis, the breadth of our comparative experiments, and the well-structured presentation of the paper. Below, we respond to all comments raised by the reviewer.

W1 Alignment Pseudolabels

Thank you for your feedback. As you indicated, label alignment across views in multi-view clustering presents challenges that could potentially limit the effectiveness of self-supervised learning methods.

To address potential issues with pseudo-label alignment, our framework uses a multi-view agreement mechanism to refine pseudo-labels and enhance their reliability. This step (elaborated in Section 4.4.2) reduces errors by focusing on consistent labels across views. Even when some pseudo-labels are incorrect, our learned representation still improves clustering. As shown in Fig. 3(i)(a), despite some level of falsely annotated pseudo-labels (24% error), the Adjusted Rand Index still improves (purple line), demonstrating the robustness of our approach. Additionally, our theoretical analysis in Sec. 4.3 provides bounds that further support the method's resilience to pseudo-label errors.

W2 Technical innovation of our work

While specific individual components like correlation maximization and pseudo-labeling have appeared in prior work, COPER introduces the integration of these components along with several additional innovations. First, we present a novel framework for multi-view pseudo labeling, which includes multi-view agreement. This agreement is essential for facilitating within-cluster permutations in the subsequent steps. Additionally, although the relationship between Canonical Correlation Analysis (CCA) and Linear Discriminant Analysis (LDA) has been previously examined, it has not been applied to multi-view clustering frameworks. In addition to our proposed method for enabling within-cluster permutations, we leverage the relationship between CCA and LDA by providing an approximation analysis and deriving new error bounds for our learned representations, utilizing matrix perturbation theory. This supports the effectiveness of our approach.

We thank the reviewer for the valuable feedback. We appreciate the recognition of our work's strengths, including the novel permutation-based canonical correlation objective and the theoretical analysis showing how the learned embeddings approximate those obtained by supervised linear discriminant analysis (LDA). Below, we respond to all comments and outline our modifications to address the raised issues.

W1 Updated multi-view applications

We have updated the text to include motivating examples of multi-view clustering applied to more recent use cases. The revised text (page 1, lines 41-44) is: "This approach has great potential in applications like communication systems content delivery [1], community detection in social networks [2,3], cancer subtype identification in bioinformatics [4], and personalized genetic analysis through multi-modal clustering frameworks [5,6]." Updated references:

References:

[1] Miguel Angel Vázquez and Ana I Pérez-Neira, "Multigraph Spectral Clustering for Joint Content Delivery and Scheduling in Beam-Free Satellite Communications," ICASSP 2020.

[2] Zhao et al., "Multi-view Tensor Graph Neural Networks Through Reinforced Aggregation," IEEE TKDE 2022.

[3] Shi, X., Liang, C., and Wang, H., "Multiview Robust Graph-based Clustering for Cancer Subtype Identification," IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2023.

[4] Wang, B., et al., "Multi-dimensional Data Integration for Personalized Analysis Using Random Walks," BMC Bioinformatics, 2023.

[5] Li et al., "netMUG: A Network-guided Multi-view Clustering Workflow for Dissecting Genetic and Facial Heterogeneity," Frontiers in Genetics, 2023.

[6] Wen et al., "Deep Multi-view Clustering with Contrastive Learning for Omics Data," Bioinformatics, 2024.

W2/W3/W4 Baselines

We appreciate the reviewer’s feedback regarding our choice of baselines and the suggestions for expanding and categorizing the methods. ICMVC [7] and RMCNC [8] were chosen since they were recently published and because of their demonstrated effectiveness and relevance in various multi-view clustering scenarios beyond their specialized design focuses. ICMVC [7] was originally designed to manage incomplete multi-view data. However, it includes robust components such as instance-level attention fusion and contrastive learning for view alignment, which remain highly effective even when all views are fully observed. Additionally, its weight-sharing pseudo-classifier allows for end-to-end representation and clustering, making it a strong competitor in our context. The evaluation results presented in Table 1 of the ICMVC paper [7] show excellent performance on standard datasets without any missing views, further supporting our choice to include it as a baseline. Similarly, RMCNC's core architecture [8] is designed to handle noisy correspondences effectively. It incorporates components such as a noise-tolerant contrastive learning framework and a unified probability computation strategy, which enhance its clustering performance across various data conditions. The evaluation results (refer to Table 2 in the original paper) demonstrate strong performance on noise-free datasets, highlighting its suitability as a comparison method. Since real-world multi-view datasets frequently contain noisy correspondences, RMCNC is a practical choice for our evaluation.

Following the reviewer's suggestion, we have expanded our experiments to include additional baselines that cover a broader spectrum of multi-view clustering techniques [9, 10, 11]. As suggested, we have categorized our baselines into CCA-derived and non-CCA-derived methods (see Tables 2 and 6). This categorization highlights the distinct contributions of our method’s novel permutation-based CCA objective, clearly assessing its advantages over similar frameworks and alternative multi-view clustering paradigms.

Our systematic experimental evaluation involves ten diverse datasets (2–6 views each). We benchmarked against ten baselines, conducted an ablation of four studies, and assessed scalability. Our method demonstrates superior performance, surpassing eight state-of-the-art deep models with up to 7% improvement in accuracy and consistently higher ARI and NMI scores across datasets. Robustness is ensured by averaging results over ten runs and reporting standard deviations, providing a comprehensive and reliable performance assessment.

References:

[7]. Chao, G., Jiang, Y., & Chu, D. (2024, March). Incomplete contrastive multi-view clustering with high-confidence guiding. In Proceedings of the AAAI Conference on Artificial Intelligence

[8]. Sun, Y., Qin, Y., Li, Y., Peng, D., Peng, X., & Hu, P. (2024). Robust multi-view clustering with noisy correspondence. IEEE Transactions on Knowledge and Data Engineering.

[9] Liu J, et al. One-pass Multi-view Clustering for Large-scale Data, ICCV 2021.

[10] Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios (CVPR 2024)

[11] Lindenbaum, O., Salhov, M., Averbuch, A., & Kluger, Y. (2021). L0-sparse canonical correlation analysis. In International Conference on Learning Representations.

W5 Minor comments and corrections

We appreciate your feedback regarding the necessary changes. We have reviewed the paper again and made adjustments based on your suggestions.