Scalable Cross-View Sample Alignment for Multi-View Clustering with View Structure Similarity

Q1: It remains unclear whether cross-view similarity between unaligned representations can reliably capture semantic correspondence under strong view heterogeneity.
A1: Thanks for your careful review. We would like to clarify that in our framework, the cross-view similarity between unaligned representations is not intended to directly capture semantic correspondence or high-level semantic consistency across views. Instead, it mainly estimates sample-level correspondence between views under unaligned scenarios. Specifically, our approach constructs cross-view similarity graphs by leveraging the view-specific graph structures, i.e., the similarity between the aligned samples and the unaligned samples for each view. These similarity measures are designed to identify samples that are likely to be counterparts across views, even in the presence of view heterogeneity. They are not responsible for learning semantic representations, but rather provide a relational criterion to guide the sample alignment process.

Q2: The scalability of the method to large-scale data is not thoroughly discussed, particularly regarding the computational implications and potential optimizations.
A2: Thanks for your constructive comments. As detailed in Section 4.3, the primary sources of computational cost in our proposed method include cross-sample similarity learning, cross-view similarity construction, and the late fusion stage. Among these, the largest computational complexity is the generation of the partition matrix $\mathbf{F}^v$ during the late fusion process, which leads to an overall complexity of $\mathcal{O}(n^2)$. Fortunately, this cost can be significantly reduced in practice. In particular, when more efficient or approximate strategies are adopted for generating the feature matrix $\mathbf{F}^v$, such as sparse graph construction, anchor-based clustering, or randomized low-rank approximations, the computational complexity can be reduced to nearly linear in the number of samples. Moreover, other components of the proposed method, such as CLM-based baseline view selection and cross-view similarity learning, scale linearly or near-linearly concerning the number of samples. In our experimental results, the proposed method has already demonstrated the ability to handle relatively large datasets within a reasonable time.

Q3: While the use of a k-nearest neighbor graph is intuitive, the sensitivity of the model to the choice of k warrants further analysis.
A3: Thank you for your valuable comment. We agree that the choice of the $k$-nearest neighbors parameter may impact the performance of the proposed method, and thereby we conduct additional experiments to analyze the proposed method's sensitivity to different values of $k$ under 50% sample alignment. As shown in the following experimental results, we evaluated clustering performance in terms of ACC, NMI, ARI, and F1score across a range of $k$ values ($k$=5, 9, 15, 19, 25) on six benchmark datasets. The results demonstrate that while minor performance fluctuations exist across different $k$ values, the overall performance remains relatively stable. For instance, on the HW and 100leaves datasets, the ACC and NMI remain consistently high regardless of the choice of $k$, suggesting that our method is robust to the $k$ setting. Although some datasets exhibit moderate variations, these changes do not significantly affect the general trend of the ranking of results. Moreover, the best performance does not concentrate on a single $k$ value, indicating that the model is not sensitive to a specific value. In a word, these observations confirm that our method maintains a stable and competitive performance across a reasonable range of $k$ values, suggesting that it is not highly sensitive to the neighborhood size. This robustness enhances the practicality of our method, especially when the optimal $k$ is not known a priori.
Effect of $k$-nearest neighbors on ACC under 50% sample alignment.

Effect of $k$-nearest neighbors on NMI under 50% sample alignment.

Effect of $k$-nearest neighbors on ARI under 50% sample alignment.

Effect of $k$-nearest neighbors on F1score under 50% sample alignment.

Q4: Without explicit correspondence, what principle underlies the computation of cross-view similarity between non-aligned samples?
A4: Thanks for your insightful comments. Our proposed method addresses this challenge through a carefully designed two-step strategy. First, within each view, every unaligned sample is represented by its affinity relative to the aligned samples in the same view, effectively embedding the unaligned data into a consistent reference criterion defined by these aligned samples. Since the aligned samples are assumed to correspond across views, cross-view similarity between unaligned samples can then be evaluated indirectly by comparing their affinities over the common aligned samples. This approach builds on the fact that non-aligned samples displaying similar affinity patterns with a common set of aligned samples across their respective views are likely to share semantic consistency, even without explicit correspondence. Furthermore, the experimental results demonstrate the effectiveness of the proposed strategies.

Q5: What is the rationale for enforcing orthogonality on the alignment matrix M? A clear justification is currently lacking and should be provided.
A5: We appreciate your insightful question. In our framework, we aim to establish soft correspondences between unaligned samples by reconstructing them based on structurally relevant information within their respective subspaces, rather than relying on hard 0-1 alignment. The orthogonality constraint is introduced to preserve the intrinsic geometric structure of the data during this reconstruction process. Specifically, orthogonality helps ensure that the transformation encoded by $\mathbf{M}$ behaves in a structure-preserving manner, avoiding distortion of sample relationships that could arise from degenerate or overly flexible alignments. This is particularly important in the presence of noise and semantic inconsistencies across views, where maintaining relative distances and semantic separability becomes challenging. Additionally, orthogonality serves as a form of regularization that stabilizes the optimization process and prevents trivial or collapsed solutions.

Q1: It would be helpful for the authors to clarify the specific motivations behind adopting the CLM mechanism for benchmark view selection. A brief discussion comparing it to alternative strategies could further strengthen the justification.
A1: Thanks for your valuable suggestion. The motivation for adopting the CLM-based mechanism lies in its ability to provide a view-independent evaluation of clustering quality. Unlike many existing methods that either randomly select a baseline view or rely on view consensus, our approach aims to directly assess the intrinsic structural information of each view's clustering. Consensus-based strategies typically identify a baseline view by measuring its agreement with other views. However, such methods may favor views that are consistent yet collectively poor in quality, leading to confirmation bias across views. Similarly, uncertainty-based methods rely on the distributional confidence, but they often lack explicit consideration of clustering structure, especially in the absence of ground truth labels. In contrast, the CLM mechanism jointly evaluates intra-cluster compactness and inter-cluster separability in a scale, shift, and class cardinality-invariant manner. This allows for fair and consistent comparison across heterogeneous views. By selecting the view with the highest CLM score, the most structurally reliable view is used in the proposed method, thereby enhancing the overall robustness of cross-view alignment.

Q2: As the alignment process is anchored on a single benchmark view, there is a potential risk that misrepresentative or noisy data within this view could influence the quality of alignment across other views.
A2: Thanks for your insightful comments. We fully agree that anchoring the alignment process on a misrepresentation or noisy baseline view could propagate errors to other views and negatively affect overall performance. To mitigate this risk, our framework employs a carefully designed selection mechanism based on the CLM score, which evaluates each view's clustering structure through a combination of compactness and robustness to data scale, shift, and class imbalance. By computing the CLM score for each view, we explicitly avoid relying on random selection. Instead, we select the view with the highest structural quality as the baseline view, ensuring that it reflects well-formed clusters and minimizes the chance of incorporating noise from poorly separated structures. Furthermore, since CLM is constructed to be scale and data cardinality-invariant, it is less sensitive to outliers or imbalanced cluster sizes, making it more robust in practice. Additionally, according to the extensive experimental results in Section 5, we can observe that it consistently achieves satisfactory alignment performance across diverse multi-view datasets.

Q3: The paper could benefit from further analysis on how initial clustering inaccuracies might affect the propagation of alignment errors.
A3: We thank your insightful comments. We agree that inaccuracies in the initial clustering stages may potentially affect the construction of similarity graphs between aligned and misaligned samples, thereby influencing the propagation of alignment errors. To mitigate this, we integrate both feature-level information and structural-level information to construct the consensus representation. This dual-perspective design ensures that the model does not rely solely on any single level of information, and enables correction and compensation for localized clustering inaccuracies during the optimization of the consensus representation. Furthermore, the late fusion clustering framework is introduced to decouple the final clustering decision from the initial alignment assumptions, providing an additional layer of robustness. Essentially, even in the presence of some local misalignment, the global fusion step can improve the clustering results by weighted integration of diverse information from multiple views, without being overly affected by any single misaligned view.

Q4: While the alignment matrix plays a key role in sample correspondence, its robustness under noisy benchmark views remains unclear.
A4: We appreciate your valuable comment regarding the robustness of the learned alignment matrix under noisy baseline views. From a theoretical perspective, our alignment matrix learning framework is designed to incorporate global structural information from multiple views rather than relying solely on local pairwise similarities. This global view enables the alignment matrix to be less sensitive to localized noise or misalignment in any single baseline view. Furthermore, the alignment objective jointly optimizes for consistency across views, implicitly enforcing robustness through structural constraints and regularization. These design choices ensure that the learned correspondences capture the intrinsic relations among samples, improving stability despite noise. Empirically, we validate this theoretical robustness by conducting experiments where the alignment matrix learned by our method is applied to align clustering results from several different algorithms. Compared to using the classical Hungarian algorithm, which is primarily a local matching approach, our alignment matrix consistently achieves better clustering accuracy across diverse datasets and algorithms. The detailed experimental results are presented in the following, and the detailed introductions can refer to Section 5.7. This demonstrates that our approach can generalize beyond the specific baseline view used during alignment learning and effectively mitigate the influence of noise views. In a word, the theoretical design and empirical evidence strongly support the conclusion that our learned alignment matrix is stable and robust under the noise baseline views.
Results of competitors on the 100leaves under a sample alignment ratio of $\rho$=50%

Q5: To what extent does this design truly exploit the structural and semantic complementarities between views?
A5: Thanks for your constructive comments. Indeed, it is critical to ensure that the late-fusion strategy fully leverages the structural and semantic information among views. To address this, we designed our framework with the following considerations. Firstly, while the fusion process is conducted at a later stage, we do not rely solely on independent view-wise clustering results. Instead, the alignment phase incorporates both feature-level information and graph-structural similarities, which are used to learn a unified latent representation space. This design allows the alignment process to indirectly capture inter-view dependencies before fusion. Secondly, to address the potential underutilization of cross-view information that only emerges through joint modeling, we introduce a weighted consensus mechanism in the fusion step. Specifically, each view contributes to the final clustering based on its alignment consistency and structural reliability, allowing high-quality views to reinforce low-quality ones and flexibly preserving multi-view complementary information. In summary, although our method does not perform explicit joint modeling in the traditional sense, it captures structural and semantic information through a combination of unified representation learning and adaptive weighted fusion, and the experimental results demonstrate the effectiveness of our proposed method.

Q6: Another intriguing aspect is the reliance on cluster-label matching for selecting a benchmark view.
A6: We sincerely appreciate your insightful comments. Unlike methods that assess each view independently, CLM measures the relative consistency of each view concerning all other views through a pairwise, normalized label-matching function. By aggregating over all $\binom{k}{2}$ cluster pairs, CLM captures global structural similarity in a redundant and smoothed manner, which effectively reduces the impact of local noise or instability in individual views. Furthermore, this strategy enables CLM to select the view that is most structurally aligned with the consensus of the remaining views, even when all views contain some level of noise. As such, the view selection process becomes inherently more robust than strategies that either randomly choose a baseline view or rely on the absolute clustering performance of a single view. Furthermore, as observed in Section 5.7, using our alignment matrix to align the clustering results of other algorithms achieves superior performance compared to the traditional Hungarian matching approach, which demonstrates the robustness and generalization of the selected baseline view.

Q1: Can the proposed method be effectively applied to cases where samples are entirely unaligned across views?
A1: Thanks for your valuable comment. Our method remains effective even in the absence of any sample-level correspondence across views. Rather than depending on explicit alignment, it captures the shared clustering structure implicitly present across different modalities. This is achieved by projecting data into a latent space where structural patterns are preserved through orthogonality constraints. These constraints ensure that the alignment process respects the intrinsic geometry of each view, enabling the model to discover coherent groupings without relying on direct sample matching. In fully unaligned scenarios (i.e., alignment ratio $\rho$=0), we adapt our strategy by discarding alignment-based similarity measures and instead computing cross-view relationships via the affinity between view-specific partition matrices. This allows the model to emphasize clustering-level consistency, which serves as a reliable proxy for alignment under severe view discrepancy. To assess the practical effectiveness of our method in such settings, we conduct comparisons with two representative models designed for unaligned multi-view clustering: VSC_mH and OpVuC. Extensive experiments on six benchmark datasets demonstrate that our method consistently achieves higher performance across most metrics. These results confirm that even in the most challenging case of complete sample misalignment, our framework successfully exploits latent structural information to produce robust clustering results.
Clustering results under fully unaligned views

Q2: The same approach is used for both unaligned sample representation and cross-view similarity—are these the only feasible choices, or can alternative methods be employed?
A2: Thank you for your insightful comment. In our method, we adopt the adaptive neighbor graph learning for both unaligned sample representation learning and cross-view similarity learning. This approach provides a unified and effective way to capture local geometric structures and inter-view relationships. Of course, we acknowledge that this is not the only feasible solution. Alternative strategies, such as metric learning techniques, multiple kernel learning, could also be employed for them. We chose the current method due to its flexibility and effectiveness in our preliminary studies. Therefore, future work could explore the impact of these alternative methods.

Q3: As alignment performance depends on the benchmark view, how is its reliability ensured?
A3: Thanks for your constructive comments. In the existing alignment-based multi-view clustering methods, the baseline view is either randomly selected or chosen heuristically without considering the intrinsic clustering quality of each view. This randomness can lead to sub-optimal alignment, especially when the selected view exhibits poor cluster structure or noise. To address this, we introduced a baseline view selection mechanism based on the CLM algorithm, and the detailed introductions can refer to Section 3.2. As presented in Eqs. (4)-(5), the measures assess the separation and compactness of cluster structures in a scale-invariant, shift-invariant, and class-size-invariant manner, enabling a fair comparison of clustering quality across unaligned views. Specifically, we compute the CLM score for each view and select the one with the highest score as the baseline view, thereby ensuring that alignment is guided by the most reliable structural information. Therefore, compared with the existing baseline view selection methods, the proposed method mitigates the limitations of random selection and enhances the stability and performance of the alignment process.

Q4: While the datasets are standard in multi-view clustering, could the authors clarify how they were preprocessed to simulate sample misalignment?
A4: Thank you for your valuable comments. As existing multi-view datasets are originally aligned across views, we simulate the sample misalignment scenario by artificially disrupting the arrangement of the samples. Specifically, we first apply the CLM algorithm to determine the baseline view. Then, we select a fixed proportion of samples with the same indices across views to serve as aligned samples for each view. The remaining samples in each view are considered as unaligned by randomly shuffling their order. Additionally, in the baseline view, we preserve the original sample order to maintain consistency with the ground-truth labels. Through this process, we construct multi-view datasets with varying degrees of sample misalignment, which allow us to effectively evaluate the robustness of our proposed method under sample misalignment scenarios.

Q5: It is recommended to compare more SOTA shallow MVC methods like tensor-based or subspace-based methods.
A5: Thank you for your insightful suggestion. In our experimental comparison, we have added two representative shallow multi-view clustering methods, i.e., TPCH [1] and ESTMC [2], and we report results under a 50% sample alignment ratio, with and without Hungarian alignment. The results demonstrate that our proposed method consistently outperforms them across all datasets in four clustering metrics. Notably, applying Hungarian alignment to these baselines does not necessarily lead to performance gains and sometimes even degrades the results, indicating that rigid post-processing alignment may be unreliable when sample correspondences are uncertain. In contrast, our method achieves significantly better and more stable performance by jointly modeling soft sample alignment and cross-view structural consistency, further demonstrating its effectiveness in handling partially aligned multi-view data.
Clustering performance under a sample alignment ratio $\rho$=50%

[1] Wang Z, Li X, Sun Y, et al. TPCH: tensor-interacted projection and cooperative hashing for multi-view clustering, AAAI. 2025.
[2] Ji J, Feng S. Anchors crash tensor: efficient and scalable tensorial multi-view subspace clustering, IEEE TPAMI, 2025.

Q1: Is the proposed method unable to address the issue of fully misaligned views?
A1: Thanks for your valuable comment. Although our method is primarily designed for partially aligned multi-view scenarios, it is also theoretically and practically applicable to fully unaligned views. Theoretically, our framework is built on the assumption that all views share a common underlying clustering structure. Even in the absence of explicit sample correspondence, samples belonging to the same cluster across different views exhibit implicit structural consistency. To capture this, we employ an orthogonality-constrained alignment matrix $\mathbf{M}$ that aligns samples based on their structural behavior in the latent space, rather than their correspondence. In practice, under fully unaligned scenarios, i.e., alignment ratio $\rho=0$, we adapt our strategy by discarding the alignment-based similarity construction. Instead, we compute cross-view similarity based on the affinity between partition matrices $\mathbf{F}^v$ of different views. This allows us to preserve and exploit high-level clustering consistency without relying on direct sample-wise alignment. The learned alignment relationship thus enables indirect alignment by capturing common clustering structures. Furthermore, to demonstrate the effectiveness of our proposed method on fully unaligned scenarios, we compare it with Vsc_mH and OpVuC, which are among the few existing methods capable of handling fully unaligned multi-view data. According to the experimental results, we can find that our method outperforms the competitors in most of the datasets. Thus, these results confirm the adaptability of our method in addressing full sample misalignment by leveraging shared structural information at the partition level.
Clustering results of compared methods under fully unaligned views (alignment ratio $\rho=0$)

Q2: Should be compared more with the SOTA methods of view misalignment instead of traditional multi-view clustering methods.
A2: Thank you for your valuable suggestion. To better evaluate the effectiveness of our method in handling view misalignment, we have included tow representative state-of-the-art methods specifically designed for unaligned multi-view clustering, i.e., TUMCR[1] and DAGF[2], and the corresponding experimental results are presented in the following, where we report ACC under two different alignment ratios: 0%(fully unaligned) and 50% (partially aligned). As shown, our method consistently achieves superior performance across six benchmark datasets in both settings. For example, under the fully unaligned scenario, our method outperforms TUMCR and DAGF by large margins on datasets such as MSRCV and 100leaves. Similar trends can be observed under the other datasets. Therefore, these results demonstrate the adaptability of our method in dealing with both fully and partially misaligned scenarios.
ACC comparison with view misalignment multi-view clustering methods under different alignment ratios

[1] Ji J, Feng S, Li Y. Tensorized unaligned multi-view clustering with multi-scale representation learning[C]// KDD. 2024.
[2] Jiang H, Tao H, Jiang Z, et al. Unaligned multi-view clustering via diversified anchor graph fusion[J]. Pattern Recognition, 2025.

Q3: Can the learned rotation matrix be used to output aligned samples?
A3: Thanks for your constructive comments. In our method, the feature rotation matrix $\mathbf{R}$ is primarily learned in the latent feature space to align the view-specific representations with the unified representation, rather than to directly realign the original samples. Specifically, $\mathbf{M}$ operates on the extracted features of samples, mapping them into a shared feature space. Therefore, while the learned rotation matrix can produce aligned feature representations, it does not yield aligned samples.

Q4: The roles of each component in Eqs. (4), (5), and (7) should be specifically explained.
A4: Thanks for your advice. The metric in Eq.(4) acts as an enhanced separation-compactness ratio that evaluates the structural quality of a clustering $\mathbf{Y}$ with respect to the entire dataset $\mathbf{X}$. Specifically, H integrates scale-invariant normalization through $\sigma_{d^2}$, exponential smoothing to mitigate the effects of data shifts, and a balanced weighting of inter-cluster and intra-cluster distances. As a result, H provides a robust and comparable measure of cluster quality across different views, even when the data distributions and cluster cardinalities vary. On the basis of Eq.(4), Eq.(5) defines the final clustering quality measure, i.e., it averages pairwise cluster scores and applies a logistic transformation to normalize the range to 0-1. This design not only ensures class-cardinality invariance but also provides a directly interpretable and bounded quality score. Therefore, with Eqs. (4)-(5), we can quantify both intra-view compactness and inter-view separability in a scale-free manner and consistently identify the view that fully preserves the intrinsic structure of the data, regardless of sample size, feature scales, or the number of clusters. In this way, we can avoid biases caused by poorly aligned or low-quality views for subsequent cross-view alignment.
After selecting the base view according to the Eqs.(4)-(5), the final clustering objective function can be formulated, as presented in Eq.(7). The first part of Eq.(7) is a sample alignment-based late fusion clustering framework, where the sample alignment matrices $\mathbf{M}^v$ and the feature rotation matrices $\mathbf{R}^v$ are jointly learned to map each view-specific clustering result $\mathbf{F}^v$ into a shared space, enabling the consensus partition $\mathbf{F}^*$ to integrate multi-view information effectively. The second part introduces a structure-guided alignment term, where $\mathbf{S}^v$ encodes the similarity between the non-aligned samples of each view and the baseline view. This regularization term drives $\mathbf{M}^v$ to capture the correct correspondence by preserving cross-view structural relationships.

Thank you for your kind response. We are delighted to hear that our rebuttal has addressed your concerns. We sincerely appreciate your thoughtful comments and the time you devoted to reviewing our manuscript. Your constructive feedback has greatly helped us improve the quality of our work. We are also grateful for your positive evaluation and the final recommendation to raise the score.

Best regards,

The Authors