Dual-level Affinity Induced Embedding-free Multi-view Clustering with Joint-alignment

Q3: How do you set the number of anchors? What is the influence of it?

A3: Thanks. In experiments, we set the number of anchors to be equal to the number of clusters. The reasons are as follows.

When updating the variable $\mathbf{T}_p$, the objective function is $\operatorname{Tr} \left( \mathbf{T} _{p}^{\top} \mathbf{G} _p \mathbf{T} _{p} \left( \lambda \mathbf{H} _{p} + \boldsymbol{\alpha} _p^2 \mathbf{M} _p - 2\lambda \mathbf{S} _{p}^{\top} \right) -2\boldsymbol{\alpha}_p^2 \mathbf{T} _{p}^{\top} \mathbf{J} _{p} \right)$, which is the form of $\mathbf{A}^\top \mathbf{B} \mathbf{A} \mathbf{C} + \mathbf{A}^{\top} \mathbf{D}$. Besides, the feasible region $\\{ \mathbf{T}_p^{\top} \mathbf{1}=\mathbf{1}, \mathbf{T}_p \mathbf{1}=\mathbf{1}, \mathbf{T}_p \in \\{0,1\\}^{m \times m} \\}$ is discrete. These cause this optimization problem being hard to solve. To this end, we adopt traversal searching on one-hot vectors to obtain the optimal solution. The traversal searching operation takes $\mathcal{O}(m!)$ computing overhead where $m$ is the number of anchors. Too large $m$ will induce intensive time cost. Therefore, in all experiments, we set $m$ to the number of clusters $k$. More genius solving schemes could be further investigated in the future.

Q4: The experimental results are not convincing. For instance, OrthNTF achieves 69.4% Acc and 68.6% NMI values on the Reuters dataset, while this paper only reports 28.67% Acc and 3.07% NMI.

A4: Thanks. We are so sorry for bringing some unnecessary confusion to Reviewer zrPV.

This is mainly due to that the default hyper-parameter settings in the released code are not consistent with that in the paper. We have carefully corrected relevant parameter settings according to the suggestions presented in their paper and re-executed them. Please check the clustering result comparison table. Especially, for OrthNTF, on the dataset Reuters, the anchor number is automatically 100 in their experiments while it is automatically 93 in our experiments. The reason is that the dataset versions adopted in experiments are different. Accordingly, the generated clustering results are different. We execute OrthNTF under anchorRate=$[0.1, 0.2, 0.3, \cdots, 1.0]$, $p=[0.1, 0.2, 0.3,\cdots, 1.0]$, $\lambda=[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100, 500, 1000, 5000, 10000]$ respectively, and report the highest results.

---

Thanks very much to Reviewer zrPV for bringing this point to our attention!

We very thank Reviewer zrPV's insightful feedback and guidance for the revision of this manuscript. All concerns have been carefully responded point by point. We sincerely hope these issues have been cleared.

Q1: The self-expression affinity learning and graph based Laplacian are widely used in existing subspace clustering works. It remains unclear why the anchor self-expression enhances the quality of anchors.

A1: Thanks. The self-expression affinity learning is utilized to construct the sample-sample affinity with full size in subspace clustering. Inspired by this, we explicitly extract the global structure between anchors via self-expression learning, and meanwhile feed that into anchor-sample so as to better exploit the manifold characteristics hidden within samples. (Kindly note that in this work, we did not calculate the sample-sample relations through self-expression learning.) In addition to this, our work also designs a joint-alignment mechanism which does not involve the selection of the baseline view and meanwhile can cooperate with the learning of anchors. Moreover, a solving scheme with linear complexity enables our framework to effectively tackle MVC tasks.

---

Anchor self-expression learning can help extract the geometric characteristics between anchors, and meanwhile facilities the learning of anchors owing to the joint-optimization mechanism. To validate this point, we organize four groups of ablation experiments, i.e., No self-expression + No leanring (NSNL), No self-expression + Having learning (NSHL), Having self-expression + No learning (HSNL), Having self-expression + Having learning (HSHL, i.e., Ours). The comparison results are reported in the following table.

As seen, HSNL is consistently preferable than NSNL, and HSHL is consistently preferable than NSHL. These demonstrate that the anchor self-expression can facilitate the clustering performance improvement. Additionally, NSHL is consistently preferable than NSNL, and HSHL is consistently preferable than HSNL. These illustrate that the anchor learning can help increase the clustering results. Therefore, we can conclude that the anchor self-expression learning can enhance the quality of anchors to increase the clustering results.

We very thank Reviewer 2FNR's profound comments and guidance for the revision of this manuscript. All concerns have been carefully responded point by point. We sincerely hope these issues have been cleared.

Q1: The novelty of this work is limited since the involved components have been widely used for anchor learning and spectral clustering.

A1(1): Thanks. We emphasize that current alignment strategies typically require selecting the baseline view, and also are separated from the anchor generation. Different from them, we learn to align based on the characteristics of respective view itself and meanwhile jointly conduct anchor generation and anchor alignment, which makes them able to negotiate with each other. Besides, we explicitly take into account the geometric properties between (aligned) anchors, and feed that into the anchor-sample affinity to extract the manifold structure hidden within original samples more adequately. Especially, we also give a feasible solving scheme with linear complexity to optimize the resulting objective.

To demonstrate the strengths of our alignment strategy, we organize the experiments compared to some remarkable alignment algorithms like FMVACC [1], 3AMVC [2], AEVC [3].

FMVACC utilizes feature information and structure information of the bipartite graph generated by fixed anchors to build the matching relationship, and regards the first view of each dataset as the baseline view.

3AMVC gets rid of prior knowledge by identifying and selecting discriminative anchors within a single view using hierarchical searching, and takes the view exhibiting the highest anchor graph quality as the baseline view.

AEVC narrows the spatial distribution of anchors on similar views by leveraging the inter-view correlations to enhance the expression ability of anchors, and treats the view concatenated by column as the baseline view.

The comparison results are shown in the following table.

From this table, one can observe that our results are more desirable in most cases, which illustrates that our proposed alignment strategy is more worthy of recommendation.

[1] Wang et al., Align then fusion: Generalized large-scale multi-view clustering with anchor matching correspondences, NeurIPS, 2022.

[2] Ma et al., Automatic and Aligned Anchor Learning Strategy for Multi-View Clustering, ACM MM, 2024.

[3] Liu et atl., Learn from view correlation: An anchor enhancement strategy for multi-view clustering. IEEE CVPR, 2024.

A1(2): Additionally, we also conduct experiments to validate the effectiveness of our alignment strategy, as shown in the following table where 'Wo-A' and 'WA' denote the results without and with involving alignment respectively.

As seen, our alignment strategy is working and can effectively increase the clustering results.

---

Further, we also do alignment separately as current methods do, and the comparison results are reported in the following table where 'SA' denotes the results based on separate alignment.

Evidently, our joint-alignment mechanism makes more impressive clustering results.

---

Furthermore, we also conduct experiments to demonstrate that the geometric features between anchors are beneficial for the clustering performance improvement. The comparison results are presented in the following table where 'NAA' denotes the results not considering the characteristics between anchors.

It can be seen that our results involving anchor-anchor characteristics are more encouraging.

Q2: The authors do not compare the proposed method with theses popular deep learning ones.

A2(1): Thanks. We organize the comparative experiments with deep learning methods AdaGAE [1], DEMVC [2], MFLVC [3], DSMVC [4].

AdaGAE utilizes a graph auto-encoder to extract the potential high-level information behind data and the non-euclidean structure, and avoids the collapse by building the connections between sub-clusters before they become thoroughly random in the latent space.

DEMVC generates the embedded feature representations by deep auto-encoders, and adopts the auxiliary distribution generated by $k$-means to refine the deep auto-encoders and clustering soft assignments for all views.

MFLVC learns different levels of features from the raw features in a fusion-free manner to alleviate the conflict between learning consistent common semantics and reconstructing inconsistent view-private information.

DSMVC concurrently exploits complementary information and discards the mean ingless noise by automatically selecting features to reduce the risk of clustering performance degradation caused by view increase.

 [1] Li et al., Adaptive Graph Auto-Encoder for General Data Clustering, IEEE TPAMI, 2022. 

 [2] Xu et al., Deep Embedded Multi-view Clustering with Collaborative Training,  Information Sciences, 2021

 [3] Xu et al., Multi-Level Feature Learning for Contrastive Multi-View Clustering, IEEE CVPR, 2022. 

 [4] Tang et al., Deep Safe Multi-view Clustering: Reducing the Risk of Clustering Performance Degradation Caused by View Increase,  IEEE CVPR, 2022.

Q3: The model is somewhat complex, introducing more variables and mathematical processes. A more detailed explanation of the transition from Eq.(2) to Eq.(3) would enhance reader understanding of the methodology.

A3: Good suggestion! We here manage to explain the objective transition as much detail as possible.

The objective function in Eq.(2) is $\sum _{p=1}^v \left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{Z} _{p} \right\|| _F^2 + \lambda \left\|| \mathbf{A} _{p} -\mathbf{A} _{p} \mathbf{S} _{p} \right\||_F^2 + \beta \sum _{i,j=1}^{m} \left\|| [\mathbf{Z} _p] _{i,:} - [\mathbf{Z} _p] _{j,:} \right\|| _2^2 [\mathbf{S} _p] _{i,j}$.

The objective function in Eq.(3) is $\sum _{p=1}^v \boldsymbol{\alpha} _p^2 \left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{B} _{p} \mathbf{C} \right\|| _F^2 + \lambda \left\|| \mathbf{A} _{p} \mathbf{T} _p - \mathbf{A} _{p} \mathbf{T} _p \mathbf{S} _{p} \right\|| _F^2 + \beta \operatorname{Tr}( \mathbf{B} _p^{\top} \mathbf{L _s} \mathbf{B} _p \mathbf{C} \mathbf{C} ^{\top} )$.

Firstly, considering that the essence of anchor misalignment is that the order of anchors on different views is not identical, we can eliminate the misalignment problem via re-arranging anchors. Specially, we associate each view with a learnable matrix $\mathbf{T}_p$ to flexibly transform anchors according to the characteristics of respective view itself. (Kindly note that owing to transforming anchors in original view space, this can well preserve the view diversity.) Accordingly, the anchor matrix $\mathbf{A}_p$ on each view is reformulated as $\mathbf{A}_p \mathbf{T}_p$, the self-expression affinity learning $\left\|| \mathbf{A} _{p} - \mathbf{A} _{p} \mathbf{S} _{p} \right\||_F^2$ becomes $\left\|| \mathbf{A} _{p} \mathbf{T} _p - \mathbf{A} _{p} \mathbf{T} _p \mathbf{S} _{p} \right\||_F^2$, and the reconstruction error item $\left\||\mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{Z} _{p} \right\||_F^2$ becomes $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _p \mathbf{Z} _{p} \right\|| _F^2$.

Then, due to variance arising from the construction of embedding, we avoid forming embedding, and choose to directly learn the cluster indicators. We factorize the anchor graph as a basic coefficient matrix and a consensus matrix, and utilize binary learning to optimize the consensus matrix. Therefore, we have that the reconstruction error item $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _p \mathbf{Z} _{p} \right\|| _F^2$ is reformulated as $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _p \mathbf{B} _{p} \mathbf{C} \right\|| _F^2$, and the point-point guidance $\sum _{i,j=1}^{m} \left\|| [\mathbf{Z} _p] _{i,:} - [\mathbf{Z} _p] _{j,:} \right\|| _2^2 [\mathbf{S} _p] _{i,j}$ is reformulated as $\sum _{i,j=1}^{m} \left\|| [\mathbf{B} _p \mathbf{C}] _{i,:} - [\mathbf{B} _p \mathbf{C}] _{j,:} \right\|| _2^2 [\mathbf{S} _p] _{i,j}$, which can be equivalently written as the matrix trace form of $\operatorname{Tr}(\mathbf{B} _p^{\top} \mathbf{L _s} \mathbf{B} _p \mathbf{C} \mathbf{C}^{\top})$. $\mathbf{L_s} = \mathbf{D}_p - \mathbf{S}_p$, $\mathbf{D} _p = diag(\sum _{j=1}^{m} [\mathbf{S} _p] _{i,j}~| i=1,\cdots, m)$. (Kindly note that the consensus cluster indicator matrix $\mathbf{C}$ provides a common structure for anchors on all views, inducing them rearranging towards the common structure.)

Finally, considering that views typically have different levels of importance, we introduce a learnable weighting variable for each view to automatically measure its contributions. Therefore, we have that $\sum _{p=1}^{v} \left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _p \mathbf{B} _{p} \mathbf{C} \right\||_F^2$ is reformulated as $\sum _{p=1}^{v} \boldsymbol{\alpha} _p^2 \left\||\mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _p \mathbf{B} _{p} \mathbf{C} \right\||_F^2$.

At this point, we can obtain the objective function like Eq.(3).

For the feasible region, $\\{ \mathbf{T} _p^{\top} \mathbf{1}=\mathbf{1}, \mathbf{T} _p \mathbf{1}=\mathbf{1}, \mathbf{T} _p \in \\{0,1\\}^{m \times m} \\}$ denotes only re-arranging anchors and does not change the anchor values. $\\{ \mathbf{S} _{p}^{\top} \mathbf{1}=\mathbf{1}, \mathbf{S} _{p} \geq 0, \sum _{i=1}^{m} [\mathbf{S} _{p}] _{i,i}=0 \\}$ denotes expressing oneself using other anchors and meanwhile avoids expressing oneself using oneself. $\{\mathbf{B} _p^{\top} \mathbf{B} _p = \mathbf{I} _k \}$ denotes learning discriminative basic coefficients. $\\{ \sum _{i=1}^{k} \mathbf{C} _{i,j} =1, j= {1, 2, \dots, n}, \mathbf{C} \in \\{0,1\\}^{k \times n} \\}$ denotes that each column has only one non-zero element, that is, a sample belongs to only one cluster. $\\{ \boldsymbol{\alpha} ^{\top} \mathbf{1} = 1, \boldsymbol{\alpha} \geq 0 \\}$ denotes normalizing and meanwhile avoids trivial solutions.

We very thank Reviewer 4XPe's thoughtful suggestions and guidance for the revision of this manuscript. All concerns have been carefully responded point by point. We sincerely hope these issues have been cleared.

Q1: It lacks complex constraints like the Schatten p-norm that could help capture deeper cross-view complementarities. How does the model handle scenarios where the quality of anchors varies significantly across different views?

A1: Thanks! This is a very promising research direction. The tensor Schatten p-norm is commonly regarded as a good means to exploit the complementary information between views, like [1], [2], [3], [4], [5], [6], etc. Our model at present dose not involve the Schatten p-norm, and adopts the shared cluster indicator matrix to capture view complementary information. Including the Schatten p-norm could help further enhance the clustering ability of our model. We will pay efforts to explore this in the future. We sincerely appreciate the Reviewer 4XPe for his/he fairly constructive suggestions!

About handling the scenarios where the quality of anchors varies significantly across different views, in this work, we utilize a learnable view-related weighting scheme to adaptively adjust the contributions of each view so as to balance the importance of anchors on the view. Perhaps, the anchor-wise weighting scheme will be more advisable since anchors on the same one view also could own diverse importance. We will further investigate this in the future. Thanks Reviewer 4XPe for bringing this to our attention.

Q2: The reliance on anchor alignment to maintain cross-view consistency introduces additional computational steps. Some methods avoid alignment through feature space fusion or shared representations. It would be useful for the authors to elaborate on the essential role of anchor alignment and under what conditions it might be adapted or simplified. Are there specific conditions or datasets where the necessity of anchor alignment might be relaxed or modified?

A2(1): Thanks. The anchor alignment indeed introduces additional computational steps due to the need for optimizing the permutation variables.

---

The works based on feature space fusion or shared representations usually extract a group of unified anchors rather than multiple groups of view-specific anchors to construct the similarity relationship. Although avoiding alignment, this paradigm could not effectively exploit complementary information between views due to the unified anchors being shared for all views. To further illustrate this point, we organize the comparison experiments between unified anchors (UA) and view-specific anchors (VSA, i.e., ours). The results are presented in the following table.

As seen, our results are more preferable than the counterparts adopting unified anchors, the reason of which could be that the view-exclusive complementary representation information (view-specific (aligned) anchors contain) outweighs the view-common consensus representation information (unified anchors contain).

[1] Guo et al., Logarithmic Schatten-p  Norm Minimization for Tensorial Multi-View Subspace Clustering, IEEE TPAMI, 2023.

[2] Xia et al., Tensorized Bipartite Graph Learning for Multi-view Clustering, IEEE TPAMI, 2023.

[3] Feng et al., Federated Fuzzy C-means with Schatten-p Norm Minimization, ACM MM, 2024.

[4] Li et al., Label Learning Method Based on Tensor Projection, ACM KDD, 2024. 

[5] Sun et al., Improved Weighted Tensor Schatten p-Norm for Fast Multi-view Graph Clustering, ACM MM, 2024. 

[6] Wang et al., Bi-Nuclear Tensor Schatten-p Norm Minimization for Multi-View Subspace Clustering, IEEE TIP, 2023.

A4(2): Subsequently, considering that the variance arises from the construction of embedding, we avoid forming embedding and choose to directly learn the cluster indicators. Specially, we factorize the anchor graph as a basic coefficient matrix $\mathbf{B}_p$ and a consensus matrix $\mathbf{C}$, and utilize binary learning to optimize the consensus matrix. Therefore, we have that the term $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{Z} _{p} \right\|| _F^2$ is reformulated as $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{B} _{p} \mathbf{C} \right\|| _F^2$. The point-point guidance term $\sum _{i,j=1}^{m} \left\|| [\mathbf{Z} _p] _{i,:} - [\mathbf{Z} _p] _{j,:} \right\|| _2^2 [\mathbf{S} _p] _{i,j}$ is reformulated as $\sum _{i,j=1}^{m} \left\|| [\mathbf{B} _p \mathbf{C}] _{i,:} - [\mathbf{B} _p \mathbf{C}] _{j,:} \right\|| _2^2 [\mathbf{S} _p] _{i,j}$, which can be equivalently transformed as the matrix trace form of $\operatorname{Tr}(\mathbf{B}_p ^{\top} \mathbf{L _s} \mathbf{B} _p \mathbf{C} \mathbf{C}^{\top})$. $\mathbf{L _s} = \mathbf{D} _p - \mathbf{S} _p$, $\mathbf{D} _p = diag( \sum _{j=1}^{m} [\mathbf{S} _p] _{i,j}~| i=1,\dots, m )$. This paradigm not only makes the consensus matrix $\mathbf{C}$ successfully represent the cluster indicators, but also provides a common structure for anchors on all views, inducing them rearranging towards the corresponding matching relationship.

At the last, due to views generally having different levels of importance, we assign a weighting variable to each view to adaptively adjust its contributions. Accordingly, the term $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{B} _{p} \mathbf{C} \right\||_F^2$ is further reformulated as $\boldsymbol{\alpha} _p^2 \left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{B} _{p} \mathbf{C} \right\|| _F^2$.

Based on the above analysis, we have that the objective is formulated as $\sum _{p=1}^v \boldsymbol{\alpha} _p^2 \left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{B} _{p} \mathbf{C} \right\|| _F^2 + \lambda \left\|| \mathbf{A} _{p} \mathbf{T} _p - \mathbf{A} _{p} \mathbf{T} _p \mathbf{S} _{p} \right\|| _F^2 + \beta \operatorname{Tr}( \mathbf{B} _p^{\top} \mathbf{L_s} \mathbf{B} _p \mathbf{C} \mathbf{C}^{\top} )$.

The first item aims at building the similarity via minimizing the reconstruction error. The second item represents the self-expression affinity of aligned anchors. The third item plays a role in feeding anchor-anchor characteristics into anchor-sample.

Further, the constraints $\boldsymbol{\alpha} ^{\top} \mathbf{1} = 1$ and $\boldsymbol{\alpha} \geq 0$ aim at doing normalization and meanwhile avoid trivial solutions. $\{ \mathbf{B} _p^{\top} \mathbf{B} _p = \mathbf{I} _k \}$ aims at learning discriminative basic coefficients. $\\{ \mathbf{T} _p^{\top} \mathbf{1}=\mathbf{1}, \mathbf{T} _p \mathbf{1} = \mathbf{1}, \mathbf{T} _p \in \\{0,1\\} ^{m \times m} \\}$ aims at rearranging anchors and meanwhile guarantees not to change the values of anchors. $\\{ \sum _{i=1}^{k} \mathbf{C} _{i,j} =1, j= {1, 2, \dots, n}, \mathbf{C} \in \\{0,1\\}^{k \times n} \\}$ guarantees that there is only one non-zero element in each column, i.e., one sample belongs to only one cluster. $\\{ \mathbf{S} _{p}^{\top} \mathbf{1}=\mathbf{1}, \mathbf{S} _{p} \geq 0, \sum _{i=1}^{m} [\mathbf{S} _{p}] _{i,i}=0 \\}$ guarantees expressing oneself through other anchors while avoiding using oneself to express oneself.

Q5: What is the difference between the anchor alignment module with those of existing works?

A5: Thanks. The main differences are as follows,

• We do not require selecting the baseline view.

• We can coordinate with the generation of anchors.

The selection of baseline view not only brings complicated solving procedure but also affects the clustering performance. If the baseline view is not well selected, the graph structure will be inaccurately fused. Unlike this, we do not require the baseline view, and can automatically rearrange anchors according to respective view characteristics.

Besides, we also can coordinate with anchors in the unified framework and thereby facilitate the learning of anchors, which makes view information interact across different levels.

Q4: Table 5 does not include all the symbols. The Methodology section might be too brief, which should be introduced with more details by explaining the reasons for the design of each component.

A4(1): Thanks. We have updated Table 5, please check it.

The symbols in this manuscript are as follows,

For the methodology, we here provide more details to explain the reasons for the design of each component.

First of all, to exploit the geometric characteristics between anchors, inspired by the concept of subspace reconstruction, we introduce self-expression learning for anchors. Specially, we utilize the paradigm $\left\|| \mathbf{A}_p -\mathbf{A}_p \mathbf{S}_p \right\||_F^2$ to explicitly extract the global structure between anchors.

After obtaining the anchor-anchor characteristic $\mathbf{S}_p \in \mathbb{R}^{m \times m}$, we need to integrate that into anchor-sample so as to exploit the manifold features inside samples. To this end, we adopt the idea of point-point guidance to adjust the anchor graph. Note that the rows of anchor graph $\mathbf{Z}_p \in \mathbb{R}^{m \times n}$ correspond to anchors, and thus we utilize the element $[\mathbf{S} _p] _{i,j}$ to guide $[\mathbf{Z} _p] _{i,t}$ and $[\mathbf{Z} _p] _{j,t}$, $t=1, \cdots n$, which can be formulated as $\sum _{i,j=1}^{m} \left\|| [\mathbf{Z} _p] _{i,:} - [\mathbf{Z} _p] _{j,:} \right\|| _2^2 [\mathbf{S} _p] _{i,j}$ and aims at restricting similar features to maintain the consistency.

Then, to alleviate the anchor misalignment, considering that the nature of misalignment is that the order of anchors on different views is not identical, we alleviate the misalignment issue by rearranging anchors. Particularly, we associate each view with a learnable permutation matrix $\mathbf{T} _p$ to freely transform anchors in the original dimension space according to the characteristics of respective view. In addition to not involving selecting the baseline view, our mechanism also can coordinate with anchors in the unified framework and thereby facilitates the learning of anchors. Correspondingly, the anchor matrix $\mathbf{A}_p$ is reformulated as $\mathbf{A}_p \mathbf{T}_p$. The self-expression term $\left\|| \mathbf{A} _p -\mathbf{A} _p \mathbf{S} _p \right\||_F^2$ and the reconstruction term $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{Z} _{p} \right\||_F^2$ are reformulated as $\left\|| \mathbf{A} _p \mathbf{T} _p -\mathbf{A} _p \mathbf{T} _p \mathbf{S} _p \right\|| _F^2$ and $\left\|| \mathbf{X} _{p} - \mathbf{A} _{p} \mathbf{T} _{p} \mathbf{Z} _{p} \right\|| _F^2$, respectively.

A1(2): Besides, we also conduct separate-alignment (SA) as current methods do, and the comparison results are summarized in the following table. (We omit the variance items since they are all zero.)

Evidently, our joint-alignment receives preferable results in most cases.

Q2: The comparison methods lack some latest works. For example, the reference Liu 2024 (in line 581) was discussed in this paper, which includes anchor alignment mechanism, but it is not compared with the proposed work.

A2: Thanks. We organize some new comparison experiments in which [1], [2] and [3] are all based on anchor alignment methods. ([3] is the reference Liu 2024 mentioned.)

Ref [1] utilizes feature information and structure information of the bipartite graph generated by fixed anchors to build the matching relationship, and regards the first view of each dataset as the baseline view.

Ref [2] gets rid of prior knowledge by identifying and selecting discriminative anchors within a single view using hierarchical searching, and takes the view exhibiting the highest anchor graph quality as the baseline view.

Ref [3] narrows the spatial distribution of anchors on similar views by leveraging the inter-view correlations to enhance the expression ability of anchors, and treats the view concatenated by column as the baseline view.

Please see the above table in A1(1) for experimental results.

Q3: In Table 1, several compared methods exhibit extremely poor performance on some datasets. It might be better if the authors could explain the possible reasons.

A3: Thanks. We sincerely apologize for causing the confusion to Reviewer Ccq1. The reasons are that the default parameter settings in the released code are inconsistent with that in the paper. We have carefully corrected these according to the guidance in their paper, and reorganized the comparison experiments. Please check it. We deeply appreciate Reviewer Ccq1 for reminding us this, which significantly helps us improve the quality of this manuscript.

Especially, PMSC, AMGL and MLRSSC still express inferior performance in certain scenarios, the reasons of which could be that PMSC reaches the consensus clustering under the premise of the basic partition realizing the ground truth and meanwhile equally treats every view, the factors generated by the cluster indicator with orthogonal constraints in AMGL impair the discriminability of some graphs, and MLRSSC linearly combines the generated representation matrices and only utilizes truncation operation to determine the penalty parameters.

[1] Wang et al., Align then fusion: Generalized large-scale multi-view clustering with anchor matching correspondences, NeurIPS, 2022. 

[2] Ma et al., Automatic and Aligned Anchor Learning Strategy for Multi-View Clustering, ACM MM, 2024. 

[3] Liu et atl.,  Learn from view correlation: An anchor enhancement strategy for multi-view clustering. IEEE CVPR, 2024

We very thank Reviewer Ccq1's constructive comments and guidance for the revision of this manuscript. All concerns have been carefully responded point by point. We sincerely hope these issues have been cleared.

Q1: A core idea of the work is to introduce an anchor permutation matrix, while this idea has been widely adopted by previous works.

A1(1): Thanks. Current permutation strategies generally require selecting the baseline view, such as [1], [2], [3]. Moreover, the anchor generation, the anchor transformation and the graph construction are separated from each other, which hinders the interaction of view information across different levels. Unlike them, in this work, we associate a learnable permutation for each view to freely rearrange anchors during their original space, successfully unify anchor generation and anchor transformation as well as graph construction within one common framework, and meanwhile provide a feasible solving solution with linear complexity. Owing to the joint-alignment mechanism, we do not involve selecting the baseline view, and meanwhile anchors can be permuted automatically according to respective view characteristics. Also, this paradigm can coordinate with the learning of anchors.

Particularly, we conduct some experiments against [1], [2] and [3] to demonstrate the advantages of our alignment mechanism, as shown in the following table.

As seen, our results are more desirable in most cases.

[1] Wang et al., Align then fusion: Generalized large-scale multi-view clustering with anchor matching correspondences, NeurIPS, 2022. 

[2] Ma et al., Automatic and Aligned Anchor Learning Strategy for Multi-View Clustering, ACM MM, 2024. 

[3] Liu et atl.,  Learn from view correlation: An anchor enhancement strategy for multi-view clustering. IEEE CVPR, 2024.