Bit-swapping Oriented Twin-memory Multi-view Clustering in Lifelong Incomplete Scenarios

Q1: Theoretical explanation on the cooperation mechanism between UAM and SBG; the assumption conditions; the limitations.

A1: Thanks! UAM and SBG collectively transform knowledge via one common learnable framework. UAM aims to do propagation at embedding level while SBG does propagation at similarity level. They seamlessly integrate representations from historical views and newly-arrived data to facilitate clustering. Kindly note that they can cooperate with each other via $\mathcal{L}_1$, or via $\mathcal{L}_2$ and $\mathcal{L}_3$ using the swapping matrix, or via both.

About the assumption conditions, we assume that views are independent of each other. This is rational since in practice, view data is usually collected from diverse sensors. Based on this assumption, we customize a guidance for each view to harmonize disparate dimensions and thereby construct transferable anchors and bipartite graph.

About the limitations, there are two hyper-parameters in our model, which could to some extent weaken its practicality. The paradigm that firstly formulates spectrum and then conducts partitioning may degenerate view data diversity. Besides, we at this point do not learn to adjust the representation contributions between newly-arrived data and historical views. As data collection progresses, inaccessible historical views will account for an increasing proportion compared to new view. Therefore, balancing them will further boost the model performance.

Q2: The influence of different hyper-parameter combinations on the convergence behavior.

A2: We employ an alternating optimization strategy to minimize the formulated loss function that is lower-bounded by zero. Under different hyper-parameter combinations, the loss still can be lower-bounded by zero. Additionally, the solutions of all variables still can be obtained by Eqs.(8), (10), (15) and (18) respectively, even both of hyper-parameters as zero (as illustrated in Steps Under Initial View). Therefore, during updating each variable, it is monotonically decreasing. Combined with monotonicity and boundedness, consequently, our algorithm is theoretically convergent for different hyper-parameter combinations. We will add the convergence curves under diverse hyper-parameter combinations in the next version.

Q3: The interactions between UAM and SBG; the performance contributions of UAM and SBG in isolation.

A3: UAM and SBG work within one integrated framework, and they negotiate with each other during the learning process. Owing to the joint-optimization, UAM and SBG establish a tightly-coupled relationship through bidirectional collaborative mechanisms. Specially, UAM guides SBG in capturing global clustering structures, while SBG refines UAM through its connection patterns to eliminate view-specific biases. This synergistic interaction enhances clustering robustness, particularly when handling incomplete views, where the memory function of UAM and the topological elasticity of SBG effectively preserve clustering performance.

About the performance contributions, the following table gives the ablation results. As seen, UAM and SBG collaboratively improve the clustering performance.

Q4: Complexity comparison between BSTM and comparison methods.

A4: Thanks! We summarize the complexity comparison in the following table. Please check it. We will add this in the next version.

Thanks for the detailed response. It has clarified most of my concerns. I would maintain my score.

Q1: The topology transformation and discrete optimization may complicate the model.

A1: Thanks! The topology transformation aims at reorganizing anchor order to alleviate the mismatching dilemma while the discrete optimization is to preserve the value consistency. They collaboratively support the anchor knowledge transfer from historical views to current view. The topology transformation and discrete optimization are equivalently converted to a linear programming, and can be easily solved by off-the-shelf software. Besides, the theoretical complexity demonstrates that they are with a linear overhead and do not bring about severe computing and storing workload.

Q2: The need for parameter searching may weaken practical performance.

A2: The hyper-parameters aim at regularizing anchor residual and graph residual to formulate desirable similarity. Although it needs parameter searching, combined with sensitivity figures, we have that the model performance is relatively stable under diverse parameter values on multiple data scenes. Therefore, we can deploy it on different scenes with the given guidelines. In the future, we will strive to devise a parameter-free version to further enhance its practicality.

Q3: The memory anchor may increase the space cost.

A3: The memory anchor matrix plays a role in propagating basis embedding from historical views to incoming view. Although it is with memory function, kindly than it has the size of $m \times m$. In general, $m$ is greatly smaller than $n$. Thus, the space cost caused by the memory anchor matrix is almost negligible compared to the overall cost.

Q4: Can the specialized guidance generalize to raw data?

A4: Yes. The specialized guidance is associated with anchor matrix (AAM) in the proposed model, and it is mainly to alleviate dimension disparity to build transferable space. When generalizing to raw data (GRD), although this is also able to construct transferable space via dimension reduction, it generally degenerates dimension diversity and brings about information loss of view data, accordingly weakening the overall clustering performance. To validate this, we organize relevant comparison experiments, as shown in the following table. One can see that GRD is suboptimal compared to AAM.

Q5: Is it possible to build one indicator matrix from the other? Are there means available to decrease its overhead?

A5: Yes. The construction of indicator matrices is based on the union set of observed sample indexes. Therefore, the indicator matrix about current view can be built through that about historical views, vice versa. Besides, although the indicator matrix about historical views is close to full size over time, kindly note that we do not explicitly build and calculate these indicator matrices, and only utilize them for illustration. In practical, we select certain columns from the shared graph via corresponding indexes to establish similarity for current view and do graph propagation for historical views. All these operations are proven to only require linear complexity.

Q6: How to calibrate the contributions between incoming new views and unavailable historical views?

A6: Due to privacy protection or memory limitation, the historical views become inaccessible over time. Consequently, it can not directly weight historical views and incoming new views. To calibrate the contributions, note that during propagation, it inherently involves anchor transfer and graph transfer from historical views to new views. These components hierarchically characterize view data representation. Therefore, instead of view data, we can calibrate anchor and graph respectively. Specially, we can adjust anchor importance in $\mathcal{L}_2$ and graph importance in $\mathcal{L}_3$ to balance view contributions. This dual-calibration mechanism ensures equitable weight allocation across views while preserving structural hierarchies.

Q1: Associating IN-1 and IN-A with $\mathbf{H}_t$ in (3) and $\widetilde{\mathbf{E}} _{t-1}$ in (5).

A1: Thanks! We will associate these in the next version.

Q2: The projectors could bring about extra overhead.

A2: According to their size $d_t \times m$, we have that storing them will take $\mathcal{O}(dm)$ memory overhead. Besides, combined with Remark 1, we have that updating them will take $\mathcal{O}(n)$ computing overhead. Since $d$ is a constant and $m$ is largely smaller than $n$, we can derive that these projectors do not impact the overall complexity.

Q3: More illustration about topology rendering.

A3: The topology rendering mechanism requires that in each column and each row, there is only one element equal to 1, while all other elements are zero. These features guarantee that it merely alters anchor order while anchor value remains unchanged. This makes learned anchors well maintain the characteristics of original data.

Q4: How to guarantee compatible graph scale?

A4: To derive compatible graph, we construct indicator matrices to pick out certain columns of the common graph for current view to do similarity calculation. Afterwards, to make this graph transferable, we utilize the indicator matrix to construct corresponding small similarity and then compare it with that constructed on all previous views. Kindly note that we do not explicitly establish these tailored indicator matrices, and use them solely for illustrative purposes. Remarkably, we adopt the element-wise value extraction operation via corresponding index to do similarity construction and comparison.

Q5: How to reduce the cost of bipartite graph to linear?

A5: Along with the view collection, the size of $\widetilde{\mathbf{E}} _{t-1}$ will be close to $n \times n$. Accordingly, computing $\widetilde{\mathbf{E}} _{t-1} \widetilde{\mathbf{E}} _{t-1}^{\top}$ will require a third-order cost. To reduce it to linear, note that $\widetilde{\mathbf{E}} _{t-1}$ consists of binary elements, and $\widetilde{\mathbf{E}} _{t-1} \widetilde{\mathbf{E}} _{t-1}^{\top}$ is a diagonal matrix. The diagonal elements are each row sum of $\widetilde{\mathbf{E}} _{t-1}$. Thus, instead of explicit calculation, we can directly construct the diagonal elements, and then multiply it by the shared graph with the size of $m \times n$. Consequently, the computing cost is linear.

Q6: How to guarantee the coefficient in (13) non-zero?

A6: $\mathbf{H}_t$ aims at measuring whether the samples are available on current view, while $\widetilde{\mathbf{E}} _{t-1}$ measures whether the samples are on historical views. Note that the current sample either appears in the current view, the historical view, or both. Therefore, we have that each row sum of $\mathbf{H} _t$ + $\widetilde{\mathbf{E}} _{t-1}$ must be non-zero. Accordingly, we can derive that this coefficient is non-zero.

Q7: The performance under heuristic search.

A7: The following table presents the performance comparison where HS and LS denote the results based on heuristic search and learning strategy respectively. As seen, HS usually is sub-optimal, possibly because the heuristic search is unable to effectively characterize original data due to the separation from graph generation.

Q1: A summary on the contribution of each item.

A1: Thanks! $\mathcal{L}_1$ aims at formulating similarity and extracting basis patterns via reconstruction error minimization. $\mathcal{L}_2$ renders anchors via current view and historical views while preserving value consistency to reach matching configuration. $\mathcal{L}_3$ guarantees graph scale to be compatible and meanwhile reorganizes graph topology. $\mathcal{L}_2$ and $\mathcal{L}_3$ collaboratively transfer knowledge at similarity level and embedding level. We will add this summary in the next version.

Q2: The inspiration about tailored indicator matrices.

A2: Due to the incompleteness, the graph size is diverse and consequently it is impossible to construct the graph shared for newly-arrived view and previous views. Besides, it also can not do knowledge transfer on graph. To overcome these difficulties, we utilize $\mathbf{H} _t$ to pick out some columns of the shared graph for current view to build similarity, and utilize $\widetilde{\mathbf{E}} _{t-1}$ to form a smaller graph to compare with previous views. Kindly note that the size of shared graph is dynamically changing as view data is collected.

Q3: The underlying principle for $\widehat{\mathbf{S}}_t$.

A3: This essentially guarantees that anchors are merely rearranged without altering their values. It mainly leverages one-hot characteristics to reorganize anchors. If violated, the learned anchors will be disturbed, potentially destabilizing the entire graph topology.

Q4: The advantages for twin-memory paradigm.

A4: Rather than extracting features by view, it builds only one graph across views, and so the computational workload is smaller than current per-view architectures. Besides, it effectively gathers view information during forming features instead of via fusion. Accordingly, the distortion induced by fusion is effectively avoided.

Q5: Could the transformation be a unit matrix?

A5: Yes. When the anchors learned on current view is the same order as those on previous views, topological rendering of anchors becomes unnecessary during transformation. Accordingly, it degenerates into a unit matrix.

Q6: Performance under guidance without column constraints.

A6: These constraints aims at discriminating projectors. Without them (WO), the dimension space could be redundant and accordingly the model performance will degrade. The following table gives the comparison results.

Q7: The reasons for hyper-parameter setting.

A7: $\lambda$ plays a role in regulating anchor residual while $\beta$ regulates graph residual. Because of the consensus structure, these residuals are generally in the same low magnitude. Besides, in experiments, we found that the reconstruction error is usually slightly larger than these residuals. Thus, we search $\lambda$ and $\beta$ in a slightly larger same scale range.

Q8: The rationale for matrix vectorization.

A8: It primarily transforms the intractable trace problem into an element-wise maximization linear programming problem, thereby enabling solutions via existing packages.

Q9: Is the transfer still valid without view intersection?

A9: Yes. The devised transfer mechanism is only related to the union between indexes. Even without view intersection, it still operates properly as long as the union is not empty.