Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization

We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.

In fact, the authors call this degradation phenomenon model collapse, which is not very accurate. Collapse should be very extreme. The performance degradation is a bit like an overfitting problem.

Your question is very helpful in understanding the model collapse of DCCA.
Previous studies have referred to weights being redundant and low-rank as an over-parametrization issue $1,3$ , and then the impaired feature quality and downstream task performance caused by this issue is called collapse $2$ . Similarly, in this paper, we call the degradation of DCCA in the downstream task, model collapse.
It is worth noting that this is not the same as overfitting in traditional supervised learning, the collapse of DCCA is not due to to it is memorizing training data, leading to a reduction in generalizability.

$1$ Wang, Zhennan, et al. "Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles." Advances in Neural Information Processing Systems 33 (2020): 19099-19110.

$2$ Jing, Li, et al. "Understanding dimensional collapse in contrastive self-supervised learning." arXiv preprint arXiv:2110.09348 (2021).

$3$ Barrett, David GT, and Benoit Dherin. "Implicit gradient regularization." arXiv preprint arXiv:2009.11162 (2020).

The method is effective, but it cannot be denied that it is too simple.

Our NR method is intuitively efficient, and it rests on a number of theoretical guarantees, including why NR guarantees CIP properties that constrain the behavior of the network (Theorem 1), and why NR guarantees that features do not degrade (Theorem 2).
Our NR method can be generalized to other methods such as DGCCA.

We believe a simple method is a plus, rather than a minus, as it can be used in different types of DCCA very easily, and it can be widely used as a plug-and-play module.

The legend may be better if it is a vector diagram.

We are grateful for your suggestion, we will improve the quality of the pictures in the next version.

The authors' theoretical analysis is based on Gaussian white noise. Is it the same for other types of noise?

Your question is very helpful in understanding NR. From our theoretical analysis, the most important feature of noise is that the sampled noise matrix is a full-rank matrix. Therefore, continuous distributions such as the uniform distribution can also be applied to NR, which demonstrates the robustness of the NR method. Here are our experiments with synthetic datasets (mean and variance are from multiple datasets (different common rates)):

As you can see, their effects on preventing model collapse and improving the performance of DCCA are almost identical:

Is NR loss also universal and effective on other DCCA methods? I think this is very important because it avoids the risk that DCCA is a special case.

We strongly agree with you that it is important that the NR method works for other DCCA methods. In particular, we test the results of NR-DGCCA in this paper and compare them with other methods such as DGCCA. Please see Appendix A.8 DGCCA and NR-DGCCA in our paper.
Moreover, we have added two new DCCA methods, DCCA_EY and DCCA_GHA, which replace the matrix factorization in CCA with matrix operations, and in the follow-up work, we can check whether NR can be used in them $1$ .

$1$ Chapman, James William Harvey, Ana Lawry Aguila, and Lennie Wells. "A Generalized EigenGame With Extensions to Deep Multiview Representation Learning."

We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.

However, as I know the relation between weight matrices’full-rank property and the overfitting phenomenon is still underexplored. The authors do not provide enough proofs to support this opinion.

Thanks for your comment. There is theoretical work devoted to the phenomenon that weight matrices tend to be low-rank in overfitting networks $1,2$ .
In the field of representation learning, $1$ mentions that due to the interaction between network weight layers, the weight matrix collapses (be low-rank), which in turn affects the quality of the representation.
We can observe very clearly that the performance of DCCA decreases with training while its NESum decreases rapidly, so we hypothesize that the weight matrix collapse (low-rank) also occurs in DCCA.
More theoretical work, such as analyzing how the layers in DCCA interact with each other and why they become low-rank, is also very important and we leave it for future work.

In this paper, we mainly provide practical solutions to model collapse, and the theoretical analysis is for justification purposes.

$1$ Jing, Li, et al. "Understanding dimensional collapse in contrastive self-supervised learning." arXiv preprint arXiv:2110.09348 (2021).

$2$ Barrett, David GT, and Benoit Dherin. "Implicit gradient regularization." arXiv preprint arXiv:2009.11162 (2020).

Besides, ‘full-rank’ is not a precise(even misleading) enough expression if the authors determine to use NESum as metric to evaluate the redundacy of weight matrices.

Indeed, there is a difference between the notion of the rank of a matrix and that of NESum. NESum was originally proposed as a measure of whether the eigenspace is dominated by a small number of very large eigenvalues. We simply follow the previous research and use this metric to measure the eigenvalue distribution and redundancy of the matrix.
In some cases where the eigenvalues are suddenly decreasing, e.g., NESum (2,1,0.1) = 1 + 0.5 + 0.05 = 1.55 (1->0.1), NESum (2,2,0) = 1 + 1 + 0 = 2 (2->0), the full-rank matrix will have a smaller NESum.
In order to better analyze the weight matrices in DNNs, we add the average of the cosine similarity between the dimensions in the weight matrices as another measure of redundancy, and we use DCCA and NR-DCCA in synthetic datasets as an example:

It can be seen that the metrics of cosine similarity and NESum are synchronized, with weight redundancy in DCCA increasing (cosine similarity increases, NESum decreases) and NR-DCCA redundancy decreasing (cosine similarity decreases, NESum increases).
In particular, we visualize the correlation matrix and the eigenvalues of the first Linear layer in DCCA and NR-DCCA. Again, it can be found that the redundancy of the weight matrices in DCCA is rising (please see the pdf material).

The theoretical proof of Theorem 2 only shows that noise regularization do not cause a serious degradation to DCCA’s performance. However, why noise regularization works in DCCA still remains unclear. The authors just write they try to enforce DCCA to mimic the behavior of Linear CCA but no more illustrations are provided.

Your suggestions are very helpful for the understanding of NR.
First of all, the key difference between DCCA and Linear CCA is that the weight matrix in DCCA cannot be guaranteed to be full-rank during the optimization process. Combined with the previous research, we believe that this difference leads to model collapse in DCCA, and therefore we need to design a method to constrain the behavior of DCCA.
As for the full-rank property of Linear CCA, we prove through Theorem 1 that it is equivalent to CIP.
So we design NR, through which DCCA has the property of CIP, and its behavior is constrained to be consistent with that of Linear CCA.
Theorem 2 shows that NR can guarantee that the generated features will not be degenerated (reconstruction and denoising) from the feature level.

Some texts in the figures are too small.

We apologize for the small size of the text in the images. Due to space constraints, we put too many images under the same figure, which will be improved in our subsequent releases.

why can noise regularization enforce DCCA to mimic the behavior of Linear CCA and therefore prevent DCCA from model collapse?

Thanks for your question. The key difference between DCCA and Linear CCA is that the weight matrix in DCCA cannot be guaranteed to be full-rank during the optimization process. Combined with the previous studies, we believe that this difference leads to model collapse in DCCA, and therefore we need to design a method to constrain the behavior of DCCA.
As for the full-rank property of Linear CCA, we prove through Theorem 1 that it is equivalent to CIP.
So we design NR, through which DCCA has the property of CIP, and its behavior is constrained to be consistent with that of Linear CCA.

The authors need to polish their expressions. According to the math formulat of NESum, a full-rank matrix with considerably different eigenvalues can still have low NESum score. For example, diag(1000,1,1) has lower NESum than diag(2,1,0), both of them are full-rank.

Your suggestion is valuable, and we have added the cosine similarity of the elements of the weight matrix as an indicator of the redundancy of the weights.

We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.

Readability. weak1

We apologize for not specifying the structure of the MLP in the paper. All our MLPs use the Leaky ReLu activation function. The first linear layer is feature_dim * hidden_dim and the final linear layer is hidden_dim * embedding_dim.

For the synthetic dataset, which is simple, we have only one hidden layer with a dimension of 256. For the real-world dataset, we have used MLPs with three hidden layers, and the dimension of the middle hidden layer is 1024.

To enhance reproducibility, we will release all the experiment settings and source codes after the blind review process.

weak2

Thanks a lot for your suggestion, which is very reasonable. It is essential to test the robustness of NR to model complexity, and we supplemented our synthetic datasets with the following experiment
(hidden_dim = 128):

Since synthetic data is simple, increasing the depth of the network makes both DCCA and NR-DCCA less effective, but the nature of NR for preventing model collapse is still maintained. We can clearly see that by increasing the network depth, the DCCA collapses more severely, while NR remains effective.

Q1 In Line 137

The questions you mentioned are very helpful in understanding CCA. The CCA family is not strong enough to deal with missing views (missing data for a view at some sample points), which requires that the data for each view of the sample is complete. The MVRL domain has a specialized approach to address the missing view scenario, which is called partial multi-view learning, and it may not be the focus of this paper.

Q2 What are the practical applications of the CCA methods?

Your suggestions are very reasonable and I will add various practical application scenarios for the CCA methods in the appendix:

The application of CCA mainly focuses on multi-view data. Multi-view data may come from multiple domains, such as computer vision, natural language, speech, and so on $1$ .
The features of different views are utilized to obtain a unified form of representation that captures the correlation between different views $2$ , which can be used in various downstream tasks such as classification, retrieval, clustering, and dimension reduction $2$ .

$1$ Yan, Xiaoqiang, et al. "Deep multi-view learning methods: A review." Neurocomputing 448 (2021): 106-129.

$2$ Hardoon, David R., Sandor Szedmak, and John Shawe-Taylor. "Canonical correlation analysis: An overview with application to learning methods." Neural computation 16.12 (2004): 2639-2664.

(No room for more references)

Q3 In line 137

I apologize for the potential misunderstanding caused. Let's take the Caltech101 dataset as an example, the training set has 6400 images, so the same image has been fed to three different feature extractors producing three features a 1, 984-d HOG feature, a 512-d GIST feature, and a 928-d 681 SIFT feature. Then for this dataset. X_1: 1984*6400 , X_2: 512*6400 , X_3: 928*6400

Q4 The experiment setting

We apologize for the misunderstanding and will add Hwang et al. (2021) in the appendix. Our experiment setting is the same, in the training set, we only utilize the multi-view features for multi-view learning and train the encoder to capture the correlation between multiple views. Subsequently, using the trained encoder, the test set features are projected into a new space, and the newly obtained features are used as a multi-view representation. This representation is then used in the downstream classification task to report the average classification metric (F1_score) through 5-fold cross-validation using the SVC linear classifier.

Q5 What is the deep network used for DCCA?

Again, we apologize for not specifying the structure of the MLP in the paper.
The first linear layer is feature_dim * hidden_dim. All our MLPs use the Leaky ReLu activation function.
For the manually generated dataset, which is simple enough, we have only one hidden layer with a dimension of 256. For the real-world dataset, we have used MLPs with three hidden layers, and the dimension of the middle hidden layer is 1024.

Q6 What if

The questions you mentioned are very helpful in understanding CCA, and it's exactly what CCA methods are trying to solve: how to utilize features generated by different pre-trained foundational encoders. DCCA uses an MLP for each view, where the first linear layer dimension is feature_dim * hidden_dim, and the structure of the MLP behind the first layer is the same.
This way our MLP will project different features onto the same size of feature space.

Q7 The NESum

Your question is very helpful in helping us understand model collapse. We believe that the addition of DropOut to DCCA_private, which is a common regularization technique, does, to some extent, prevent redundancy among network weights (NESum rises), but it is clear that its effect is not stable for DCCA, especially as seen in (a), where the performance variance of different datasets is very large, which means DCCA_private is highly dependent on the dataset and is not generalizable.

We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.

Adding regularization has been fully explored in different machine learning scenarios. To this end, the proposed method is lack of research novelty, which may diminish the paper contribution.

In the machine learning area, adding regularization is a common setup. However, the exploration of new regularization approaches is always needed. To be specific,

Our motivation is new, we first time demonstrate the difference in behaviors between DCCA and Linear CCA, and our regularization forces DCCA to mimic the behavior of Linear CCA in order to take on the properties of CIP, thus mitigating the problem of model collapse of DCCA.

Our approach is new in that noise regularization previously had to rely on the Autoencoder architecture to implicitly regularize the network, but we now rely on (G)CCA loss to regularize the network from a different perspective using noise, which opens up the possibility of wider use of noise regularization.

The compared methods are relatively old, adding more recent publication for comparison helps to support the draft.

Thank you for your suggestion. We add two more methods: DCCA_EY and DCCA_GHA, the latest two DCCA-based methods for comparisons $2$ . Due to time constraints, we conducted experiments on synthetic datasets (mean and variance are from multiple datasets (different common rates)):

DCCA_EY and DCCA_GHA use an efficient algorithm for matrix eigenvalue decomposition in CCA loss that can be replaced by inter-matrix operations. This algorithm is fast and does not depend on a large batchsize (no gradient bias).
In terms of effectiveness, they are no better than DCCA in terms of generating feature quality (we all used large batchsize 2000). There is still model collapse, but they collapse slower than DCCA.

$1$ Hwang, HyeongJoo, et al. "Multi-view representation learning via total correlation objective." Advances in Neural Information Processing Systems 34 (2021): 12194-12207.

$2$ Chapman, James William Harvey, Ana Lawry Aguila, and Lennie Wells. "A Generalized EigenGame With Extensions to Deep Multiview Representation Learning."

$3$ Ke, Guanzhou, et al. "Rethinking Multi-view Representation Learning via Distilled Disentangling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Numerical results only contain some small datasets, using more large-scale ones will be helpful, especially in the current large-scale learning era.

Thanks to your suggestion, the largest dataset used for the current MVRL method is the PolyMnist dataset with 5 views, 60,000 images per view, and 10 categories.
However, it is really necessary to test MVRL in larger and more complex scenarios. To this end, we constructed 2 views of data in Cifar100 (100 categories, 60,000 images) by CLIP and BLIP pre-training image feature encoders respectively. Using the same PolyMnist experimental settings as in the paper, except that the embedding dimension was changed to 400, the 5-fold F1 score results are as follows:

We can see that although DCCA starts with better results than concat, it quickly collapses, while NR-DCCA shows stable performance. As to whether DCCA can be pre-trained for large-scale multimodal data like CLIP, this is an interesting question that we leave for future work.