Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization

We really appreciate your recognition and positive feedback of our work! The itemized responses to your questions are as follows:

One may argue why the approach has been casted solely for Multi-View Representation Learning, since it could have a broader scope, but this is not a minus.

Thanks for the thoughtful comments, and we believe this comment echoes Comment 3 by Reviewer 5U9E. The proposed Noise Regularization (NR) is indeed a way to regularize the neural network, and in general, it can be widely useful for different tasks.

The developed “full-rank” property of the neural network stems from the full-rank of the transformation W_k, so this paper mainly focuses on associating DCCA and CCA. Current theoretical derivations only apply to CCA, and it is still challenging to provide a good explanation of why such NR would work in other tasks. This also echoes Comment 3 by Reviewer w9Bv and Comments 2 and 3 by Reviewer w9Bv. We do think this is an interesting area worth further exploration, and we will leave it for future study.

In general I think that the paper can be squeezed a little, in order to host some of the experiments of the additional material. In particular, I found fascinating the experiment reported in Fig.12, about the the correlation between unrelated data. This is a further proof of the goodness of the idea. I would also report the tsne visualization in the main paper.

Thanks for your kind suggestions. We have squeezed the paper and merged the duplicate original Figures 2 and 5. We have also shown and analyzed the correlation of unrelated data in the revised section 5.2. We have mentioned the T-SNE results could also be evidence of model collapse in revised section 3.4 and pointed the readers to Appendix A.9 for details. We will further re-organize the paper to highlight the points the reviewer mentioned if it is accepted.

Overall, we sincerely appreciate your positive feedback and hope that our responses have sufficiently answered your questions. Should there be any queries or points of discussion, we stand ready to provide further clarifications.

We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.

The proposed method involves adding a regularization term to the loss function. The choice of the optimal value for the hyperparameter α in Equation 6 is crucial. If α is set too large, the model may tend to behave like an identity mapping function, potentially reducing its effectiveness. Conversely, if α is too small, the model may not maintain a "full-rank" property. The authors should provide guidance or suggestions on how to select an appropriate α value.

Thanks for the thoughtful comment. Yes, the choice of the hyperparameter α is essential in NR-DCCA. However, different from the conventional hyperparameter tuning, the determination of α is actually simpler, as we just need to search for the smallest α that can prevent the model collapse, and the model collapse can be directly observed on the validation data.

In Appendix A.6.2, we present performance curves of NR-DCCA in CUB under different α. if α is too large, the performance will converge slowly; if it is too small, the model collapse issue may remain. We increase the α adaptively until the model collapse issue is tackled, i.e., the correlation with noise will not increase or the performance of DCCA will not drop when the training progresses. The optimal α values of NR-DCCA for the CUB, PolyMnist, and Caltech datasets are found to be 1.5, 5, and 15, respectively. Additionally, one can see the NR-DCCA outperforms DCCA robustly with a wide range of α.

The reviewer is interested in whether there is any theoretical analysis of the generalization ability of the proposed method. An exploration of how NR-DCCA's performance might extend to new, unseen data or domains would add depth to the paper.

Thanks for the comment. The proposed Noise Regularization (NR) is indeed a way to regularize the neural network, and in general, it is widely acknowledged that regularization on neural networks is helpful for generalization ability [1,2].

The common regularization methods include two categories: explicit methods (i.e. regularizing the parameters, hidden features, or gradients in the neural networks) [1] and implicit methods (e.g., noise regularization) [2]. It is clear that the proposed NR falls in the second category, which lacks comprehensive studies in representation learning.

Actually, the implicit methods usually “regularize the behavior” of neural networks. We believe this is a new perspective and our numerical experiments also indicate its outperformance. However, it might be challenging to analyze what has been regularized in neural networks, and hence it would be difficult to provide theoretical analysis regarding the generalization ability, due to the intricate neural network structures. Here we actually observe a trade-off between interpretability and performance in regularization methods (explicit vs. implicit). Instead, what we did in the paper is to show the analogy with the classic CCA, and we let the neural network mimic the good behaviors of CCA, given the fact that CCA has good generalization ability.

We appreciate that the reviewer pointed out the right direction for potential future work. The primary objective of this paper lies in addressing the model collapse issue within DCCA, while we believe analyzing the detailed properties of such NR would be impactful.

[1] Huang, Lei, et al. "Normalization techniques in training dnns: Methodology, analysis and application."

[2] Poole, Ben, Jascha Sohl-Dickstein, and Surya Ganguli. "Analyzing noise in autoencoders and deep networks." arXiv preprint arXiv:1406.1831 (2014).

The paper showcases the performance improvement of the proposed method in downstream tasks using off-the-shelf methods like Support Vector Regression (SVR). However, the reviewer is curious about whether there are any significant performance differences when fine-tuning the system using downstream tasks. An analysis of how NR-DCCA performs in this scenario would be informative.

Thanks for the thoughtful comment. It is possible to also fine-tune the representation learning during the training of the downstream tasks (like SFT in LLM), while this approach deviates from the standard validation procedure of multi-view representation learning (MVRL).

In MVRL, the predominant validation procedure is first to generate representations, and then use the representation directly. To ensure a fair comparison with existing methods, we still follow this procedure.

Indeed, we believe that if we further fine-tune the representation for specific tasks, then the performance will definitely improve, but the representation will lose generality.

This actually resembles other supervised multi-view problems, such as multi-view classification [3]. Investigating how NR-DCCA performs under these problems might be interesting and it is left for future work.

Overall, we sincerely appreciate your insightful feedback and hope that our responses have sufficiently addressed any concerns raised. Should there be any queries or points of discussion, we stand ready to provide further clarifications.

[3] Han, Zongbo, et al. "Trusted multi-view classification." arXiv preprint arXiv:2102.02051 (2021).

The intuition behind the proposal is not super clear to me. If the goal is to have full-rank f(X), so that the covariances are better conditioned, the abovementioned covariance regularization approach already achieves the same effect. If I look at Proposition 4, it essentially says CCA is invariant to linear transformations of input; but this is well-known and easy to see from the original formulation, even without complicated linear algebra. The more interesting analysis would be to explain why the correlation between signal and random noise is a good quantity for deep neural networks to mimic; there must be more structure than saying that fk is full-rank in my opinion.

We apologize for the potential misunderstanding caused, and we have updated the paper to make it clearer (in Section 3.4). Actually, we define the “full-rank” property of f_k and it is different from the full-rank property of features f_k(X_k). In CCA, the two concepts are nearly the same, as features W_kX_k are forced to be full-rank by the loss function, and W_k is naturally full-rank (refer to Lemma 2 in our paper). However, it is difficult to define what is a “full-rank” property of neural networks since it is not a matirx. Because the current DCCA has no regularization on the neural networks, we believe such inconsistency is the potential reason for the model collapse in DCCA.

We agree with the reviewer that the Proposition looks straightforward. To be precise, the correlation with noise is invariant to full-rank linear transformation. However, it is the first time that the connection between the correlation with noise and the full-rank property of W_k is built. In terms of DCCA, as it is difficult to directly define the “full-rank” property of f_k, we first find the equivalent condition of the full-rank property of W_k in CCA, and then we can transplant this condition to DCCA. Specifically, because of Theorem 1 (Proposition 3 + Proposition 4), we can equivalently define the “full-rank” property of f_k based on the noise in Definition 2.

We do agree with that reviewer that more structure of the neural networks should be explored to further understand what has been regularized by our proposed Noise Regularization. However, we have to admit that interpreting the parameters in neural networks is challenging. Our way to tackle this problem is to look at the behavior of the neural networks. Through numerical experiments, we find that the CCA never collapses and the DCCA always collapses. Then we design the experiments more extremely to let both CCA and DCCA learn the correlation between two independent random noises. The results indicate that CCA can identify the invariant correlation between the noise, while DCCA cannot (we have moved related experiments into Section 5.2 ). We believe this is mainly because the powerful neural networks have “created” correlation itself. Therefore, we enforce the neural networks to preserve the correlation for random noise in DCCA, and this motivates our idea of noise regularization.

We call f_k “full-rank” to help the reader associate with the full-rank property of W_k, as the full-rank transformation matrix would not “create” correlation. But this name might cause the reviewer’s misunderstanding. We apologize and we have incorporated more discussions to elaborate on the “full-rank” property of neural networks in the updated paper to eliminate confusion (in Section 4.2).

We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.

Important references and discussions are missing. The fact that powerful neural networks could lead to degenerate feature space is known. In the original deep CCA paper [Andrew et al, 2013] there was already ridge regularization of the auto-covariance matrices (see their eqn 5) to avoid degeneracy and improve numerical stability. And that regularization technique was already studied for linear CCA...

Thanks for the thoughtful comment. The reviewer mentioned the ridge regularization, which is a widely used technique to regularize the features (W_kX_k in CCA and f_k(X_k) in DCCA using our notations), and it is a default setting across almost all (D)CCA methods. In our experiments, we have actually included ridge regularization during hyper-parameter tuning for all the methods with and without noise regularization.

However, the ridge regularization mainly regularizes the features (i.e., W_kX_k in CCA and f_k(X_k) in DCCA), rather than the transformations (i.e., W_k in CCA and f_k in DCCA). W_k in CCA does not need any extra regularization as it is automatically full-rank, while the f_k is still unconstrained and unregulated in DCCA. Therefore, our study mainly focuses on the regularization of the transformation in DCCA. That is why we did not compare with the ridge regularization in the paper.

The reviewer agreed that “powerful neural networks could lead to degenerate feature space”. However, the degenerated features produced by neural networks in DCCA can still be full-rank, and simply enforcing the covariance matrix to be full-rank by adding ridge regularization is not even helping, because the collapse of neural networks is intricate. Therefore, we develop the concept of the “full-rank” property of f_k (note this is an analogy of the full-rank property of W_k), and it is different from the full-rank of the features f_k(X_k). We apologize for the potential confusion raised by this definition.

Overall, we believe that ridge regularization is mainly for combating noise and improving numerical stability, it cannot prevent the neural networks from degenerating (i.e., model collapse). In contrast, we should directly study the “full-rank” property of the neural networks f_k and we believe the relevant studies are lacking.

To further support our arguments, we provide the experimental results with different ridge parameters on a real-world dataset CUB at 500-th epoch as follows:

One can see that the ridge regularization even damages the performance of DCCA. In our NR-DCCA, we actually set the ridge parameter to zero. We conjecture the reason is that the large ridge parameter could make the neural network even “lazier” to actively project the data into a better feature space, as the full-rank property of features and covariance matrix are already guaranteed, and this is also evidenced by the “Square sum of feature covariance” shown in the table.

We apologize that we should have included the above discussion in the paper, and we have now incorporated the above results in Appendix A.6.1 for clarification.

The paper is not purely about optimization and numerical stability. And I expect to see different inductive bias from the proposed regularization. There shall be investigation of the feature quality on real-world datasets, as shown by prior deep CCA-based papers.

Thanks for the comment. Indeed, our work is not about optimization and numerical stability. We focus on imposing the inductive bias on the neural networks for DCCA. The correlation with random noise is invariant to full-rank linear transformation, and hence we hope the correlation is also invariant to “full-rank” non-linear transformation. Hence we define such “full-rank” property as the inductive bias of the neural networks.

In terms of the investigation of the feature quality on real-world datasets, our evaluation of feature quality is mainly guided by visualization techniques (as demonstrated by TSNE in Appendix A.9 ). Additionally, the performances across a multitude of downstream tasks, encompassing both synthetic and real-world datasets, demonstrate the consistent outperformance of the proposed method. Moreover, guided by our theory, we reported the correlation between the outputs of unrelated data, which provides another way to observe feature quality and model collapse issues.

I hope to see non-trivial analysis regarding the effect of noise regularization.

Please check our response to your Comments 1 and 2.

I hope to see comparison of feature quality against alternatives.

Please check our response to Comment 3.

Overall, we sincerely appreciate your insightful feedback and hope that our responses have sufficiently addressed any concerns raised. Should there be any queries or points of discussion, we stand ready to provide further clarifications. In light of the revisions and discussions, we kindly invite you to reconsider your score on this paper.

Dear Reviewer w9Bv,

We are aware that after tomorrow, the opportunity for further discussion will no longer be available in this year's ICLR. We wonder if we have adequately addressed the concerns and clarified the misunderstanding you previously raised?

It is our earnest hope to continue our discussion, should there be any lingering questions or points needing clarification.

I appreciate the authors providing details on the tuning of ridge parameters for deep CCA. I am OK on that aspect.

I understand at the high level that the intuition is to not create non-existent or spurious correlation between the views, when we use powerful neural networks for parameterizing feature mappings. But still

• I feel the desired property is more than the "full-rank" property proposed in this paper, and this property has to be characterized more concretely (for example, similarly to how adding noise to data is shown to be equivalent to L2 regularization under assumptions). This is important since the authors claim their method to be advantageous over ridge regularized DCCA, which adds isotropic noise to data to ensure covariance is well-conditioned.
• It is not very clear how the additional loss on matching correlation obtain with DNNs with linear mappings achieves the desired property. In fact, because noise is independently generated from data, the true correlation between noise and data should be zero (with infinite data). Why not instead minimize correlation between data and noise for regularizerization?

I appreciate the authors providing details on the tuning of ridge parameters for deep CCA. I am OK on that aspect. I understand at the high level that the intuition is to not create non-existent or spurious correlation between the views, when we use powerful neural networks for parameterizing feature mappings.

We appreciate your feedback on our response and thoughtful comments. Great to know that the reviewer understands the key idea in this paper, and we also understand that the reviewer request a deeper theoretical understanding of the proposed noise regularization (NR) approach. We actually thought about this, but indeed the further exploration of such a “full-rank” property was non-trivial, because our NR approach is not simply adding noise to the input, and f_k can be any form of neural networks. However, we still believe our NR approach is quite new, and it achieves state-of-the-art performance consistently, so we would like to share this approach with the community.

I feel the desired property is more than the "full-rank" property proposed in this paper, and this property has to be characterized more concretely (for example, similarly to how adding noise to data is shown to be equivalent to L2 regularization under assumptions). This is important since the authors claim their method to be advantageous over ridge regularized DCCA, which adds isotropic noise to data to ensure covariance is well-conditioned.

Thanks a lot for the thoughtful comments! As the reviewer concluded, our desired property of neural networks is to “avoid creating non-existent correlation between views”, and this property is actually intricate because it is challenging to regularize the weights in neural networks to obtain this property. In contrast, this paper enforces the property directly, while it is indeed non-trivial to connect this property to the weights of neural networks.

Our NR approach differs from the standard NR approach, as we did not add noise to the data directly and the standard NR approach requires strong assumptions (e.g., autoencoder structures) for analysis, so it is challenging to obtain similar properties like L2 regularization in our approach. As we mentioned earlier, we believe our NR approach mainly regularizes the behavior of the neural network, and how the parameters are regularized actually depends on the structure of the neural network, which we do not restrict directly.

It might be possible to derive some analytical properties of our NR approach if we assume specific structures of the neural networks. For example, [1] assumes an autoencoder structure with a linear decoder, and [2] relies on the linear transformation in CCA. However, this paper aims to develop a generalized NR approach for deep CCA, so we did not touch on this aspect. We do think this will be a great direction for further exploration.

Overall, we still believe our NR approach is quite new, as it provides a new aspect to regularize the neural network, and its performance is consistently state-of-the-art, so we would like to share this finding with the community. Thanks again for your thoughtful comments and for pointing out the potential directions for further investigation!

[1] Poole, Ben, Jascha Sohl-Dickstein, and Surya Ganguli. "Analyzing noise in autoencoders and deep networks." arXiv preprint arXiv:1406.1831 (2014).

[2] De Bie, Tijl, and Bart De Moor. "On the regularization of canonical correlation analysis." Int. Sympos. ICA and BSS (2003): 785-790.