$\gamma$-Orthogonalized Tensor Deflation: Towards Robust \& Interpretable Tensor Decomposition in the Presence of Correlated Components

We thank the reviewer for their comment. We address their concerns as follows

Outline. Indeed, it was a real challenge for us to present our work in such a way that outlines the motivation behind our approach, illustrates the main steps for developing the theory and constructing the algorithm, and at the same time extensively evaluate our claims empirically. In this spirit, we have substantially revisited our manuscript and list the main changes in a general comment.

Experiments We have further benchmarked our proposed algorithm against prevalent spectral tensor decomposition methods that are widely used in practice, CANDECOMP/PARAFAC (CP) and Tucker. We show that the latter collapse in the presence of higher levels of correlation and fail to recover the signal of interest. We outline the details of that in Figure 5 and its subsequent explanation in Section 4/Benchmarks. Regarding evaluation on real-world data, we defer that to subsequent work. Yet, we would like to outline that our model encompasses a rich class of problems, as outlined in the Introduction Section.

We invite the reviewer's feedback and suggestions, which we will gladly consider to further improve our paper.

We thank the reviewer very much for their valuable feedback. We address their questions and comments as follows:

Outline. We have significantly revisited our manuscript, as outlined in our general comment.

• The main algorithm is motivated and presented in details in Section 3.4, which follows a thorough and complete theoretical construction of our algorithm. We list the main assumptions and theorems, outline their value within our framework and visualize them with adequate simulations.
• (Seddik et al, 2021) lays the ground to a new angle to approach spectral tensor decomposition problems. Our paper follows the same path to construct our algorithm. While the reader is referred to check that paper for further details about random tensor theory based spectral tensor decomposition (proof of Theorem 1 in Section 3.2.1 of our paper for instance), our paper outlines the detailed construction of all our results in Section 3 of the main paper and provides a self-contained review of concepts of random matrix/tensor theory (RMT/RTT) in Appendix B to guide the unfamiliar reader.

Experiments. We update our experiments section, as outlined in our general comment. This further addresses the reviewer's specific questions. We list the relevant section again here for their convenience:

• We now focus our empirical evaluation on two representative experiments and provide detailed explanations of both. In particular, we outline how our algorithmic approach performs compared to state-of-the-art and how our developed theory matches our predictions in practice with extensive experiments. In addition, we have also added separate visualizations for our theorems and assumptions throughout Section 3 and provided a detailed set of supplementary experiments in Appendix E.

Application. We agree with the reviewer that the rank-2 order-3 model limits the breadth of applications of the considered model in general. However, this work only presents the rank-2 order-3 model for its accessibility and to show the potential of the considered analysis. For higher ranks/orders, a similar construction can be done (hints on that in Appendix G), which was previously developed for a similar model- Hotelling-type tensor deflation (See Theorem 3.4 in [1])

Questions.

• We thank the reviewer for their great question! We have further benchmarked our proposed algorithm against prevalent spectral tensor decomposition methods that are widely used in practice, CANDECOMP/PARAFAC (CP) and Tucker. We show that the latter collapse in the presence of higher levels of correlation $\alpha$ and fail to recover the signal of interest. We outline the details of that in Figure 5 and its subsequent explanation in Section 4/Benchmarks. Initially, we have not considered this benchmark highly relevant, as we only considered spectral decomposition algorithms.
• Our work, for now, only considers spectral decomposition algorithms given the rich theoretical basis in RMT/RTT. Tensor Train (TT) and Tucker are, however, compression-based decomposition algorithms.

We invite the reviewer's feedback and suggestions, which we will gladly consider to further improve our paper.

[1]: Seddik, M. E. A., Guillaud, M., Decurninge, A., & Goulart, J. H. D. M. (2023). On the Accuracy of Hotelling-Type Asymmetric Tensor Deflation: A Random Tensor Analysis. arXiv preprint arXiv:2310.18717.

We thank the reviewer very much for their positive feedback and valuable suggestions.

We answer the questions as follows:

• For the symmetric case where $u=v=w$, this indeed seems a simpler case to consider. However, when considering such assumption, we impose further constraints on the structure of the noise tensor $\mathcal{W}$. The noise entries $\mathcal{W}_{i,j,k}$ are not i.i.d anymore and further steps would be needed to estimate the variance given a certain entry.
• We thank the reviewer for bringing this point to our attention. Indeed, at the second deflation step of the $\gamma$-orthogonalized tensor deflation algorithm (in Appendix F, Algorithm 2), we are jointly optimizing $\gamma$ and the alignments. Furthermore, we empirically show that the different alignments are concave functions of $\gamma$ in Appendix E.4 and emphasize that in Section 3.4, which further explains the importance of our approach and provides theoretical guarantees on the quality of our recovery given $\gamma^*$.

We thank the reviewer very much for their valuable feedback and suggestions, which has opened many avenues for the improvement of our work. We have substantially revisited our paper and we list the most important changes as a general comment.

To address the reviewer's specific questions:

1. Extension to Higher-Rank and Higher-Order Problems. In addition to the Generalized Deflation Step Procedure in Section 3.1, we now motivate every step in our developed theory/algorithm in detail in the main text. We have considered the extension to higher ranks and/or orders in more detail in Appendix G, where we also list some of the key lemmas to producing our results. In addition, we point the reader to Theorem 3.4 in [1], where the extension follows a similar approach to ours and is done in full and in a high level of detail for the Hotelling-type deflation.
2. Experiments Section. We focused our experimental evaluation on 2 representative experiments and provide detailed explanations of both. In particular, we outline how our algorithmic approach performs compared to state-of-the-art and how our developed theory matches our predictions in practice with extensive experiments. In addition, we have also added separate visualizations for our theorems and assumptions throughout Section 3 and provided a detailed set of supplementary experiments in Appendix E.
3. Outline of the Paper. We substantially revisited our manuscript, as outlined as part of our general comment. The proposed algorithm is constructed and explained in a high level of detail in Section 3.
4. Experimental Setup. We provided more details of our experimental setup in Section 4 (Experimental Setup). In addition, an easy-to-follow code sample is provided in Appendix E.1 to show exactly how our data is generated. A code repository will also be provided to ensure reproducibility.

We welcome the reviewer's additional questions and/or suggestions!

[1]: Seddik, M. E. A., Guillaud, M., Decurninge, A., & Goulart, J. H. D. M. (2023). On the Accuracy of Hotelling-Type Asymmetric Tensor Deflation: A Random Tensor Analysis. arXiv preprint arXiv:2310.18717.