Sailing in high-dimensional spaces: Low-dimensional embeddings through angle preservation

The authors do not give the response for this period and further discussion is not available.

W1 Visual inspection of embedding visualizations "worse" than tSNE This is exactly the unconscious bias we are aiming to address, and that has been discussed in the literature before. tSNE and UMAP are delivering results that not properly reflect the original data. They are pretty to look at, because they produce clusters, but they do NOT produce necessarily meaningful clusters, but rather artifacts of the optimization. To go beyond this subjective view that evaluates on what looks pretty, we evaluated a set of complementary quantitative metrics on a substantial amount of different datasets. In examples such as the cell cycle data, it is also clear from a qualitative perspective that the embeddings are wrong, as they do not align with the data (in this case, it is a cyclic structure which tSNE or UMAP do not recover at all).

W2 Terminology Faithfulness “faithful” is used in its original semantic meaning, commonly used for example in XAI. We already give a meaning in line 39-40 in the introduction: “... LDEs have to be faithful to the original data: local structures should be perceivable, but also global relationships of these structures should be appropriately reflected”

W3 Complexity of method, We properly discuss this in the text, with both theoretical as well empirical results, that the subsampling procedure is efficient and does not harm performance, bringing the computation down massively (O(n) in landau notation, faster than tSNE), which the reviewer seems to have overlooked.

Q1 PCA and high frequency components We acknowledge that PCA may not be able to capture all the information contained in data. But for high-dimensional data, as we extensively argued in Section 3.4 backed up with literature from statistics, directly working on the noisy datasets without proper denoising may lead to severely biased outcomes. If the reviewer disagrees with that literature, we would be happy to discuss this in terms of science, with proper references backing up the made claims.

Q2 Supposed loss of information due to subsampling See comment for W3. We extensively discussed the subsampling and showed that both theoretically and empirically it does not lose much information.

Q3 Clarification of PCA initialization We assume you are talking about the PCA initialization, which we explain in line 226-227 and footnote 1 on page 5. In a quick summary: The optimization is slower around the poles when using gradient descent, as loss landscape is flat around them due to the way longitude and latitude are defined (involving cosines).

Q4 Where are definitions of performance metrics In Appendix C2 as also referenced in the main paper.

Q5 Additional reference Thank you for the suggested paper, we added it to our related work.

We thank the reviewer for their feedback and address the raised each individual concern below.

(1) Term low-dimensional embedding Most existing work on low-dimensional embeddings is indeed interested in visualization (including tSNE, UMAP, hMDS) and thus discusses the 2- or 3-dimensional case. In principle our method, as well as these other methods, also generalizes to higher-dimensional settings, where then we compute angles between geodesics on the hyper-sphere. While the nomenclature is standard in this literature, we see that it might lead to confusion to people from different fields (e.g., matrix factorization or similar), where low-dimensional embeddings or representation are typically of higher dimension. We add this as a discussion point.

(2) Split up related work We are happy to do that.

(3) PCA initialization Indeed, the PCA might yield a suboptimal embedding, since it is just a linear projection. But if it would already be good in general (especially for your manifold example), we would not need other methods such as tSNE, UMAP, or Mercat. In principle we do need something that is giving a rough ordering of the data relative to each other, so that we do not have too many repulsive forces at once that break the optimization at the start. We do still see lots of change during optimization away from the PCA initialization, and just found that it recovers a good embedding faster than starting with a random initialization, although both work. An ideal initialization would make for a great further study, but since PCA initialization yielded good results already, we decided to not further pursue this research path. We visualize the change away from the PCA initialization, also in response to your other question, in the updated Appendix Fig. 7.

(4) Further theoretical investigations Identifying the family of data structures where angle-preserving embeddings outperform distance-preserving ones is a theoretically fascinating yet highly challenging topic. We must leave that for future work and would be grateful for any references where this was studied in the context of other contemporary methods, such as tSNE or UMAP.

(5) Proof of Eq 4 We have added the proof of Equation 4 in the Appendix B.

(6) Discussion on used metrics Unfortunately we had to restrict ourselves in length due to the page limit, but indeed a brief discussion benefits the paper. We added this in the beginning of the Experiments section, thanks for the suggestion.

(7) 2D vs 3D embeddings Note that our algorithm embeds in a 2-dimensional space. A sphere, as a mathematical object, is 2-dimensional and can be described by two variables, longitude and latitude. Hence, a comparison to 3-dimensional embeddings would be unfair. Note that we also compare to the (2D) sphere embedding by UMAP (here called SphereMap) to give a direct comparison to an approach embedding on the same mathematical object.

(8) Divergence from initial PCA embedding Indeed, in the simple case of 2D input data, PCA is similar. In fact, by definition, PCA yields a perfect reconstruction of the original data in this simple case, as we map from 2D to 2D. So, us not diverging from the initial PCA in this particaular 2D case is good. A more interesting case is on real data with complex, high-dimensional relationships. We now provide a visualization of optimization over time of our method on single cell data in App. Fig. 7, starting with t=0 (the original PCA). As mentioned also in the answer to question (3), our embedding diverges significantly from the original PCA.

(9) Comparison to Laplacian Eigenmaps and Diffusion Maps To address these concerns, we added additional experiments comparing to Laplacian Eigenmaps and Diffusion maps on the low-dimensional, synthetic, and smaller real world data to which these algorithms could readily scale to. We updated the tables and corresponding Figures in the Appendix. Consistent with the literature, these approaches are outperformed by newer approaches such as tSNE, UMAP, or Mercat.

(10) Reported results on cluster data This must be a misunderstanding, Table 3 and Figure 8-10 in the Appendix of the original submission (Fig. 9-11 in updated draft) provided performance and visualization of both low-dimensional as well as cluster data. Please let us know if you still can’t see them as we would need to contact the AC to sort out whether there is an issue with the platform here.