Metric Space Magnitude for Evaluating Unsupervised Representation Learning

Novelty of magnitude:

• It is worth noting that metric space magnitude has not been widely applied in neither mathematics nor machine learning. In the broadest context, existing applications of metric space magnitude are limited to a handful of papers tackling specific data analysis tasks such as biodiversity estimation (Meckes 2015), boundary detection (Bunch 2020), outlier detection (Bunch 2020), convex hull approximation (Fung 2019), edge detection in images (Adamer 2021) or neural network learning theory (Andreeva 2023).

• Although magnitude weights and functions are known in the mathematical literature, they have received little attention from both computational mathematicians and machine learning practitioners. While alternative geometric methods and multiscale summaries, such as persistent homology, are relatively well established tools appearing in hundreds of publications, applications of magnitude are considerably less explored.

• Even in terms of theoretical work, key results that would enable successful applications of metric space magnitude to real data do not yet exist. In particular, proofs on the stability of magnitude, its statistical properties or efficient approximation methods have not been developed.

---

Novelty of our contributions:

• We aim to address the aforementioned gaps in existing research by being the first to propose a distance between the magnitude weights and functions of different metric spaces and applying these for evaluating embedding quality, which is the first application of magnitude to unsupervised representation learning. However, these are not our only contributions.

• The novelty in our work also lies in speeding up magnitude computation, making it more accessible for larger datasets. This adresses the main bottleneck for wider adoption of magnitude in practice, which has been the computational complexity of inverting a matrix.

• In terms of technical difficulty, we agree that there is room for more theoretical advancements. Some straightforward corollaries in the paper (e.g. corollary 4.1 and 4.2) have been chosen to illustrate the impact of re-scaling a space and make the material more accessible and focus on motivating the proposed indicators.

• We would also like to note that the stability result introduced in Lemma 4.2 and 4.3 is novel. Even in theoretical literature, although showing robustness to noise is vital for data analysis tasks, no known stability guarantee exists. Indeed, proving the stability of magnitude is not a trivial conclusion.

---

Motivation and experiments:

• While algorithms’ original loss functions can also be used to evaluate their performance, these measures will of course be biased towards the models trained on them and do not always constitute suitable choices for secondary evaluation. The use of embedding quality measures extends beyond these as a guide to inform about the preservations of properties not explicitly involved in model training.

• Indeed, we have demonstrated cases where magnitude difference is superior to alternative embedding quality measures by capturing geometric information beyond, for example, distance or density preservation.

• In the case of visualization, for example, magnitude is aware of “zooming” and tracking the “effective number of points” corresponding to an intuitive understanding of what makes a good picture. Indeed, the experiments show that magnitude can help choose better visualizations. A particularly relevant application for visualisation is in single cell data analysis, where the use of embedding quality measures is important for understanding the amount of both local and global distortion introduced by dimensionality reduction.

---

Integrating magnitude into unsupervised learning:

• We are excited about the reviewer's interest in further applications of magnitude to unsupervised representation learning and the integration of magnitude into model training itself.

• Incorporating magnitude into the model’s loss is not a straightforward task and requires more theoretical advancements, which will be the subject of future work. However, this paper is meant as a first exploration of metric space magnitude as an indicator of embedding quality to first overcome some of the practical hurdles on computing magnitude on real data, and second, to present an investigation on why smaller magnitude differences correspond to better representations.

Statistical analysis:

• We agree that it would be desirable to add secondary statistical analysis to the main results reported in the paper. However, there are some open problems with regards to this analysis.

• Confidence intervals would be an obvious choice for estimating the statistical uncertainty of the reported magnitude differences. However, the true distribution of magnitude and magnitude differences is not known, prohibiting the use of parametric methods.

• In terms of non-parametric statistics, variations of bootstrapping methods are most commonly used for estimating confidence intervals from empirical distributions. As a core idea, bootstrapping relies on random sampling with replacement. Unfortunately, metric space magnitude does not allow for this sampling strategy as it cannot be computed from data with duplicate points (as then the similarity matrix will not be invertible and magnitude will not exist).

• Estimating uncertainty via subsampling could be an alternative, but is considerably less statistically sound as resamples would not be independent and identically distributed. Further, whether magnitude could be approximated via subsampling has not yet been investigated.

• Given these limitations, estimating the statistical uncertainty of magnitude remains a challenging task and will be the topic of further work.

• We further refrain from reporting p-values for the magnitude differences. This is motivated by well-known criticisms on the overuse of p-values and the fact that reporting statistical significance for embedding quality measures would require a suitable null model. In practice however, the ideal $d \ll D$ dimensional embedding is generally unknown and it is not at all obvious how to fulfill the null hypothesis.

Main claim:

• The main claim of our work is indeed that a good representation is one that better resembles the geometric structure of the original higher dimensional space by faithfully preserving magnitude in lower dimensions and achieving smaller magnitude differences.

• Motivated by this, the use of magnitude to indicate embedding quality is not limited solely to isometric embeddings. Indeed, we see magnitude as a tool for describing the preservation of geometric structure more generally.

• While it is hard to uniformly define what makes a good representation for any application, different embedding quality measures exist with different purposes after all - magnitude difference offers a novel take on the importance of geometric information for embedding quality.

• It is superior to alternative quality measures when preserving both the local and global structure of the data is important (Swiss Roll), when preserving the effective number of clusters across scales is relevant (MNIST/PBMC) or for the purpose of assessing lower dimensional visualizations.

---

Novelty:

• This work is the first to:

• apply magnitude to the task unsupervised dimensionality reduction,
• define differences between magnitude weights and functions of two different metric spaces,
• link these differences to embedding quality,
• evaluate the performance of these indicators experimentally,
• improve the computational efficiency of calculating magnitude weights and functions using Cholesky inversion,
• automate magnitude computation for any given dataset by automating the choice of scale parameters,
• prove the theoretical stability of magnitude,
• and evaluate the empirical robustness of the proposed magnitude difference experimentally.

• Indeed, both the application of magnitude to unsupervised dimensionality reduction as well as the definition of a difference between magnitude functions/weights and the proposed framework to systematically compare magnitude across different metric spaces are novel. Altogether the experimental and theoretical work presented in Sections 4 and 5 are new results. In this light, we will clarify our contribution in the introduction.

---

Readability:

• We will clarify the definition of the proposed difference between re-scaled magnitude functions, which we term magnitude profile difference and clarify this both in Def 4.2 and Figure 2.

• While both Def 4.2. and Def 4.3. assume that magnitude has been evaluated across the same scaling factors, we will highlight that the magnitude profile difference (4.2.) could also be calculated for unaligned spaces while magnitude weight difference (4.3.) requires aligned observations.

---

Scale-invariance:

• Scale-invariance refers to the invariance to re-scaling the distances in a metric space by a scalar as introduced in Corollary 4.2.. That is, re-scaling distances does not impact how well an embedding is perceived.

• For the re-scaling introduced in Def 4.1 notice that the variable $s\in[0,1]$ is introduced and the re-parameterization is given by $Mag_X(s) = Mag_X(s*t_{conv})$ so that $Mag_X(s)\rvert_{s=1} = Mag_X(t_{conv})$. This then allows for comparing magnitude functions across comparable scaling parameters $s\in[0,1]$.

• For $s \in [0, 1]$, $s/t_{conv}$ would indeed be incorrect as $Mag_X(s/t_{conv})\rvert_{s=1} \neq Mag_X(t_{conv})$.

---

Magnitude as a multi-scale summary: • The benefit we wanted to highlight is that by numerically approximating the area between the magnitude curves, we get a holistic summary of the dissimilarity between spaces across different scales of the distances. We argue that this gives a more complete summary of the space and removes the need to manually choose a fixed scale parameter / neighborhood size.

• One could also get multi-scale summaries by comparing different filtration curves - this idea is not exclusive to magnitude. It is the same concept as comparing other multi-scale summaries across multiple choices of hyperparameters (e.g. neighborhood sizes or scaling parameters).

• We agree how the definition of scale-invariance vs. multi-scale summary can be confusing and will clarify this in the text.

---

Magnitude difference under affine transformations:

• The proposed framework has been formulated in order to be able to ignore the impact of re-scaling the distances via representation learning. As such, the proposed magnitude differences are invariant under certain affine transformations such as uniform scaling, dilation, rotation, translation and reflection.

• This already allows for ignoring expansion and contraction automatically when comparing two metric spaces and this property is referred to as scale-invariance in the text.

• Regarding other affine transformations, linear scaling in only one direction, if too extreme, will not be an ideal representation and not exactly preserve magnitude. Similarly, shear mapping could introduce unwanted distortion.

Computational complexity:

We agree that an extensive computational benchmark is desirable and will therefore add this comparison to the manuscript.

The main contribution of Bunch et. al. towards computing magnitude is the development of an inclusion / exclusion principle for magnitude. This can speed up computations in settings where $X \subset Y \subset \mathbb{R}^n$, $Mag_X$ is already known and one wants to use this knowledge to more efficiently compute $Mag_Y$.

However, they do speed up the step of inverting the similarity matrix and we lack reason to believe their implementation speeds up the calculation of magnitude in general.

Moreover, they do not compute the magnitude function, but the magnitude at a specified value of t, so they also do not use the full expressive power of the magnitude function and their results depend on the choice of scaling parameter.

---

Magnitude as a multi-scale summary:

For answering “Is it an effect of the introduction of Magnitude that it can both eliminate the need for hyperparameter design and obtain a complete summary of the space?”, I am afraid I might still not understand the purpose of the question. Yes, magnitude functions and weights are comprehensive summaries of a space that do not require setting fixed scale parameters or tuning hyperparameters. This is a beneficial property of magnitude functions, which are natural extensions of the single-scale version of magnitude. It is also true that there exist other summaries that fulfill the same multi-scale property, but do not use magnitude.

---

Scale-invariance and magnitude under affine transformations:

The main proposal in Def 4.2 defines a difference not between the original magnitude functions but between their re-scaled i.e. re-parametrised versions. Similarly, Def 4.3. uses the difference between the re-scaled weight vectors. We do not propose nor use a difference between the non-rescaled magnitude functions or weights in any of the experiments. In response to this feedback, we will highlight and clarify this distinction in the manuscript.

We can also show the magnitude profile distance is invariant to linear re-scaling of the distances. If we take a metric space $X=(X,d)$ and its scaled version $Y=cX=(X, c \cdot d)$ for some scalar $c \in \mathbb{R}$. Say Mag_X and Mag_Y are their magnitude functions and call their re-scaled magnitude functions $M_X$ and $M_Y$. We get $\Delta Mag_{XY}= \int_S |M_X(s)-M_Y(s)| ds = \int_S |Mag_X(s \cdot x_{conv})-Mag_Y(s \cdot y_{conv})| ds = 0$

---

Novelty:

Studying unsupervised dimensionality reduction and the visualization or interpretation of learned representations in general, falls within the scope of representation learning and the relevant topics set out by ICLR. The introduction of magnitude to evaluate embedding quality is not merely a side though, but the main purpose of this work and the proposed indicators.

---

Main claim:

We choose to study representations via magnitude, because it makes sense to do so. Just like preserving other geometric properties, such as distances, density, curvature, local neighborhoods, clustering patterns or intrinsic dimensionality are obviously important, preserving magnitude is a logical extension of these aims.

In particular, the goal of representation learning that was explored in our experiment is learning lower-dimensional for the task of visualisation, which is an open problem in particular when analysing single cell data. Throughout the experiments, we introduced magnitude difference as a qualitative measure for assessing 2D representations as well as used magnitude to describe the information loss introduced by projecting onto lower dimensional PCs.

Given any potential gaps in our analysis, we are of course keen to try out suggestions on extended experiments that could give a more extensive picture of the expressiveness of preserving magnitude for representation learning. Nevertheless, for the purpose of preserving geometric structures important for visualisation, magnitude difference fulfills this original goal of indicating better representations across the experiments we have considered.

While the isometricity of expressive learning wasn’t our main focus - we agree that it shows interesting parallels to the isometry invariance of magnitude. For our proposed magnitude differences, this invariance can easily be extended to invariance under specific types of quasi-isometries given by maps which multiply all distances by the same positive scalar i.e. $\Delta Mag_{XY}=0$ for $Y$ isometric to $cX$ for any $c \in \mathbb{R}_+$. Further, it is also of interest to study how this fits in with other notions of isometricity used in expressive learning, such as nonlinear scaled isometric embeddings.