Identifying Metric Structures of Deep Latent Variable Models

We thank the reviewer for thoughtful feedback, as well as their support for acceptance.

However, I find that the experiments do not strongly validate the hypothesis that geodesic distances are truly identifiable. The main result, particularly Figure 7

We emphasize that the main result of the paper is Theorem 4.5. It shows that under any (injective) generative model, the indeterminacy transformations of the true latent space will automatically respect the true geometry of the data. I.e. it proves that any geometrical information extracted from that model is identifiable. We achieve this without placing (notable) restrictions on the model, architecture, or training methods.

We motivate our paper and focus our experiments within the point of view of geodesic distances which is an example of geometrical information used in practice. This leads to Theorem 4.7 proving that these are identifiable.

there is considerable overlap between the two histograms. [..] I would argue that this falls short of demonstrating that they are identifiable in a definitive sense

We want to clarify that identifiability is a theoretical question and that our present theorems are the definite evidence.

The experiments demonstrate that the asymptotic property in Theorem 4.7 is practically achievable in standard models using off-the-shelf methods on finite data.

However, we understand the concern about overlapping histograms, but emphasize that such is to be expected and is fully in line with the theory:

• Proposition 4.8 shows that a (scaled) Euclidean distance can be identifiable if the model behaves in a Euclidean way in a region.
• Riemannian geometry is locally Euclidean, implying that local Euclidean distances can be expected to be robust when points are close. Consequently, neighboring points can have robust Euclidean distances (low CoV), implying overlapping histograms. Finite data further implies uncertainty compared to theoretical treatment and makes the estimation of the manifold stochastic. As the reviewer points out, optimization of geodesics is noisy and may lead to a distorted picture (no efforts were made to counteract this on specific datasets or models).

While the focus of our work is theoretical, we acknowledge the reviewers’ requests for additional experiments, and have added results for FMNIST and CIFAR10. See link for plots and tables. FMNIST results are similar to previous results, while CIFAR10 shows the clear separation of histograms that you have requested.

there is a strong link between this work and the literature on lie group latent spaces [5-8]

We do not claim to be the first to explore transformations of latent space; however, we are the first to apply them in the context of identifiability using Riemannian geometry.

While previously used for different purposes, latent space transformations have proven valuable in various areas, including disentangled and equivariant learning, highlighting their mathematical and conceptual connections. The mentioned literature assumes a disentangled latent space where transformations decompose into individual factors of variation and the goal is to find representations that respect this structure (roughly speaking, equivariance of $A_{a,b}$ (Def 4.2)). Instead of enforcing specific properties, our theory analyzes the natural properties of $A_{a,b}$, and we find that just by learning a generative model, $A_{a,b}$ will automatically respect the latent Riemannian geometry (Theorem 4.5). Thus, our theory is valid regardless of whether a disentangled latent space exists in the sense of [5].

previous works on computing and analyzing geodesics must be cited

We already cite [4] and will include [1-3] as appropriate.

The claim "[...] most data is equipped with units of measurement[...]." does not hold in most vision and nlp tasks

We acknowledge that there are cases where picking a suitable metric in data space is not trivial, but emphasize that this metric should only be meaningful infinitesimally. E.g. Euclidean distances are generally unsuited for images, but infinitesimally they are perfectly reasonable. Our theory also applies when pulling back ‘perceptual distances’, e.g. using features from pre-trained neural networks, see e.g. this paper.

using Euclidian distance in x-space, we implicitly assume that the dataset is dense

We disagree with this statement as we are not measuring 'isomap-style' geodesics. We measure geodesics along the manifold spanned by the model which does not require dense data to be identifiable.

The main weaknesses of the paper are: (1) Not discussing the connection between this work and all the disentanglement work (2) The results sadly are not very promising.

To conclude:

1. see the discussion above,
2. our main results are theoretical, and the experiments are fully consistent with the theory. Our released code provides a path for turning theory into practice.

We sincerely appreciate the reviewer's valuable feedback and support for acceptance.

I believe there can be more benchmarks included in this paper. Other commonly used image datasets, such as Fashion-MNIST, SVHN, Cifar-10, should also be computationally cheap to run.

While the focus of our work is theoretical (see reply to reviewer ev8z for more details), we acknowledge the reviewers’ requests for additional experiments.

In particular, addressing this review we run experiments of FMNIST and CIFAR10. Following the approach in the main paper, we split them into an experiment satisfying the injectivity constraint (CIFAR10) and an experiment that does not satisfy the constraint (FMNIST). The histograms and an updated table are at this link and we provide the table here as well.

Table: One-sided Student's t-test for the variability of geodesic versus Euclidean distances.

The findings are similar to those reported in the submitted paper, which supports the presented theory.

We will further extend the paper with an appendix including extra results and details on implementations and model choices.

1. Why do the authors only use 3 classes of MNIST? Is this cherry-picked?

As mentioned in the submitted paper, the choice of 3 classes for MNIST was to simplify plotting. We point to the extra experiments above and code in the submitted supplementary material (CelebA) to document absence of any cherry-picking.

2. I don't see why we should expect that the digit class 0, 5, and 7 are naturally close to each other. I hope the authors can explain it in more details.

We do not expect any classes to be naturally close to each other. The digits 0,5 and 7 were picked randomly and their placement in Fig. 5 is merely a consequence of optimization of the model fitting.

We hope to have addressed your concerns and sincerely thank you again for your review.

We are grateful to the reviewer for valuable feedback and favoring acceptance.

There is scope to be much more comprehensive in the experiments - for e.g. I would be very glad to see a table similar to Figure 7 for the transcriptomic data example from Figure 1 and that would solidify the main message of the paper across different models and types of data.

While the focus of our work is theoretical (see reply to reviewer ev8z for more details), we acknowledge the reviewers’ requests for additional experiments and will extend the paper with an Appendix including extra results and details on implementations and model choices.

In particular, addressing this point we run experiments of FMNIST and CIFAR10. Following the approach in the main paper, we split them into an experiment satisfying the injectivity constraint (CIFAR10) and an experiment that does not satisfy the constraint (FMNIST). The histograms and an updated table are available at this link. The findings are similar to those reported in the submitted paper, which supports the presented theory. Furthermore, we share the sentiment that the transcriptomic data example is underexplored and plan to add a distance matrix for the geodesic distances to Figure 1.

I am tempted to make the conclusion that generative models when trained well, tend to produce representations that preserve "intrinsic" distances on the data-manifold.

Indeed, the main result of the paper, Theorem 4.5, shows that under any well-trained (injective) generative model, the indeterminacy transformations of the true latent space will automatically respect the true geometry of the data. I.e. it proves that any geometrical information extracted from the model is identifiable.

For example - If I leave out the decoder 'f' completely and simply use my training dataset with a k-nearest neighbor graph and then compute a Dijkstra-like shortest path distance on this graph - I suspect it would correlate quite well with the construction of the pullback metric and computation of the geodesic distance from (35) which very much depends on the chosen model 'f'.

We appreciate and share your intuition. This paper considers geodesics under such an approach. However, we should emphasize that this is heuristic and is not strictly tied to our theoretical results on identifiability. Practically, we expect the approach to work well, though.

It would be nice to have some experiments where the variability in the geodesic is also visualized somehow (like Fig 5 but with a band instead of just one curve) - especially in comparison to this model-independent geodesic distance

We agree that different optimisations can lead to different geodesics between the same two points, and acknowledging that a geodesic in itself is not unique. However, in this paper we address the variability of the distance measure across retraining of the models themselves. In our experiments, this means that we compute the geodesic distance between the same two points across 30 different models with 30 different latent spaces. Fig.5 shows just one geodesic in one latent space. Therefore plotting the band would be concerned with a different kind of variability.

Is Table 1 reporting the values for the geodesic or Euclidean distance? I am unable to parse the message here

Building on the above, for each pair of points we have 30 measurements of distances according to both Euclidean and geodesic measures. To demonstrate that the geodesic distance is more stable, we compute the coefficient of variation (CV) of both distance measures for each point pair. These are then plotted in Fig. 7 and Table 1 reports the Student’s t-test for the CVs with one sided null hypothesis that geodesics are more stable, i.e. has a smaller CV. The message of Table 1 is that geodesic distances exhibit significantly less variation than Euclidean distances.

We acknowledge that the table caption should be improved. We will make these details more explicit and will further add an appendix detailing the experiments.

Is the familiarity in the trajectories reported in Figure 6 always the case? I was expecting that the trajectory of the Euclidean geodesic to give images with comparatively more "abrupt" changes in each step in comparison to the geodesic which should give a more seamless transformation. It is somewhat visible already - although not strikingly.

Your expectation is correct. We often see that geodesics provide more ‘smooth’ interpolations while Euclidean interpolations are more ‘abrupt’. This happens because geodesics move with constant speed in data space. This trend is, however, more evident, e.g., in MNIST than in CelebA. We’re happy to include such MNIST examples in the appendix if they are deemed interesting.

We hope that our clarifications and additional experiments strengthen your view of the paper, and we thank the reviewer for the thoughtful feedback.