A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data

We thank the reviewer for their overall positive reception of our work and insightful comments. We are pleased to note that you found the asymptotic error bounds for the estimated WAE to be clearly presented with thorough proofs. We agree that these bounds are crucial in guaranteeing the asymptotical convergence of the WAE model, provided the assumptions about the target distribution and true encoder/decoder regularization hold.

We appreciate the constructive criticism and insightful comments provided by the reviewer. We agree with the reviewer that assumption A1 is quite strong as highlighted in the paragraph following A1 on page 6 of the manuscript. Certainly, in the context of GANs, a less stringent assumption can be employed since the learning task becomes considerably simpler. A parallel challenge arose in the analysis of autoencoders, as elucidated by Liu et al. (2023). They addressed this issue by exploring chart-autoencoders, which incorporate additional components in the network architecture compared to conventional autoencoders. As already observed by the reviewer, the empirical evidence presented in Section 2 illustrates a polynomial-like decay pattern as hinted at by Theorem 8 and its corollaries. Experiments on different other classes of Imagenet suggest a similar trend showing roughly a polynomial decay. In response to the reviewer's recommendation, we will broaden the experimental study to encompass additional datasets.

Though the main theorem is not directly applicable to GAN-WAE, one can think of the case when the dissimilarity measure is taken to be the Wasserstein-1 distance to be a close replica of GAN-WAE. This is because when the discriminator is large, one can expect that after weight-clipping, any Lipschitz function can be closely realized by them. Thus, when optimizing on the discriminator, one can assume that the supremum is roughly attained (as the discriminator is assumed to be a large network). However, when working with a fixed class of discriminators, one can possibly prove analogous theorems in the so-called neural network distance. Exploring this avenue further would be an intriguing direction for future study. We appreciate the reviewer for raising this thought-provoking question and will incorporate this insightful discussion into the revised manuscript.

I thank the authors for their kind replies. My questions are all responded, and I will retain my scores.

We thank the reviewer for the encouraging remarks and insightful comments. We are glad to note your positive evaluation of the paper's quality of writing, and we appreciate your favorable comments regarding its robustness, clarity, and meaningful contributions.

We agree with the reviewer that the proof techniques presented in this paper are largely built upon the works done in the recent literature. However, as already noted by the reviewer, applying the proof technique, particularly in the MMD context requires additional intricacies and has to be carefully dealt with. Indeed as already pointed out on page 6, A1 is a strong assumption. Certainly, in the context of GANs, a less stringent assumption can be employed since the learning task becomes considerably simpler. As observed by the reviewer, a parallel challenge arose in the analysis of autoencoders, as elucidated by Liu et al. (2023). They addressed this issue by exploring chart-autoencoders, which incorporate additional components in the network architecture compared to conventional autoencoders. We thank the reviewer for the suggestions. Indeed compact $\tilde{d}$-dimensional differentiable manifolds have a Minkowski dimension of at most $\tilde{d}$. Thus, the main result, i.e Theorem 8 and the subsequent hold with $d_\mu$ replaced with $\tilde{d}$, when the data-support is a $\tilde{d}$-dimensional differentiable manifold. A similar result holds for other examples such as affine convex sets, self-similar sets, etc. as highlighted in Proposition 9 of Weed and Bach (2019). We thank the reviewer for stimulating discussion and will include the following discussion in the revised manuscript.

We recall that we call a set $\mathcal{M}$ is $\tilde{d}$-regular w.r.t. the $\tilde{d}$-dimensional Hausdorff measure $\mathbb{H}^{\tilde{d}}$ if $\mathbb{H}(B_\varrho(x, r)) \asymp r^{\tilde{d}},$ for all $x \in \mathcal{M}$ (see Definition 6 of Weed and Bach (2019)). It is known (Mattila, 1999) that if $\mathcal{M}$ is $\tilde{d}$-regular, then the Minkowski dimension of $\mathcal{M}$ is $\tilde{d}$. Thus, when $\text{supp}(\mu)$ is $\tilde{d}$-regular, $d_\mu = \tilde{d}$. Since compact $\tilde{d}$-dimensional differentiable manifolds are $\tilde{d}$-regular (Proposition 9 of Weed and Bach (2019)), this implies that for when $\operatorname{supp}(\mu)$ is a compact differentiable $\tilde{d}$-dimensional manifold, the error rates for the sample estimates scale as in Theorem 8, with $d_\mu$ replaced with $\tilde{d}$. A similar result holds when $\text{supp}(\mu)$ is a nonempty, compact convex set spanned by an affine space of dimension $\tilde{d}$; the relative boundary of a nonempty, compact convex set of dimension $\tilde{d} + 1$; or a self-similar set with similarity dimension $\tilde{d}$.

We thank the reviewer for the overall positive feedback of our work. We appreciate the time and effort you invested in reviewing our work. We are pleased to see that you find the paper well written, and we are grateful for the positive remarks on its soundness, presentation, and contribution.

We acknowledge your insightful comments regarding the strengths and weaknesses of our paper. Your positive evaluation of the originality of our ideas and the importance of statistically analyzing Wasserstein Autoencoder (WAE) is encouraging. We also appreciate your recognition of the thorough discussion on error and convergence, as well as the comprehensive consideration of various factors such as sample size, neural network size, optimization oracle, and dimensionality dependence.

Regarding the weaknesses you pointed out, we agree that the single question of sample complexity is interesting on its own. The convergence of $\hat{G}$ to $G$ is guaranteed in Theorem 15, under additional assumptions of uniform equicontinuity. For uniformly Lipshitz generators, the convergence rate is upper bounded by the rates derived in Theorem 8, as stated in Remark 16. Similarly, Proposition 13 deals with $id - G \circ E$. We agree with the reviewer that a more focused discussion on decoding (Proposition 13) and data generation guarantee (Theorem 15) will benefit the readability of the paper. In the final version of the manuscript, we will formally state Remark 16 as a corollary for further clarification and streamline Section 5.5 to predominantly cover these two aspects. In response to your thoughtful feedback, we will omit Remark 11 as it diverges from the primary theme of the paper.

We thank the reviewer for their insightful comments and are glad to hear feedback that the work is significant, with an insightful theoretical contribution extending previous work, and has a potential impact on understanding the intricacies of WAEs. The theoretical results, particularly Theorem 8, are indeed crucial in providing a deep understanding of encoding and decoding guarantees, emphasizing the connection between the encoded distribution and the target latent distribution. We are pleased that you find value in our insights into encoding and decoding guarantees, as well as the effectiveness of the generator in accurately mapping encoded points back to their original positions.

We value your constructive critique of our paper. The empirical evidence presented in Section 2 illustrates a polynomial-like decay pattern as hinted at by Theorem 8 and its corollaries. It is important to highlight that such empirical validation is notably absent in the existing literature on GANs or WAEs including the very recent works of Chakrabarty and Das (2021), Dahal et al. (2022), and Huang et al. (2022). The primary contribution of our paper lies in comprehending how the convergence rates of sample estimates rely solely on the intrinsic dimension of the data. In this regard, the experiment in Section 2 demonstrates that the error rate of sample estimates decays at a rate depending on the Minkowski dimension of the data. In this regard, Theorem 8 provides a recommendation for the appropriate network size based on the number of samples and the underlying distribution. However, practically, the population quantities, as assumed in A1, remain unknown, making it challenging to ascertain whether the derived rates are optimal in a minimax sense. In response to the reviewer's recommendation, we will broaden the experimental study to encompass additional datasets to show that a similar trend as shown in Fig. 1, holds across multiple datasets. This expansion will enrich the empirical validation and strengthen the practical implications of our findings. To maintain conciseness, we intend to include this expanded analysis in the supplement for space efficiency.