A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models

We thank the reviewer for their insightful and constructive feedback, and we sincerely appreciate their recognition of the strengths in our likelihood‐based framework and its theoretical underpinnings.

Claims and Evidence: We thank the Reviewer for highlighting these key aspects of our theoretical analysis, the primary focus of our work. The simulations and MNIST experiments are designed to validate our statistical insights rather than introduce a new algorithm. Moreover, our framework is broad and equally applicable to other likelihood‐based methods—such as Normalizing Flows—as demonstrated in related empirical works [A] and [B].

Our theory indicates that a small noise perturbation is needed to smooth nearly singular data distributions. This is demonstrated with our synthetic manifold example and further supported by additional simulations (please see end of Rebuttal 1) which show how noise levels affect the MNIST W1 distance. In theory, this noise level is guided by the data distribution properties, and practically often chosen via cross validation as explored in [I], balancing minimal distortion with stable optimization

Theoretical Claims: While it is possible to derive the exact constants in our proofs, doing so can be cumbersome and distract from the main insights. Nevertheless, we will make every effort to track the constants that are tied to key problem characteristics. Our focus remains on clearly illustrating how these constants depend on critical variables—such as the intrinsic dimension and smoothness parameters—while remaining independent of the sample size n. As is common in the literature (e.g., [J, K]), tight upper bounds suffice to capture convergence behavior without specifying exact numerical values.

We consider our assumptions to be reasonably mild. In particular:

• Assumption 1 guarantees a density exists for both noisy and noiseless data on the manifold—which is widely acceptable and foundational in statistics and machine learning.
• Assumptions 2 and 3 are practically motivated. For example, [C] provides empirical evidence for lower-dimensional data structures, and Assumption 2 generalizes common structures like multiplicative and additive models [M].

Experimental Designs Or Analyses:

• Our experiments with synthetic data and MNIST illustrate and validate the key theoretical insights of our approach. While further empirical validation on diverse real-world datasets and architectures would better demonstrate robustness and practical utility (see [A, B]), a comprehensive evaluation is beyond our current scope. We expect these results to generalize to more complex image datasets, as benchmark data typically have intrinsic dimensionality on the order of 10¹ [L].
• We have added quantitative analysis on MNIST to assess how the W1 distance varies with different noise levels (please see at the end of Rebuttal 1). Additionally, we evaluated sample size effects by comparing performance at 25%, 50%, and 75% of the data. In full-dimensional cases (FD1, FD2, FD3) where smoothness is critical, performance improves significantly with increased sample size. Although our current experiments show minimal variations with network depth and sparsity, we acknowledge their importance and plan to explore these further.
• Since our framework is built on Wasserstein and Hellinger distances, our analysis focuses on these metrics. Nonetheless, exploring additional evaluations—such as uncertainty quantification and further baseline comparisons—would provide valuable insights. We view this as a promising direction for future work aimed at supporting optimal theoretical rates for these additional approaches.

Essential References Not Discussed: We shall include Han, Zheng, and Zhou (2022) in our references. While CARD employs a score‐based, empirically focused approach, our work establishes nearly tight convergence rates for likelihood‐based methods (including VAEs and normalizing flows) in conditional density estimation, offering new insights into overcoming the curse of dimensionality.

Other Strengths And Weaknesses: We have improved the resolution and clarity of Figures 2 and 3, ensuring clear fonts and shapes. These enhancements will be in the final manuscript, subject to permission.

We hope these explanations, along with the additional quantitative analyses, address your concerns and further strengthen our manuscript. We welcome any additional questions or further discussions.

References: Please see at the end of Rebuttal 2.

We sincerely thank the reviewer for their thoughtful and constructive feedback. While our theoretical contributions are inherently technical, we have made every effort to present them in a manner accessible to a broader audience. Below, we address each of the reviewer’s points in detail:

Claims And Evidence: Our work establishes nearly tight convergence rates for conditional density estimation on low-dimensional manifolds by analyzing deep generative models. The proofs throughout the paper follow a unified strategy that addresses two primary sources of error. First, we bound the statistical (estimation) error using entropy bounds (Eq. 10 and 11). Second, we quantify the approximation error—the bias incurred when approximating the true generator with our network class. Theorem 1 provides a general convergence rate, while Corollary 1 specializes the result to sparse and fully connected architectures, directly linking the rate to network parameters (depth, width, and sparsity). This adaptive behavior lets the estimator depend primarily on the intrinsic dimension and smoothness of the true model, mitigating the curse of dimensionality.

For nearly singular distributions, where data lie extremely close to a low-dimensional manifold—Corollary 2 demonstrates that adding a small noise perturbation smoothens the gaps. This noise injection smooths the data distribution and enables a favorable Wasserstein convergence rate, thus extending the ideas from Theorem 2. In essence, the noise helps overcome challenges inherent in estimating distributions that do not have full-dimensional support.

Extension to More General Settings: Theorem 3 broadens our analysis to generators with differing smoothness across dimensions. By combining our new approximation result (Theorem 5) with modified entropy bounds, we show that the overall convergence rate depends on a combined measure of smoothness and intrinsic dimension, reinforcing the robustness and adaptability of our framework

Methods And Evaluation Criteria: We acknowledge the reviewer’s concerns regarding the empirical evaluation. Our focus is primarily theoretical; the experiments were chosen to validate the statistical insights of our approach. Although MNIST is a high-dimensional dataset (D = 784), it naturally exhibits a low intrinsic dimension [C]. The presented simulations and MNIST experiments illustrate key trends predicted by our theory. Moreover, the framework we develop is general and applies equally to other likelihood-based methods, such as Normalizing Flows, as highlighted in related works [A] and [B].

Questions For Authors:

• While our theoretical analysis is carried out for sparse and fully connected feedforward networks, our tools are sufficiently general to extend to more advanced architectures—such as convolutional networks, GANs, and diffusion models. Our approximation-theoretic tools (e.g., Theorem 5) are potentially applicable to these architectures [F, G]. Future work will aim to incorporate these more complex models, further bridging the gap between theory and state-of-the-art empirical performance.
• Additional quantitative MNIST experiment results have been added (please see end of Rebuttal 1).
• For more complex image datasets, we expect these results to generalize as empirical evidence suggests that benchmark datasets typically exhibit intrinsic dimensionality on the order of $10^1$ [C].

Essential References and Other Weaknesses: We appreciate the reviewer’s suggestions. In the revised manuscript, we will include the recommended reference (arxiv:2003.13913). Regarding notations, most are standard in the literature (e.g., [F, G, H]).

We hope that these clarifications, together with the additional analyses, address your concerns and further strengthen our manuscript. Should you have any further questions or require additional details, we are happy to discuss them further.

All Rebuttal References

• [A] Conditional variational autoencoder for neural machine translation. arXiv:1812.04405.
• [B] Learning likelihoods with conditional normalizing flows. arXiv:1912.00042.
• [C] The intrinsic dimension of images and its impact on learning. arXiv:2104.08894.
• [D] Vector diffusion maps and the connection Laplacian. arXiv:1102.0075.
• [E] Estimating the reach of a manifold. arXiv:1705.04565.
• [F] Diffusion models are minimax optimal distribution estimators. arXiv:2303.01861.
• [G] Adaptivity of diffusion models to manifold structures. PMLR:238.
• [H] Smooth function approximation by deep neural networks. arXiv:1906.06903.
• [I] A likelihood approach to nonparametric estimation of a singular distribution. arXiv:2105.04046.
• [J] Statistical theory for image classification using deep convolutional neural networks. arXiv:2011.13602
• [K] Nonparametric regression using deep neural networks. arXiv:1708.06633.
• [L] Generalized Additive Models: Some Applications (JASA, 1987)

We thank the reviewer for their detailed and constructive feedback, and we appreciate the positive remarks on the main results and the comprehensibility of our proofs.

Our derivations beyond Theorems 1 and 2 follow a unified strategy controlling two sources of error. First, we bound the statistical (estimation) error using entropy bounds (Eq. 10 and 11), which essentially capture our function class’s complexity via covering or bracketing numbers. Second, we quantify the approximation error—the bias incurred when approximating the true generator with our network class. Theorem 1 establishes a general convergence rate by balancing these errors, while Corollary 1 specializes the result to sparse and fully connected network architectures. With appropriate tuning parameters such as depth, width, and sparsity, the estimator adapts to the intrinsic dimension and smoothness of the true model, crucial for mitigating the curse of dimensionality.

For nearly singular distributions, where data concentrate near a low-dimensional manifold, Corollary 2 demonstrates that a small noise perturbation smooths distribution gaps, achieving a favorable Wasserstein convergence rate and extending Theorem 2. Theorem 3 further expands our analysis to generators with varying smoothness across dimensions. By combining our new approximation result (Theorem 5) with adjusted entropy bounds, we established that the convergence rate in these settings depends on a composite measure of smoothness and intrinsic dimension.

Methods And Evaluation Criteria and Experimental Designs Or Analyses: We would like to emphasize that our work is primarily theoretical. The simulations and MNIST experiments serve to validate our statistical insights rather than introduce a new algorithm. While some experiments use low-dimensional data in the experiments, MNIST (D = 784) is high-dimensional yet exhibits low intrinsic dimensionality (around 10). Additionally, our framework is broad and applies equally to other likelihood-based methods—such as Normalizing Flows—as demonstrated in related empirical works [A] and [B].

Other Comments: We appreciate the suggestion. In the revised manuscript, we will add a toy example—using a one-dimensional generator map map that transforms a uniform variable into one with a β-Hölder smooth density: $u\mapsto u^{1/(\beta+1)}$. Setting $\beta^*$ = 1 and $t^*$ = 1 in our framework reinterprets our results in this concrete setting, clearly illustrating the theoretical ideas.

Other Strengths And Weaknesses: Assumption 3 holds in practice, ensuring the manifold’s closure is well defined—a common requirement in manifold learning [E]—and is crucial for maintaining low-dimensional structure. If the manifold were space-filling in $\mathbb{R}^D$, the data would lose their low-dimensional properties. The concept of reach, related to the manifold’s condition number, is well established in the literature [C, D, E] and ensures our analysis remains robust even when the support is singular.

We hope these clarifications, along with the additional analyses (see below) and proposed examples, address your concerns and strengthen our manuscript. We welcome any further questions or discussion.

Additional Experiment Addressing All Rebuttal Feedback

To address concerns about the empirical quality of our MNIST images, we computed the W1 distance metric (W1 ± SD). For each digit, we averaged the W1 distances over 50 images and aggregated the results. For context, the average distance between two test images—reflecting baseline variability—is 2.0219 ± 0.745. Below is a table summarizing the empirical results (noise denotes the $\sigma$ of $N(0,\sigma^2)$ added to the training data):

At zero noise, both architectures yield W1 distances slightly lower than the baseline, indicating that the generated samples align well with the test images. As noise increases, the W1 distance grows steadily, showing that higher noise degrades image quality by pushing samples further from the true distribution. Both network types exhibit similar performance across noise levels, suggesting robustness to architecture choice, although subtle differences hint at variations in noise sensitivity. While extracting precise convergence rates from these experiments is challenging, the observed trends validate the large-sample properties for manifold-supported data and align with our theoretical expectations.

We propose including this analysis in the appendix alongside our MNIST experiments.