A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models

We sincerely thank the reviewer for their thoughtful comments and constructive feedback. We greatly appreciate their recognition of the novelty and rigor of our approach.

Below, we provide detailed responses to each of the points raised.

1. We assume the data-generating process $Y = G_*(Z, X) + \varepsilon$, where $G_*$ maps latent variables $Z$ and covariates $X$ to the manifold, while $\varepsilon$ captures noise. The objective is to recover $G_*$ to generate data that lies exactly on the manifold. However, real-world data is inherently noisy and does not perfectly sit on the manifold, making assumptions that embed noise into the latent space overly restrictive.

   Ignoring $\varepsilon$ and embedding it into the generator would blur the distinction between the signal and noise, undermining the deconvolution objective. For instance, consider data near a 2-sphere in 3D, where a binary covariate $X$ indicates the upper ($X = 1$) or lower ($X = 0$) hemisphere. While $G_*$ should map $Z$ and $X$ to points on the 2-sphere, observed responses deviate due to noise. The key challenge is disentangling $\varepsilon$ while ensuring $G_*$ faithfully captures the manifold structure.

2. A simple example is the generator map $u \mapsto u^{1/(\beta+1)}$, which transforms a 1D uniform random variable into one with a $\beta$-Hölder smooth density. Here, setting $\beta_* = \beta$ and $t_* = 1$ allows us to reinterpret the results in our framework, offering a clear illustration of the theoretical findings.

3. When the noise level $\varepsilon$ approaches zero, likelihood-based approaches face significant challenges due to the singularity of the support space for $Y$, requiring precise approximations and making optimization difficult. Controlling the $W_1$ distance is equally non-trivial, as it involves solving computationally intensive likelihood functions and requires fundamentally different theoretical tools, including optimal transport. We appreciate this insightful point and are actively working on a long-term project to derive convergence rates and minimax bounds for likelihood-based methods under the $W_1$ distance.

   This is a complex yet promising direction, and we are also exploring diffusion- and flow-based models that leverage score-based and velocity field techniques to address these challenges.

4. Our work focuses on conditional models, with all results holding uniformly for all $x \in \mathcal{X}$. Lemma 3 (Noise Outsourcing) establishes the existence of a generator within the distribution regression framework. Section 2.4 characterizes the class of conditional distributions achieved via the push-forward of the generator $G_*$, showing that the learnable distribution class is broad and includes many commonly used distribution classes, demonstrating the generality and flexibility of our approach.

5. In our experiments, we used two conditional manifold examples—the ellipse and two-moon examples—to validate the trends in Corollary 2.

   We agree that exploring intrinsic dimensionality with simple examples and pretrained models for generating MNIST images would enhance clarity and validate theoretical bounds. However, due to time constraints during the rebuttal period, we defer this to future work and look forward to incorporating it into our ongoing research on conditional flow matching. Thank you for the insightful suggestion!

6. Please refer to the definition of the integrated Wasserstein distance in Section 2.3, just before Theorem 2. This metric is defined as the expectation over the distribution of $X$. It corresponds to the same metric as Proposition 1.(i) in [4], with the specific case of $p = 1$.

7. Thank you for highlighting these approaches. While score-based methods were briefly mentioned in our introduction, to our knowledge, there are no theoretical results addressing composite structures in flow-matching or diffusion-based models. On a related note, our group is exploring how to integrate composite structures into flow-matching frameworks, particularly investigating the smoothness properties of linear flow vector fields for composite-structured target distributions.

8. Two practical examples illustrating the smoothness disparity framework are Heart Rate Variability and Air Quality Modeling: In heart rate variability, the cardiac response conditioned on physical activity exhibits irregular, non-smooth patterns, while physical activity transitions are smoother. Similarly, in air quality modeling, particulate matter (PM) concentrations conditioned on temperature often show sharp spikes (e.g., from pollution or fires), while temperature changes are generally smoother.

Dear Reviewer VL1U,

As we approach the final stages of revising our rebuttal PDF, we wanted to gently remind you of our responses to your valuable feedback.

We have carefully addressed all your comments and questions and hope that our efforts meet your expectations. If you have any remaining concerns, we would be happy to discuss them further during the extended discussion period. Otherwise, if our updates satisfactorily address your feedback, we kindly invite you to consider raising your score.

We truly appreciate the time and effort you have dedicated to reviewing our paper, especially given its depth and complexity. Regardless of the final outcome, we are deeply grateful for your thoughtful insights, which have significantly contributed to improving our work.

Thank you again!

Sincerely,
The Authors

We sincerely thank the reviewer for for their thoughtful and detailed feedback and recognizing the relevance of our framework. We appreciate their acknowledgment of the theoretical results, error analysis, and empirical findings.

Some of the theoretical results seem hard to verify in practice. It would be interesting for some contrived examples if the authors can show the convergence of the distances as a function of the number of points.

We may consider a simple representative example to better understand the obtained rates:

Take the function with density $f(y) = (\beta + 1)y^{\beta}$, supported on $[0,1]$. This density is Hölder smooth with parameter $\beta$, and the ambient dimension is $1$. When $\alpha = 0$, the rates outlined in Corollary 1 simplify to $n^{-\beta/(2\beta+1)}$. This corresponds to the minimax optimal rate for estimating $\beta-$Hölder density function.

Note that the paper does not appear to be in the right ICLR format since the margins are considerably smaller.

We apologize for the formatting issue in the original submission and have made the necessary corrections in the revised version. For a detailed overview of the changes, please refer to the global rebuttal, which outlines the specific revisions and adjustments made.

How easy are some of the conditions to verify for the convergence theorems to hold?

• Assumption 1 ensures the existence of a density for both the noisy data and its noiseless counterpart on the manifold. This is a widely accepted and foundational assumption in statistical and machine learning literature.
• Assumptions 2 and 3 can be practically motivated in various ways. For instance, [H] explores the existence of lower-dimensional structures in data, providing empirical evidence for such assumptions. Assumption 2, in particular, generalizes common data structures like multiplicative and additive models [I].

For specific problems, certain aspects of these assumptions can be known or approximated. For example, in the MNIST dataset, the ambient dimension $D = 784$ is readily available. However, determining intrinsic characteristics, such as the intrinsic dimension $d$, remains a difficult task.

To the best of our knowledge, there is no existing result that provides a framework for consistent testing of these assumptions while simultaneously achieving efficient estimation. This dual goal represents a challenging open problem that warrants further research.

The theorems contain some constants that do not have a clear way of being estimated making some of the errors hard to use in practice.

How do you interpret the constants in the various theoretical statements? For example, some are dependent on the domain or other parameters, in what sense do they grow for perturbations of their dependencies? Can these be estimated in practice?

When analyzing the large sample properties of estimators, the explicit dependence on constants is often deliberately hidden. This approach allows us to focus on the obtained rates without being distracted by the cumbersome and sometimes "ugly" expressions of these constants.

The constants in our theoretical statements depend on fixed model parameters (e.g., $\beta_*$, $t_*$, or $\sigma_{\max}$), which do not grow with sample size. While they can, in principle, be traced through the proofs, estimating them in practice is challenging due to unknown parameters. In specific cases, such as when the data-generating process is known (e.g., $f(y) = 6y^5$), these constants can be directly identified. However, adaptively estimating these parameters with theoretical guarantees in a non-parametric setting remains a complex open problem. In practical applications, partial knowledge—such as ambient dimensions (e.g., $D = 784$ in MNIST)—can provide some guidance, but estimating intrinsic dimensions like $d$ is notably harder and has been the focus of significant research such as [H].

• [H] The Intrinsic Dimension of Images and Its Impact on Learning (arXiv:2104.08894)
• [I] Generalized Additive Models: Some Applications (JASA, 1987)

We thank the reviewer for their valuable feedback and sincerely appreciate their positive remarks on the clarity of the presentation and the rigor of the proofs in our manuscript.

1. The method seems to be extending the techniques of previous work and filling in some details with approximation.

Our work aims to address the gap in understanding why conditional deep generative models perform well by examining the problem within a statistical framework. Beyond bridging this gap, our cohesive framework not only provides key insights but also generalizes several existing works. A notable special case of our results is Chae et al. (2023), where the model complexity is significantly smaller, as they only consider the case when $|\mathcal{X}| = 1$. In contrast, our work allows for infinitely many co-variates simultaneously. Another special case (derived from Theorem 3) is Li et al. (2022), which examines sequential smoothness structures but in a non-generative model framework.

From a technical standpoint, we have introduced several novel tools in our proofs, which are elaborated upon in the appendix, while also leveraging established techniques with proper citations to the best of our knowledge. Notable contributions include the integration of two distinct classes of neural networks—sparse and fully connected—into a unified framework, the derivation of a general convergence result for distribution regression as stated in Theorem 1, and the development of a novel approximation result, which is rigorously established in Theorem 5.

2. Question on the applicability to multiple chart manifolds.

1. In regards to the discussion in 2.4.3 and Appendix C with the multiple manifold case: If a partition of unity is used to define local transport maps, we require that the local distributions on each chart need to have density lower bounded. However, a partition of unity will have its weights vanish to 0 on the boundary, so the induced local distribution would have density that vanishes to 0 on the boundary. How can you apply optimal transport regularity theory in that case, when your target has density that is not lower bounded?

Thank you for pointing this out. We have made a small modification in the proof to address the issue (please see the updated manuscript). This correction ensures that the transport map properly handles densities that are both upper and lower bounded, maintaining the desired properties throughout the mapping process.

3. Numerical examples are very simple.

The focus of this work is primarily theoretical in nature. The examples provided are intended to validate the proposed methodology and offer statistical insights in different settings. We utilize the (conditional) VAE architecture to demonstrate the applicability of our theoretical findings but do not propose any new algorithms. The developed theory is equally applicable to other likelihood-based methods, such as Normalizing Flows.

On the algorithmic and computational side, there are existing works, such as [A] and [B], that incorporate more advanced examples; however, these primarily focus on computational performance and offer limited statistical insights.

• [A] Conditional Variational Autoencoder for Neural Machine Translation (arXiv:1812.04405)
• [B] Learning Likelihoods with Conditional Normalizing Flows (arXiv:1912.00042)

Thank you for your response and updating the manuscript. I had a question about Appendix D of the revision. While the formula for the density $q$ in line 945 follows from the partition of unity (as it involves a sum of scalar valued quantities, i.e. the probability density), I am confused about the for the formula for $G$ in line 964. Correct me if I am wrong, but the co-domain of $\varphi_k \circ g_k$ should be the patch of the manifold $U_k \subset \mathcal{Y}$. Reducing to the case of $K=2$, the formula is (where $h_k = \varphi_k \circ g_k$)

$G(z,x) = \pi_1 \tau_1(h_1(z, x)) h_1(z, x) + \pi_2 \tau_2(h_2(z, x)) h_2(z, x)$

$h_k(z, x)$ is an element of $U_k$, and the quantity $\pi_k \tau_k(h_k(z, x))$ is a scalar, so the RHS involves scaling elements of $U_1$ and $U_2$ and adding them together. I think the adding together part doesn't matter since the domains $\mathcal{V}_k$ are disjoint, but I do not know how to scale an element of a manifold. In other words, I am confused about how to interpret the value $\tau_1(h_1(z, x)) h_1(z, x)$ if $\tau_1(h_1(z, x)) \neq 1$. This makes me think that the formula in line 964 is not correct.

We sincerely thank the reviewer for their feedback. We greatly appreciate their recognition of our detailed analyses across various setups.

The paper is not very easy to follow... . It is relatively difficult for me ...

Thank you for your suggestion. The focus of this paper is on characterizing the statistical foundations of conditional deep generative models by rigorously studying their statistical properties. We aim to understand why these models perform so well by providing a unified framework that extends and generalizes several previous studies, such as the work by Chae et al. (2023) which focuses on the case where $|\mathcal{X}| = 1$.

Our approach differs from existing works like Schmidt-Hieber (2020), which uses sparse neural networks for regression in a non-generative setting. In contrast, we address a broader problem and incorporate additional mechanisms like the manifold hypothesis to handle the curse of dimensionality. Our framework provides rigorous guarantees for both sparse and fully connected networks.

Additionally, Tang and Yang (2023) rely on adversarial or GAN-based methods with strict assumptions that the data lies exactly on the manifold and has a lower-bounded density. Our approach is more flexible, allowing noisy data close to the manifold without requiring a lower-bounded density, and offers practical advantages over adversarial-based methods.

Practical impact ... assumptions.

Please allow me to illustrate the technical assumptions and their practical implications.

• Assumption 1 ensures the existence of densities for both the noisy data and its underlying noiseless counterpart on the manifold, which is a foundational and widely accepted assumption in statistical theory.
• Assumption 2 introduces structural constraints to overcome the curse of dimensionality. This assumption leverages inherent data structures, such as additive models, which are widely used in practice [E].
• Assumption 3 employs the manifold hypothesis to further reduce the curse of dimensionality, with applications in image data, which naturally reside on low-dimensional manifolds within high-dimensional spaces [C]. The concept of reach, as the inverse of the manifold's condition number, is crucial here, and while no consistent manifold estimation methods exist for manifolds with zero reach, this challenge is well understood in manifold learning literature [F,G].

Numerical results loosely connect to the theoretical results.

Thank you for your comment. The primary focus of this work is theoretical, with the numerical examples serving to illustrate and validate the proposed methodology in practical settings. We provide three types of examples to demonstrate different aspects: the FD examples validate the methodology in full-dimensional settings, the toy manifold examples confirm the trends outlined in Corollary 2, and the MNIST example highlights a scenario where the intrinsic dimension is much smaller than the ambient data dimension ($D = 784$). These examples are designed to offer statistical insights and confirm the broader applicability of our theoretical results.

The paper is not prepared in the standard ICLR style ...

We apologize for the formatting issue in the original submission and have corrected it in the revised version. Please refer to the global rebuttal for a detailed overview of the revisions made.

Do numerical results support the rates proved in theory? Maybe we can take logarithm on the sample size and inspect the slope of the error.

Following your suggestion, we extended our analysis to examine how the $W_1$ distance varies with sample size, while keeping the noise level fixed at $\sigma_* = 0.01$. Below is a summary table showing the median empirical Wasserstein distances for different sample sizes. The experimental setup remains consistent with the manifold case described in the manuscript.

While extracting exact rates can be challenging, the results in the table validate the large-sample properties for manifolds. These empirical findings align well with the theoretical expectations, further confirming the consistency and convergence trends of our framework.

Does Assumption 3, which excludes interior points, hold (approximately) in practical applications?

Yes, Assumption 3 holds in practical applications. It ensures that the closure of the manifold is well-defined, which is a common requirement in manifold learning literature [D]. This assumption is crucial for maintaining the low-dimensional structure of the data. If the manifold were space-filling in $\mathbb{R}^D$, the data would lose its low-dimensional properties. In practice, it implies that data points lie close to a manifold with a well-defined structure, as discussed in [C].

Dear Reviewer Qywh,

Thank you for your thoughtful comments and for raising your score. We sincerely appreciate your acknowledgment of our theoretical contributions and statistical analysis.

While our study focuses on foundational theory rather than immediate practical applications, we hope it serves as a stepping stone for future research to bridge the gap between theory and empirical advancements. We are confident that our work can inspire further exploration, leading to algorithmic innovations and architectural designs.

Sincerely,

The Authors