Sample and Map from a Single Convex Potential: Generation using Conjugate Moment Measures

Dear Reviewer,

Thank you for taking the time to review our paper. We did our best to answer some of your concerns / questions below.

That being said, I believe the experimental section could be slightly improved. In my opinion, it lacks commentary on failure cases, such as the checkerboard patterns in 2D, for instance.

We agree that CMFGen does not perform perfectly on the 2D checkerboard dataset and that we could comment more on that. However, note that the generative ICNNs also fail in this experiment (see Figure 13). More generally, the box plots on Figure 13 shows that in low-dimensional settings, our method performs as well as generative ICNNs, and outperforms them on higher-dimensional datasets such as MNIST and Cartoon (see Figures 5, 14, and 15). Since ICNNs are known to be challenging to train [Korotin et al. 2021], we believe the failure on the checkerboard dataset is primarily due to the limitations of the ICNN itself rather than the methodology we propose. We will clarify this briefly in the main paper and provide details in the appendix.

Additionally, the ICNN architecture used for the MNIST experiment seems overly simple compared to standard models typically applied to this dataset. It would have been helpful to test a more expressive architecture—at least a convolutional neural network (CNN)—to better assess the potential of the method for generative modeling.

Thank you for the suggestion. We tested convolutional ICNNs, which have been used, for example, in [Korotin et al. 2021] to generate clean images from degraded inputs. However, to our knowledge, they have not been applied in generative modeling settings where the starting distribution is a noise prior. In our experiments, convolutional ICNNs did not outperform plain MLP-based ICNNs. We believe our method could benefit from improved integration of convolutional layers into ICNN architectures.

Lastly, I think the paper would benefit from a brief discussion that situates the contribution within the broader field of generative modeling. In particular, it would be useful to outline the potential advantages of this method over existing alternatives.

At this moment, we do not see the conjugate moment factorization as a direct competitor to current state-of-the-art generative models, as its performance lags behind. The main contribution of our approach lies in linking the sampling process of the prior distribution with the transport of this prior to the data distribution. At the moment we see no other approach than the (conjugate) moments approaches presented in the paper. While this may seem like a conceptual curiosity, we hope the originality of our idea can trigger new ideas in the field.

Compared to standard generative models like diffusion models, our method offers the following advantages:

• it does not require arbitrarily choosing a prior distribution
• it provides a fast access to the generated log-probability, enabling applications such as image completion.

The main limitation is that CMFGen relies on Input Convex Neural Networks (ICNNs), which require further improvements for the method to compete with state-of-the-art generative models.

1/ If $\rho = \nabla u$ # $\mathfrak{B}_u = \nabla w^*$ # $\mathfrak{B}_w$, do we have $\mathfrak{B}_u^*= \mathfrak{B}_w$?

No, this is not the case. Note that in both formulations, the potential appears both in the transport map—$\nabla u$ for the moment measure factorization and $\nabla w^*$ for the conjugate moment measure—and in the associated Gibbs distribution, namely $\mathfrak{B}_u$ and $\mathfrak{B}_w$ respectively. Therefore, the first equation does not imply the second. For instance, in the Gaussian case (see Propositions 1 and 2), we have $\mathfrak{B}_u^* = \mathcal{N}(0, \Sigma)$, while $\mathfrak{B}_w = \mathcal{N}(0, \Sigma^{1/3})$. We did not manage to establish any theoretical relationship between $u$ and $w$, or between $\mathfrak{B}_u$ and $\mathfrak{B}_w$.

2/ What could be the impact of the fact that your results only hold for absolutely continuous measures, given the manifold hypothesis, which suggests that real data distributions are highly degenerate?

While the existence is guaranteed for densities, the neural networks must be trained on a stream of i.i.d. samples. This is the same type of result that allows for instance the Brenier theorem to guarantee the existence of an optimal map, and numerical methods to try recovering them. While degeneracies might happen if e.g. the samples do clearly lie on a thin submanifold, as is the case for images if the background is always the same, they can be corrected by adding noise.

For these reasons, we do not consider this assumption to be particularly limiting for practical applications.

3/ Is $\nabla w^*$ still an OT map?

Yes, this follows from Brenier's theorem: for any convex function $f$ and any probability measure $P$, the gradient map $\nabla f$ is the optimal transport map from $P$ to the pushforward measure $\nabla f$#$P$ under the squared Euclidean cost. Since $w*$ is convex, it follows that $\nabla w^*$ is indeed an optimal transport map.

4/ From the experiments, it appears that the results are often noisy. Do you have an intuition to explain this?

We added low-variance Gaussian noise to the dataset to ensure absolute continuity of the target probability measures. This explains why the generated samples also appear a bit noisy. Moreover, since our networks do not use convolutional layers, this may further amplify the impression of noise in the generated samples.

Many thanks for taking the time to review our paper and for your valuable feedback.

The main weakness of this work is that the computational cost is not dsicussed in detail. For sampling sampling from ρ using its Conjugate Moment Factorization, it requires concave maximization problem:∇w∗(y) = arg supx ⟨x, y⟩ − w(x) in each iteration. It is not how expensive this is computaitonally, especially in high-dimensional settings.

We agree that this aspect is not discussed in the paper and we will add a paragraph in the revised version.

As you noted, the problem is a well-posed concave maximization that must be solved at inference time but also during training at each gradient step — not by differentiating through the solver, but by using the optimal solution to get the gradient.

This challenge is inherent to neural OT solvers ([Makkuva 2020], [Amos 2023]). The recent solver we use from [Amos 2023] amortizes the cost of computing $\nabla w^\*$ by training an MLP that, given $y$, predicts $\nabla w^\*(y)$ L.687 in order to initialize the solver. This prediction is then refined by running 30 optimization steps (as proposed in [Amos 2023]) to improve accuracy. This refined value is used both in the loss (step 5 of Algorithm 1), applying jax.lax.stop_gradient to treat it as constant, and to train the MLP for better approximations. At inference, we can again amortize the cost by using the MLP prediction refined with these optimization steps to get a fast and accurate approximation of $\nabla w^*$.

Second, there is no discussion on the advantages/disadvantages of the proposed method relative to the existing generative methods, such as diffuion and continuous normalizing flows.

At this moment, we do not see the conjugate moment factorization as an immediate competitor in the field of generative modeling, and do not wish to present it that way, for the obvious reason that our performance is lagging behind. While we believe this is very likely due to the limitations arising when training ICNNs. Our paper simply asks whether one can propose a generative model that is able to include both noise distribution and transport in a single parameterization. At the moment we see no other approach than the conjugate moments approaches presented in the paper. While this might be a curiosity, we hope the originality of our idea can trigger new ideas in the field.

Finally, the experimental results are not convicining. While the results show some promise of the method, more experiments with higher-dimensional image data will significantly strengthen the paper.

We agree that the current method does not yet compete with state-of-the-art generative models. However, relying solely on ICNNs, it improves upon results generated by generative ICNNs. See Figure 5 for comparisons on MNIST, and Figures 14 and 15 for results on the Cartoon dataset. For a discussion on the limited expressiveness of ICNNs in generative modeling, see [Korotin et al., 2021].

(1) Is the conjugate moment measure factorization given in Theorem 1 unique?

We did not manage to answer this question in the general case. However, for the Gaussian case, we proved the existence of a unique fixed point within the class of quadratic functions (whose Gibbs distributions are Gaussian). This is a challenging theoretical question that we are currently investigating.

(2) What is the computational complexity the proposed algorithm?

For CMFGen, the algorithm complexity per gradient step is O(d PB(T_{LMC}+T_{conj})) with B the batch size, P the number of parameters, T_{LMC} the number of steps used for LMC, and T_{conj} the number of steps used for computing the convex conjugate d dimension. For CMFMA the complexity is O(d PB(T_{LMC}+T_{conj}+P)) due to hessian computations. For comparison, a supervised training of a NN is typically O(PB).

(3) How scalable the algorithm is?

This question can be interpreted in multiple ways.

Perhaps the most obvious bottleneck in our implementation lies in data dimensionality, for two reasons:

• $w_{\theta}$ is parameterized as an ICNN. In high $d$, ICNNs are known to be hard to train. While they encode desirable convexity properties, the constraints in their parameters prevent established optimizers (e.g. ADAM) from converging quickly. While we do obtain reasonable results (our paper is, to the best of our knowledge, the first to train ICNNs in the pixel space of MNIST) there is still a gap w.r.t non-constrained models.
• Our approach requires evaluating conjugate functions at all points in a mini-batch. While the function itself is convex and the optimization problem is therefore well posed, increase in dimension will typically mean that more effort is needed to evaluate these conjugate functions $w_{\theta}^\*$.

(4) I wonder if the authors can comment on the statistical errors of the generated distribution based on the proposed method.

Practically speaking, the box plots on the Sinkhorn divergence (Figure 13) demonstrate that the generated distribution was close to the target, indicating good empirical performance. Theoretically, the statistical error depends on how well the method approximates the conjugate moment potential, which in turn hinges on whether the fixed point is found—something we have not yet proven. In theory, at least, the use of ICNNs is not a limitation here, as they are universal approximators of convex functions.

We are very grateful for your time spent reading our paper, as well as for your detailed review. We did our best to address your concerns and questions below.

The argument that moment potentials are unsuitable for practical sampling tasks and that conjugate moment potentials offer a better alternative is primarily based on a relatively narrow set of examples -- Gaussians and Gaussian mixtures in one and two dimensions. I am not convinced that the behavior in these examples generalizes to non-Gaussian (mixture) and higher-dimensional distributions based on the arguments given in the paper.

If you were not convinced by our Gaussian experiments, we suggest that you read the argument put forward in L.119-131, which helped us understand better the limitations. In the moment measure factorization, $\nabla u$ must take values, by construction, on the support of the measure 𝜌 to be factorized. If 𝜌 is narrow and concentrated, so must $\nabla u$ be. If $\nabla u$ moves slowly, $u$ looks like a linear potential. If 𝜌 is narrow and concentrated around 0, then $u$ is almost constant. In both cases, $u$ would be fairly degenerated and extremely spread on its support.

Moreover, moment measures/potentials are not typically used for practical sampling tasks, so from that standpoint the idea that we need an alternative to the moment potential factorization isn't credible.

We agree that moment measures were never highlighted as a means to generate data better. Given the importance of generative models in society as a whole, we believed that exploring original ideas to define new generative models was a worthwhile avenue. We looked for models that include, holistically, a parameterized noise description, and naturally found inspiration in moment factorizations (L3-4). Moment measures are not widely known within the machine learning community, and one of our contributions is to introduce them to Neurips, and propose an alternative factorization.

The numerical results include no comparisons to widely used or state-of-the art sampling methods. The only comparisons are to ICNNs trained to push forward a Gaussian to $\rho$ directly, whereas state-of-the-art neural network transport approaches usually invoke some kind of dynamics, rather than being "one-shot".

It is true that our algorithm does not match the performance of progressive generative models. However, our approach addresses a different aspect of the problem—namely, finding a suitable prior that facilitates the transport by the transport map. We will clarify.

The pipeline for generative modeling with conjugate moment measures is similar. There are many sampling methods that enable the user to generate new samples directly after the training of a neural network (neural ODEs, flow-matching, using PINNs to solve continuity equations, etc.), without the post-processing steps involved in the proposed method.

Your parallel with Neural ODE is interesting!

In that sense the LMC effort is indeed comparable to the ODE solver effort (parameterized by # steps), with the nuance that LMC uses the gradient of a time-independent potential, vs. a time parameterized velocity field for ODE.

For that reason, the convergence of LMC is conceptually simpler than the integration of a vector field with a rotational component.

In the density driven-setting, one could run LMC on $\rho$ directly. [...] Have you compared the quality of samples generated by the conjugate moment measure with those generated by a trained neural network (for dynamic transport) directly or by LMC?

In the case of non–log-concave distributions such as $\rho$, the LMC algorithm lacks theoretical guarantees [Roberts and Tweedie, 1996; Cheng and Bartlett, 2018; Dalalyan and Karagulyan, 2019]. In particular, when the probability density function has multiple local maxima, the generated samples tend to be highly correlated, as particles from the LMC algorithm often become trapped in local basins [Pompe et al. 2020] [Gabrié et al. 2022].

In the variational inference framework used in Section 4.3, our focus is on learning the conjugate moment potential, rather than designing a practical algorithm to sample from unnormalized densities. For this reason, we did not compare to other sampling algorithms. To make the method practical, a possible direction would be to combine our approach with MCMC methods to improve the current uniform sampling over the optimization space, in the spirit of the algorithm proposed by [Gabrié et al., 2022].

Theorem 1 guarantees existence of a conjugate moment factorization for absolutely continuous measures $\rho$ with compact, convex support. Many of the distributions in the numerical examples do not satisfy these assumptions. Is this a problem?

While the existence is guaranteed for densities, the neural networks must be trained on a stream of i.i.d. samples. This is the same type of result that allows for instance the Brenier theorem to guarantee the existence of an optimal map, and numerical methods to try recovering them. While degeneracies might happen if e.g. the samples do clearly lie on a thin submanifold, as is the case for images if the background is always the same, they can be corrected by adding noise.

How do you know, for example, that the fixed-point iteration in CMFGen will converge? Have you tried any examples where it didn't converge?

We did not manage to theoretically prove the convergence of the fixed-point iteration in CMFGen. Note that in practice (except in 1D), we run a smoothed version where we do not compute the exact OT map at each iteration but instead take a single gradient step. With this strategy, the algorithm always converged for all the experiments. In 1D, where the OT map is fully estimated at each iteration, we encountered convergence issues in some experiments (distinct from those presented in the paper). However, these were easily resolved using a simple heuristic: averaging the OT maps computed at iterations $n-1$ and $n-2$. This is a failure case that we omitted and will add to the Appendix. It may in fact suggest that using momentum based corrections could be beneficial.

It appears that noise is added to the generative modeling training images to guarantee existence of an OT map. Computationally, does the pipeline break down if noise is not added?

Following your question, we tested our algorithm without adding noise on the MNIST and Cartoon datasets. While the pipeline remained stable and did not break, the visual results were slightly degraded. These cases are interesting because they are not covered by Theorem 1, which assumes absolutely continuous probability measures.

The face image inpainting example seems relatively easy because the mask covers one vertical half of the the image and faces are symmetric about the vertical axis. [...] Is the quality of the in-painting as good, or does it get degraded?

It is not clear whether the experiments mentioned are more challenging for the algorithm. We conducted the two experiments you suggested—masking the top half and masking the bottom half of the image—and in both cases, the algorithm consistently filled in the masked regions. The results were comparable in quality to those obtained with horizontal masking, as shown in Figure 17. When masking the bottom part of the image, the algorithm tends to produce samples that closely match the original. However, when masking the top part, features such as hair color can become ambiguous—especially in cases where the cartoon character has very short hair—leading the algorithm to generate cartoons with different hair colors or hairstyles. Unfortunately we cannot share these images due to the rebuttal process in place this year.

Do the authors see a way that this step could be avoided? Or do they see a way that their conjugate moment factorization could play an intermediate role in generative modeling pipelines which ultimately produce noiseless images?

We currently do not see a way to remove the added noise without degrading performance, but this is intrisically due to the degeneracy (uniform background) of the data. A possible direction to improve scalability would be to use our method to estimate the conjugate moment potential of the distribution $\rho$, and then retain only the corresponding Gibbs distribution as a learned log-concave prior. Flow Matching combined with mini batch Optimal Transport ([Tong et al. 2023], [Pooladian et al. 2023]) could then be used to learn a generative model between this prior and the target distribution. This approach would approximate our current method and may offer better scalability. This however would involve two neural networks (one for learning the prior, one for transport), and thus move slightly away from the original theory of conjugate moment measure factorization, which relies on a single convex potential.

Many thanks you for your encouraging comments. We appreciate that you have recognized our contribution, which proposes to reassess the link between noise distribution and transport.

Can you please provide a complete and formal statement of Brenier's theorem that is being used either in the main paper or in Appendix?

We will include a formal description of the Brenier theorem in our appendix.

From what I understand, although the theory holds for convex potentials, none of the methods are actually using convex potentials. Is that correct? If not, how are you ensuring that $w_\theta$ are convex? If yes, how is convex conjugate / legendre transform interpreted for a non-convex function?

Thank you for the question. In our case, the potentials $w_\theta$ are indeed convex with respect to their input by construction, as we rely on Input Convex Neural Networks (ICNNs) [Amos et all. 2017], this is mentioned L.170. We will add a more detailed explanation of the ICNN construction in the appendix for clarity. Regarding the convex conjugate: since our potentials are convex by design, the Legendre transform is well-defined and its gradient can be reliably computed by maximizing the concave problem defined L.181 using a solver. We use an off-the-shelf L-BFGS implementation (L.182).

Would it be possible to compare performance with other energy-based modelling methods (like say score matching) in a synthetic setting for example over an exponential family?

Our method does not currently outperform state-of-the-art generative models, and we expect that advanced score-matching-based models will achieve better performance than ours. We believe this is primarily due to our reliance on ICNNs, which, while advantageous for parameterizing convex functions, are known to be challenging to train ([Korotin et al. 2021]). For this reason, we compare our method to other generative ICNNs using the Sinkhorn divergence as a metric (see boxplots in Figure 13). Make the method scalable will require improving further the training of ICNN architectures, an important research topic.

Line 214: How is $w_\theta$ initialized to be roughly $\lVert \cdot \rVert^2$? It does not feel necessary to have this initialization for the proposed method, can you explain choice of this initialization?

This initialization was proposed in [Bunne et al. 2022] and is akin to initializing with the identity map, but you are correct, one can consider other more data informed initializations, and this topic has been the subject of great research in [Hoedt, Klambauer, 23] but this is not yet useful for OT maps. [ Hoedt, Klambauer, Principled Weight Initialisation for Input-Convex Neural Networks, Neurips 23]

Line 404: $w$ is convex only if $\Omega$ is convex. Is this assumed?

Yes, $\Omega$ is assumed to be convex, as stated in Theorem 1. We will also clarify this in the appendix when presenting the proof.

Line 405: How is $\nabla w$ defined? $w$ seems to be discontinuous on boundary of $\Omega$.

The function $w$ is differentiable in the interior of $\Omega$, so taking its gradient there is well-defined. On the boundary, $w$ is discontinuous by definition, as it takes infinite values outside $\Omega$, and thus its gradient is not defined there. However, note that the boundary of the bounded convex set $\Omega$ has Lebesgue measure zero, so this does not affect the proof.

Formatting issues

Thanks for spotting these typos — we will take them into account in the revised version.

Lines 91: Does not supported on hyperplane mean that support is not contained in any hyperplane?

It means that the support does not strictly include any hyperplane (or, more generally, does not put positive mass on a subset of lower dimension than the ambient space, or, more loosely, that it has a density w.r.t. the Lebesgue measure) This is a classic assumption to obtain results in OT.

Isn't boundary a measure zero set, implying that if $u$ is only discontinuous at the boundary then it is still essentially continuous? I do not see how this is relevant to rest of the paper.

The boundary of the bounded, convex set $\Omega$ is a null set for the $n$-dimensional Hausdorff measure, but the definition of essential continuity in this context requires the discontinuity set to have zero $(n-1)$-dimensional Hausdorff measure. Since the boundary has positive $(n-1)$-dimensional measure, $u$ is not essentially continuous. This result is not directly relevant to our contributions; it was intended to introduce [Cordero-Erausquin and Klartag, 2015], and the problem they studied, as they made a significant contribution to the moment measure factorization with this result.

From what I understand LMC only ever provides an approximation, these lines seem to suggest that it matches exactly to the final distribution after a warm-up period.

You are right, the LMC algorithm is the Euler discretisation of the Langevin diffusion process and this discretization induces a bias that can be controlled for smooth log concave distributions [Dalalyan 2017]. We will add a note on that bias in the revised version.

Line 428: Is $\mathscr{C}(\Omega)$ the set of convex functions on $\Omega$? $\mathcal{C}(\Omega)$ is used for same purpose on line 71

It is actually the Banach space of continuous functions over a compact set, so no convexity assumption is involved. While we used a different font, we will introduce explicit notation such as $\operatorname{cvx}(\Omega)$ to clarify these distinctions.

Many thanks you for your time and for your kind comments on the writing, novelty, and theoretical grounding of our work. We address your questions and concerns below.

The notation can at times be difficult to read -- for example $*$ is used both to denote the optimizer and the convex conjugate, and there are several usages of related letters such as $\mathcal{B}$ and $\mathfrak{B}$ for distinct concepts.

We agree that the $\mathcal{B}$ - \mathcal{B} notation for the Brenier potential and $\mathfrak{P}$ - \mathfrak{P} for Gibbs distribution look similar. Following your comment, we have changed the notation of the Gibbs potential to $\mathfrak{G}$ - \mathfrak{G}.

Regarding the star notation, we make a distinction between the optimizer denoted $\star$ and the convex-conjugate operator, denoted *. While we agree that the symbols look similar (less so in the paper than in markdown), these are standard notations, and we prefer to keep them as they are.

The results on higher-dimensional datasets like images are unfortunately quite bad, even for simple cases such as MNIST

We acknowledge that our results are far below from the state-of-the-art in generative modeling. However, compared to the existing OT approaches that use ICNNs to generate data from Gaussian (e.g. [Amos’23], [Makkuva et al. 2020], [Korotin et al. 2020]) our method improves the results on the 2D datasets, MNIST and Cartoon, while also relying exclusively only on ICNNs. The improvement is particularly visible on MNIST (see Figure 5 for comparison).

Very minor comment, but the authors miss a few of the major initial references on flow matching

Thank you for these additional references, we have added them. Please feel free to suggest any others you believe could be relevant.

1. In the introduction, the authors comment that $\omega^*$ is the Brenier potential linking $e^{-\omega}$ to $\rho$. Can the authors describe, then, why the method is different from optimal transport? Is it simply because we also learn the source distribution?

Yes, exactly, in the classical optimal transport (OT) problem the source and target distribution are fixed and the objective is to estimate the OT map between these distributions (gradient of a convex potential). In the (conjugate) moment factorizations, both source distribution and transport map are uniquely parameterized by the same convex potential (L.37, L.206)

We know that the optimal transport map can be hard to learn, and so many generative modeling approaches are intentionally based on sub-optimal transport to simplify the estimation problem. Can the authors explain clearly why it is a "good idea" to try to estimate the optimal transport map if the goal is "only" generative modeling?

This is a great point. In this paper, our hands are tied with the “ideal” OT parameterizations of transport, by a convex potential, because this is what the factorization theorem dictates.

While having a convex potential helps with LMC sampling, it is indeed a burden when computing the map. We leave the door open on using our method as a first pre-processing step, in order to learn first a “good” prior, before using a more advanced generative tool.

2. Can the authors explain the computational efficiency of their approach? The iterative optimal transport computation in Sec. 4.2 seems very expensive.

The proposed method inherits the efficiency (and limitations...) of neural OT solvers.

Algorithm 1 mirrors the standard procedure for estimating the OT map between two distributions [Amos’23]. The only difference is that, in our case, the (noise) source distribution is re-updated at each iteration. Specifically, in line 4 of Algorithm 1, instead of drawing a minibatch from a fixed distribution, we sample from the evolving distribution using the Langevin Monte Carlo (LMC) algorithm. Re-ampling from this moving, log-concave distribution is the only overhead compared to a standard OT solver. To mitigate this cost, we speed up LMC by reusing particles from the previous iteration as a warm start (L.219 in the paper). Furthermore, to compute $\tilde{x}$ at line 5, we leverage the latest OT solver by [Amos’23], which warm-starts the conjugate solver using predictions from a trained MLP, significantly accelerating the overall algorithm, as noted in Appendix (L.693).

3. Can the authors comment on the feasibility of their sampling method from Sec. 4.3? The sampling method requires Langevin samples from the target measure $\rho$ to learn. This seems to be a chicken and an egg problem, because the method's goal is to generate samples from $\rho$. From this perspective, I'm having trouble seeing if the method will really work in practice. Is there a way to make it iterative and sample from the current estimate of $\rho$, using the fact that $\omega$ is convex? Will this suffer from mode collapse?

In the variational inference framework used in Section 4.3, we do not require samples from $\rho$. Instead, we assume access to the unnormalized densities associated to $\rho$ i.e. for a given sample $x$, we can require access to $q(x)$ where $q(x) \propto \rho(x)$. The algorithm does not train on samples from $\rho$ but rather on couples $(x, q(x))$ where $x$ is sampled in a more or less clever way. In our experiments, $x$ is sampled uniformly over the domain, and Section 4.3 focuses on learning the conjugate moment potential in this context. To make the method practical, an idea would be to combine this algorithm with MCMC methods to draw samples $x$ smartly in the same spirit as the methods explored in [Gabrié et al. 2022]. We will include these clarifications in the revised version.

4. Can the authors get their method to work better in the numerical examples? I'm a firm believer that we shouldn't require SoTA image results in every paper on generative modeling, but I do think if the method doesn't really work on the 2d checker, it indicates a problem. For publication, I think the authors need to have compelling results on some simple examples.

We agree that CMFGen does not perform perfectly on the 2D checkerboard dataset. However, note that the generative ICNNs also fail in this case (see Figure 13). More generally, the box plots on Figure 13 shows that in low-dimensional settings, our method performs as well as generative ICNNs, and outperforms them on higher-dimensional datasets such as MNIST and Cartoon (see Figures 5, 14, and 15). Since ICNNs are known to be challenging to train [Korotin et al. 2021], we believe the failure on the checkerboard dataset is primarily due to the limitations of the ICNN itself rather than the methodology we propose. In response to your comment, we experimented with larger ICNN architectures—specifically, models with 6 hidden layers of size 128 and 5 hidden layers of size 512—to improve performance. For now the gains were not significant, but we plan to continue exploring this.

They should also try to quantify their results in some way, such as FID on images and metrics such as KL or Sinkhorn in low-d. The authors should also tone down the claims in the text that the method works well, which are sort of contradicted by their figures.

Note that we use the Sinkhorn divergence to compare our method to generative ICNNs (see Figure 13), where boxplots provide a visual summary of the results. From our perspective, the method performs well—this is, to our knowledge, the first time MNIST samples have been successfully generated using ICNNs (see Figure 5 for a comparison between our results and samples generated by a generative ICNN). That said, we acknowledge that, when compared to state-of-the-art generative methods, the results are not impressive. We will make this distinction clearer in the revised version.

We are grateful for your support and constructive criticism.

We agree that our method is not immediately competitive, but we hope that the originality of our mathematical result and proof of concept implementation can also encourage others to reconsider and explore alternatives to the Gaussian + ODE/SDE flow paradigm that has come to completely dominate generation. In particular, we genuinely believe that the choice of the sampling distribution is an understudied area, and we believe our idea is quite innovative in that sense.

Since you mention normalizing flows, we also noticed recently a resurgence of such approaches, see e.g

arxiv/2412.06329 Normalizing Flows are Capable Generative Models

We will carefully incorporate all of the points we have discussed in this rebuttal to clarify the message of our draft. Many thanks for your time and consideration, and for helping us improve our paper.

thanks!

the authors

PS: we will explore a simpler notation than mathfrak for the Gibbs distribution.