On the Stability of Iterative Retraining of Generative Models on their own Data

We would like to thank the reviewer for highlighting the clarity of the manuscript, the importance of the topic, and the significance of the results. We implemented most of the propositions of Revierwer LGsW, which significantly improved the quality of the paper

Exact Likelihood Model

We would like to thank Reviewer LQsW for their insightful comments. In the new version of Figure 3, we now include a state-of-the-art flow model: conditional flow matching [1,2]. This corresponds to a continuous normalizing flow, which is an exact likelihood model as the reviewer requested. This model was trained on the image dataset CIFAR-10, and we observe the exact same trend with $10$ iterative retraining steps (by the end of the rebuttal we hope to update this plot with 20 steps).

More Epochs and Retraining

As stated in the global response to all reviewers, we have added more epochs (from 5 or 10 to 20) and more retraining (from 5 to 20). This yields much clearer results: one can see a stabilization after 10 to 15 retraining, especially when compared to retraining on fully synthetic data. This is also supported by the visual inspection of the samples added in Appendix G.3 which shows almost no degradation when retraining on a mix of synthetic and generated data.

Seeds

We acknowledge the reviewer’s comment regarding the number of seeds used in our iterative retraining experiments for natural image datasets. We first highlight that we have improved this experiment in the updated version of the manuscript but substantially increased the number of iterative retraining steps ($5$ vs $20$) while adding a fully synthetic baseline ($\lambda = \infty$). In our plots, we see that the margin of difference between the fully synthetic case and the more stable regime of $\lambda > 0$ of iterative retraining is quite dramatic. While the reviewer has requested more random seeds, we wish to politely push back due to the high computational cost of running each experiment. For example, just for FFHQ-64, conducting iterative retraining using the SOTA EDM model amounts to a full week of experimentation using 40 RTX8000 GPUs. Unfortunately, running additional random seeds for all models and datasets is beyond the computational budget within the period of this rebuttal. In addition, as required by reviewer bfKK, we have added quantitative measurements for the experiments on the 2D synthetic data and were able to repeat this experience multiple times and provide statistics such as confidence intervals and error bars.

We thank the reviewer again for their review and detailed comments that helped strengthen the paper. We believe we have answered to the best of our ability all the great questions raised by the reviewer. We hope our answer here and in the global response allows the reviewer to consider potentially upgrading their score if it they see fit. We are also more than happy to answer any further questions.

[1] Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M. and Le, M., 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747.

[2] Tong, A., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Fatras, K., Wolf, G. and Bengio, Y., 2023, July. Improving and generalizing flow-based generative models with minibatch optimal transport. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems.

We thank the reviewer for their warm comments on the updated manuscript.

Plot in the parameter space

We are grateful for the reviewer's insightful feedback and ongoing engagement in this rebuttal process. First, we would like to say that ideally, Theorem 1 and 2 would have been proved in the distribution space, i.e., on the distribution $p_{\theta}$, not the parameters $\theta$. However, the infinite dimension of the probability space currently makes it out of reach for our current proof.

To further substantiate our theory—on $\theta$—and as the reviewer rightfully encouraged—we have included a new appendix G.5 with plots of the direct model distance in parameter space for the CIFAR10 – EDM, FFHQ-64 EDM, and CIFAR-10 OTCFM experiments. In particular, we plot the parameter distance between an iteratively retrained model at timestep $t$ to the initial model $\theta_0$, $|| \theta^{\lambda}_t - \theta_0 ||_2 / \sqrt{n}$ for $t \in [0, 20]$. We further ablate the influence of $\lambda$, with $\lambda = \infty$ corresponding to the fully synthetic setting in our paper. Finally, we compare $\theta_0$ with a randomly initialized (and untrained network) to ground this study.

Our results in Appendix G.5, for EDM, Figure 7 shows that when the number of iterative retraining increases, the average parameter $L_2$ distance stays in a range between $0.0005$ and $0.001$. In contrast, the randomly initialized network has a distance order of magnitude larger than $|| \theta^{\lambda}_t - \theta_0 ||_2 / \sqrt{n}$. This means a random network is extremely far away in average $L_2$ distance to $\theta_0$. These results further corroborate the results in Figure 3. The main theory of our paper is that for stable regions, the average parameter distance for iterative retraining to $\theta_0$ remains small for small values of $\lambda$.

Git-rebasin

We thank the reviewers for their insightful suggestions. Because of the multitude of local minima, we also think that git-rebasin-like is the way to compare the convergence in the parameter space properly. Unfortunately, the current public implementations are model-specific (e.g., specific to a ResNet 100) and do not work directly for our models. This would require significant additional time to adapt git re-basin for our architectures.

Thank you again for your suggestions and prompt response. We hope that the additional Figure (Fig. 7 in Appendix G.5) provides the desired level of clarity and information for the convergence in the parameter space, and convincingly answers any lingering doubts. We appreciate your time and effort in this rebuttal period and hope that our answers are exhaustive enough to convince you to upgrade your score.

Experimental Validation

We concur with the reviewer that our experimental validation of our theoretical results can be improved, especially since stability is guaranteed in the infinite iterative training regime. To more convincingly demonstrate this claim we have updated our previous Figure 3 to go from $5 \to 20$ iterative retraining steps and more epochs per retraining, from $5/10 \to 20$ for the SOTA EDM model on CIFAR and FFHQ. In addition, we have also added another run to this Fig 3 plot which uses $*only synthetic*$---i.e. $\lambda=0$ samples. This allows us to see a very clear signal where we can now experimentally see a clear convergence towards a stable regime in ALL metrics (FID, Precision, Recall). This is in stark contrast to the fully synthetic line in the Fig 3 plot which exhibits a linear rate of degradation. We hope this update to Fig 3. helps alleviate the reviewer's concern regarding the extrapolation behavior for $\lambda > 0$.

We value the reviewer's opinion that providing a quantification of uncertainty in the vein of standard deviation would improve the empirical caliber of the results. Unfortunately, for the image experiments, this is computationally challenging. For example, the bottom row of Figure 3 (FFHQ – EDM with 20 fine-tuning epochs and 20 retraining) takes one full week of training on 40 RTX8000 GPUs. Running this experiment multiple times and for multiple models and datasets is currently beyond our computational budget given the stringent time constraints of the rebuttal period. However, we still wish to take the reviewer's suggestions into consideration and we have updated our synthetic experiments. Specifically, we included quantitative metrics such as the Wasserstein distance between the generated samples and samples from $\hat{p}_{\text{data}}$ for multiple values of $\lambda$ and also corresponding standard deviations in this setting. We hope this allows the reviewer to be more convinced by our overall experiments and how they substantiate our theory.

Over Simplifying Assumptions

Assumptions 1 and 2 (local Lipschitzness + strong convexity) “We basically assume convexity of the loss function through assumptions 1 and 2”. We would like to highlight an important point: assumptions 1 and 2 imply local Lipschiptness and local strong convexity, which is much weaker than global Lipschitness and strong convexity, which would be indeed much stronger and unrealistic. To the best of our knowledge, local Lipschitness and strong convexity are among the mildest assumptions required for understanding the local landscape of neural networks in terms of optimization [1,3] or generalization [2]. We would be eager to consider other sets of assumptions if the reviewer can point us toward such works. The question on Assumptions 3 (generalization error) is answered as a joint response to reviewers

[1] Nesterov, Y., & Polyak, B. T. Cubic regularization of Newton method and its global performance. Mathematical Programming, 2006.

[2] Keskar, N.et al. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017

[3] Jin, Chi, et al. "How to escape saddle points efficiently." ICML. PMLR, 2017.

Minor points

We are grateful to the reviewer for pointing out the small typos and mistakes in our manuscript. We have fixed this in our updated rebuttal PDF. The PDF size of the figures has been reduced, and the PDF should render smoothly now.

We thank the reviewer again for their valuable feedback and great questions. We hope that our rebuttal addresses their questions and concerns and we kindly ask the reviewer to consider upgrading their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise.

We would like to thank the reviewer for their time and constructive feedback. We appreciate the fact that the reviewer felt our paper was “well structured” in how simple cases are presented first to build intuition before more complex and practical examples. Moreover, we are encouraged to hear that the reviewer considers the topic of our paper an “interesting and timely problem”. Finally, we thank the reviewer for highlighting the “technical rigor” of our work. We now address the key clarification points raised by the reviewer.

Improving the clarity of notation:

We acknowledge the reviewer’s comments regarding the comprehensibility of our mathematical notation which we now clarify. $\succeq$ operator for matrices “I suppose it is the usual condition for $\Sigma$ to be positive definite and not that each individual element of sigma should be larger than zero.” Yes, this is exactly what we mean, positivity in the matrix, sense, i.e., $A \geq 0$ if and only if for all $x$, $x^\top A x \succeq 0$. In other words, the matrix $A$ is positive semidefinite. “On the other hand, the authors write $H_f \leq - \alpha I_d$ , comparing two matrices. suppose that it now constrains each element. Can the authors please clarify?” This choice of notation here does not constrain each element, but rather $\leq$ in the matrix sense, i.e., $A \preceq B$ if and only if $A - B \preceq 0$. This notation is sometimes referred to as Loewner order (https://en.wikipedia.org/wiki/Loewner_order), the reader can also refer to Example 2.15 of Section 2.4.1 of the textbook [1] for more details. We believed this was conventional notation when working with matrices, but understand how this was not immediately obvious. This clarification has been added in the notation section to improve readability.

Description of Precision and Recall

We answer this question in detail in our global response to all reviewers. In summary, the precision/recall we use here differs from those used for evaluating a classification model. Intuitively, at the distribution level, precision and recall measure how the training and the generated distributions overlap. Precision can be seen as a measure of what proportion of the generated samples are contained in the “support” of the training distribution. On the other hand, recall measures what proportion of the training samples are contained in the “support” of the generated distribution. If the training distribution and the generated distribution are the same, then precision and recall should be perfect. If there is no overlap—i.e. disjoint between the training distribution and the generated distribution, then one should have zero precision and zero recall More details and references are provided in the joint answer to reviewers and in Appendix G.2. Feel free to ask any clarification questions if needed.

Improving the clarity of the proofs. We agree with the reviewer that some of our proofs could benefit from additional clarity to ease the verification of their claims. To improve readability the proofs in contention as highlighted in the review have been encapsulated into simpler lemmas, with additional intermediate steps, which we believe are much easier to follow. The key changes with respect to the initial submission have been highlighted with blue text in the updated PDF. More precisely, Lemma A.1 and Lemma A.2 in Appendix A.1 and A.2 should now provide detailed guidance on the proofs in the Gaussian case. We thank the reviewer again for helping us strengthen this aspect of our submission and are we more than happy to add anything else to these proofs should the reviewer deem it necessary. [1] Boyd, S.P. and Vandenberghe, L., 2004. Convex optimization. Cambridge University Press.

[2] M. S.-M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. Neurips 2018

[3] Kynkäänniemi, Tuomas, et al. "Improved precision and recall metric for assessing generative models." Advances in Neural Information Processing Systems 32 (2019).

I thank the authors for their detailed response. Here are detailed follow-ups to the points raised:

Clarity: In thank the authors for improving the clarity of the theory part in the revision and I increased my ratings for Presentation and Soundness. In particular, I now see how the operator $\succeq$ is to be interpreted. I also understandand how the precision / recall metric works. I checked the Proofs of Lemma A.1. and Lemma A.2 again and found that they are better to follow now. I still think that it could be more explicitly stated that the expression involving the $\Gamma$-function in Eqn. (26) comes from the analytical solution of the mean of the chi distribution, but the proof seems solid now overall. I am still mildly confused why there is an $\mathbb{E}$ around $\Sigma_0$ in the theorem. Is $\Sigma_0$ initialized randomly? As I read Proposition 1, $\Sigma_0$ is an arbitrary, but constant value?

Assumptions: As pointed out by the authors, the theoretical analysis in this paper is concerned with strongly convex regions of the loss landscape. I think this should be explicitly stated in the paper, as the Assumptions 1+2 appear very technical, but their main effect seems to be the constraint of convexity in the region around the optimum, that is considered. Do the authors have any evidence that it is possible to find an initialization in the convex region around the optimum in practice?

Experimental Evaluation: Thank you for updating the experiment to feature 20 retraining steps (which is still far from the infite regime but should still help to solidify the evaluation). The plot of the fully-synthetic regime also helps. However, my main concern still subsists. The experiments seem to show a smooth interpolation between the divergent regime with all-synthetic data and the stable case with no synthetic data. Nevertheless, while training stays stable for more iterations when more of the original data is re-used, the infinite regime is considered in the paper. This could be similar to adding two sequences, a divergent (that can even be multiplied with a very small $\lambda$) and a non-divergent one, which in summary would still be divergent. To me, the empirical evaluations in Fig. 3 still don't strongly support the theoretical claims of the paper that there is some value for $\lambda > 0$, for which training will also be stable in the infinite regime. What are the authors' thoughts on this?

Thank you again for your detailed review and comments. We believe we have implemented all of your great suggestions in the revised manuscript. In particular,

• The notation and proofs should now be more straightforward (Page 3, 13, and 14),
• Figure 3 displays now more epochs and more retraining, which should be more compelling, clearly illustrating Theorems 1 and 2.
• Precision and recall are now carefully explained.
• Details and comments on each assumption have been provided (Assumption 3 in the global response and assumptions 1,2 in the individual answer).

As the end of the discussion period is fast approaching, we would love the opportunity to engage with you if there are any remaining questions (in particular, concerning the clarity of our proofs and assumptions). Otherwise, we politely encourage you to consider upgrading your score. Thank you again for your time and energy in reviewing our paper.

Thanks for the clarifications. I think the proof of Proposition 1 has been solidified now. In case of acceptance, I encourage the authors to carefully go through all proofs again, check their accessiblity and potentially add necessary intermediate steps.

The results in Figure 3 appear like this to me: Suppose there is a divergent sequence $(a_n), {n \in \mathbb{N}}$ and a convergent sequence $(b_n), {n \in \mathbb{N}}$. We can now generate linear interpolations between these sequences as $c_n := \lambda a_n + (1-\lambda)b_n$. For every $\lambda > 0$ the resulting sequence will diverge.

Looking at the plots, I have the impression that something similar is happening there. I would have expected a regime change for some $\lambda$, where training then becomes stable. However, it now seems like we are just interpolating between the regimes, which, in the sequence example, would always result in unstability. Although I am fully aware that the example is more complex here, I am not entirely convinced that something similar to the interpolation between sequences would not be happening if we perform more retraining iterations of the generative models. In the sequence example, we could also choose a very small $\lambda$, and divergence would not be visible after 20 steps.

Overall, I think the paper's theory part is sound now, but relies on some strong assumptions which are (1) everything on the training path from initialization to the optimum being contained in the convex region around the optimum, and (2) the models being perfectly able to fit the distribution (i.e., being universal function approximators). The empirical results in the main paper have not entirely convinced me that these assumptions hold in practice and the convergence trend is hard to see due to statistical noise in these results, which unfortunately feature no measures of dispersity (e.g., std. errors).

If these assumptions were clearly conveyed in the paper, I suppose we could leave it up to the reader to judge whether the theory applies to their model or not. As this has been done now, I am still not championing this paper but I have no major reservations either.

We are thinking of investigating other sampling procedures for the generated samples for future work. A way to do it would be to score each sample, using for instance reinforcement learning techniques. It has been explored heuristically and provides improvement of the generative models on language tasks [1]. One other way to score generated samples could be to do conditional generation, use a classifier, and look at the last layer (after the softmax). This would give a score to each sample. Then, one can each filter the examples or weigh them according to their score, as done in very recent works [2]. As mentioned by the reviewer, an additional challenge in this task is to select diverse samples, which are not copied from the training set. To the best of our knowledge, this is currently an active area of research, and are currently investigating it.

This is an excellent question! There are numerous possible ways to change the inference procedure for generative models such as diffusion and flow-matching. Common examples include classifier-based and classifier-free guidance that achieves low-temperature sampling to enhance sample fidelity. More generally, this type of sampling falls into the larger goal of conditional generation which is beyond the scope of our presented theory as we are concerned with stability with respect to $p_{\text{data}}$. Another exciting direction for sampling is to score each sample, using for instance reinforcement learning techniques. This has been explored in the context of language modeling tasks [1] and has shown tangible improvements in domain-specific metrics. This direction and developing appropriate theory remain exciting directions for future research and as such are not contained within our iterative retraining framework.

We thank the reviewer again for their time and valuable feedback. We hope our rebuttal succeeded in addressing all the salient points of the review and we kindly request the reviewer to consider upgrading their score if they so deem it. We are also available to answer any further questions that the reviewer may have.

[1] Gulcehre, C., Paine, T.L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C. and Macherey, W., 2023. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998.

[2] Hemmat, R.A., Pezeshki, M., Bordes, F., Drozdzal, M. and Romero-Soriano, A., 2023. Feedback-guided Data Synthesis for Imbalanced Classification. arXiv preprint arXiv:2310.00158.

We thank the reviewer for their time and positive appraisal of our work. We are thrilled that the reviewer viewed our work as “one of the best-written theoretical papers I’ve ever read”. Moreover, we are glad that the reviewer finds our paper “well-motivated”, and tackles an “interesting problem” in which our paper “sheds a new light into the topic of continual retraining of generative models”. We wholeheartedly agree with the reviewer that our investigation in this paper has the potential to “impact” neighboring fields such as the continual learning of generative models and also tangibly impact the practical aspects of deploying large generative models. We now address the main questions raised by the reviewer.

Assumptions used in theory “setups with normally hard or impossible to achieve assumptions (e.g. an infinite number of rehearsal samples generated by the model” We acknowledge that the infinite number of rehearsal samples is infeasible in practice,, however, we would like to point out that we provide results with a finite number of data (Theorem 2), provided that the generative models learn “well enough” (Assumption 3) the true distribution.

Empirical evaluation of simpler models Quantitative metrics have been added for the synthetic experiments (see Figure 6 in Appendix G.4). The Wasserstein-$1$ and $2$ distances between the true data distribution and the generated one is displayed as a function of the number of retraining. The distances are averaged over $50$ runs, the line corresponds to the mean, and the shaded area to the standard deviation. When the model is fully retrained only on its own data (Fully Synthetic, dashed red line with triangles), the distance to the true data distribution diverges. When the model is retrained on a mixture of real and synthetic data ($\lambda=0$, $\lambda=0.5$, $\lambda=1$), then distance between the generated samples and the true data distribution converges.

Evaluation over seeds We acknowledge the reviewer's comment regarding the number of seeds used in our iterative retraining experiments for natural image datasets. We first highlight that we have improved this experiment in the updated version of the manuscript but substantially increased the number of iterative retraining steps (5 vs 20) while adding a fully synthetic baseline ($\lambda = \infty$). In our plots, we see that the margin of difference between the fully synthetic case and the more stable regime of $\lambda > 0$ of iterative retraining is quite dramatic. While the reviewer has requested more random seeds, we wish to politely push back due to the high computational cost of running each experiment. For example, just for one curve of FFHQ-64, conducting iterative retraining using the SOTA EDM model amounts to a full week of experimentation using 4 RTW8000 GPUs. This yields to $40$ GPUs over one week for Figure 3 itself. Unfortunately, running additional random seeds for all models and datasets is beyond the computational budget within the period of this rebuttal, and even beyond our academic resources. However, to promote reproducible research we commit to releasing the full training code of all our experiments such that one can readily verify the stability and collapse property of iterative retraining.

We would like to thank the reviewer again for their enthusiastic and constructive comments, which gave us the opportunity to improve our work significantly. In particular, quantitative synthetic experiments with confidence intervals have been added in Figure 6, in Appendix G.4, corroborating Theorems 1 and 2. Figure 3 displays now more epochs and more retraining, which showcases stronger evidence of stability.

We want to thank Reviewer pzxF for their feedback. We are glad that Reviewer pzxF found our paper to be “clearly presented”, with our theory bringing “clear answers” to the iterative retraining problem. We also appreciate that the reviewer found our experiments to be conclusive. We now address the specific comments raised by the reviewer below.

We note that each of the reviewers questions has also been addressed in detail in the global response. Here we summarize these points but kindly ask the reviewer to also read our global response to all reviewers.

• Assumption 3 implies sample complexity generalization bounds on the parameter for generative models. From a theoretical perspective, numerous works have proved sample complexity bound on the distribution, more precisely universal density approximation and function approximation capabilities of model classes like diffusion models, affine coupling and residual normalizing flows.
• Precision and Recall Intuitively, at the distribution level, precision and recall measure how the training and the generated distributions overlap. Precision can be seen as a measure of what proportion of the generated samples are contained in the “support” of the training distribution. On the other hand, recall measures what proportion of the training samples are contained in the “support” of the generated distribution. If the training distribution and the generated distribution are the same, then precision and recall should be perfect. If there is no overlap—i.e. disjoint between the training distribution and the generated distribution, then one should have zero precision and zero recall More details and references are provided in the joint answer to reviewers and in Appendix G.2. Feel free to ask any clarification questions if needed.

We hope that our responses here in conjunction with global response help answer the great questions raised by the reviewer. We politely encourage the reviewer to continue asking more questions or if possible consider a fresher evaluation of our paper with a potential score upgrade.

We would like to thank you again for your positive and constructive comments. The global response should provide clarification on Assumption 3, precision, and recall. In addition, the updated experiments in Figure 3 now strengthen our contribution and corroborate better with our theory (Theorems 1 and 2). We appreciate your time and effort in this rebuttal period and hope our answers are detailed enough to convince you to upgrade your score.

We acknowledge the reviewers's comments regarding the number of seeds used in our iterative retraining experiments for natural image datasets. We first highlight that we have improved this experiment in the updated version of the manuscript but substantially increased the number of iterative retraining steps ($5$ vs $20$) while adding a fully synthetic baseline ($\lambda = \infty$). In our plots, we see that the margin of difference between the fully synthetic case and the more stable regime of $\lambda > 0$ of iterative retraining is quite dramatic. While the reviewer has requested more random seeds, we wish to politely push back due to the high computational cost of running each experiment. For example, just for FFHQ-64, conducting iterative retraining using the SOTA EDM model amounts to a full week of experimentation using 40 RTX8000 GPUs. Unfortunately, running additional random seeds for all models and datasets is beyond the computational budget within the period of this rebuttal. However, to promote reproducible research we commit to releasing the full training code of all our experiments such that one can readily verify the stability and collapse property of iterative retraining. In addition, as required by reviewer bfKK, we have added quantitative measurements for the experiments on the 2D synthetic data and were able to repeat this experience multiple times and provide statistics such as confidence intervals and error bars.

Empirical evaluation of toy models

We thank the reviewer for their great suggestion. We have now included a plot of the Wasserstein 1 and Wasserstein 2 distance between the generated samples and the ground truth dataset as a function of the number of retraining steps (Figure 6 in Appendix G.4). These quantitative results add further support to the observed visual phenomena of collapse and stability highlighted in Fig 1 and 2.

We acknowledge the reviewer's healthy skepticism on the applicability of Assumption 3 in practice. Assumption 3 implies the existence of sample complexity bounds (on the parameters) on the considered generative models. From a theoretical perspective, there are numerous works that prove sample complexity bound on the distribution, more precisely universal density approximation capabilities of model classes like diffusion models (Oko et. al 2023), and affine coupling normalizing flows ([5], Teshima et. al 2020). Universal function approximation capabilities were also proven for residual flows ([6], Zhang et. al 2020). We would like to note that all of our experimental results use diffusion models or normalizing flows and thus from a theoretical perspective is in line with existing theory.

From a practical point of view, our paper is motivated by the existence of web-scale models that are trained on the Internet. Examples of these models include Dalle-3, MidJourney [8], and StableDiffusion [9]. We argue that it is hard to debate the impressive power of these models, especially their ability to handle potentially never-before-seen prompts. That is the generalization ability of these models as evidenced by their sample quality reduces the concern that diffusion—and more generally likelihood-based generative models—given sufficient, data, and can capture the natural image manifold. Consequently, we believe that while Assumption 3 is needed for our theory, we can reasonably expect to satisfy it in practice when working with large foundation models that consume web-scale data.

[4] K. Oko, S. Akiyama, and T. Suzuki. Diffusion models are minimax optimal distribution estimators. arXiv preprint arXiv:2303.01861, 2023.

[5] T. Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems, 33:3362–3373, 2020.

[6] Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, pp. 11086–11095. PMLR, 2020.

[7] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. PMLR, 2021.

[8] Midjourney. https://www.midjourney.com/home/, 2023. Accessed: 2023-09-09.

[9] Stability AI. https://stability.ai/stablediffusion, 2023. Accessed: 2023-09-09.

We thank the reviewer for their question regarding the exact computation of the Precision/Recall metric used for evaluation.

We first highlight the notion of precision and recall used for generative models is different from the one used in standard supervised learning. It was first introduced in “Assessing Generative Models via Precision and Recall” [1] and refined in “Improved Precision and Recall Metric for Assessing Generative Models” [2], which we use in practice. Intuitively, at the distribution level, precision and recall measure how the training and the generated distributions overlap. Precision can be seen as a measure of what proportion of the generated samples are contained in the “support” of the training distribution. On the other hand, recall measures what proportion of the training samples are contained in the “support” of the generated distribution. If the training distribution and the generated distribution are the same, then precision and recall should be perfect. If there is no overlap—i.e. disjoint between the training distribution and the generated distribution, then one should have zero precision and zero recall.

In order to adapt these metrics for empirical distributions that have finite samples customary of training deep generative models, multiple numerical tricks are required. Specifically, for an empirical set of samples, the support of the associated distribution is approximated by taking the union of balls centered at each point with radius determined by the distance to the k-th nearest neighbor. This is generally done in some meaningful feature space (here we use the Inception network [3]). In order for our work to be self-contained, the remaining details on the precision and recall are now added in a new Appendix for additional clarity).

​​We appreciate the insightful feedback and excellent inquiries from the reviewer. Our rebuttal aims to comprehensively address the key issues highlighted. Should our answers meet your expectations, we would be grateful if you could reconsider and potentially upgrade your evaluation. Please do let us know.

[1] M. S.-M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. Neurips 2018

[2] Kynkäänniemi, Tuomas, et al. "Improved precision and recall metric for assessing generative models." Advances in Neural Information Processing Systems 32 (2019).

[3] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two-time-scale update rule converge to a local Nash equilibrium. NeurIPS 2017.