Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences

We thank the reviewer for their constructive comments and are pleased that they find the question raised "highly original" and "very clearly" motivated. We further appreciate that they find it "significant" and "more realistic" than previous works. We now address the key clarification points raised by the reviewer.

Presentation of statements of the theorems

We acknowledge the reviewer's concern that the dissemination of all of our assumptions may not have been sufficiently clear. Below, we present updated assumptions and theorems that we hope address the clarity issue.

We propose to restructure assumption 2.1 to sub-assumptions of increasing strength:

Assump. 2.1-A: The distribution $p \in \mathcal{P}(\mathbb{R}^d)$ has a density w.r.t. Lebesgue measure and $E_p[e^{r(x)}] < \infty$.

Assump. 2.1-B: The distribution $p\in \mathcal{P}(\mathbb{R}^d)$ has a density w.r.t. Lebesgue measure and there exists $r_* \in \mathbb{R}$ such that: (a) $p$-almost surely, $r(x)\leq r_*$ and (b) $p$ puts positive mass in a neighborhood of $r_*$ i.e., $\forall \epsilon > 0, P_0(r(x)\geq r_* - \epsilon) > 0$.

Assump. 2.1-C: Assum. 2.1.B and $P(r(x) = r^*) > 0$.

Using these assumptions, our results of section $2$ will have the following overheads:

Thm. 2.1: Let for all $t\geq 0$, $p_{t+1}$ be the distribution induced from a discrete choice model on $p_t$ (4) where $\mathcal{P}= \mathcal{P}(\mathbb{R}^d)$ is the set of probability distributions on $\mathbb{R}^d$. If $p_0$ satisfies Assump 2.1-C, then we can define $p_*(x):=\frac{p_0(x)1_{r(x)=r_*}}{P_0(r(x)=r_*)}$ and the self-consuming loop on curated samples $p_t$ converges to $p_*$: $KL(p_*||p_t) \xrightarrow{t\rightarrow \infty} 0$

The other results have been updated similarly. If the reviewer would like to see the other exact statements, we can share them in another response.

Contextualization of our results in the literature on model collapse

We agree with the reviewer that better contextualization of our results on whether the retraining is performed from scratch or via fine-tuning can improve the clarity. We will update the paper to clarify this and present below how we would proceed.

On retraining from scratch vs iterative fine-tuning:

1.) Experiments: All retraining step in experiments on mixture of Gaussians (MoG) and two moons are performed from scratch, whereas in the case of CIFAR dataset, due to the high compute cost of retraining the model from scratch (20 hours on an A100 GPU) we performed fine-tuning at each step. Fine-tuning is always performed on $10^6$ images. We use the same amount of images for fair comparison between different proportions of real data injected. In contrast, in [1], the collapse is shown when the model is retrained from scratch at each iteration. In [3], the experiments are performed using retraining from scratch for VAEs and GMMs and sequential fine-tuning for LLMs. In [2], toy experiments on two moons and MoG are performed by retraining from scratch, while experiments on CIFAR10 and FFHQ are performed using iterative fine-tuning. We agree with the reviewer's remark that stability using real data in the setting of iterative fine-tuning is easier to obtain than when retraining from scratch since the model parameters are initialized around a good potential set of parameters. However, we would also like to point out that model collapse occurs also in the setting of iterative fine-tuning as shown in Figure 2 of [2] (red curves).

2.)Theory: Finally regarding our theoretical results, only thm 2.2 happens in the setting of iterative fine-tuning (since it uses the same setting as in [2]). However, all our other results and in particular thm 2.1, 2.3, 2.4 do not explicitly assume a special learning algorithm in parameter space. Instead, we consider having a perfect learning model and consider the evolution of the expected reward for such a learning model. In that sense, it applies to learning from scratch with the additional assumption that the model attained perfectly fits the curated distribution. We will make this point clear in the updated draft.

On fresh real vs fixed real data:

[1] studies the self-consuming loop in three different settings where the model is retrained a) only on synthetic data b) on a mixture of synthetic data and a fixed set of real data samples b) on synthetic data and a fresh set of real data samples at each step. In setting (a), the retraining loop collapses. In (b), it collapses, too, but with some delay related to the amount of fixed real data. In (c) the retraining loop does not degrade performances provided there is enough fresh real data at each step. [2] proved stability in setting (b) under some theoretical assumptions and in the iterative finetuning framework. Comparatively, our experiments on MoG are performed using fresh real data at each step while the CIFAR experiments are performed in the fixed real data framework.

Experiment clarification:

We thank the reviewer for highlighting a relevant point on the presence of multiple spots with different densities of fig. 4 at iteration 4. These are remaining errors due to the non-fully convergence. To improve clarity, we ran the experiment for more iterations (global response fig 3) in which case it is clearer that the final distribution is a unique renormalized Gaussian distribution restricted to the ball centered at $x_*$ of radius $r_{min}$.

If the reviewer finds our updates agreeable, we would appreciate it if the reviewer considers championing/upgrading their score for this paper, as mentioned in the original review. We are also happy to answer any further questions that the reviewer may have.

[1] Alemohammad et al Self-consuming generative models go mad. ICLR 2024

[2] Bertrand et. al On the Stability of Iterative Retraining of Generative Models on their own Data. ICLR 2024

[3] Shumailov et al. The curse of recursion: Training on generated data makes models forget. Nature 2024

I would like to thank the authors for the thoughtful reply to my review.

In particular, I appreciate the additional contextualization of works [1] and [2]; I think that the literature would greatly benefit from having that explanation from the section "On fresh real vs fixed real data". I also appreciate the more clear statement of Theorem 2.1.

Would it be possible to share here the other exact statements as well--namely, how the authors would propose to update the other exact statements of the other theorems, 2.2, 2.3, 2.4--in another response?

I have updated my score to above the acceptance threshold, with the expectation that the authors will update the camera-ready version's main result statements with more clearly stated hypotheses and clearly referenced terms, similar to the rebuttal above. Although my concern was mainly with Theorem 2.1 (it contained the least context out of all the other theorems), in my opinion, Theorems 2.2, 2.3, 2.4 should each be improved via clearer context, and I would want to see the updated statements if possible.

I strongly believe that having these statements stated that clearly would better allow the community to benefit from the authors' work.

We thank the reviewer for their time and constructive comments. We appreciate that the reviewer finds the problem we tackle "important and interesting" and that our theory in connection to preference optimization “interesting and reasonable”. We now address the key points raised in the review:

Plackett-Luce model

We thank the reviewer for referring us to the Plackett Luce model. We would like to mention that we were aware of the Luce choice rule model that we cited on lines 87-90. We will change our notation from $BT$ to $PL$ referring to the Plackett-Luce model since it provides a more unified framework consistent with the literature.

Normalization of $p_t$

We agree with the reviewer on the fact that if $p_t$ is not normalized, then $p_{t+1}$ is not normalized either. However we do not foresee a problem since in practice we apply our results to an initial normalized probability distribution. We hope this answers the reviewer’s question and are happy to clarify further.

Related Work

We are grateful to the reviewer for referring us to the two important references [1, 2] that we will add to our related work section. In [1], the authors tackle the problem of generating synthetic DNA sequences using GANs. They introduce an external function analyzer to rate synthetic samples from the generator and add the highest-scored ones into the discriminator training set. This work [1] is mostly experimental and tied to the GAN architecture while we adopt a more theoretical framework to understand the self-consuming loop without specifying any architecture.

Furthermore our study takes the point of view of the recently developed model collapse literature and relies only on $K$-wise preferences as in [2]. In [2], the authors propose a new GAN framework to incorporate user’s preferences in the training. They show state of the art results in generating the user-desired data distribution and theoretically prove the convergence of their method. The key difference to our work is that they aim to generate a diversity of samples that are desired by users and their focus is therefore not on the collapse of their method to a maximal reward set.

Closeness with [3]

We acknowledge the reviewer’s comment that our proof of theorem 2.2 is adapted from the proof of [3], since our goal was to improve their paper’s main result in the same setting. Note that despite the similarity in the proof technique the theoretical improvement we provide is significant.

Moreover, we believe that it does not consist in our main theoretical contributions, and was serving as an introduction to the rest of the section on how real data provides stability to the retraining loop. Instead, our major theoretical results, i.e. theorem 2.1, 2.3 and 2.4 are proved in a different setting and with proof techniques that completely differ from [3].

Clarification of the $p_{ref}$ notation

We acknowledge the reviewer's concern about the notation $p_{ref}$ and we will clarify this notation in the updated draft. Namely, we denote $p_{ref}$ any probability distribution with density with respect to Lebesgue measure. It may indeed be of interest to apply thm 2.3 to other cases than retraining on a mixture of synthetic data at iteration $t$ and the data distribution. For example, it may be of interest to retrain on a mixture of synthetic data at iteration $t$ and of the synthetic data at initialization. In such a case, $p_{ref}$ would be identified to $p_0$. It bridges the gap with RLHF, as the KL regularization in the RLHF objective is done with respect to the supervised fine-tuned policy (see equation 3 in the DPO paper [5]). This makes our result more general as we show that stability occurs around any reference distribution that is injected in the retraining loop (either real data samples or samples from the initial model). It also provides an avenue to study the retraining process over a mixture of all previous iterations of the generative models.

Clarification on the assumption on $\lambda$

We respectfully disagree with the reviewer that our assumptions $L\epsilon < \alpha \quad \text{and} \quad \lambda < \frac{\alpha}{2L\epsilon}$ are in conflict with the claim that previous work was restricted to $\lambda < \frac{1}{2}$. Indeed, the necessary conditions for theorem 1 of [3] are $\lambda \leq \frac{1+ \frac{L\epsilon}{\alpha}}{2}\quad \text{and}\quad L\epsilon< \alpha$ which necessarily requires $\lambda < \frac{1}{2}$. These conditions are more restrictive than ours. In particular, we see that for fixed $\alpha$, if $L\epsilon\rightarrow 0$ then both our conditions are satisfied (in particular $\lambda$ bigger than $\frac{1}{2}$ will be allowed).

Clarification of $r_*$ notation

We agree upon the reviewer's remark that our definition of $r_*$ could be clarified and we will improve this paragraph in the updated paper. In a nutshell, $r_*$ should be thought as the smallest number that upper-bounds the random variable $p_0$ with probability $1$. For example, a Uniform distribution on the interval $[0, 10]$, $r_*=10$ whereas for unbounded distributions such as $\mathcal{N}(0, 1)$, $r_*$ does not exist. As suggested by the reviewer, an equation that could clarify this is $r_* = inf \\{r \in \mathbb{R}, \mathbb{P}(r(x)\leq r_*) = 1\\}$.

We thank the reviewer for their valuable feedback and great questions. We believe we have answered to the best of our ability all the great questions raised by the reviewer and we kindly ask the reviewer to potentially upgrade their score if they are satisfied with our responses. We are also more than happy to answer any further questions that arise.

[4] Gerstgrasser, Matthias, et al. "Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data." arXiv 2024 [5] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." NeurIPS 2023

We thank the reviewer for their nuanced review and constructive feedback. We appreciate that the reviewer has found our paper "well-written" and the integration of preference learning to the model collapse literature to be "highly sensible". We took note of the reviewer's concerns and provide clarifications:

Accounting for the accumulation of data

We thank the reviewer for referring us to the paper [1] and pointing out this interesting extension. We believe, together with the reviewer, that the setting of [1] could be adapted to show that accumulating data provides additional stability to the retraining loop and avoids collapse. However, we believe such a study is out of the scope of our work whose aim was to introduce and theoretically develop a new research question of model collapse from the view of preferences. We will, therefore, update our draft to provide additional clarification of this aspect and mention it as an exciting future direction.

Clarification on Thm 2.1

We understand the reviewer's natural concern. The crucial point is that the curated distribution at time $t+1$, $p_{t+1}$ is issued from learning a modified distribution from $p_t$ constructed by sampling from $p_t$ and curating samples using preferences. This implies that if a set has probability $0$ for $p_t$, it will never be sampled and therefore never be preferred over other samples. This means that this set will also have probability $0$ for $p_{t+1}$.

Finally, we note that the reviewer’s intuition is valid when the retraining step is not perfect (for example due to bounded expressivity of the class $\mathcal{P}$ involved in equation 4), or when noise is injected in the process. Then the support of $p_{t+1}$ is not necessarily included in the support of $p_t$ anymore. In that case, the self-consuming loop iteratively explores regions that had probability $0$ at initialization and converges to the maximal reward possible, validating the reviewer’s intuition.

Improvement of the experiments on MoGs

We thank the reviewer for their suggestion which will help clarify the theoretical results. We provided in the additional pdf for figures a heat map of the level set of the reward when using $4$ centroïds (Fig 1a). We additionally provided a heat map of the raw mixture of Gaussians (MoG) distribution (Fig 1b) and the corresponding limit distribution defined L154, as the renormalized MoG restricted to the set of maximal reward at initialization (Fig 1c). We also re-ran our experiments for this reward model and observed that the learned distribution converges as expected to $4$ Gaussian restricted to $4$ balls around the designed centroïds (Fig 2a). We additionally plotted the reward variance and showed that it vanishes in the purely synthetic setting. This demonstrates that the reward variance can vanish independently of the overall distribution variance. We will clarify this in the updated manuscript as it is central to our analysis.

Clarification of Thm 2.3, 2.4

The goal of these two theorems is to respectively provide a lower bound on the expected reward and an upper bound on the KL divergence of a self-consuming generative model loop with respect to a reference distribution $p_{ref}$. These theorems formally connect our results to the KL regularization in RHLF ensuring that the KL divergence between the aligned model and the reference model is not too large. This is why only $p_{ref}$ and not $p_t$ appear on the right-hand side. The reviewer’s observation is, however, insightful as our proof proceeds by an induction argument in which each induction step uses Lemma A.1 (see line 563) and hence a right-hand side dependent on $t$. We finally achieve to make this term not dependent on $t$ by induction.

Related work

We thank the reviewer for pointing us to these important references that we will cite and discuss in our revision. We will especially provide more discussion on the relationship between RLHF and rejection sampling fine-tuning. We now discuss the most relevant works mentioned by the reviewer (because of the space constraint we do not have space to discuss them all extensively here): Note that [2,3] are concurrent work according to NeurIPS guidelines:

papers that appeared online within two months of a submission will generally be considered "contemporaneous" The key difference between [2] and our work is that they show the benefit of using feedback on the quality of a sample to prevent model collapse.

In [3], the authors investigate how the different stages of alignment affect a model’s generalization capabilities and output diversity. They empirically show that the output diversity of the RLHF policy is decreased w.r.t. the supervised finetuned policy, which is consistent with our theoretical insights (e.g. lem. 2.2, thm 2.1). However, there are major differences with our setting as their contribution is empirical, and we investigate the impact of iteratively retraining a model several times on synthetic samples while they study a single training round using RLHF. [4] experimentally investigates the self-consuming loop, specifically in the case of LLMs, and evidences model collapse in that setting.

We thank the reviewer again for their review and detailed comments that helped strengthen the paper. We hope our answer here and in the global response allows the reviewer to consider potentially upgrading their score if they see fit. We are also more than happy to answer any further questions.

[1] Gerstgrasser, Matthias, et al "Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data" 2024

[2] Feng, Yunzhen, et al "Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement" 2024

[3] Kirk, Robert, et al "Understanding the effects of RLHF on LLM generalisation and diversity" 2023

[4] Briesch, Martin, et al "Large language models suffer from their own output: An analysis of the self-consuming training loop" 2023

We thank the reviewer for referring us to the paper [1] and pointing out this interesting extension. We believe, together with the reviewer, that the setting of [1] could be adapted to show that accumulating data provides additional stability to the retraining loop and avoids collapse. However, we believe such a study is out of the scope of our work whose aim was to introduce and theoretically develop a new research question of model collapse from the view of preferences. We will, therefore, update our draft to provide additional clarification of this aspect and mention it as an exciting future direction.

I feel like the authors and I miscommunicated here. The point I was trying to raise is: what are realistic assumptions to make about how model-data feedback loops should be modeled? I wasn't so much interested in that other paper as much as I was interested in whether this paper faithfully captures the settings we care about ("I do not think that Equation (3) faithfully models reality. Specifically, it assumes that the -th model is fit to a mixture of (1) real data and (2) preference-filtered data from the -th model. But realistically, synthetic data should amass over time, as I believe is the case with the datasets that the authors mention in their introduction, e.g., LAION-5B.")

My thinking is that the assumptions of this paper are not especially realistic because synthetic data should increase over time and the total amount of data should increase over time too.

TLDR: I think your assumptions are not realistic and I think this harms the significance & relevance of your paper.

If I missed the response to this point by the authors, I apologize and I would appreciate being pointed in the correct direction. Thank you!

Thank you to the authors for their response!

Improvement of the experiments on MoGs

We provided in the additional pdf for figures a heat map of the level set of the reward when using centroïds (Fig 1a).

This is wonderful. Thank you for running these additional experiments - I really like them, and I think they'll improve the paper (in my opinion; if you disagree, you don't need to include them).

Note that [2,3] are concurrent work according to NeurIPS guidelines:

Yes, then it's fine to not cite them. I tried to allude to that above ("Some may be concurrent with yours") but apparently didn't finish the sentence. I appreciate your care to the other citations.

We understand the reviewer's natural concern. The crucial point is that the curated distribution at time $t+1$, $p_{t+1}$ is issued from learning a modified distribution from $p_t$ constructed by sampling from $p_t$ and curating samples using preferences. This implies that if a set has probability $0$ for $p_t$, it will never be sampled and therefore never be preferred over other samples. This means that this set will also have probability $0$ for $p_{t+1}$.

I'm not sure I buy this argument. I agree that if a set has probability $0$ for $p_t$, it will never be sampled and therefore will never be preferred over other samples. But you lose me for two reasons:

1. Depending on the choice of realizable distributions $\mathcal{P}$, the support might be the entire space of outcomes e.g., if $\mathcal{P}$ is the set of Gaussian distributions. In this case, no set would have probability $0$ for $p_t$. Your explanation then seems to hinge on a condition "if" that might not be applicable.

2. Even if the conditional statement is true i.e. there is some set with mass/density 0 under $p_t$, why is $p_{t+1}$ prohibited from extending its support to this set? In general, probabilistic models are often capable of placing mass/density on sets that were not in their training data.

Could the authors please clarify?

We are grateful to the reviewer for their time and engaging with us during this rebuttal. We answer below the reviewer’s additional questions.

On whether the support of $p_{t+1}$ is included in the support of $p_t$

We acknowledge the reviewer's comments regarding the fact that the support of $p_{t+1}$ is included in the support of $p_t$. We understand the two points provided by the reviewer as

1. Depending on the choice of realizable distributions $\mathcal{P}$, the support might be the entire space of outcomes e.g., if $\mathcal{P}$ is the set of Gaussian distributions.

Note that having that the support is the entire space of outcome is not a problem at all for our theory. Our only requirement is that the reward is bounded over the support of $p_0$ (at initialization) by $r_*$ (Assumption 2.1) which implies that the reward is bounded for all timesteps (since the support of $p_t$ is included in the support of $p_0$). There exist many functions with unbounded support that are bounded (e.g. the sigmoid function). We believe that it is reasonable to assume that the intrinsic human reward is bounded.

2. In general, probabilistic models are often capable of placing mass/density on sets that were not in their training data (because they generalize)

Our setting prevents this scenario from happening: we are working in a setting were we neglect the errors due to the finiteness of model’s capacity and training data. In other words, we work in a setting where we have:

i. infinite capacity regime: the set $\mathcal{P}$ of achievable distributions is the entire set of probability distributions (lines 112-113 in lemma 2.1). In that case, equation 6 holds and hence the support of $p_{t+1}$ is included in the support of $p_t$.

ii. infinite training data regime: This comes from the fact that we are assuming that the next distribution minimizes the population likelihood (and not the empirical one) in equations 3 and 4 ($p_{t+1}$ is defined using an expectation on the distribution $p_t$ and $p_{data}$).

Finally, i), ii) together prevent the points 1) and 2) from happening in our setting. Note that we mentioned i) explicitly in the statement of Lemma 2.1 “If $\mathcal P$ is the set of probability distributions on $\mathbb{R}^d$” and on Line 174. The point ii) was indicated by the fact that we were using expectations and not finite sum in our paper.

We acknowledge that we should have made i) and ii) appear more clearly in our paper to avoid any confusion. We will clarify and highlight our setting in the updated manuscript. Current generative models are getting larger and larger, and are trained on more and more data with more than billions of parameters and datapoints. That is why, we believe it is reasonable to neglect the errors due to the finiteness of model’s capacity and training data in our theory and that our results satisfyingly capture the reward maximization phenomenon induced by human curation.

---

edit: we updated this comment to enhance clarity of the response

We thank the reviewer for finding the problem we tackle “interesting” and to be “an accurate description of what happens when new generative models are trained nowadays”. We also appreciate that they find our experiments “interesting”. We now address below the key points raised in the review:

Extension to mixture of reward

Extending our results to hold for a mixture of rewards is a very interesting point that cannot be straightforwardly tackled in our framework. We believe it is an exciting avenue for future work and refer the reviewer to our global response for a more detailed discussion. The crucial difference with our framework is that while we showed that the learned distribution concentrates around the maximal level set of the reward, in the presence of multiple rewards it is not clear to which level set the learned distribution will concentrate.

Larger scale experiment

We acknowledge the reviewer's remark that experiments on larger scales dataset would be interesting to shed light on practical implications of such a collapse in terms of the expected reward. However due to the computational cost of such an experiment, we couldn't perform during the rebuttal period. We would however like to point out that existing studies already show how the style of samples of a generated model evolves when fine-tuned on synthetic and real data with preference optimization. For example, in [1] the authors retrain a generative model by iteratively fine-tuning using Direct Preference Optimization on a mixture of synthetic and real data and where the reward model favors real data. It is demonstrated in Figure 5 [1] that the style of large scale vision models can be shifted by retraining on synthetic data using a preference model. In particular, the model’s interpretation of “a very cute boy” drastically changes, illustrating how the reward influences the realistic biases that are picked up.

Relationship with rejection sampling fine-tuning

We thank the reviewer for encouraging us to strengthen the discussion between iterative finetuning of language models and retraining with data curation. While there is already a large literature on RLHF to iteratively finetune LLMs the reviewer highlights rejection-sampling as one way to optionally see finetuning as a sampling problem amenable to probabilistic inference. Indeed, recent works [2,3] frame iterative finetuning as drawing samples—using rejection sampling, Twisted Sequentional Monte Carlo etc…—from the unnormalized posterior distribution $p(x) \propto e^{r(x)} p_0(x)$,where $p_0(x)$ is an initial generative model trained on real data. From this perspective, our framework studies the case where we curate data by using human reward and obtain $x \sim p(x)$ which are samples from the posterior, without access to the density. This allows us to then finetune $p_0(x)$ to approximate the posterior $p(x)$, which in our notation is a step of iteratively finetuning on curated data. We will include a small discussion on this connection in our updated paper.

We thank the reviewer again for their valuable feedback and great questions. We hope our answer here and in the global response allows the reviewer to consider potentially upgrading their score if they see fit. We are also more than happy to answer any further questions that arise.

[1] H. Yuan, Z. Chen, K. Ji, and Q. Gu. Self-play fine-tuning of diffusion models for text-to-image generation, 2024.

[2] Zhao, Stephen, et al. "Probabilistic inference in language models via twisted sequential monte carlo.", 2024.

[3] Kong, Lingkai, et al. "Diffusion models as constrained samplers for optimization with unknown constraints.", 2024.

We thank the reviewer for their time and review of our paper. We are glad that the reviewer finds our paper "well-written" and our connection with RLHF to be "novel" in the "extremely important" area of work on synthetic data. We address below the key points raised by the reviewer:

Extension of the theory

Theory for conditional distribution $p(x|y)$

As remarked by the reviewer, our theoretical results focus only on an unconditional distribution $p(x)$. However we believe that our theory can be easily applied to a conditional distribution $p(x|y)$. Indeed for any sampling of $x_1, \dots, x_K$ given a prompt $y$, and a subsequent curation by humans of these samples, we can apply our theoretical results conditioned on $y$ (especially because the reward $r(x,y)$ also depends on the prompt $y$). In particular, our results state that the expected reward conditioned on $y$, i.e. $E_{p_t(x|y)}[r(x,y)]$ will be maximized and converge to a variable $r^*(y)$ where $r^*(y)$ is given as in our Assumption 2.1 (but now conditionally on $y$). Note that this is valid for any $y$, hence our framework extends to conditional generation.

Finite samples and optimization:

We agree with the reviewer that we work under the assumption of perfect learning. We believe that accounting for finite sample optimization is a crucial extension, but that it would necessitate a more involved framework beyond the scope of our paper, especially in the case of deep generative models optimization. However, we note that a starting point to such analysis could be similar to the assumption 3 made by [5] to include finite sample optimization to their framework.

Mixture of reward

We thank the reviewer for referring us to an interesting related work [1] which investigates the use of rich human feedback to improve text-to-image models. This enhanced human feedback is no longer a single scalar score but consists of multiple evaluation such as delimiting part of an image with implausibility content, or labels on prompts words that are misrepresented in the image. While we proved convergence of $p_t$ to the maximal level set in the presence of a single reward, this raises an interesting question on what dynamics would arise when the maximal level sets of each reward from the mixture are disjoints. Please refer to our global response for more details on mixtures of rewards.

Can curation prevent collapse?

The reviewer highlights an important point related to the difference between collapse of the distribution (i.e. the model only generates a single sample $x$ given $y$), and collapse of the expected reward to its maximum level set.

• Retraining with curated samples can avoid the former but not later as mentioned L71 of our paper:

Retraining with curated samples both maximizes an underlying reward whose variance collapses and converges to maximum reward regions.

• We also refer the reviewer to the new experiments of the global response, especially Figs.1 and 2 that illustrate how reward’s variance may vanish but not the overall distribution’s variance.
• However, such a collapse of the expected reward will induce a decrease of diversity as the samples with high but non-maximal reward will not be generated. This is additionally shown in our Figs. 7 where we show on CIFAR that, even if the average reward increases, the FID consistently increases.

Finally, we mention that we believe that *both the reward and the quality are related since it is expected that low-quality images would have low reward following human standards and understanding to what extent is an interesting avenue for future works. We will make sure to clarify this point in the updated draft and discuss the two important works [2, 3] mentioned by the reviewer as follows:

In [2], the authors empirically investigate the linguistic diversity of generated text in the self-consuming loop. They consistently observe a decrease in diversity across the retraining iterations. However, unlike our work, they don’t study the impact of curation. In [3], the authors compare the diversity of essays written by humans, assisted either by GPT3, a feedback-tuned LLM (InstructGPT) or without LLM assistance. They found that when using InstructGPT, the overall diversity decreased. As pointed out by the reviewer, this shows that feedback-tuned models have a diversity decrease with respect to the original distribution and that human supervision is not sufficient to compensate for it. It is consistent with our theoretical insights: we show how retraining with curation incurs a collapse of the reward’s variance similar to feedback fine-tuning.

Sec. 2.2.1

As rightfully pointed out by the reviewer, Sec. 2.2.1 revisits the framework proposed by [5] which differs from the rest of the section. We chose to incorporate this section as a preliminary to the rest of the section for the following reasons:

• It introduces the reader to how reusing real samples in the retraining loop can provide stability.
• It formulates the stability results of [5] using a Kullback-Leibler divergence term, which is new and is the same formulation as in Thm 2.4.
• It yields a significant improvement on prior work for the upper bound on the parameter $\lambda$ which [5] left as future work.

[5] Bertrand, Q., et. al. On the Stability of Iterative Retraining of Generative Models on their own Data. In ICLR 2024

We will make the setting distinction clearer in the updated draft to avoid confusion.

Related work

We thank the reviewer for this reference to [4]. We note that this work is presently cited on line 47 but are happy to provide more discussion.

We are grateful to the reviewer for their great questions and hope that our answers in conjunction with the global response clarified them. We politely encourage the reviewer to ask any further questions they may have and, if they are presently satisfied with our responses, to consider a fresher evaluation of our paper.