Self-Consuming Generative Models with Adversarially Curated Data

We thank the reviewer for their time and encouraging feedback, especially for recognizing the novelty and effectiveness of our empirical validation. We address the reviewer’s concerns below:

“The gradient-based attack methods are computationally expensive, potentially limiting their practical deployment.”

We agree with the reviewer that gradient-based adversarial curation methods can be computationally expensive, especially in large-scale or real-time settings. However, we emphasize that we were aware of this limitation and have addressed it in the paper by proposing alternative attack strategies that do not rely on gradient computations (See Section 4.2. Heuristic methods). While each approach has its own limitations, all were shown to be effective in the experiments.

Our work is the first to study curation attacks in the self-consuming generative retraining process. We believe that designing more efficient or scalable attack algorithms is an important direction for future research.

“Experiments are primarily demonstrated on CIFAR-10 and synthetic data; further validation on larger-scale or more diverse datasets would strengthen the claims. Further validation on larger-scale or more diverse datasets would strengthen the claims.”

As noted by the reviewer, we currently demonstrate results on CIFAR-10 and synthetic data. We chose these datasets to align with prior works [1, 2], which evaluate the self-consuming retraining loop under benign data curation. We fully agree with the reviewer that evaluating on larger or more diverse datasets would strengthen our findings. However, due to time and compute constraints, we were unable to complete experiments on very large datasets during the rebuttal period.

Instead, we have extended our experiments to CIFAR-100, a more diverse dataset with 100 classes. The results can be found at [3], which validate the theorem and demonstrate the effectiveness of the proposed algorithm. The observations are consistent with our findings: benign curation gradually steers the class distribution toward user preferences, leading to progressively increasing reward scores. In contrast, adversarial curation with the gradient-based attack disrupts this alignment, depresses reward growth, and drives the model away from the user preferences.

"Have the authors considered strategies to mitigate or detect such adversarial curation attacks in practical scenarios?"

While this work focuses on analyzing the vulnerability of self-consuming retraining loops to adversarial data curation, we have indeed considered potential mitigation strategies:

• Adding real data: This is a common strategy used in prior works to stabilize the self-consuming retraining loop of generative models [4, 5]. We have evaluated this method in our paper. As shown in the experiments (Fig. 3), adding real data only partially mitigates adversarial effects by driving the model closer to the true data distribution $p\_{data}$. However, it does not fully prevent misalignment.

• Anomaly detection methods: Although approaches like outlier detection may help identify and eliminate adversarially curated samples, they can inadvertently remove genuine preferences. When users are heterogeneous and come from multiple groups, removing genuine preferences from minority groups may potentially introduce biases. Additionally, our attack algorithm already considers such defense mechanisms: when formulating optimization (10), we impose a penalty term $\text{dist}(R\_{\theta},\widetilde{R}\_{\widetilde{\theta}})$ to prevent the adversarial behavior from being easily detected as anomalous (see line 261, left column).

We believe that designing effective defense mechanisms is an important direction for future research. We would be happy to include the above discussion in the revised version of the paper.

[1] Ferbach, D., Bertrand, Q., Bose,A.J., and Gidel, G. Self consuming generative models with curated data provably optimize human preferences, 2024.

[2] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.

[3] https://anonymous.4open.science/r/Anonymize_ICML2025-4CC5/CIFAR100.jpg

[4] Alemohammad, S., Casco-Rodriguez, J., Luzi, L., et al. Self-consuming generative models go mad, 2023.

[5] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.

We thank the reviewer for the detailed feedback and positive assessment of our work. We appreciate the recognition of our theoretical analysis, experimental design, and the contribution of introducing a new adversarial model in the context of iterative retraining. We address the reviewer's concerns below:

"One minor note is that the theory seems to assume that training converges to the global optimum. It is unclear how crucial that condition is to the overall theoretical analysis."

The reviewer is correct. Our theoretical analysis assumes that each model update converges to the global optimum of the training objective. (i.e., $p\_t$ maximizes data log-likelihood). This is a standard simplification in prior works [1, 2] to facilitate the analytical analysis.

However, we agree that this assumption may not always hold in practice. Nevertheless, we emphasize that our main theoretical insights remain valid as long as each model update sufficiently approximates the optimum. In fact, our empirical results show that the observed behaviors persist even when training is noisy or approximate. We appreciate the reviewer for highlighting this point, and we will clarify this assumption and its implications in the revised version.

“Lemma 3.3 doesn’t necessarily suggest that the distribution will converge to the optimal value.”

Yes, Lemma 3.3 does not claim that the iteratively retrained model converges to the optimum; instead, it only characterizes the relationship between the expected reward $\mathbb{E}\_{p\_{t}}\left[e^{r(x)}\right]$ over two consecutive time steps. As discussed in lines 172-185 (left column), whether the expected reward converges to the maximum depends on the covariance term $\operatorname{Cov}\_{p\_{t}}\left[e^{r(x)},e^{\widetilde{r}\_{t}(x)}\right]$. When the covariance is positive at every step, the expected reward increases and the variance decreases, indicating convergence toward the optimal distribution (which aligns with the findings in [1]). However, if the covariance becomes negative, the expected reward may oscillate and deviate from the maximum value. Indeed, our attack algorithm is developed based on these theoretical insights: when misaligning the model from human preferences, optimization (10) aims to flip human preference labels such that the covariance is as negative as possible.

About $\kappa$: the success rate

The reviewer’s understanding is correct. In real-world settings, the attacker typically does not have full control over which adversarial samples are ultimately curated on the target platform. This is exactly why we avoid referring to $\kappa$ as budget, and instead interpret it as the success rate of perturbing the data on the target platform.

In the experiments, we treat $\kappa$ as a controllable parameter to analyze how different attack intensities impact model alignment. We will add this clarification in the revised manuscript.

[1] Ferbach, D., Bertrand, Q., Bose,A.J., and Gidel, G. Self consuming generative models with curated data provably optimize human preferences, 2024.

[2] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.

We thank the reviewer for the thoughtful and encouraging feedback, especially their positive comments on the theoretical analysis, experimental design, and for describing our paper as “well written and very clear”. We provide clarifications to address the main concerns:

"Do we know how frequently theses curation attacks are likely to happen?"

In practice, the frequency of curation attacks depends on how often a model is iteratively retrained using human-curated feedback. Precisely quantifying this frequency requires collecting real-world data from the target platform, which is an interesting direction for future research. Nevertheless, we emphasize that the proposed scenario is realistic: as generative models increasingly rely on human feedback for training and updates, it opens an opportunity for curation attacks.

For example, InstructGPT [1] and LLaMA-2-Chat [2] are fine-tuned using curated human preferences; Pick-a-Pic [3] and Rich Feedback for Text-to-Image [4] demonstrate how human judgments can guide and refine image generation. If users can influence model updates through their preferences, then malicious feedback, which is curated by adversarial users such as those employed by competing platforms, may steer the model away from genuine user intent. Our work addresses a timely and critical question: under what conditions does adversarial curation impact the iterative training of generative models intended to align with benign user preferences?

"Do the authors think about Carlini 2024 implementation of attacks?"

We thank the reviewer for suggesting [5], a valuable and highly relevant reference. Their work demonstrates that poisoning large-scale training datasets is not merely a theoretical concern but a practical threat. They show how adversaries can inject poisoned examples into datasets at minimal cost by exploiting web-based data collection mechanisms.

While we did not cite [5] in the current version of our paper, we have discussed related dataset poisoning studies in the background and related work. We will revise the paper to include [5] and clarify the differences between their work and ours.

Unlike [5], which focuses on poisoning static datasets during pretraining, our attack operates in an iterative retraining setting, where models continuously adapt based on user feedback. Notably, our attacker does not require access to data collection pipelines or backend systems; instead, they can act entirely through public feedback mechanisms, such as voting or ranking systems.

This makes our approach more practical and harder to detect, as it unfolds gradually over time without direct data manipulation. Whereas traditional poisoning attacks often aim to induce outright model failure, our objective is more subtle: gradually misaligning the model from genuine user preferences. In competitive settings, such gradual misalignment can be highly damaging while remaining difficult to trace.

"The conclusion feels a bit 'obvious'"

While the conclusion that adversarial curation can lead to misalignment may seem intuitive, our work goes further by formally characterizing when and how this misalignment occurs. First, we clarify that such misalignment is not inevitable. According to Lemma 3.3, misalignment arises only under specific conditions: $\operatorname{Cov}\_{p\_{t}}\left[e^{r(x)},e^{\widetilde{r}\_{t}(x)}\right] < 0$. When $\operatorname{Cov}\_{p\_{t}}\left[e^{r(x)},e^{\widetilde{r}\_{t}(x)}\right] \geq 0$, adversial curation can still lead the model converging to the optimum that maximizes reward (at slower convergence rate); this is verified in Fig. 5 and highlights the inherent robustness of the iterative retraining process.

Another less intuitive finding is that, while prior work on self-consuming generative models suggests that adding real data can effectively stabilize training [6, 7], we demonstrate that simply incorporating real data does not mitigate the effects of adversarial curated data.

[1] Ouyang, L., Wu, J., Jiang, X., et al. Training language models to follow instructions with human feedback, 2022

[2] Touvron, H., Martin, L., Stone, K., et al. Llama 2: Open foundation and fine-tuned chat models, 2023.

[3] Kirstain, Y., Polyak, A., Singer, U., et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023.

[4] Liang, Y., He, J., Li, G., et al. Rich human feedback for text-to-image generation, 2024

[5] Carlini, N., Jagielski, M., Choquette-Choo, C. A., et al. Poisoning web-scale training datasets is practical, 2024.

[6] Alemohammad, S., Casco-Rodriguez, J., Luzi, L., et al. Self-consuming generative models go mad, 2023.

[7] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.

We thank the reviewer for the constructive comments and for recognizing our work as "a new method in generative models' adversarial attack field." We now address the reviewer's concerns regarding the experiment:

On dataset diversity and model generality

Our experimental setup follows prior work [1, 2], which also uses synthetic and CIFAR-10 datasets to analyze self-consuming retraining. In addition to reproducing the expected reward maximization under benign curation, our experiments also demonstrate the impact of adversarial curation, compare different attack strategies, and show that the model outputs remain visually plausible under attack.

We acknowledge that including a broader range of generative models, such as GANs and VAEs, would make our evaluation more comprehensive and convincing. While our experiments present results using DDPM, we adopt a theoretical framework that does not depend on specific architectures. Our analysis applies to any likelihood-based model, including VAEs. Extending this work to GANs would require reformulating the training dynamics, which we leave for future work.

Similarly, we agree that evaluating on more complex datasets (such as ImageNet) would strengthen the empirical validation of our method. However, due to time and compute constraints, we were unable to complete these experiments during the rebuttal period.

Instead, we have extended our experiments to CIFAR-100, a more diverse dataset with 100 classes. The results can be found at [3], which validate the theorem and demonstrate the effectiveness of the proposed algorithm. The observations are consistent with our findings: benign curation gradually steers the class distribution toward user preferences, leading to progressively increasing reward scores. In contrast, adversarial curation with the gradient-based attack disrupts this alignment, depresses reward growth, and drives the model away from the user preferences.

[1] Ferbach, D., Bertrand, Q., Bose,A.J., and Gidel, G. Self consuming generative models with curated data provably optimize human preferences, 2024.

[2] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.

[3] https://anonymous.4open.science/r/Anonymize_ICML2025-4CC5/CIFAR100.jpg