Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification

We thank the reviewer for their thoughtful review and for specifically appreciating our analysis from theory to empirical settings, as well as the quality of our appendix. Below, we provide our responses.

Novelty of results. From a practitioner's point of view, the conclusion from these results is simple: when it is possible to evaluate a verifier on labeled data, picking a more accurate verifier gives better results on that same task. This seems to be obvious / empirically quite clear in existing work (e.g. Appendix A.1), and I'm not sure if the specific definition of $p_*$ offers gains over just picking by existing accuracy metrics.

Intuitively, the general trend should hold that a more accurate verifier will lead to better results. However, as shown in our news summarization task, also as highlighted by reviewer rwoT and reviewer mXV4, a better model in evaluation metrics does not necessarilly translate into a model more capable of selection. In our setup, evaluting with the same test set and the rouge score, the finetuned Llama-3 model exhibits higher performance compared with the finetuned Llama-2 model. However, when using the Llama-3 model to select the same amount of data compared with using the Llama-2 model, the model trained from the Llama-3 selected set is actually worse than the model trained from the Llama-2 selected set. In this case, an existing accuracy metric does not help select a better model while our $p_*$ metric could correct do so. Specifically, $p_*$ could be estimated with human / experts looking at a set of generated data and how the data servive during selection. It can be estimated without training and thus offers a cheap measure on how the trained model behaves.

How might this generalize to language modeling / task-agnostic pretraining, where there is not necessarily a sense of "correctness" for verification?

Thank you for raising this important question. We hope our discussion in General Response 1 provides helpful insights.

Lines 390-395 point out that when using a weak verifier, there's a dependence on $n'$, the amount of synthetic data to train on. This seems practically quite important to establish. Can the authors comment more on how to "take into account the quantity of data selected" (line 395)?

Thanks for raising this point! We hope our General Response 2 address your question.

suggestions for a final version: 1. I would suggest moving Appendix C into the main text. If necessary for space, I would move Section 3 to the appendix, since these results are reasonably well-known from the test-time compute literature. 2. https://arxiv.org/pdf/2209.03942 is a related reference 3. Line 365 typo: "and the ground truth $w_*$. 5Having the verified" 4. Line 12 typo: "other LLM[s]"

Thank you for your valuable suggestions. We have moved Appendix C into the main text before the news summarization and corrected the typos. The paper you sent is of high relevance and we have included it prominently in the introduction. We plan to include a discussion: Model-generated data has also been shown to amplify dataset biases through feedback loops.

Please let us know if our discussion addresses your concerns regarding the novelty of our work. If there are no remaining issues, we kindly request that you consider raising your score.

Thanks to the authors for the response to the review. The response makes a lot of sense, and I'm satisfied with the quality of the paper. I maintain my original rating.

One final suggestion for the camera-ready: the observation that a more accurate verifier does not necessarily result in better downstream performance has previously been empirically shown in https://arxiv.org/abs/2309.17425. This submission offers some insight as to why; it would be lovely to mention this connection explicitly in the paper.

We thank the reviewer for the thoughtful evaluation and for recognizing the practical applicability and effectiveness of our proxy $p_*$. Below, we provide our responses to your questions.

Model collapse not only happens when trained on synthetic data but also occurs when synthetic data is involved in the training set iteratively. This paper fails to address model collapse from this dimension.

Indeed, model collapse can occur across iterations, but we show that it is possible to prevent collapse at each iteration.

In the worst-case scenario, without mixing in real data, [1] demonstrates that using synthetic data generated by the previous model alone causes the subsequent model to perform worse than the generator. Over iterations, this leads to a compounding effect where the model becomes progressively worse.

Our paper addresses this issue by examining each iteration, which consists of one generation and one training process. We demonstrate that with proper verification, the model trained on synthetic data does not perform worse than its generator. When applied iteratively, proper verification ensures that the model does not degrade over time. In fact, the model can improve with each iteration. Thus, our approach effectively mitigates model collapse.

As stated in the paper, Llama 3 performs worse than Llama 2 when as a verifier. Therefore, an advanced model is not necessary to be a good verifier. There could be more discussion on the reason behind it and how to choose verification models.

The reason can be elaborated even in linear settings. In Equation (6) and Appendix E.6, we introduced three angles between the generator, verifier, and the ground truth, and derived the relationship between $p_*$ and these angles. The theoretical insight is that the performance and $p_*$ depend on all three angles and cannot be reduced solely to the angle between the verifier and the ground truth (i.e., the performance of the verifier).

Intuitively, a verifier may exhibit strong performance but could learn knowledge that is entirely different from the knowledge embedded in the generator. As a result, it may fail as a good selector because it cannot effectively distinguish the responses produced by the generator.

We have included the discussion.

This paper primarily focuses on accuracy for analysis, yet it would be insightful to consider cases where accuracy alone is insufficient for assessing model performance, such as alignment in large language models (LLMs), and explore how the synthetic data selection strategy could be applied in such complex scenarios.

Thank you for raising this point! We hope we have addressed the question in General Response 1. Additionally, we would like to comment further on alignment in large language models, as we have discussed this topic with Reviewer CePm in connection to RLHF. We believe that verification can play a valuable role in improving optimization for alignment. Verification could be applied both before and after establishing preferences.

For instance, when all sampled generations are equally poor, it may be preferable to discard them entirely. Verification could help improve the generation of responses for exploration and exploitation. Similarly, after preferences are established, response pairs with weak reward signals could be excluded from the DPO/RLHF training data. Verifiers provide a practical advantage in these scenarios by enabling selective filtering, thus preventing low-quality samples from propagating into the reward and policy models.

The main challenge lies in constructing an effective verifier. A potential starting point could involve combining the reference model with the current reward model. However, this remains an open question and requires a completely different analysis for RLHF compared to the Vicuna-style fine-tuning we did. We leave this as a promising direction for future work.

"We validate our theoretical findings in three empirical settings" and "We conduct two large-scale experiments to test our theoretical insights :" are somewhat at odds. And if they aren't, then the writing for this part should be clarified.

Thank you for raising this point. We used the term "two large-scale experiments" because one of the three experiments involves linear models on Gaussian data, which we did not consider as a large-scale experiment. In the updated version, we have removed the reference to "two" for clarity.

Ranking is only mentioned once, at the very end of the intro. It is unclear how ranking fits into the verification process -- data selection does not necessitate ranking, although ranking is indeed a way to perform data selection. What is meant by this?

We completely agree and have changed rank to distinguish.

Is verification really all we need? Some form of "verification is all you need" is mentioned throughout the paper, but it seems like there is a long list of things that we need.

We have removed the expression to enhance formality of the scientific writing.

While I imagine that this would be computationally expensive, it would be useful for the authors to include a scaling law analysis under synthetic data, and for that matter, an example of pretraining on synthetic general data. The proposed technique indeed improves over the collapsed baseline, but it does not necessarily approach the performance of training on real data. There are many questions related to scale that I don't think this paper addresses, but I am not going to factor this into my evaluation, as this seems like obtaining an answer to this would be quite expensive. In any case, I would be interested to hear any speculative thoughts that the authors might have about this.

We thanks the reviewer for raising such insightful question. We totally agree that scaling is an important factor here which is not fully addressed. We discuss this point in detail in General Response 1.

We would like to thank you once again for reviewing our work. We have further improved the introduction beyond the advice provided. We hope our responses meet your approval and encourage you to consider raising your score, if so.

We thank the reviewer for recognizing our analysis as fundamental and for finding our observation that a better model does not necessarily lead to a better verifier to be intriguing. We also appreciate the valuable advice on improving the writing of the introduction. Below are our responses.

The theoretical analysis could be improved with some form of a finite-sample result. The current result only applies in the infinite sample limit in a simplified setting, although the result is validated empirically (using finitely-many samples, of course).

Thank you for highlighting this important extension. We hope our discussion in General Response 2 helps address it.

The writing in Section 3 can be improved substantially. The purpose of this section and its contribution to the overall narrative are currently unclear, which can be mostly fixed by leading with a crisp motivational statement. More generally, I'm still not sure that I understand the takeaways of this section -- why is it that the model "lacks the inherent capability to autonomously select the best predictions," when using beam search? Is this a statement about decoding, or about the model? Earlier, in the intro, the authors say that models "cannot intrinsically identify the best solution using perplexity alone," yet beam search is essentially based on maximizing the likelihood of outputs.

Thank you for raising this point. Please allow us to clarify this section: the main takeaway is that while the model can generate good samples, as indicated by the accuracy of all the samples in the beams, the sample selected as the "best" based on the lowest perplexity—the model's most confident prediction—is not necessarily of high quality. As the beam size increases, the accuracy of the beam search solution (as chosen by the model) does not improve at all.

From this, we conclude that the model lacks the inherent capability to autonomously select the best predictions for a given question. This underscores that beam search is not an effective method for improving the quality of generated synthetic data. While many tasks benefit from solutions with higher confidence (e.g., greedy decoding often outperforms nucleus sampling with temperature 1), this correlation does not hold for beam search in our setting. High-quality data do exist within the beams, but the model fails to select them effectively.

We hope this clarifies the intent of this section, which we hope demonstrates this point in a clean synthetic data setting that allows for full experimental control.

4.1 in the theoretical analysis says that the  y’ variables "[have] been generated by an AI model," but it's unclear what this statement means mathematically. This should be more formal, particularly since it's part of the theoretical analysis.

Thank you for pointing that out. We intended to emphasize that there is no additional assumption beyond that the synthetic labels being generated as independent and identically distributed. We have updated the version to clarify this point.

Assumption 4.1 introduces "bits" $q_1, \cdots, q_N$. I assume these represent whether or not a sample has been pruned, however this is not defined until after it's already mentioned, and it is up to the reader to infer what these bits represent.

The bits are introduced in Section 4.1, "Downstream Model and Pruning," where we first present the pruning strategy and then explain how the downstream models are trained on the pruned dataset. Specifically, we wrote: "We will model our data selection (whether with or without feedback) via a {\em pruning strategy} $q=(q_1, \ldots, q_N)$, where $q_i$ is a bit that indicates whether the $i$th training example from $D_N'$ has survived pruning." This notation is also introduced mathematically in Equation (1), prior to Assumption 4.1.

The authors compare to "selection with weak supervision," but there is little elaboration about what this actually means. Weak supervision is an overloaded term and refers to many things -- one reading of this could be that the authors refer to programmatic weak supervision, which could make sense because programmatic weak supervision in some sense generates synthetic training data and verifies it via abstain votes.

Thank you for pointing that out. We used "selection with weak supervision" to describe the selection process using the Llama-3 model on the news summarization task, intending to highlight the difference between selecting with a model and selecting with the ROUGE score, which assumes access to the ground truth. However, we agree with the reviewer that the term is overloaded. For clarity, we have updated it to "Llama-3 selection."

Thank you for appreciating our writing, theoretical contributions, and the connection with experimental evidence! We also greatly value the connection with RLHF that you highlighted. Here are our responses.

In terms of LLMs, qualitative examples showing model collapse in some of the tasks may benefit the delivery further.

Thank you for the suggestion—it significantly improves the delivery! We have included additional qualitative examples from the testset of the news summarization task to compare how the model performs with each selection strategy on the same news (see additions to Appendix D). We specifically highlight the collapse behavior and how selection strategies mitigate it. Here, we provide one example but omit the original news text due to its length.

Ground truth: The International Monetary Fund (IMF) has announced that China's currency, the yuan, will join the fund's basket of reserve currencies.

Generation of the model trained without selection: 2015 will be a big year for the Chinese yuan.

Generation of the model trained with Self-selection: The International Monetary Fund (IMF) has approved the inclusion of the Chinese yuan in its basket of reserve currencies.

Generation of the model trained with Llama-3 selection: 2015 was a good year for China's currency.

Generation of the model trained with oracle selection: China's yuan has been added to the International Monetary Fund's basket of reserve currencies, the IMF has announced.

Notice how the generation without selection collapses: it heavily relies on a clichéd format that lacks specific information. While this approach might work for simple news by piquing curiosity, it fails to convey the depth or significance of the current news. The model trained on generated samples collapses to these formats. The Llama-3 selection does not address this issue, aligning with our results that Llama-3 selection performs poorly despite using a better model.

In contrast, generations with self-selection and oracle selection produce summaries that are specific, informative, and directly address the key event, significantly improving the quality of the output.

I am interested in whether the proposed methods can be applied to other modalities (e.g., VLMs, Vision-Language-Action Models in robotics) - many problems face more challenges in data annotation cost and also resort to synthetic data.

We completely agree that this is an interesting direction! We hope our General Response 1 provides some tangible insights.

Synthetic data is widely leveraged in multi-modal tasks to tackle challenges such as reducing data annotation costs or augmenting specific modalities. For example, in vision-language models (VLMs), synthetic captions are commonly used to expand datasets for unlabeled images. Similarly, in robotics, synthetic simulations are utilized to train policies for complex action sequences. We believe that implementing proper verification methods can greatly enhance the effectiveness of these approaches. For example, in generating diagnostic reports for medical images, researchers or LLMs might incorrectly use synonym substitution as a form of augmentation, potentially violating domain-specific knowledge. Leveraging LLMs coupled with retrieval-augmented generation (RAG) techniques could help produce a verifier to ensure the accuracy and appropriateness of diagnostic reports.

In settings involving multiple modalities, the theory and the proposed metric $p_*$ may not directly apply, as it primarily captures mappings from prompts to responses. Developing a robust theoretical model for selection in multi-modal tasks and understanding selection mechanisms in this context remain promising directions for future work.

Many LLMs are trained with synthetic data specifically on preferences (e.g., RLHF, DPO). I wonder if parallels can be drawn between the verifiers and a reward model and if model collapse is particularly severe with Vicuna-style finetuning.

Thank you for the insightful question. We agree that model collapse is particularly severe in Vicuna-style fine-tuning. In this setting, the model is trained directly on generated synthetic data without any external signals. However, in RLHF, generated responses are labeled as 'chosen' or 'rejected' using external signals, which help mitigate collapse in a manner similar to a verifier. Comparing the efficiency and effectiveness of a verifier versus a reward model under the same amount of information or labeling is an intriguing direction for future research.

Verification of general data is a critical and challenging problem. Recently, startups have emerged with a strong focus on addressing this issue through advanced engineering solutions. Datalogy have demonstrated significant success in data selection with synthetic data incorporated into workflows, particularly in reducing the volume of required data for tasks involving both text and language-label pairs [9]. These promising achievements have the potential to inspire further research and innovation in this direction.

We sincerely thank the reviewer for their insightful questions, which have prompted us to formulate this discussion. The ideas outlined here not only expand on the theoretical framework of our paper but also open avenues for application in diverse domains. We will include this extended discussion in the appendix.

[1] Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.

[2] Saha, Swarnadeep, et al. "Branch-Solve-Merge Improves Large Language Model Evaluation and Generation." Proceedings of the 2024 Conference of the Association for Computational Linguistics. 2024.

[3] Pang, Richard, et al. "Iterative reasoning preference optimization." arXiv preprint arXiv:2404.19733 (2024).

[4] Wu, Tianhao, et al. "Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge." arXiv preprint arXiv:2407.19594 (2024).

[5] Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Wang, Yiping, et al. "Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning." Advances in Neural Information Processing Systems 37 (2024).

[7] Dong, Yijun, et al. "Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning." arXiv preprint arXiv:2407.06120 (2024).

[8] Dohmatob, Elvis, et al. "A Tale of Tails: Model Collapse as a Change of Scaling Laws." Forty-first International Conference on Machine Learning.

[9] https://www.datologyai.com/post/productionized-multimodal-data-curation-at-the-billion-sample-scale

2. Extending the results to finite-sample settings. Reviewer mXV4 suggested that 'the theoretical analysis could be improved with some form of a finite-sample result,' while Reviewer rFSV emphasized the need to address 'how to take into account the quantity of data selected'.

Indeed, we limited our theoretical analysis (Theorem 4.2 and Theorem E.3) to the asymptotic regime where the sample size $n$ is cranked to infinity. In this limit, a few important analytical simplifications prevail; the most important being the passage from equations (31) to (42) of the appendix, where we only retain the tail behavior of the sigmoid function, allowing us to obtain fixed-point equations which ultimately lead to our theorem. This asymptotic regime is still rich enough to provide theoretical coverage for our experimental findings.

Though our asymptotic results are indeed experimentally confirmed even in the finite-sample setting, a theoretical analysis of this regime, while still technically possible, will require considerably more work. This will be pursued in a future paper. We can already say that such an analysis would use ideas from random matrix theory and so-called Gaussian comparison inequalities like Gordon's inequality and generalizations thereof.

In summary, we are grateful for the constructive feedback provided by the reviewers. The questions raised are crucial for advancing the effective use of synthetic data in the whole field. We will include these discussions in the appendix and hope that our paper, along with the insights shared here, can serve as a catalyst for further research in this direction.