Self-Verification Provably Prevents Model Collapse in Recursive Synthetic Training

Thank you for your thorough review and constructive comments. We have carefully addressed all your concerns and thoroughly revised the paper, and sincerely hope our responses fully answer your questions.

Q1: Empirical Support and Simulation: Though theoretical results are promising, the main paper lacks empirical findings showing error control in finite samples or visualizing error growth. Are there toy simulations or experiments (even in the appendix) supporting the tightness of your bounds? What are the practical limits for deploying self-verification in LLMs?

A: Thank you for recognizing the value of our theoretical contributions, which are central to the learning theory area. Following your suggestion, we conducted experiments during the discussion period to validate our results. We also promise to include additional, more comprehensive experiments in the final version.

Specifically, we follow Fu et al. (2025)’s model collapse setting, training a 12-layer, 8-head GPT-2 (hidden size 256) to recursively in-context learn linear functions:

For each prompt, we sampled $x_1, \ldots, x_k, x_{query}$ and $w$ independently from $\mathcal{N}(0, I_d)$. The model predicts $y_{query} = w^\top x_{query}$. We compared three training strategies:

• Full Synthetic: Trained solely on synthetic data.
• Mixed: Fresh and synthetic data were mixed in a 0.5 ratio.
• Verification-Based Filtering: Sampled 20 candidates, selected the highest-confidence prediction ($\gamma = 0$), and trained on verified data exclusively.

The results of these experiments are summarized below:

As observed, the error accumulates progressively with more generations, particularly in the full synthetic case, where the error grows rapidly. In contrast, incorporating real data and applying the verification mechanism effectively reduce the loss, consistent with our theoretical findings.

Regarding the practical limits of self-verification, we agree that scaling self-verification in LLMs presents practical challenges. A key constraint is the computational overhead introduced by candidate sampling—for instance, generating $N = 20$ outputs per input to enable verification.

To address this, we outline potential mitigations in the revised manuscript, such as adaptive sampling (e.g., reducing $N$ for inputs with low predictive uncertainty) or approximate filtering strategies, which aim to strike a balance between efficiency and verification performance.

Q2: Limited Discussion of Assumptions' Strength and Realism: The theoretical results rely on confidence calibration and internal agreement assumptions (e.g., Assumption 1), but the main text rarely discusses empirically or interpretively how easily these hold for real-world LLMs.

A: We appreciate the reviewer’s focus on Assumption 1. We have strengthened the manuscript to address this by clarifying both the intuition behind the assumption and its alignment with real-world LLM behavior.

At its core, Assumption 1 formalizes a confidence-calibrated agreement condition: high-confidence model predictions are likely to be correct. This is consistent with standard assumptions in self-training (e.g., Assumption 2 on page 14 of [1]; the abstract of [2]) and aligns with active research on confidence calibration in LLMs—a well-studied area (see [3] for a survey).

Critically, empirical work shows modern LLMs often exhibit strong calibration. For example, [4] reports small expected calibration errors for pre-trained models like Llama-3-8B (3.52%), Qwen-2.5-7B (5.41%), and DeepSeek-V2-Lite (3.39%) on MMLU—demonstrating close alignment between confidence and accuracy.

To further validate this, we conducted experiments on the MATH dataset using Phi3.5-Mini, analyzing log-probabilities of correct vs. incorrect responses:

Correct responses consistently show higher log-probabilities, with their distribution stochastically dominating that of incorrect ones. This confirms the positive correlation between confidence and correctness, directly supporting Assumption 1.

Q3: Robustness to Calibration Errors: The key error bounds depend on well-calibrated base verifier $\mathcal{G}_0$, but LLMs often have calibration issues, worsened by distribution shifts. How sensitive are theoretical guarantees to $\mathcal{G}_0$'s miscalibration or adversarial initial models? What guidance exists for identifying self-verification failures due to poor calibration?

A: We thank the reviewer for raising this important point. Our theoretical framework explicitly controls the dependence on the calibration quality of the base verifier $\mathcal{G}_0$ via the term $ε(γ)$, which quantifies the correctness of high-confidence predictions. For instance, in Theorem 4 (Error Bound for Transformer-based LLMs with Self-Verification), the leading error term is:

which implies that poor calibration in $\mathcal{G}_0$ (i.e., large $ε(γ)$) will directly degrade the final bound. In this case, the guarantee gracefully degrades to an $O(ε(γ))$ bound, which signals when self-verification may become unreliable.

Notably, model collapse research focuses on scenarios where the initial model performs well (to isolate degradation from recursive training), rather than adversarially poor $\mathcal{G}_0$. Assuming a severely miscalibrated $\mathcal{G}_0$ would conflate inherent model failure with collapse due to recursive dynamics—contradicting the core motivation of studying collapse.

In practice, recent studies have shown that modern pre-trained LLMs are often well-calibrated. For example, [4] reports small expected calibration errors (ECE) on MMLU: LLaMA-3-8B (3.52%), Qwen-2.5-7B (5.41%), DeepSeek-V2-Lite (3.39%), indicating strong alignment between confidence and accuracy. As noted in Q2, our additional experiments with Phi-3.5-Mini on MATH further validate this assumption, showing that high-confidence predictions are typically correct.

For extreme miscalibration, while our bounds loosen, confidence calibration in LLMs is well-studied with effective solutions (see [1] for a survey). To guide practice, we recommend measuring ECE on a validation set and applying calibration (e.g., temperature scaling) if it exceeds a reasonable threshold. These steps have been incorporated into the manuscript to help identify and mitigate calibration issues prior to deploying self-verification.

Q4: Absence of Empirical Visualizations: The main text lacks figures/tables, hindering intuitive understanding and theory-practice connection. For example, a figure on Theorem 1’s exponential error growth or a table comparing theoretical vs. practical error rates under varied $N$, $C_0$ would add significant value.

A: We apologize for the absence of figures and tables in the main text, which hinders intuitive understanding. Per NeurIPS rebuttal policies (prohibiting PDF/link uploads), we cannot include visualizations here, but we firmly commit to adding the requested elements in the final version.

Specifically:

• For illustrating error growth in Theorem 1: The experiment in our Q1 response shows rapid error escalation in the full synthetic setting (consistent with predicted collapse) and that our verification mechanism effectively prevents this. We will present these as figures (error curves across training rounds for different regimes) and add more comprehensive results.

• For clarifying mechanisms: We will add a schematic diagram of the verification architecture to the main text, aiding intuition about how filtering operates.

Regarding error rates under varying $N$ (number of candidates) and $C_0$:

• $C_0$ is a data- and model-dependent quantity (not tunable) characterizing worst-case confidence, consistent with standard model collapse literature.

• $N$ is adjustable, and we conducted additional experiments (same setup as Q1: 12-layer GPT-2 on regression) with $N = 1, 10, 20, 30, 40, 50$. Note that $N=1$ means using only synthetic data per round without verification, equivalent to the Full Synthetic baseline (naive recursive training without filtering).

The table confirms that error decreases with increasing $N$, validating the efficacy of self-verification. The drop is steepest from $N=1$ to 20, then plateaus, suggesting $N=20$ balances cost and effectiveness. These results will be visualized as error curves in the final manuscript, showing the relationship between $N$ and performance.

References

[1] Self-evolved reward learning for LLMs.

[2] Self-improvement in language models: The sharpening mechanism.

[3] Uncertainty quantification and confidence calibration in large language models: A survey.

[4] Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator.

Thank you for your constructive comments and kind support! We've carefully addressed all concerns and revised the paper accordingly.

Q1: Although the verification mechanism is more complex, it still relies on the first model as a fixed verifier—functionally external throughout the recursive process.

A: We appreciate this insight. Though $\mathcal{G}_0$ originates within the system, its frozen status makes it effectively an external verifier in the recursive process.

That said, our design offers a practical and principled alternative to traditional external supervision. Standard verification methods typically depend on large-scale human annotations or auxiliary fine-tuned models—both costly in terms of labor and computation, and increasingly impractical as high-quality real data becomes scarce.

In contrast, our framework avoids such overhead by reusing $\mathcal{G}_0$ as a stable verifier anchored in real data. This preserves the benefits of external verification (e.g., consistency and filtering) while maintaining the self-contained, scalable nature of self-learning.

Q2: The criticism of Bernoulli variables seems misplaced—they're just a tool to measure synthetic-real data agreement, also central to the authors' results (see $ε(γ)$).

A: Thanks. We have revised the manuscript to clarify the role of Bernoulli variables in Feng et al., and to position our $ε(γ)$ as an analogous construct within our framework.

To clarify, both approaches share analogous roles: Feng et al. use Bernoulli random variables to model synthetic-real data agreement, abstractly capturing alignment likelihood with ground-truth labels. In our setting, $ε(γ)$ is analogous—it characterizes the probability that when the model assigns most probability mass to high-quality verification-filtered data, its prediction matches the true label.

Q3: The authors avoid specifying a data-generating process, using TV distance instead—though this quantity is harder to reason about.

A: Thanks. We agree TV distance is hard to operationalize, but we use it deliberately as it has become a standard analytical tool in recent model collapse theory (e.g., Dohmatob et al. 2024c; Fu et al. 2024, 2025; Seddik et al. 2024), enabling study of recursive training dynamics without restrictive assumptions on the data-generating process.

That said, we fully recognize the importance of grounding such abstract quantities in practical settings. Specifically, in Theorem 4, our manuscript leverages PAC-Bayesian analysis tailored to transformer-based LLMs to derive a finite-sample bound for the TV distance term, connecting our theoretical bounds to state-of-the-art architectures with concrete guarantees.

Q4: Could you clearly define $\mathcal{G}_0(\cdot | x)$ in line 117? It is described as a function mapping but later as a probability measure, which is confusing.

A: Thanks. We have revised the definition of $\mathcal{G}_0(\cdot | x)$ to clarify that $\mathcal{G}_0: \mathcal{X} \to \Delta(\mathcal{Y})$ denotes a conditional distribution mapping an input $x \in \mathcal{X}$ to a distribution over responses—i.e., $\mathcal{G}_0(y | x)$ is the probability assigned to response $y$ given $x$. This correction aligns the formal definition with its use as a probability measure. Line 117 has been updated accordingly.

Q5: Assumption 1 is hard to parse: Is $τ$ the same for all $γ$ or $γ$-dependent? How credible is it? Also, since $y^*(x)$ seems deterministic, the assumption appears to rule out randomness in $y | x$.

A: Thanks. We address each point and summarize the revisions below.

1. Intuition and Credibility of Assumption 1

This assumption encodes a confidence-calibrated agreement: predictions with sufficiently high confidence are likely to be correct. Specifically, if the model assigns probability mass $\ge 1 - τ$ to the verified high-quality set, then the probability that its prediction matches the ground truth $y^*(x)$ is at least $1 - ε(γ)$.

This aligns with standard self-training assumptions (e.g., Assumption 2 on page 14 of [1]; the abstract of [2]) and active research on LLM confidence calibration (surveyed in [3]). It is realistic: modern LLMs often show strong calibration, with [4] reporting small expected calibration errors for Llama-3-8B (3.52%), Qwen-2.5-7B (5.41%), and DeepSeek-V2-Lite (3.39%) on MMLU, indicating close confidence-accuracy alignment.

2. Parameterization and Determinism in Assumption 1

Assumption 1 treats $τ$ as a fixed constant for notational simplicity, but our analysis holds for any $τ \in (0, 1)$. Specifically, the bound for

(see Equation 31, page 22) remains valid regardless of the choice of $τ$. Moreover, $τ$ does not need to depend on $γ$, since variation in verification strictness across $γ$ is already captured by $ε(γ)$.

Assumption 1 also implicitly assumes a deterministic ground-truth label function $y^*(x)$. This reflects a realizable setting commonly used in self-training theory (e.g., [2]), enabling a clean and tractable convergence analysis. This clarification has been added to the manuscript.

3. Empirical Support for Assumption 1

To strengthen the credibility of Assumption 1, we conducted additional experiments analyzing the relationship between model confidence and prediction correctness. Using the Phi3.5-Mini model on the MATH dataset (following [2]), we sampled responses and measured log-probabilities:

These results confirm that correct responses consistently receive higher log-probabilities, validating the positive correlation between confidence and correctness. Formally, the log-probability distribution for correct responses stochastically dominates that for incorrect responses.

References

[1] Self-evolved reward learning for LLMs.

[2] Self-improvement in language models: The sharpening mechanism.

[3] Uncertainty quantification and confidence calibration in large language models: A survey.

[4] Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator.

Q6: Please define the key coverage coefficient outside Theorem 1 (hard to find) and standardize its indexing.

A: Thanks. We now define the recursive coverage coefficient early in the technical section as

with $t$ indexing the training iteration. To clarify the notation in Theorem 1: The coefficient $C_{γ}$ referenced there corresponds to $C_{γ,0}$. These revisions improve clarity and consistency.

Q7: I am completely open to adjusting my score upwards if the authors adjust their comparison to previous work. I think this work would be a clear accept if it had empirical confirmation of a new claim resulting from this theory.

A: Thanks. We clarified our comparison and conducted additional experiments to support our theoretical claims.

1. Clarifying Our Comparison with Prior Work

We have revised the manuscript to emphasize two core contributions and clarify three key comparisons:

• General and Practical Framework: Unlike prior work (Feng et al., 2025; Firdoussi et al., 2025) focused on Gaussian mixtures or linear classifiers, our framework extends to Transformer-based LLMs, addressing realistic settings.

• Recursive vs. Non-Recursive Training: While prior work studied non-recursive setups ($T = 1$), we focus on recursive training—where model collapse primarily occurs—explicitly modeling its iterative dynamics.

• On the verifier $\mathcal{G}_0$: As noted in Q1, it acts as an external verifier but avoids the need for additional annotations or stronger auxiliary models, enhancing the practicality of the framework.

• On Bernoulli variables: As discussed in Q2, we now clarify their alignment with our $ε(γ)$—both quantify agreement—while acknowledging their utility in prior work.

• On TV distance: As explained in Q3, we recognize its practical challenges but emphasize our finite-sample bounds (Section 5) tailored for Transformer-based LLMs, bridging theory and practice.

2. Experimental Validation of Theoretical Results

To validate our theory, we follow Fu et al. (2025)’s model collapse setting, training a 12-layer, 8-head GPT-2 (hidden size 256) to recursively in-context learn linear functions:

For each prompt, we sampled $x_1, \ldots, x_k, x_{query}$ and $w$ independently from $\mathcal{N}(0, I_d)$. The model predicts $y_{query} = w^\top x_{query}$. We compared three training strategies:

• Full Synthetic: Trained solely on synthetic data.
• Mixed: Fresh and synthetic data were mixed in a 0.5 ratio.
• Verification-Based Filtering: Sampled 20 candidates, selected the highest-confidence prediction ($γ = 0$), and trained on verified data exclusively.

The results of these experiments are summarized below:

As observed, the error accumulates progressively with more generations, particularly in the full synthetic case, where the error grows rapidly. In contrast, incorporating real data and applying the verification mechanism effectively reduce the loss, consistent with our theoretical findings.

Furthermore, we promise to include additional experiments to further support our results in the final version.

Thank you for addressing the clarity concerns raised in the original review. I appreciate the thorough response, though it is still not clear to me that this experiment motivates all of the theory presented in this paper. To summarize my understanding, the core idea of this paper is that using a good verifier for synthetic data keeps the DGP from drifting too far from the ground-truth. There's some limited empirical support for this strategy presented in the rebuttal and a fair amount of theoretical exposition. I'm still not fully convinced of its operational value - your theory makes a very specific claim relating calibration to accumulated error. So, it seems like it would be interesting to see if the effect of the different verifiers you test is predicted by their well-calibratedness? I would suspect that this relationship is somewhat weak in practice - surely I could use an uncalibrated and post-trained GPT-5 to verify data much more effectively than a calibrated small model, but there may be some subtlety here re: the original DGP. Either way, I think this empirical validation is a fine start, but insufficient to justify raising my score further. That being said, my quibbles aside, I think this is a solid paper and I am supportive of its publication.

Thank you for your constructive comments and kind support! We have carefully addressed all concerns below and thoroughly revised the manuscript. We hope our responses resolve your questions.

Q1: The motivation for removing external verification isn't clear - it cost? GPU hours?

A: We appreciate this question and have clarified the motivation in the revised introduction.

As highlighted in our revisions, external verifiers inherently depend on two scarce resources: large-scale human annotations (becoming increasingly scarce as high-quality real data is depleted) and additional fine-tuned auxiliary models. This incurs significant labor costs and computational overhead (e.g., GPU hours for continuous training/retuning), which is exacerbated in recursive scenarios where distribution shifts demand constant adaptation.

Our self-verification framework avoids these issues by using the initial model as a stable anchor, eliminating the need for ongoing annotations or auxiliary models to bypass external verification’s resource constraints in recursive training.

Q2: Confidence Scores: How is $s_{verify} = \log \mathcal{G}_0(y|x)$ calculated—similar to perplexity? When generating candidates for scoring, are they from greedy sampling or other methods?

A: We appreciate this question, as it helps clarify a key technical detail.

The verification score $s_{verify}(y|x) = \log \mathcal{G}_0(y|x)$ uses $\mathcal{G}_0$ to evaluate outputs $y$ from later models: $x$ and $y$ are concatenated into a sequence fed to $\mathcal{G}_0$, whose token-level log-probabilities (from position logits) are summed for total log-likelihood.

Regarding its relation to perplexity: While both use log probabilities, perplexity measures average uncertainty over a dataset or sequence, whereas $s_{verify}$ ranks or filters individual outputs based on alignment with $\mathcal{G}_0$.

Regarding the generation process: Candidate outputs are sampled from $\mathcal{G}_{t-1}(\cdot | x)$ via stochastic decoding (e.g., temperature, top-p sampling), not greedy decoding. This randomness promotes diversity, enabling effective verification-based filtering.

Q3: Filtering Mechanism: Keeping candidates within margin $γ$ of the best score resembles top-p sampling for longer generations. Is this to align the new model’s distribution with $\mathcal{G}_0$?

A: We agree our filtering shares top-p sampling’s core intuition (retaining high-probability outputs while promoting diversity) but with key differences, outlined below.

Standard top-p sampling selects the next token from high-probability options (token-level). Ours filters full candidate sequences (sequence-level). Also, top-p uses a fixed number of options, while our $γ$-margin filtering is score-adaptive—retained candidates depend on how many fall within the best sample’s high-quality neighborhood, adapting to input contexts and score variances.

As noted, its core purpose is aligning the new model’s distribution with $\mathcal{G}_0$: filtering out candidates that deviate too far from $\mathcal{G}_0$’s judgments prevents cumulative drift from real data, critical for avoiding collapse in recursive training.

Q4: Teacher-Student Analogy: Is it fair or useful to view $\mathcal{G}_0$ as a fixed "teacher" and subsequent $\mathcal{G}_t$ as "students" learning its distribution?

A: We appreciate this insightful analogy, which captures a key dynamic of our framework.

Your view of $\mathcal{G}_0$ as a fixed teacher and $\mathcal{G}_t$ as students captures our setup: $\mathcal{G}_0$, trained on real data, guides via verification score, indicating how well student outputs align with its knowledge.

There are two critical distinctions from traditional teacher-student frameworks:

1. Architectural parity over superiority: Unlike traditional setups requiring a stronger teacher, $\mathcal{G}_0$ and $\mathcal{G}_t$ share the same architecture. $\mathcal{G}_0$’s authority stems from its real-data training, not model size or performance.

2. Indirect judgment vs. direct instruction: Traditional teachers guide students via direct supervision (e.g., distillation), while $\mathcal{G}_0$ acts as a reference judge—evaluating outputs via verification scores without participating in training.

This analogy highlights how $\mathcal{G}_0$ anchors the system against drift. We’ve added it to the revised discussion—thank you for the insightful framing.

Q5: Reusing Real Data: In Remark 3, you note reliance on an "ongoing stream of high-quality real data." What if the same initial real data is reused for every generation? Is new data strictly necessary to prevent collapse?

A: We appreciate this question, as it highlights a critical aspect of our framework. To directly address your inquiry: reusing the same initial real data in every generation is insufficient to prevent collapse, and an ongoing stream of new real data is essential for long-term stability. This requirement arises from both theoretical insights and empirical evidence:

Theoretical Perspective: If each generation’s training data uses only original real data $D_0$ plus synthetic data $D_{s,t-1}$ (generated by prior models), this leads to:

1. Reduced Information Diversity: As generations progress, the synthetic data becomes a degraded copy of the original real data, with diminishing novel information. This effectively reduces the usable diversity of the training set, gradually forcing the model to train on increasingly lower-quality data.

2. Error Accumulation: Any biases or imperfections in early synthetic data are amplified in subsequent generations, causing the model's distribution to drift away from the true data manifold. Over time, this drift accumulates, leading to catastrophic forgetting of rare but critical patterns in the original data.

Empirical Evidence: Prior work on recursive training (e.g., [1] with diffusion models and GANs) shows reusing initial real data only delays degradation—models eventually collapse. In contrast, ongoing new real data mitigates this by injecting novel patterns and correcting drift.

We’ve clarified these points in the manuscript, with relevant citations. Thank you for prompting this deeper explanation.

Reference:

[1] Alemohammad et al. Self-consuming generative models go mad. ICLR 2024.

Q6: Training Regimes: What are the three regimes you mention? Briefly naming/describing them in the introduction would help.

A: We appreciate this question. Defining these regimes in the introduction improves clarity, and we’ve revised the section as follows:

1. Unverified Full Synthetic Regime: Trains exclusively on predecessor-generated synthetic data (no real data/verification). This naive setup is prone to rapid drift and collapse from unconstrained error accumulation.

2. Fresh Real Data Augmented Regime: Uses mixed synthetic (from prior models) and new real data, without verification. Fresh real data delays degradation but is unsustainable long-term due to costs and data depletion.

3. Verified Full Synthetic Regime: Trains solely on synthetic data filtered via self-verification (retaining $\mathcal{G}_0$-aligned candidates). This avoids ongoing real data reliance while preventing collapse by anchoring to initial real data.

Thank you for enhancing clarity.

Q7: Data Source: Just out of curiosity, the framework uses data from a model’s direct predecessor. Have you considered training on data from a stronger model instead?

A: We appreciate the reviewer’s question. Our choice to use the model’s direct predecessor is driven by a practical motivation: for frontier models, there may be no stronger model available, and training one would require significant compute and fresh real data, which are often already exhausted. This makes self-learning from one's own generations a necessary and realistic paradigm.

That said, if a stronger external expert model were available, our framework could naturally incorporate it—resembling a teacher-student setup and potentially improving performance. However, our focus is on the more challenging and practical case where no stronger teacher exists.

Q8: The limitations (e.g., real data dependency) are scattered; a dedicated section on practical issues—such as the computational cost of sampling and a poor initial $\mathcal{G}_0$—would be more impactful.

A: Thank you for this constructive suggestion—we agree consolidating limitations into a dedicated section enhances clarity, and have revised the manuscript accordingly.

The new "Limitations" section addresses key practical constraints:

1. Computational Overhead of Candidate Sampling: Generating $N$ candidates per training point (for verification) introduces additional computational costs, especially at scale. We discuss potential mitigations, such as adaptive candidate selection (e.g., reducing $N$ for low-uncertainty inputs) or leveraging approximate filtering to balance efficiency and performance.

2. Vulnerability to a Poor Initial Anchor: The framework’s stability hinges on $\mathcal{G}_0$ being a reliable encoder of real-data patterns. If $\mathcal{G}_0$ is poorly trained (e.g., due to biased or limited initial data), its verification scores may misguide filtering, leading to compounding errors. We note scenarios where this risk is heightened (e.g., small initial datasets) and propose future work on robustifying $\mathcal{G}_0$ via techniques like data augmentation or uncertainty quantification.

By consolidating these points, we aim to provide a clearer roadmap for both practical deployment and future research. Thank you for strengthening the manuscript.