Escaping Collapse: The Strength of Weak Data for Large Language Model Training

We thank the reviewer for their positive assessment of our paper. We appreciate their recognition that (paraphrasing from their response) our paper is well-written, theoretically solid, and contains experiments that confirm the theoretical analysis.

We respond to their concerns below.

Weakness: Only one model (Gemma2 2B) and only one prompting method (when fine-tuning) - 3-shot prompting.

• Response: In Appendix F in the supplement (Figure 7), we experiment with other models, including Gemma 1 2B and Gemma 7B.

For the filter-only setup, is the model trained on less samples (than w/ boosting) since we filter the bad ones out?

No, in all settings considered, there are more correct examples than (training steps * batch size). The difference between the setups is not in the quantity of examples, but the source (either from the weak learner or from the model itself).

We thank the reviewer for their positive assessment of our paper. We appreciate their recognition that (paraphrasing from their response) our paper addresses a critical and urgent problem, offers a deep and insightful explanation of our method, and contains experiments which validate our theoretical claims.

We respond to their main concern below.

Weakness: The framework assumes a binary (good/bad), unambiguous, and efficiently evaluable quality function q. This is true for the tested tasks of math and coding, but for more subjective tasks like creative writing or summarization, quality is nuanced and hard to define, limiting the direct applicability of the theory.

• Response: In fact, we relax this assumption in Appendix A.2 in the supplement, where we show that our results straightforwardly extend to a setting where filtering according to the quality function can only be performed imperfectly.

We thank the reviewer for their engagement with our paper. We appreciate their recognition that (paraphrasing from their review) we provide a unifying formalism that connects recent self-improvement heuristics (STaR / ReST) to classical boosting, and that our experiments reproduce the model-collapse phenomenon while showing how our proposed variant can mitigate it.

We respond to their concerns and questions below, nearly all of which seem to be based on misunderstandings:

Weakness: Boosting improves training accuracy and slightly increases test accuracy compared with (a) do nothing and (b) filter-only baselines, but the gains over the best supervised baseline are small.

• Response: In fact, the gains are large. Table 1 (“GSM8k test” column) shows that boosting leads to substantial improvements in test accuracy over (1) do nothing baseline (29.3% absolute, 191.5% relative); (2) filter-only baseline (7.5% absolute, 20.2% relative); (3) supervised baseline, i.e., Gold SFT (7.7% absolute, 20.9% relative).

Weakness: Experiments simulate the labeler with an already pretrained Gemma 2B, i.e. they assume access to an LLM stronger than the initial model. No evidence is given that a genuinely weak...

• Response: In fact, none of our experiments use a labeler that is stronger than the initial model. In Section 7, both the initial model and labeler are Gemma 2 2B PT, which gets a test accuracy of 21.9%. In Appendix F in the supplement (Figure 7), the initial model is also Gemma 2 2B PT, but the labeler is Gemma 1 2B PT, which gets a test accuracy of 11%. We see improvements across rounds compared to the initial model (red dotted line). This demonstrates the success of our approach with genuinely weak labeler.

Weakness: No error bars, confidence intervals, or significance tests are reported.

• Response: We do report them. In Appendix F in the supplement (Table 4 and Figure 5) we report mean and standard deviation for all results in the main text (Table 1 and Figure 1). We ran 3 trials per experiment, which is more trials than the STaR/ReST_EM papers.

Weakness/Question: The proof requires every generation to exactly fit the weighted dataset (Definition 1) ... How sensitive is convergence if the model only approximates the empirical distribution?

• Response: While our theory assumes an idealized strong learner, in our experiments we used a real learner instead of a strong learner, and our algorithm outperformed several baselines. So we think we have demonstrated that our theory is practically relevant, as it led to the development of a high-performing algorithm. Furthermore, our assumption is motivated by the widely observed phenomenon that LLMs memorize their training data. We agree that further weakening of the assumption would be an interesting extension of our theory, and view spurring research in that direction as a primary benefit of publishing our paper.

Question: Does the algorithm still converge if the labeler’s errors are systematic (biased) rather than random?

• Response: We do not assume that the labeler’s errors are random. In Section 4.2 (line 181) we say that they can be arbitrary.

Question: What is the wall-clock time and GPU budget per round for k = 8 on GSM8K?

• Response: For GSM8K, and k=8, we sample 112k responses each round, which dominates the accelerator time compared to training, and requires approximately ~16 H100-equivalent hours. Scaling up to a 35x larger model and 142x larger dataset would suggest a 5000x increase in required compute, applying the flops = 2 * params * tokens estimate (assuming inference is batched enough to be compute bottlenecked).

We thank the reviewer for their positive assessment of our paper. We appreciate their recognition that (paraphrasing from their review) our algorithm is general enough to encompass existing frameworks, enabling diagnosis of their weaknesses, and provides one of the first guarantees of convergence for self-improvement.

We address their concerns and questions below:

Weakness: The work rests on some strong assumptions for the involved LLMs. Most notably the derivation assumes that the LLM can be considered as a strong learner powerful enough to interpolate a given conditional probability (which might be unrealistic in practice).

• Response: While our theory assumes an idealized strong learner, in our experiments we used a real learner instead of a strong learner, and our algorithm outperformed several baselines. So we think we have demonstrated that our theory is practically relevant, as it led to the development of a high-performing algorithm. Furthermore, our assumption is motivated by the widely observed phenomenon that LLMs memorize their training data. We agree that further weakening of the assumption would be an interesting extension of our theory, and view spurring research in that direction as a primary benefit of publishing our paper.

Weakness: A perfect filter is also assumed.

• Response: In fact, we relax this assumption in Appendix A.2 in the supplement, where we show that our results straightforwardly extend to a setting where the filter is imperfect.

Question: I think a valuable addition would be to explicitly compare the existing frameworks mentioned in the paper (STAR, REST, REST-EM) with the proposed framework.

• Response: In fact, we explicitly compare to STaR/ReST. See line 275.

Question: Can removing the focus on hard examples allow faster improvement over the first few rounds (as hinted in Fig.1 left), hence potentially be preferable under smaller computation budgets?

• Response: We agree with the reviewer that the benefit of focusing on the hard examples appears to mostly be realized in the later rounds of our algorithm, and thus might only be worthwhile when computational resources are adequate.

Question.: A useful addition would be some further studies on the impact of the involved hyperparameters (e.g., the number of samples $k$, and fractions $\alpha$ and $\beta$).

• Response: As $\beta$ is a quantity determined by the quality of the weak labeler, we implicitly vary this parameter in our off-policy experiments in Figure 7 of the Appendix, where we experiment with different labelers and find faster improvement with larger $\beta$ as the theory predicts.