Improving the Scaling Laws of Synthetic Data with Deliberate Practice

We thank the reviewer for their thoughtful and detailed feedback.

---

On theory

Relation to Active Learning. Though related at a high level, our work differs from standard active learning, which focuses on querying labels for unlabeled data. We instead focus on generating useful training examples. The papers mentioned by the reviewer establish bounds which indicate that aggressive pruning restricted to a well crafted region leads to improved sample complexity for active learning. Our contributions differ: (1) We show that DP is equivalent to pruning. (2) We analyze this pruning in a toy setting using random matrix theory, deriving exact error curves. (3) This analysis helps explain why DP improves sample efficiency in practice.

Connection between theory & algorithm. The theory offers a principled explanation for DP’s success: it effectively samples from a pruned data distribution, without over-generating and discarding samples. We focus on high-dim regression as it’s analytically tractable yet captures the key aspects of the problem. While simplified, the goal of the theory is to isolate and analyze core components of the algorithm, helping explain its empirical effectiveness. We will clarify this in the revision.

It just seems to be a linear classifier over gaussian input data? No. The theory involves linear classification over "pruned" Gaussian data, which significantly alters the data distribution. For example, pruning distorts the standard Marchenko-Pastur law (Lemma 4), making the analysis technically involved. This required careful use of non-standard random matrix theory tools (see Appendix). We'll clarify this in the manuscript.

Use of the validation set

Using a validation set for early stopping and hyperparameter tuning is standard practice [1-4]. We follow the same principle: validation is used only for model selection, not training. Like early stopping, we detect when performance saturates, but instead of stopping, we generate new data and continue. This can be seen as repeated early stopping with dynamic dataset expansion.

Fair evaluation. In Table 1, we report on both the standard ImageNet val set (IN real Val.: commonly reported, e.g., [1, 2] since the actual test set is not public) and on the held-out training data (IN real tr.), which is untouched by the selection or training process in DP and serves as a true test set.

Validation use is helpful but not essential. While we use the validation set to determine when to generate more data, DP does not depend on it fundamentally. Alternatives include:

a. Internal signals, like training loss flattening (see Figure 5 in this anonymous link https://drive.google.com/file/d/1XbkVVsHDQyhfSJfqGkaFLC7Polb0dHZR/view).

b. Predefined schedules, e.g., adding data every X epochs.

c. A synthetic validation set

In short, our use of validation aligns with common practice and is just one possible design.

[1] Sarıyıldız et al. 2023. [2] Fan et al. 2024. [3] Dosovitskiy et al., 2021. [4] He et al. 2022.

Comparison with Hemmat et al.

While we build on feedback-guided generation, the two works differ significantly in their motivation, setup, and methodology. Our goal is to improve scaling laws in a zero-shot setting with dynamic data generation. In contrast, Hemmat et al. focus on class imbalance, using a static, one-time rebalancing step with image-conditioned generation.

Table 2 and Figure 2

In Tab 1, columns 2, 3 report baselines with varying data sizes and iterations. We include these to ensure a fair comparison with our dynamic setup, which increases data over time.

In Fig 2, the x-axis shows the final data size. E.g., to train with 2^10 examples using the "top 10%" (red), we generate 2^11 and keep the hardest 10%. The "select all" (black) strategy generates and uses all 2^10.

---

DP as the acronym

Thanks for the suggestion. We are happy to adopt the acronym to SDP (Synthetic data with Deliberate Practice). It has a nice ring to it! :)

---

Thank you again for your constructive comments. If there's anything in particular that’s holding you back from potentially increasing your score, let us know.

We thank the reviewer for their detailed and encouraging review. We’re glad to hear that you found the method to be conceptually intuitive, the theoretical analysis to be sound, and the experimental setup and evaluation thorough.

Qualitative Analysis of Hard Samples:

Thank you for raising the point about qualitatively analyzing the generated hard samples. While we do include some visual examples in Figures 6 and 10, in response to your suggestion, we have conducted additional visual analysis available at this anonymous link (https://drive.google.com/file/d/1XbkVVsHDQyhfSJfqGkaFLC7Polb0dHZR/view) (can be access in incognito mode and requires no logging in):

In the newly added Figures 1 and 2, we start with a pool of examples and compare the following sampling setups visually:

1. Uniform random selection from the pool
2. High entropy selection from the pool
3. Directly generating high entropy samples with DP

We visually observe that:

1. Uniform selection does not change the distribution of the examples.
2. High-entropy selection results in some ambiguous or atypical examples, often featuring occlusions, rare viewpoints, complex backgrounds, or low-contrast textures; factors that increase classification uncertainty.
3. DP’s direct generation produces the most visually diverse and semantically rich set of samples. These include unusual lighting, texture variation, and sometimes near-failure cases; all of which challenge the model and encourage more robust learning.

Figure 3 compares the following setups visually:

1. Initial samples with no entropy-guidance
2. DP samples with entropy-guidance earlier during training
3. DP samples with entropy-guidance later during training

We observe that early-stage generations show mainly color diversity, while later stages exhibit a richer set of transformations, aligning with the classifier’s evolving uncertainties.

Figure 4 compares the following setups visually:

1. Random samples from the initial data at the beginning of training for the class ‘fox’
2. Random samples from the final accumulated data at the end of training.

The initial training data (visually) lacks diversity. By the end of training, the accumulated dataset contains progressively harder/diverse examples.

---

Thank you again for your thoughtful comments and for recognizing the contributions of this work. We hope the addition of qualitative examples will further strengthen the final version. If there's anything in particular that’s holding you back from potentially increasing your score, we’d really appreciate hearing about it. We're happy to clarify or address any remaining concerns.

Thank you for the thoughtful and encouraging review, we are very pleased that you found the paper intuitive, practical, and effective.

Related Work and Data Pruning:

Thanks for pointing out these relevant papers. While our method focuses on improving training via generation of informative synthetic data rather than selection or pruning of real data, both lines of work share the goal of increasing data efficiency by concentrating training on examples that contribute most to learning. In particular, works like Coleman et al. (2019) and Mindermann et al. (2022) develop data selection strategies based on proxies for example utility, such as gradient norms or learnability. Similarly, Pooladzandi et al. (2022) and Yang et al. (2023) propose coreset-based methods that aim to retain the most informative or representative points. Joshi et al. (2024) further emphasize the importance of data quality over quantity, which aligns closely with our motivation, though we operationalize this through generative modeling and entropy feedback rather than dataset filtering as discussed in Sec 5.3. We view our approach as complementary: instead of selecting from a fixed pool of data, we dynamically generate examples in regions where the model exhibits high uncertainty, effectively synthesizing the types of data these methods might prioritize. We will include this discussion in the revised related work section.

DDIM Background:

We agree that the paper would benefit from a more complete and self-contained explanation of DDIM. Due to space constraints, we had to keep the background section concise, but we will expand it in the final version. We also want to clarify that while our experiments use DDIM, the proposed method is not limited to this specific sampler. Our approach relies on accessing an approximation of the denoised sample x_0 at each step, which is also available in other diffusion samplers such as DDPM and its variants. As long as the sampler provides a usable x_0 ​ estimate, the entropy-guided feedback mechanism remains cheap and can be applied efficiently. We will emphasize this more clearly in the revised version.

Language Model Extension:

This is a very relevant direction which we have been pondering about. While our current work focuses on image generative models, the principle of DP can be extended to language. One key reason this approach works efficiently with diffusion models is that we have access to an approximation of the clean sample x_0 at each step, which enables us to steer the sampling process before the generation ends. This intermediate visibility during generation allows us to influence the generation process. Extending this idea to autoregressive language models is less straightforward since generation typically proceeds token-by-token with limited ability to revise earlier outputs. However, the emergence of language diffusion models opens up a promising path [1]. These models offer a latent trajectory similar to image diffusion models and may allow for entropy-guided sampling. We see this as a compelling direction for future work and are actively thinking about how DP could be incorporated into language diffusion models for synthetic data training.

[1] Nie, Shen, et al. "Large Language Diffusion Models." arXiv preprint arXiv:2502.09992 (2025).

We are very open to further discussion and would be happy to address any other questions.

Thank you for your review.

The concept of "deliberate practice" has indeed been borrowed from psychology and has thematic connections to "curiosity". However, to the best of our knowledge, we are the first to adapt and implement this principle in the setting of training classifiers entirely on synthetic data generated from diffusion models. Rather than generating a large static dataset or pruning post hoc, we propose a dynamic approach that modifies the generative process itself to prioritize informative, high-entropy examples.

Our work goes beyond showing isolated gains, it demonstrates that deliberate practice shifts the scaling laws of training with synthetic data, enabling significantly better performance with fewer generated examples.

---

We were puzzled by the low overall rating, especially given that the review doesn’t raise specific methodological, theoretical, or experimental concerns. If there are particular issues we may have missed, we’d be happy to address them. Otherwise, if the current score was entered in error, we would be grateful if you would consider updating it.

We thank the reviewer for the detailed feedback. We're glad that you found the method conceptually clear. Also, thank you for the helpful proofreading suggestions. We will incorporate these corrections in the final version.

---

Connection Between Theory and Practice: We agree that the linear regression setup in Section 4 is a simplified version of the full setup used in practice. The goal of this section is not to precisely model the real-world setting, but to isolate and analyze the principle of prioritizing difficult examples based on entropy or uncertainty. We will include a short discussion highlighting the limitations and interpretability benefits of the theoretical analysis in the simple setup. Also, note that even though loss used in training the linear model is regression loss (squared L2), the learned model is used as a classifier and we analyze classification accuracy (see Section 4). The use of linear classifiers fitted with squared loss is common in ML theory (Couillet & Liao, 2022; Liao & Mahoney, 2021; Firdoussi et al., 2024).

Unpacking Theorem 1. The point of the theorem is to showcase that the test error can be written analytically as a function all problem parameters: parametrization rate $\phi$, regularization parameter $\lambda$, pruning ratio $p \in (0,1)$, cosine of angle between the pruning direction and the ground-truth labelling function $\rho$, etc. Analytic expression provided is defined via concrete functions like $\Phi$, and the spectral functions (linked to Marchenko-Pastur Law) introduced earlier and can be used to numerically compute / simulate the shape of the theoretical error curve in different regimes corresponding to different settings of the problem parameters. We agree with the reviewer that expanding the Phi function can provide some quantitative insights, at least in setting regimes of its argument, especially: large +ve, large -ve, and small values.

Finally, note that in the unregularized (ridgeless) case $\lambda \to 0^+$, the aforementioned analytic expression simplifies drastically, as shown in Corollary 1 (Appendix).

In Appendix A.1, the order of the limits is: first let $d,n \to \infty$ such that $d/n \to \phi$. Then, let $\lambda \to 0^+$.

Proportional scaling in linear regression. This is actually standard in high-dimensional statistics and learning theory. Here, such a scaling allows us to capture important regimes: interpolation threshold (corresponding to $\phi=p$), under-parametrized (phi < p), over-parametrized ($\phi > p$), extreme under-parametrized ($phi \to 0^+$), extreme over-parametrized ($\phi \to \infty$) regimes while keeping the analysis tractable (via the tools of random matrix theory).

The reviewer’s statement that we are “only scaling data” is inaccurate. Observe that $d,n \to \infty$ such that $d/n\to \phi$ can be equivalently written as $n = \phi d$ (rounded to an integer), $d \to \infty$. Therefore, we can fix $d$ to a large value and study the effect of $\phi$, as done in the experiments. Other large values of $d$ would yield a similar experimental picture. Everything about scale is completely captured in the $\phi$ parameter. We shall make this clearer in the manuscript.

---

Experimental Diversity:

We appreciate your suggestion to evaluate DP across a broader range of generative models and datasets. While we included comparisons in Appendix B.1 using other diffusion models, and have already reported results on two datasets (ImageNet-100 and ImageNet-1K) we acknowledge that this could have been more clearly emphasized in the main paper. We will revise the manuscript to better highlight the diversity already present in our evaluation.

We also want to clarify that LDM-1.5 was not chosen because DP performs best with it. Rather, we selected it because it consistently provides the strongest baseline performance in static data generation setups, independent of DP (as shown in Appendix B.1). This makes it a compelling and fair testbed. This model choice is in-line with prior work (e.g., Astolfi et al., 2024 and Fan et al., 2024), which uses LDM-1.5 as it produces more diverse samples than alternatives which is a desirable property when training classifiers on synthetic data.

More broadly, we do not expect DP to be limited to a particular diffusion model or sampling strategy. Our approach is a lightweight modification of the sampling process, similar in spirit to classifier guidance, that leverages the model's entropy to influence which samples to be generated. As long as the sampler provides an approximation of the denoised sample x_0, DP can be applied by using the downstream classifier to adjust the sampling trajectory. We will clarify this generality in the revised version.

---

Thank you again for your constructive comments. If there's anything in particular that’s holding you back from potentially increasing your score, let us know.