LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

Thanks for your helpful comments and feedback.

[EDA1 - Conclusions are subjective]: We now quantify our findings in two ways:

1. We quantify the goodness of fit of the loss-to-loss power laws as $R^2$. We show this in our revised Fig. 1 (note that Fig. 1 has also received other updates; please refer to our reply to Reviewer he3E). Generally, the goodness of fit is very close to $1$. We will similarly update all figures for the camera-ready.
2. We quantify the impact of different interventions as the area between fitted curves in a newly added Tab. 2. Pretraining data clearly has the biggest impact on the scaling laws. Please also refer to our reply to Reviewer he3E for our updated conclusion on the impact of the tokenizer.

[Conventional vs. loss-to-loss scaling laws]: The distinction between compute-to-loss and loss-to-loss scaling laws may not have been sufficiently clear in the text and caused some confusion; we have updated our introduction and related work sections to explain this better. To reiterate, we focus on loss-to-loss scaling laws (i.e., train-to-train, train-to-test, test-to-test), not compute-to-loss scaling laws (e.g., Kaplan et al., 2020; Hoffmann et al., 2022). While compute-to-loss scaling laws are primarily used to find optimal compute budgets, loss-to-loss scaling laws can help study generalization, i.e., how performance transfers from training distributions to downstream tasks (Brandfonbrener et al., 2024; Du et al., 2025). Although compute-to-test scaling can be informative, they do not allow for connecting training loss to downstream performance, as you pointed out. This is where loss-to-loss scaling laws become a fascinating object of study, as they explicitly model how training/validation loss is converted to test loss, a proxy for test performance. Please also see our response to reviewer fuqS on the utility of loss-to-loss scaling laws.

[W1, W2 - Findings are implied by existing scaling law formulations]: We respectfully disagree. As the reviewer points out correctly, conventional scaling laws (i.e., compute-to-loss scaling laws) are a function of model and data size. That formulation alone does not automatically imply that changing data quality without affecting data size should impact the scaling law.

Having said that, for compute-to-loss scaling laws, it is of course well known that data quality does play a role. However, other factors like architecture, optimizer, and learning rate scheduler also matter for compute-to-loss scaling laws (Li et al., 2025). It is not at all “inherently implied in existing scaling law formulations” which factors impact scaling laws and which don’t – after all, none of these factors are quantifiable and explicitly enter the scaling law.

Now, loss-to-loss scaling laws are a relatively recent object of study. Prior to our study (Takeaway 1), it was unclear whether loss-to-loss scaling laws could generally be described by shifted power laws at all, since previous studies like Brandfonbrener et al. are very limited in their settings. On top of that, it is unclear which factors, if any, play a role. Again, it is not at all “inherently implied in existing scaling law formulations” whether pretraining data (quality, not size), architecture, tokenizer, or other hyper parameters might impact the scaling law. If we were to go by the experience of compute-to-loss scaling laws, we might assume that most of these factors matter. In contrast, we find that loss-to-loss scaling laws are notably insensitive to many factors, while strongly depending on the pretraining data. As reviewer EE76 points out, “this is of broader interest to the community, in terms of advancing our understanding of the role of architecture, optimization, etc. in downstream properties”.

[W3b - Mamba & Llama perform similarly]: As you note, determining whether drastically different architectures like transformer-based Llama and state-space-based Mamba achieve similar downstream performance given identical training setups is a significant open question.

We have to emphasize the distinction between compute-to-loss and loss-to-loss training laws. While compute-to-loss training laws can answer whether two models converge to the same training loss with increasing compute, loss-to-loss training laws answer whether models reach the same downstream performance given a training loss.

In this sense, yes, it is precisely the case that Mamba and Llama models “converge to similar downstream performance across tasks” given they reach the same training loss. To further illustrate this, we present in Tab. 3 multiple Llama and Mamba models trained under identical conditions, which show nearly indistinguishable downstream performances.

We hope this addresses your concerns and encourages you to raise your score.

(Li et al, 2025) (Mis)Fitting: A Survey of Scaling Laws

We thank you for your helpful comments and feedback. We address each of your questions and concerns below.

[Q1 - Models trained and evaluated and complete results]: Thank you for noticing this; we have made amendments in multiple places: First, we now mention in Sec. 4 the size of not only our models, but also the models sourced from HuggingFace. Second, we have expanded on App. A and Tab. 1 to clarify the models we trained and created a new table for models we evaluated from HuggingFace (see Tab. 2). Third, we will release the entire data frame with all model settings and evaluation results for the roughly 6000 checkpoints and our complete code with the camera-ready version.

Here’s an abridged overview of the evaluated models:

• We trained Llama and Mamba models ranging from approximately 60M to 420M parameters from scratch. We have also added some 7B models now, see below.
• We evaluated pretrained models from HuggingFace: Pythia (70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B), GPT-Neo models (125M, 1.3B, 2.7B), GPT-NeoX (20B), and GPT-J (6B), all trained on variants of The Pile. These models are included in the right-most columns of Figs. 4–6, 16–18, and 22–27.
• We also evaluated FineWeb ablation models (1.7B) from HuggingFace trained on The Pile, C4, and FineWeb-Edu. These models are also included in Figs. 4-6 (e.g. Fig. 4 column 3).

Note that although we fit scaling laws using all available checkpoints, we display a randomly selected subset in figures for readability. We have made this more evident in the text and figure captions.

[CE1 - Training/Evaluating large models]: As explained above, our analysis already includes publicly available pretrained models with up to 1.7B parameters for Llama, up to 20B parameters for GPT, and up to 2.7B parameters for Mamba. We apologize for this not being evident in the original manuscript. As illustrated in Figs. 4-6 (especially in the right-most column showing the largest GPT models), these models generally achieve lower loss on validation and test sets compared to our models trained from scratch, but still adhere to the power-law trends and confirm our key insights. For example, the right-most column in Fig. 6 contains all of the largest HuggingFace models and clearly demonstrates that architecture does not affect loss-to-loss scaling laws even at these scales.

Following your suggestion, we revisited available pretrained models. However, most other publicly available checkpoints of large scale models (≥7B) are unsuitable for our analysis as they are trained on (often undisclosed) data mixtures that prevent us from directly comparing them to other models (.e.g, the Falcon series).

All that said, we agree that especially the impact of the pretraining data in Fig. 4 is not conclusively shown for larger models, since the largest HuggingFace models included in our analysis were all trained on variants of The Pile (see Fig. 4, right-most column). To remedy this, we have now trained additional 7B models for 1B tokens ourselves. Due to compute and time constraints we have limited this to three settings:

• Llama-7B (tiktoken tokenizer) on FineWeb-Edu / on C4
• Llama-7B (GPT-2 tokenizer) on FineWeb-Edu

We’ve included all these models in our revised Fig. 1. All models follow the established power-law scaling curves and confirm our main conclusions.

[OCS1 - Clarifying taxonomy used in paper]: Thanks for the suggestion. We will state and clarify the taxonomy used in the paper early in the manuscript. We have also added an appendix section that explains the scaling laws in more detail, which we describe in our reply to reviewer EE76.

[OCS2 - Normalizing tokenizer comparisons]: Thank you for this excellent suggestion! We performed the experiment and have updated Fig. 1 for BPB. Indeed, after normalizing negative log-probability by bytes, the previously distinct tokenizer lines collapse onto each other. This confirms even more strongly that architecture, tokenizer, and optimization hyperparameters minimally affect loss-to-loss scaling—only pretraining data distribution truly matters. We will update other figures and sections of the manuscript accordingly.

We hope this addresses your concerns fully and encourages you to raise your score.

Thank you for your helpful comments and feedback. We address your concerns below:

[Q1, Q3, OSW Clarity 1 - On the construction and fitting of scaling laws]: We have added two Appendix sections that (1) detail the scaling law formulation and (2) explain how parameters are estimated. Here’s a slightly abridged version:

Scaling Law Details We adopt the compute-to-loss scaling law formulation from (Brandfonbrener et al.) Eq. (4): $L\left(f^{(N, D)}\right) = E + \left( \left( \frac{A}{N} \right)^{\frac{\alpha}{\beta}} + \frac{B}{D} \right)^\beta,$ where $f^{(N, D)}$ is a model with $N$ parameters trained on $D$ tokens and $E, A, B, \alpha, \beta$ are parameters to be fit. Notably, the irreducible error $E$ captures the minimum loss possible for model $f$ in the limit of infinite model and data size.

By default, this is fit using the training or validation loss. However, as demonstrated by (Brandfonbrener et al.) and our experiments, we can alternatively predict the loss $L_x$ on dataset $\mathcal D_x$ achieved by model $f_p^{(N, D)}$ trained on the pretraining set $\mathcal D_p$: $L_x\left(f_p^{(N, D)}\right) = E_{x|p} + \left( \left( \frac{A}{N} \right)^{\frac{\alpha}{\beta}} + \frac{B}{D} \right)^\beta.$ As in (Brandfonbrener et al.) Eq. (7), the irreducible error $E_{x|p} = L_x(f_p^*)$ then captures the minimum possible loss on $\mathcal D_x$ of a model trained on $\mathcal D_p$.

With that, we can formulate the loss-to-loss scaling law for arbitrary combinations of pretraining data and two test or validation sets, as stated in Eq. (1).

Fitting Details For each line in a plot corresponding to a loss-to-loss scaling law from Eq. (1), we first fit the two compute-to-loss scaling laws $L_x(f_p^{(N, D)})$ and $L_y(f_p^{(N, D)})$. This yields estimates for the irreducible errors $E_{x|p}, E_{y|p}$, which correspond to the minimum x- and y-value of the loss-to-loss line. We use SciPy's default curve_fit optimizer for fitting. In rare cases when all checkpoints have the same number of parameters $N$ or same number of tokens $D$ (this is the case only for a small subset of the HuggingFace models) and a compute-to-loss scaling law cannot be fitted, we instead estimate the irreducible error as the minimum loss achieved: $E_{x|p} = \min_{N,D} L_x\left(f_p^{(N, D)}\right)$. With $E_{x|p}, E_{y|p}$ from the compute-to-loss fits, we again use SciPy's curve_fit to fit $K, \kappa$ for the loss-to-loss scaling law from Eq. (1).

We add examples of compute-to-loss fits for some of the loss-to-loss scaling laws from Fig. 2 in this section; see new Fig. 10. Note also that while Figs. 4-6 show averaged eval/validation performance, Figs. 22-27 in App. F includes results for specific validation and evaluation datasets (C4, The Pile UC, ARC-Easy, and HellaSwag).

[OSW Clarity 1a - More details on experimental setups]: We have updated all the figure captions, Tab. 1, and Tab. 2 in the appendix to include more details for the models used.

[OSW Clarity 1b - Clearer axes labels]: Thank you for pointing this out. In our case, plot colors correspond to different dimensions in different plots. In Fig. 1, the x- and y-axis correspond to two specific datasets, and colors denote different interventions. In Fig. 2, the x-axis is fixed, and colors denote the dataset of the y-axis (most similar to Brandfonbrener). In Figs. 4-6, axes are again fixed (and y-axis reports an average over multiple datasets), and colors denote different values for the specific intervention. In all cases, the training set is specified in the caption. We have now updated the figure captions and legend layout to clarify this.

[W1 and Q2 - Modeling the influence of pretraining data]: Quantitatively modeling the influence of data distributions is an intriguing open problem. The central issue is the difficulty in mapping differences in data distributions onto a simple feature space (e.g., scalar or vector values). In Data Mixing Laws [2], the mixing ratio serves this purpose effectively. However, we face the additional complexity of quantitatively comparing disjoint pretraining distributions with unknown compositions, a challenge beyond the scope of this work. We also do not believe multi-loss-to-loss scaling laws offer a satisfying solution. Without a reliable way to quantify diverse large-scale distributions and their relationships, we would need separate multivariate power laws for each combination of losses, again without a direct means for meaningful comparison.

[OSW2 and Q4 - Practical utility of the paper]: We refer you to our response to Reviewer fuqS (section: [W1 & Q2b - On the utility of loss-to-loss scaling curves]) for an abridged version.

We hope this addresses your concerns and encourages you to raise your score.

We thank you for the helpful feedback. We address your concerns as follows:

[W1 & Q2a - On comparing raw numbers]: Comparing individual performance metrics like MMLU is indeed possible and can illustrate the effectiveness of an intervention for a specific model scale and setting. For example, we added Table 3 to the appendix, which shows the raw numbers for MMLU and several other tasks. Note that Figs. 22-27 in App. F already shows versions of Figs. 4-6 for specific validation and evaluation datasets (C4, The Pile UC, ARC-Easy, and HellaSwag). That said, comparisons on a single model scale and setting or on a single test set fall short in understanding whether the effectiveness of an intervention is dependent on model size, dataset size, downstream task or other factors. Additionally, evaluations on a single test set like MMLU are noisy, as evident from Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, and Fig. 15, which we have added to the appendix. Our comprehensive study addresses this by systematically evaluating performance across multiple scales, factors, and downstream tasks while rigorously controlling all training parameters (including learning rate, optimizer, context length, tokenizer, model and dataset sizes).

[W1 & Q2b On the utility of loss-to-loss scaling curves]: Loss-to-loss scaling laws across datasets provide critical insights beyond single-setting performance comparisons, as detailed in our related work and discussion sections. We reiterate the most important points here and have updated the corresponding sections in the paper to state this more clearly:

• Generalization: Loss-to-loss scaling laws (train-to-train, train-to-test, or test-to-test) provide insight into how performance transfers across datasets (Taori et al., 2020, Fang et al., 2022, Brandfonbrener et al., 2024). Specifically, these scaling laws help answer the generalization question: if a model achieves a certain performance on a given dataset (generally training dataset), how well does it do on another task/dataset? As Reviewer EE76 rightly points out, this can be of “broader interest to the community, in terms of advancing our understanding of the role of architecture, optimization, etc. in downstream properties.”

• Compute Budget Translation: By combining train-to-test scaling laws with compute-to-train scaling laws, we can more precisely understand how compute budget translates into downstream task performance (Brandfonbrener et al., 2024), and also help uncover emergent model abilities (Du et al., 2025).

• Factor Decomposition: Separately analyzing compute-to-train and train-to-downstream scaling laws helps isolate factors influencing each step. For instance, in our work, we identify that architecture and optimizer settings do not influence loss-to-loss scaling laws — but they do affect compute-to-train scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022; Brandfonbrener et al., 2024; Porian et al., 2024; Li et al., 2025). As a result, architectures and optimizers can be independently optimized to enhance compute scaling without negatively impacting downstream task performance.

• Limitations of Single-Dataset Validation: As Reviewer EE76 highlights, our findings caution against relying solely on one validation dataset's loss (as typical in data curation methods, e.g., (Liu et al., 2024)) — achieving a particular loss on one validation set can correspond to varying downstream task performances. Thus, a single validation dataset may not comprehensively indicate downstream efficacy. By illuminating the factors that consistently impact loss-to-loss scaling laws, we enable practitioners and researchers to use them as a tool for analyzing and optimizing model training.

[Q1 - On the irreducible errors]: We have added two Appendix sections that detail the scaling law formulation and explain how parameters are estimated; please refer to reply to Reviewer EE76 for more details.

[Regarding the referenced paper]: We are happy to include the suggested reference in our camera-ready version. As you rightly point out, the impact of pretraining data on loss-to-loss scaling laws has been shown before (Taori et al., 2020, Fang et al., 2022, Brandfonbrener et al., 2024), and our results further confirm this finding. However, to the best of our knowledge, our study is the first to show this comprehensively for a large number of datasets and model settings and is the first to analyze the impact of other factors like architecture, optimizer settings, and context length.

We hope this addresses your concerns fully and encourages you to raise your score.

(Li et al, 2025) (Mis)Fitting: A Survey of Scaling Laws