Predictable Scale (Part II) --- Farseer: A Refined Scaling Law in LLMs

Thank you for your efforts and contributions in reviewing the paper. Below are our responses to the relevant issues; we hope they will address your concerns and improve the quality of the manuscript.

W1: Clarity on the transition from the formula on line 92 to line 97

Thank you for pointing out this lack of clarity. You are correct that the transition was abrupt and lacked proper explanation. The formula on line 92 is the general, high-level form that we hypothesize. The formula on line 97 is the specific, empirically-derived functional form that resulted from our analysis in Section 3. Specifically, through our data exploration, we found that the general terms could be effectively approximated as follows:

• The term V(D)+H(N,D) is approximated by $f_B(N;θ_{B∗})D^{f_A(N;θA∗)}$
• The term E+U(N) is approximated by $f_U(N;θ_{U∗})$ We improperly present the final form on line 97 before detailing the derivation and justification. This section will be revised in the new version of the manuscript to ensure a logical flow, in which introduce the general scaling law and then systematically explain how our empirical findings lead to the specific functional forms, defining all terms before they are used.

W2: Why H(N, D) is not included in model-dependent terms

This is an excellent and perceptive question.Our choice to group H(N, D) with the data-dependent term V(D) is motivated by both theoretical precedent and strong empirical evidence from our own analysis. On one hand, our formulation is inspired by the theoretical analysis in the Chinchilla paper(Appendix D.2.) [1], which we use as a foundational starting point. In their work, the total loss is decomposed into:

1. A term limited by the model size (N), which is $L(f_N) - L(f^\star)$
2. A term, which we denote as H, that is related to the data size (D) and the training dynamics, which is $L(\bar{f}_{N,D}) - L(f_N)$

On the other hand, our decision to treat the term $H(N, D)$ as a power-law function of the data size, $D$, was fundamentally an empirical one, guided by the results of our analysis. Our first-order difference analysis (Section 3.1) provided strong evidence for this modeling choice. Specifically, as shown in Figure 3a, the analysis of $\Delta_D L(N,D)$ versus $D$ revealed a remarkably stable and clear power-law relationship, evidenced by a high $R^2$ value of 0.9807. This empirical observation led us to model $H(N,D)$ as a power law with respect to $D$. Crucially, we recognize that this power-law relationship is not independent of the model size, $N$. Our findings indicate that the parameters of this power law, most notably the exponent, are themselves a function of $N$. Our key contribution, therefore, is demonstrating that the complex behavior of the $H(N,D)$ term can be effectively captured by modeling it as a power law in $D$, whose characteristics are modulated by $N$. We will clarify this distinction in the revised manuscript.

We admit that the original manuscript did not explain this motivation clearly. In the revised version, we will explicitly state that we adopt the loss decomposition from Chinchilla and then clearly articulate that our primary contribution is demonstrating and modeling the previously overlooked dependence of the H term on N.

W3: Increased degrees of freedom, fitting cost

Regarding the additional parameter: Our parameters is intentionally introduced and is crucial for capturing the interaction effect between model size (N) and data size (D), which, as our results show, is a key factor that existing laws fail to model. Regarding the fitting cost:

1. Farseer achieves a comparable and evne more accurate with a small number of data points. While more parameters can sometimes require more data, our Farseer law demonstrates greater efficiency in capturing the true scaling behavior. In fact, our Figure 8 already provides a preliminary illustration of this, showing that Farseer achieves a more accurate fit even with a small number of data points.
2. The error of Chinchilla becomes divergent, while Farseer's remains stable. This is a key contribution of our work. It is the empirical demonstration that Chinchilla's formulation, which lacks the higher-order interaction term H(N, D), leads to divergent prediction errors, as shown in Figure 1a and Figure 2. In contrast, Farseer's error remains convergent and stable across the full range of N.
3. Farseer is closer to the true nature of Scaling Law, not just a simple accuracy-cost trade-off. The primary reason for Chinchilla's divergent error is that it extrapolates using a single, average rate of data scaling derived from its fitting set to predict performance for all other model sizes. This means that as the target model size N for prediction grows larger and deviates from the fitting range, Chinchilla's error is expected to increase and diverge. Farseer, however, is designed to capture this variance by modeling the interaction between N and D, leading to higher predictive confidence for larger models. This is critical because in most industrial and research applications, the goal of using scaling laws is precisely to forecast the performance of these larger, more resource-intensive models. From another perspective, Chinchilla does not inherently benefit from being fitted on a wider range of model sizes (N). Farseer, on the other hand, leverages the additional data points across N to build a more accurate model of how data scaling dynamics change with model size. Therefore, this is not a simple trade-off between prediction accuracy and fitting cost. Instead, it concerns the fundamental capability of the scaling law to generalize reliably to the large-scale regimes that are of greatest practical interest. 4 . A scaling law is only valuable if its prediction error is within a reasonable range. Here, we wish to add an observation from our industrial practice. When we need to use large-scale models to determine key conclusions, such as the effectiveness of different Attention structures, experiments are often conducted at the scale of tens of billions of parameters (e.g., >20B) and hundreds of billions of tokens. In this context, a performance difference between two approaches must typically exceed 0.6% for us to consider it a significant gap, while differences below this threshold are often attributed to noise or engineering implementation details. In our experience, the performance gap between two distinct, viable solutions rarely exceeds 3%. However, as shown in Figure 8, Chinchilla's prediction error for a 25B model already surpasses 2.5%. An error of this magnitude severely diminishes its practical value, as it can obscure the true performance differences between models and lead to incorrect conclusions.

About limitations:

We had previously included an extensive two-page discussion of "Discussion and Future Work" in Appendix J. However, we completely agree that a more thorough and centralized discussion of limitations is needed. Prompted by your suggestion, we recognize that this topic deserves a more prominent treatment in the main paper. To that end, we have added a dedicated "Limitations" section to the revised manuscript. In this new section, we will not only summarize the various aspects we originally mentioned in Appendix J, but we will also condense and integrate our points regarding the "fitting cost" of Farseer(as mentioned above). We agree that adding this section to the main body makes the paper stronger, and we appreciate you pushing us to improve its clarity and completeness.

[1] Training Compute-Optimal Large Language Models

We sincerely thank you for your thorough review and positive assessment of our work. We are very encouraged by your recognition of our paper's clarity, novelty, and significance. Your insightful suggestions are invaluable for improving our manuscript. Below are our responses to your specific points.

W1 & Q1: More baselines and downstream tasks

Thank you for these excellent suggestions.

Regarding the inclusion of more baselines, we focused our comparison on the Chinchilla scaling law because it represents the most widely recognized and dominant formulation in this area. Many other proposed scaling laws are often modifications of the Chinchilla framework or are tailored for very specific, narrow tasks. Therefore, we believe that a direct and thorough comparison with Chinchilla provides the clearest demonstration of Farseer's advancement over the current state-of-the-art.

Regarding the prediction of downstream task performance, we agree this is a very important research direction. We did not include it in this work because establishing a precise and stable scaling law for downstream benchmarks is a substantial research challenge in its own right, due to several sources of high variance in their evaluation.

Specifically, we identify three main issues:

1. Stochasticity in Decoding: Most benchmarks rely on an LLM decoding process followed by an answer-matching script, a method that is inherently probabilistic. A model's score can fluctuate significantly across identical runs. While this can be partially mitigated by averaging over multiple decoding attempts (i.e., avg@N), it introduces measurement noise that complicates the modeling of a clean scaling signal.
2. Benchmark Sensitivity and Scope: The performance on any single benchmark can be volatile. This arises because (a) benchmarks often have a limited number of test items, making scores susceptible to high variance, and (b) they are extremely sensitive to the model's output formatting and its ability to follow instructions. A pretrained model's instruction-following capability can change drastically over very few training iterations depending on recent data exposure. This can cause large swings in benchmark scores that reflect shifts in instruction adherence rather than fundamental changes in the model's core capabilities. To obtain a more stable signal, one might need to cluster similar benchmarks and predict an aggregate metric, which adds its own layer of modeling complexity.
3. Confounding Effects of Fine-Tuning: To fairly compare the underlying capabilities of different pretrained models, one cannot simply test them directly. A rigorous evaluation would require first aligning each model with a standardized, lightweight instruction-following fine-tuning. This would normalize their instruction-following abilities to a similar level before evaluation. However, designing and constructing such a benchmark instruction-following dataset is a highly demanding task in itself. For these reasons, directly modeling downstream performance introduces significant confounding variables that would obscure the scaling dynamics related to pre-training. We believe that rigorously addressing these measurement challenges to isolate a clean scaling law for downstream tasks is a substantial endeavor that warrants a dedicated future study. We have added a discussion of this as a promising direction for future work in our revised manuscript.

W2: Minor comments on typos and formatting

Thank you very much for your meticulous reading and for pointing out these issues. We appreciate you catching the formatting suggestions for Figures 2 and 7, the LaTeX error in line 130, the syntax in lines 146-147, and the missing bolding in line 168. We have corrected all of these in the updated version of the paper.

Once again, we thank you for your valuable feedback and strong support for our paper. We hope our responses have satisfactorily addressed your questions.

Thank you for your detailed, thoughtful, and constructive review. We are encouraged by your positive assessment of our work's significance, originality, and the value of our open-source contributions. We appreciate the specific questions and suggestions for improvement, which will help us strengthen the paper. Below, we address the weaknesses and questions you raised.

W1: Prediction precision and cost.

You are right to point out that in Figure 8, Farseer's error is comparable to Chinchilla's when fitting on a very small number of models. This is an expected behavior. Because our formula is designed to capture the learning dynamics across a wider range of model sizes (N), it requires more data points across N to realize its full predictive power. When using only a few fitting points from smaller models, its performance naturally converges to the Chinchilla baseline. Its performance is largely comparable to the Chinchilla baseline when using only a few fitting points, and while it shows a very slight disadvantage at the initial point in Figure 8b, its superiority becomes clear as the number and scale of models used for fitting increase. Furthermore, a key contribution of our work is the empirical demonstration that Chinchilla's formulation, which lacks the higher-order interaction term H(N, D), leads to divergent prediction errors, as shown in Figure 1a and Figure 2. In contrast, Farseer's error remains convergent and stable across the full range of N. The primary reason for this is that the Chinchilla model extrapolates using a single, average rate of data scaling derived from its fitting set to predict performance for all other model sizes. This means that as the target model size N for prediction grows larger and deviates from the fitting range, Chinchilla's error is expected to increase and diverge. Farseer, however, is designed to capture this variance, leading to higher predictive confidence for larger models. This is critical because in most industrial and research applications, the goal of using scaling laws is precisely to forecast the performance of these larger, more resource-intensive models. From another perspective, Chinchilla does not inherently benefit from being fitted on a wider range of model sizes (N). Farseer, on the other hand, leverages the additional data points across N to build a more accurate model of how data scaling dynamics change with model size. Therefore, this is not a simple trade-off between prediction accuracy and fitting cost. Instead, it concerns the fundamental capability of the scaling law to generalize reliably to the large-scale regimes that are of greatest practical interest.

W2: On defining a "low enough" loss prediction error and performance on model selection tasks like DataDecide.

Our definition of "low enough" is based on the relative error at scale, which approaches the inherent noise from factors like training stability or data parallelism. The key finding is the significant improvement over the Chinchilla baseline within this context. Thank you for the excellent suggestion to evaluate on predictive model selection tasks like DataDecide. This is a great way to measure the practical impact of improved loss prediction, and we will explore incorporating such an evaluation to more directly demonstrate the benefits of our method for development decisions. To supplement this, we can share an observation from our industrial practice. When making critical architectural decisions, such as comparing different Attention structures, experiments are typically conducted with models in the scale of tens of billions of parameters trained on hundreds of billions of tokens. In this context, a performance gap between two methods often needs to exceed 0.6% to be considered significant, while the entire performance difference between two competing approaches is rarely more than 3%. However, as shown in Figure 8b, Chinchilla's relative prediction error at the 25.1B scale already exceeds 2.5%. This level of inaccuracy severely undermines its practical value, as the prediction error is nearly as large as the performance margin one hopes to distinguish.

W3, Q2, Q4, Q8, Q9, Q11, Q15: On notation, clarity, and presentation of the formulas and methodology.

Thank you for this collection of detailed and very helpful suggestions regarding the clarity of our paper. We sincerely appreciate your careful reading and feedback. We agree that the presentation can be significantly improved. In the revised version, we will perform a thorough update on our notation and presentation based on your feedback. This includes:

• Adopting more mnemonic function names (e.g., your suggestion of L_N(N) instead of U(N)).
• Adding direct, plain-language explanations of terms like A(N), B(N), G(.), and U(.) to the figure captions to make them self-contained.
• Moving the overview of experiments (Figure 7) earlier to clarify the data requirements for our fitting method from the outset.
• Explicitly stating that all parameters other than N and D are fitted.
• Restructuring the beginning of Section 3.2 to more clearly state the motivation and logic of our fitting procedure. We believe these changes will substantially improve the paper's readability and logical flow.

Q1 & Q3: On the release of training code and the curated validation set.

Thank you for your interest. We are open-sourcing the Farseer fitting code. The training code, which is a modified version of the open-source project Megatron-LM, will not be released. Our primary modifications involve custom implementations to accelerate Data Parallelism (DP) and Tensor Parallelism (TP). For the smaller models in our study, which do not require complex parallel strategies, we believe that training with the standard community version of the codebase would not significantly impact the results.

Furthermore, ensuring the validity and unbiased nature of the validation set was a resource-intensive endeavor, requiring substantial manual effort to maintain impartiality in the data curation process. For instance, the book portion of the set was fully annotated by human reviewers, rather than relying solely on machine-based OCR. We understand the value that releasing this validation set would provide to the community for reproducing our work. Consequently, we are actively seeking approval to make this dataset publicly available in the future.

The 2024 data cutoff for the English validation set is indeed intended to ensure it contains data unseen by the models, as its academic paper sources are from arXiv publications after 2024. We can confirm that we constructed similar validation sets for Chinese and code, and the fitting performance was also strong on these, though the English validation set was the primary one used for the results reported in the paper.

Q5: Regarding the specific slice shown in Figure 2.

In Figure 2, the model size is fixed at N=6B. We chose this specific slice to provide a clear, direct comparison between the Farseer and Chinchilla functional forms against the ground-truth loss points. This visualization effectively highlights the improved fit our proposed form provides for a given model size across varying data sizes.

Q6 & Q7: On the role of scaling laws and the meaning of the "scaling gap".

To clarify our statement that LM progress is "attributed to scaling laws," we mean that the discovery and use of these laws have given researchers the confidence to invest massive compute resources, knowing they can be predictably converted into model capability. Scaling laws have been essential for guiding and evaluating this process. By "scaling gap," we refer to the phenomenon where insights and conclusions drawn from small-scale experiments (e.g., identifying a superior dataset or config) do not consistently hold true when scaling up to much larger models. A more accurate scaling law, like Farseer, can help mitigate this by providing more reliable predictions of large-scale performance, thereby making small-scale exploratory experiments more predictive and valuable. In fact, Appendix E provides a simple example from this scaling-law perspective.

Q10: Regarding the x-axis in Figure 6 and its relation to Figure 8.

You are correct. The x-axis label in both Figure 6 and Figure 8 is "Largest model size used in fitting". The figures do show similar trends but are complementary as they show extrapolation to different target model sizes. We agree Figure 8 is more informative with the Chinchilla baseline and will consider merging or refining the presentation to avoid redundancy.

Q12, Q13, Q14, Q16: On various typos and clarifications.

Thank you for spotting these. You are correct on all points.

• Line 86: The reference should indeed point to Appendix A, which details hyperparameter and architecture ablations. Appendix F contains the specific model settings.
• Figure 3: We will increase the tick density on the axes. The y-axis direction indicating faster loss improvement will be clarified. The R² value shown is the average R² across each of the fitted lines. We will clarify the caption and define terms before they are used.
• Line 130: We will correct the LaTeX typo.
• Line 162: You are right, the reference should be to Figure 5(b).

We will correct all of these in the revision. Thank you again for your meticulous review.

We sincerely thank you for your thorough review and positive evaluation of our work. We appreciate your recognition of Farseer's innovation, extensive validation, and practical value. Your questions are insightful, and we would like to address them below.

Disadvantage 1 & Q1: Regarding the theoretical explanation for the formula's exponential form

You correctly pointed out that our paper lacks a deep theoretical explanation for the chosen exponential form. We agree that a complete theoretical justification for a scaling law's functional form is a challenging but important goal. While a first-principles derivation remains an open problem, we did empirically validate our choice. To help the community tackle this open problem, we have open-sourced our extensive experimental data, models, codes and training curves. As you suggested, we compared our exponential form against other functional forms. Due to space constraints, these results were included in Appendix C.3, Table 2. The results clearly demonstrate that our proposed formula provides the best fit among the alternatives we tested, justifying our choice from an empirical standpoint.

Disadvantage 2 & Q2: Regarding the maximum validation scale of 25.1B parameters

You raised a valid concern about the extrapolation of Farseer to models larger than our maximum validated size of 25.1B parameters. We acknowledge this limitation. This was primarily due to constraints on computational resources and the increasing engineering complexity of stable, large-scale parallel training. We believe our experiments up to 25.1B provide strong evidence for Farseer's effectiveness, and we acknowledge that validating on even larger models is a critical next step as resources permit.

Q3: Regarding the scope of model architectures and data types

Thank you for this excellent suggestion. We agree that testing across diverse architectures and data modalities is crucial for generalizability. We are pleased to clarify two points:

• Data Types: In Section 4.1 and Figure 6, we demonstrate Farseer's robustness across different data distributions. Furthermore, as part of our ongoing work, we have conducted a detailed study of the code domain, including the effects of various code-mixing ratios. These experiments confirm that the Farseer scaling law continues to yield highly accurate predictions. We believe a full analysis of these results is an important direction for future work and plan to report them separately.
• Architectures: As for architectures like Mixture-of-Experts (MoE), they introduce additional complexities and variables (e.g., sparsity, fine-grained routing) that require more detailed modeling and experimentation. This is an exciting direction that we have planned for our future work.

We hope these clarifications address your concerns and further highlight the contributions of our work. Thank you once again for your valuable feedback.