We sincerely appreciate your thoughtful suggestions and valuable feedback.

Response to "Claims and Evidence about sentences that are hard to understand":
We apologize for misunderstanding caused by our use of the term "overlap." Our intended meaning was that the curves from different transfer starting points "coincide". In the subgraph of Figure 2, we show the distribution shift curves of different transfer starting points and the horizontal axis represents the number of steps in CPT. These distribution shift curves almost "coincide" and could be fitted with a power-law form.

Response to "Choosing Knowledge-Pile":
First, we do not exclusively utilize Knowledge-Pile as the CPT dataset. We also employ Pile-of-Law [1] as a domain-specific CPT dataset and demonstrate the fitting and predictive capabilities of our scaling law in Appendix D. Second, the data in Knowledge-Pile [2] is derived and filtered from existing data sources, such as Wikipedia, arXiv, Semantic Scholar, without any atypical components. From the components [2], Knowledge-Pile is a good combination with higher STEM knowledge density, which can be used to enhance domain-specific abilities. Third, Our law is independent with the CPT dataset. When the CPT dataset distribution or quality change (better or worse), the form of our law remains applicable while the coefficients differ.

Response to "Weakness":
W1: "LRS" and "WSD":
We apologize for the reduced readability caused by the abbreviation of certain concepts in Figure 1. LRS refers to learning rate schedules, and WSD [3] denotes the Warmup-Stable-Decay schedule. We will implement these clarifications in the final version.
W2: Explain "transfer curve":
The term "transfer curve" is actually the CPT curve. As stated in line 96, "the CPT loss curve is a transitional curve on both D_pt and D_cpt validation set." The CPT curves in Figure 1 (b), (c), (e), and (f) illustrate that the curves deviate from the blue dashed line, and finally align with the orange dashed line.
W3: Explain "transfer point" and "Hidden Pre-training":
The "transfer starting point" represents the starting step point of CPT. We formally define the "Hidden Pre-training" curve in Section 2.2 and represent these trajectories with dashed lines in Figure 1.
We acknowledge that the definations of some terms were not placed appropriately, and we commit to refining these definitions in the final version.
W4: Small font size:
We apologize for the small font size in the figures and we will increase the font size in the final version.
W5: Explain "loss potential":
In the final version of the manuscript, we will incorporate a comprehensive description of the "loss potential" concept within the introduction section to establish a clearer theoretical foundation from the outset.

Response to "Comments and suggestions: The wrong caption in Figue 5 (b)(c)(e)(f)?":
We sincerely apologize for unclear captions of Figure 5 due to our intention to save space during the writing process. The complete caption in the Figure 5 is:
(b) D_cpt true loss v.s. CPT step of different loss potentials (w/o re-warmup setting)
(c) D_cpt predicted loss v.s. loss potentials of different CPT steps (w/o re-warmup setting)
(e) D_cpt true loss v.s. CPT step of different loss potentials (w/ re-warmup setting)
(f) D_cpt predicted loss v.s. loss potentials of different CPT steps (w/ re-warmup setting)

Response to "Questions: is the hidden pre-training the second epoch of the same pre-training data?":
No. The hidden pre-training is not from the second epoch. it is based on the different data from the same distribution. For example, in our experiment, we continually pre-train the model on the remaining PT dataset Fine-web.

[1] Henderson, Peter, et al. "Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset." Advances in Neural Information Processing Systems 35 (2022): 29217-29234.
[2] Fei, Zhaoye, et al. "Query of cc: unearthing large scale domain-specific knowledge from public corpora." arXiv preprint arXiv:2401.14624 (2024).
[3] Hu, Shengding, et al. "Minicpm: Unveiling the potential of small language models with scalable training strategies." arXiv preprint arXiv:2404.06395 (2024).

We sincerely appreciate your thoughtful suggestions and valuable feedback.

Response to "Comments Or Suggestions 1 about loss potential and turning length":
We apologize for any poor reading experience caused by organizational issues about these terms in our paper and will address these concerns in the final version.
For loss potential, we define in line 187 "captures the potential of future loss drop via LR annealing.", which corresponds to the height in Figure 4(c). For example, if the pre-training implements substantial annealing and end at the valley bottom in Figure 4(c), the remaining annealing room or potential for CPT is diminished. Quantitatively, we define loss potential as the ratio of the final learning rate of the pre-training annealing phase to the initial or maximum learning rate in the pre-training phase: $loss\ potential = \frac{Final\ LR\ of\ PT}{Maximum\ LR\ of\ PT}$, which is same as the legend in our Figure 5(b)(e) and x-axis of Figure 5(c)(f). In Figure 5(a)(d), if the pre-training model anneals from 2e-4 to 1e-4 using a linear method, the loss potential for this model is 50%.
For turning length, we define it in line 315 as "The minimum training steps required to achieve a lower loss are designated as the turning length." We also illustrate the meaning of this term in Figure 7(c). Qualitatively, when the CPT steps once reach the turning length, the validation loss on pre-training dataset reverts to the same loss as the very beginning of CPT as shown in Figure 7(c).

Response to "Comments Or Suggestions 2 about Figure 13":
We will implement appropriate revisions for Figure 13 in the final version of the paper.

Response to "Q1: Why is models with higher loss potential achieve lower final losses":
The statement "models with higher loss potential achieve lower final losses" indicates that pre-training models with higher loss potential can attain lower CPT validation losses and demonstrate superior adaptation to the CPT dataset. The intuitive understanding is that models with higher loss potential can allocate more annealing to the CPT phase, which enable to achieve lower D_cpt validation loss. The more specific reason of this finding is explained in line 255-265 in our paper. This finding is substantiated by the truth training loss trajectories of different loss potential models, as illustrated in Figure 5(b) and 5(e), as well as by the predicted loss values derived from our CPT law, as depicted in Figure 5(c) and 5(f).

Response to "Q2: What do you mean by releasing the high loss potential version":
Based on the above explanation, the models with higher loss potential could better adapt to the CPT dataset. However, current open-source models, such as Qwen or Llama, typically employ extensive learning rate annealing that reduces the learning rate to near-zero or minimal values, resulting in low loss potential. If releasing high loss potential variants of these models specifically (i.e., those trained without learning rate annealing), researchers and practitioners could adapt these models for CPT or downstream tasks.

Response to "Q3: Explain Finding 4":
The turning length is affected by: (1) the distribution distance between PT and CPT data; and (2) the sufficiency of pretraining. If the distribution distance is large or the pre-training is sufficient, the turning length becomes larger and even to potentially infinite. The detailed explanation is in the line 308-315 and Figure 7(c). If you have further questions about the Finding 4, feel free to further comment.

Response to "Q4: How do you set lambda_1 and lambda_2 in equation 5?":
It can be considered in two scenarios. In some situations, we could allocate a percentage ratio for validation losses on PT and CPT datasets based on our prior knowledge with the importance between general and downstream performance. Otherwise, for a specific test set that we focus and optimize, we could always precompute the linear coefficients based on dataset similarity, as demonstrated in Section 5.2.

We sincerely appreciate your suggestions. We would also like to acknowledge the thorough and constructive feedback provided which help strengthen our work, which we summarize and respond as follows.

Response to "Our work is empirical":
Our work is indeed empirical. However, most scaling laws in LLMs are empirical, such as the OpenAI [1] scaling law and the Chinchilla [2] scaling law. We believe that it is common practice in developing scaling laws of LLMs. Moreover, our scaling law can be applied to many practical scenarios We have conducted extensive experiments to demonstrate the validity of our empirical law, including different learning rate schedules in Figure 3 and Figure 10, various model sizes in Appendix E, and different CPT datasets or replay ratios in Appendix D.

Response to "No baselines":
We are the first to propose the CPT law that traces the learning dynamics of LLMs throughout the CPT process and we model the full loss curve during CPT. Our CPT law considers numerous variables that affect CPT performance, which was not addressed in previous works. Consequently, we cannot compare the accuracy of fitting or prediction with other baselines due to the absence of comparable approaches. Nevertheless, to validate our CPT law, we have compared it with alternative law formulations as baselines, as detailed in Appendix I.

Response to "Contribution of laws", "Why this may be useful", "What problem may be solved", "No extant articulated problem" and "Questions: what is a practitioner who reads this work and is now familiar with the proposed 'laws' to do differently?":
Although our scaling law is empirical, it could help researchers understand the dynamics of continual pre-training, as noted by several reviewers. Reviewer nT6s states: "This work sets a stage for subsequent research for scaling to larger model scales during continual pre-training." Similarly, Reviewer 5BNQ observes: "Overall, this is an educational paper to understand how continual pre-training works," while Reviewer oHxM acknowledges: "These contributions are unique and not seen in prior works." Our scaling law especially assists researchers in understanding the impact of various hyperparameters on the complete learning dynamics. More specifically, our scaling law could fit by very few CPT data points and accurately predict the loss across diverse hyper-parameters (e.g. LR schedule) as shown in Figure 10, which enables to efficiently optimize the training hyperparameters (Figure 8) without extensive search procedures, thereby conserving computational resources.

Response to "Findings seems to be a series of uncontroversial statements which are so general as to be obvious":
We believe that some findings in our paper may not be so obvious to certain researchers. Our scaling law validates these obvious findings, which further demonstrates the correctness of our scaling law and reliability of these findings. Most importantly, our scaling law not only provides the qualitative analysis about these findings but also enables quantitative analysis for each finding. For example, it can predict the optimal loss potential, peak LR, and other hyperparameters through the scaling law as shown in Figure 8.

Response to "Loss Potential" and "Turning Length":
We apologize for any poor reading experience caused by organizational issues about these terms in our paper and will address these concerns in the final version.
For loss potential, we define in line 187 "captures the potential of future loss drop via LR annealing.", which corresponds to the height in Figure 4(c). For example, if the pre-training implements substantial annealing and end at the valley bottom in Figure 4(c), the remaining annealing room or potential for CPT is diminished. Quantitatively, we define loss potential as the ratio of the final learning rate of the pre-training annealing phase to the initial or maximum learning rate in the pre-training phase: $loss\ potential = \frac{Final\ LR\ of\ PT}{Maximum\ LR\ of\ PT}$, which is same as the legend in our Figure 5(b)(e) and x-axis of Figure 5(c)(f). In Figure 5(a)(d), if the pre-training model anneals from 2e-4 to 1e-4 using a linear method, the loss potential for this model is 50%.
For turning length, we define it in line 315 as "The minimum training steps required to achieve a lower loss are designated as the turning length." We also illustrate the meaning of this term in Figure 7(c). Qualitatively, when the CPT steps once reach the turning length, the validation loss on pre-training dataset reverts to the same loss as the very beginning of CPT as shown in Figure 7(c).

[1] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).
[2] Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

We sincerely appreciate your recognition of our work.
Response to "Limited theoretical foundation for the derived scaling laws":
Our work is indeed empircial. However, most scaling laws in LLMs are empirical, such as the OpenAI [1] scaling law and the Chinchilla [2] scaling law. We believe that it is common practice in developing scaling laws of LLMs.

[1] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).
[2] Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

Response to "Smaller model scales (0.1-1B parameters)":
The largerest model size in our experiment is 1.7B parameters without embedding, and we also conduct experiments with LLaMA-3.2-1B in Appendix G. We believe this model size is reansonable in the current context of large language models. Due to resource constraints, we are indeed unable to train larger models. Our scaling law has demonstrated consistent patterns across these model sizes (0.1-1.7B), so we believe our law is also applicable for larger models.

Response to "Comments Or Suggestions 1" and "Q1" about "Loss Potential":
We apologize for any poor reading experience caused by organizational issues about these terms in our paper and will address these concerns in the final version.
For loss potential, we define in line 187 "captures the potential of future loss drop via LR annealing.", which corresponds to the height in Figure 4(c). For example, if the pre-training implements substantial annealing and end at the valley bottom in Figure 4(c), the remaining annealing room or potential for CPT is diminished. Quantitatively, we define loss potential as the ratio of the final learning rate of the pre-training annealing phase to the initial or maximum learning rate in the pre-training phase: $loss\ potential = \frac{Final\ LR\ of\ PT}{Maximum\ LR\ of\ PT}$, which is same as the legend in our Figure 5(b)(e) and x-axis of Figure 5(c)(f). For example, in Figure 5(a)(d), if the pre-training model anneals from 2e-4 to 1e-4 using a linear method, the loss potential for this model is 50%.

Response to "Comments Or Suggestions 2: typo in figure 5":
We apologize for the typo in the legend of Figure 5(a); the correct value is 10% loss potential, not 0%. We will revise it in the final version.

Response to "Q2: loss potential is relative to D_pt validation loss or D_cpt validation loss":
Based on the defination of $loss\ potential$ above, the loss potential is neither related to D_PT validation loss nor D_CPT validation loss but rather represents the annealing potential of the models. Our findings indicate that pre-trained models with higher loss potential can achieve lower D_CPT validation losses, suggesting that these models could better adapt to the downstream datasets.