We sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We especially appreciate the recognition of its novelty and relevance, as well as the clarity of our presentation. We also appreciate the constructive suggestions. Let us clarify some of the questions and comments raised in the following.

Question on Relevance to transfer learning and other scaling laws

We have briefly discussed connections to transfer learning (and cited relevant two-stage training methods like model growth and pretraining/SFT) in Section 7, but we agree that this deserves a more in-depth discussion, which we describe below.

• Prior work (e.g., Mikami et al 2022) on transfer learning has proposed scaling laws using multiplicative or additive forms involving $D_1$ (pretraining data) and $D_2$ (fine-tuning data).
• However, as far as we know, there has been no attempt to incorporate an interaction term such as ours, namely $D_1\log D_2$. We believe this interaction is novel and meaningful: it is both empirically validated and theoretically motivated by our derivation in Section 4.1 (see also Appendix C.3).
• While our focus is on upcycling MoE models, we expect the interaction term to also be applicable in transfer learning settings. For example, transfer learning literature has observed ossification effects (Hernandez et al 2021), where models pretrained on large datasets can become resistant to adaptation during fine-tuning.
• The diminishing returns we observe from increasing $D_1$ align with this notion. Our interaction term $D_1 \log D_2$ captures precisely such diminishing gains and may offer a useful framework for modeling ossification quantitatively in transfer learning scaling laws (and other two-stage training methods).

We will revise the discussion to emphasize this connection.

Comment on Limited Generalizability to Larger Models

• We acknowledge the concern about generalizability to larger models; however, we note that this limitation is common across empirical studies of scaling laws, including highly influential ones.
• Training and evaluating models at the 70B+ scale requires orders of magnitude more computational resources which is currently infeasible in an academic setting. We estimate this to require 5,000 times more FLOPs (70x larger model and 70x more tokens) than what we have run (see Appendix A.10), amounting to an additional cost of 25 million USD (even assuming 1 USD per GPU hour) .
• Despite this, we provide empirical evidence supporting the robustness of our findings: in Figure 6, we show that scaling behavior observed in sub-0.5B parameter models can reliably predict the performance of larger models (up to 1B parameters).
• This suggests that the scaling laws identified are stable across model sizes, and we expect the trends to extend to even larger models, consistent with patterns observed in prior work (e.g., Chinchilla paper).

Comment on theoretical explanation

• While we indeed have not included formal theoretical proofs, we emphasize that the proposed scaling law is not purely empirical. Its functional form is guided by well-motivated principles, as detailed in Section 4.1, and was noted by two other Reviewers to be a reasonable theoretical contribution to empirical scaling law studies.
• Our work also follows the tradition of impactful scaling law work (e.g., OpenAI, Chinchilla paper) that focused on empirical observations and practical utility, without extensive theoretical justification.
• That being said, we agree that theoretical understanding is important, and although it is not the primary focus of this paper, we provide related references in Section 7.

Comment on Limitation to upcycled models

We respectfully disagree with the characterization that our work is limited in scope due to its focus on upcycled MoE models.

• In practice, both MoE architectures and upcycling strategies have become central components of modern LLM development. Recent state-of-the-art models, including DeepSeek, Qwen, Skywork-MoE, and Mixtral, adopt MoE architectures while also leveraging dense-to-sparse upcycling. This growing trend highlights that upcycling is not just a research curiosity but a practical and widely adopted technique in the current LLM landscape. Our findings are therefore both timely and relevant, offering insight into how/when upcycling can be efficient.
• Although our experiments specifically focus on upcycling into MoE models, the core insights, such as the interaction between dense and upcycled training budgets, are useful for formulating two-stage training regimes more broadly, including transfer learning as mentioned above, and potentially model growth and pretraining-SFT framework as cited in the main text.

We hope these address your concern. Please let us know if further clarification would be helpful.

We sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We are especially grateful for the recognition of our experimental design, the quality of the fitted scaling laws, and the clarity of our empirical methodology. We also appreciate the constructive suggestions. Let us clarify some of the questions and comments raised in the following.

Question on initial parameterization

We appreciate the reviewer’s insightful observation on whether modifying initialization can improve performance, and indeed we have studied this and mentioned it in Appendix A.6. Let us elaborate on it further:

• To investigate whether this effect could be mitigated, we experimented with several initialization modifications for the MoE experts. These included (i) adding Gaussian noise to the expert weights of the upcycled model, (ii) partially randomizing the MLP weights, and (iii) applying low-rank approximations to encourage expert divergence. However, across all these approaches, we observed no significant improvement in training dynamics or final performance.
• These results are consistent with findings from prior work, including Komatsuzaki et al (2022), He et al (2022), and Muennighoff et al (2023), who similarly explored various re-initialization or weight-perturbation strategies in MoE settings, with limited success. Overall, our experiments support the conclusion that the vanilla upcycling strategy (without additional modification) is the most effective approach currently known.

Comment on Figure 3

The reason some training curves begin from intermediate number of tokens (e.g., around 4.1 B tokens) stems from our use of the Warmup-Stable-Decay (WSD) learning rate schedule combined with a checkpoint reuse strategy for efficiency (explained in Section 3.1).

• Specifically, for runs with smaller $D_2$ budgets (e.g., $D_2$ = 4.61B), we initialize from an intermediate checkpoint of a longer run ($D_2$ = 9.23B) and continue pretraining from that point to reach the desired total token count, without needing to retrain from scratch for every token budget.
• Therefore, the curve that appears to start around 4.1B tokens means that it has already undergone 4.1B tokens of training along the “main branch”, and the red line shown corresponds to additional continued pretraining to reach a total of 4.61B tokens.

Comment on different text domains

Thank you for the insightful suggestion. We agree that understanding how scaling laws vary across text domains is an important direction. In fact, we already have some results on this in Appendix B, where we replicate our experiments on two additional datasets, Japanese and code, to test the generality of our findings. Let us elaborate:

• We observe that the scaling relations introduced in the main text hold consistently, regardless of dataset.
• The Japanese dataset has higher validation loss (harder task), while the code dataset results in lower loss, making it a relatively easier task in terms of cross-entropy loss.
• Interestingly, the Japanese dataset is harder to saturate with increasing $D_1$, meaning upcycled training remains effective. In contrast, the code dataset saturates more quickly, making upcycling less beneficial.

This is a promising evidence that the core scaling behavior we identify is robust across diverse domains. However, the more fine-grained question of how mixtures of pretraining data impact respective downstream performances is an orthogonal direction. While interesting, it is beyond the scope of our current work. For readers interested in downstream impacts, we do include results in Appendix A.9 and Table 7, where we evaluate downstream performance on various tasks for models trained with SlimPajama.

On analyzing the threshold

• In Section 5.1, we define the threshold $D^*$ as the token count at which training a model from scratch matches the performance of an upcycled MoE with the same total token budget using derived scaling laws with fixed model configuration. We solve this equation numerically and also provide an analytic approximation (Equation 2) to aid practical use.
• We also note in the text that covering all possible configurations and settings would require exponentially more compute, which is infeasible in academic environments. As such, we focus on a widely used MoE configuration (Mixtral) to make the analysis tractable.
• Finally, we offer guidance on how to apply this threshold in practice, highlighting the need to balance model size, compute, and token budgets in the beginning of Section 5 and Section 7. Once the configuration is set, one can run the scaling law experiments and use the procedure mentioned above to get the threshold. We will revise the text to make these points clearer.

We hope these addresses your concern. Please let us know if further clarification would be helpful.

We sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We especially appreciate the recognition of our empirical approach, the soundness of our experimental design, and the validation of our scaling law derivation in Section 4.1. We also thank the reviewer for acknowledging the practical value of techniques such as the WSD learning rate schedule. The comments regarding the justification of Equation 11 and the desire for more concrete experiments around compute optimality are very helpful, and we address them in detail below.

Comment on Equation 11

We acknowledge that a derivation and ablation study of Equation 11 were lacking, an oversight on our part. We appreciate the opportunity to address it.

• During the rebuttal period, we conducted a principled analysis to derive and validate the appropriate functional form similar to what is done in section 4.1.
• Recall that we wish to understand the cross-entropy loss as a function of sparsity, defined as $P= N_{\rm total}/N_2$ and the number of active parameters, $N_2$.
• Starting from the power-law ansatz, we require that the loss satisfies $L(P,N_2)= L_{P}(N_2) = A N_2^{-\beta_1} + E$ and $L(P,N_2)= L_{N_2}(P) = A P^{-\beta_2} + E$. This is reasonable as $P$, when fixing $N_2$, is the total number of model parameters, which we expect to satisfy the power-law ansatz. Analogously, $N_2$ corresponds to the number of dense model parameters, which should satisfy the power-law ansatz as well.
• We then consider, as before, both additive and multiplicative functional forms, with and without interaction, satisfying the above requirements, and evaluate them using leave-one-out RMS error. We obtain

i.e., additional form without interaction provides the best fit to the data. We will revise the paper to include the corrected form, along with a derivation and ablation comparison of alternative models.

Comment on more experiments

• We agree that additional experiments on compute optimality could further strengthen the claims in Section 5.2. However, this would require sweeping over many configurations with comparable FLOP budgets, which is not feasible under our current computational constraints. Such experiments would require multiple times the total GPU budget we already spent (estimated in Appendix A.10), amounting to over 10,000 USD even under conservative assumptions (e.g., 1 USD per GPU hour).
• Moreover, while compute optimality is important, it is not the primary focus of this work. Our main goal is to understand how upcycling performance depends on data and model sizes. That said, our formulation naturally supports compute analysis, and we use it in Section 5.2 to identify regimes where upcycling becomes less compute-efficient than training from scratch. Note that our findings are robust and extrapolatable: in Figure 6, we show that scaling behavior observed in sub-0.5B parameter models can reliably predict the performance of larger models (up to 1B parameters).
• Our approach is also in line with prior work, e.g., Krajewski et al. (2024) derived a data–model size scaling law and use it to predict FLOP-optimal behavior, rather than performing exhaustive FLOP-based sweeps, emphasizing interpretable scaling relationships under realistic compute budgets.

We will further correct typos and figure presentation as suggested in the revised version.

We hope these address your concerns. Please let us know if further clarification would be helpful.