Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families

Thank you for your time and dedication to our paper! We have responded to your concerns below and have updated the paper to accommodate suggestions by reviewers. Please let us know if you have further questions.

"The authors conduct experiments using only three open-source dense LLMs": We use 30 LLM families in our paper; please check Section 4.1 for more details.

Inclusion of MoE models: We do not include MoE models because of data limitations. For example, training tokens for Mixtral models are unknown and DeepSeekMoE is only available at one size on HuggingFace.

Post-training factors (STF, RLHF, etc): Our findings do not contradict the referenced papers; the reason is the following. In our analysis, the number of tokens and parameters are only related to pre-training. All the post-training factors, including more training tokens during SFT, have their effect taken into account by the family-specific intercepts, i.e., $\alpha_i$’s. For example, if a certain family went through a post-training procedure that made these models strong in reasoning, their values for $\alpha_i$ in reasoning will be relatively large (recall that we consider base and instruct models to be from different families). We made this point clear in Section 3.1.

With the discussion period concluding in a few days, we want to ensure we address any remaining questions or concerns you might have about our paper. During the rebuttal process, we have updated the manuscript to (i) clarify how our work differs from Ruan et al.'s and highlight that they do not explore performance prediction from compute (see lines 134-139 and 471-475 in the revised paper), (ii) include a list of the models and families used (Appendix F), and (iii) emphasize that the intercept $\alpha_i$ accounts for hidden factors such as post-training adjustments (lines 184–187). For a detailed explanation of the rationale behind our family-specific intercept, please refer to our response to reviewer DNEX.

Thank you for your work on our paper! We have responded to your concerns below and have updated the paper with some of your suggestions. Please let us know if you have any questions.

Regarding Ruan et al (2024): The ideas proposed by Ruan et al (2024) cannot be used to predict skills or performance from compute and we consider that to be the main difference between our papers; see below for a detailed explanation. As we comment in our Section 3.1, LLMs low-dimensional latent skills have been studied before by many papers but we are the first ones to propose and successfully achieve a scaling law for those skills. For example, the focus of Ruan et al (2024) is building a scaling law for complex downstream tasks from latent skills instead.

In this paragraph, we explain why Ruan et al (2024) scaling cannot be used to predict skills or performance from compute, implying a major difference from our work. The observational scaling law proposed by Ruan et al (2024) is used to predict the performance of LLMs in complex downstream tasks from their PC scores (extracted from observed benchmark data); please check their Section 3.4 for more details. This implies that, for example, to predict the performance of LLaMa-3-70B in a certain downstream task, they first have to observe the scores of LLaMa-3-70B in a bunch of benchmarks. That is, it is not possible to predict the performance of LLaMa-3-70B in downstream tasks using their approach before the LLM is released (i.e., making predictions from compute); this is one of the gaps we attack in our paper. This limitation was confirmed by Tatsunori Hashimoto in a conversation we had at ICML 2024 (T. Hashimoto, personal communication, July 2024). Moreover, based on Hashimoto's suggestion, we tried to extend their method to make performance prediction from computing possible; as we show in Figure 1, that approach ("PCA+FLOPs") did not work well.

Section 3.1:

• Economics model: We use that specific formulation for two main reasons. First, it is a generalization of models previously used in the scaling law literature (e.g., the model used by Owen(2024)). Second, it accounts for the interaction of log(s)log(t), which we found to work well (check our Appendix D), and incorporates family effects (intercepts) in a parsimonious manner.

• Family-specific intercept: We use family-specific intercept and not slopes mainly because of a practical reason: our model has three covariates in $x$, therefore it would be impossible to fit a scaling law with family-specific slopes without observing many models from the same family in our training set. The family-specific intercept ends up being a parsimonious way of dealing with models produced with better technology (e.g., algorithms, data formatting/quality) and has good predictive power as we see from Figures 1 and 2.

• "why does x(s,t) not include log(st)?": it includes. This is because log(st)=log(s)+log(t).

Sections 3.2 and 3.3:

• Assumption 3.1: This assumption is only used to describe to what degree the model's parameters are identifiable. This type of assumption is widely used and accepted in the field of Psychometrics (please see [1], for example).

• Design matrix and $p$: the design matrix $X\in \mathbb{R}^{n \times p}$ is a matrix of covariates $x(s,t)$ specified in Eq 3.2 concatenated with a matrix of zeros and ones (to account for the family intercepts). Each row of $X$ represents a different LLM and $p=3 +$ #families. We made this explicit in Section 3.2.

• Huber loss: this loss is a smoothed version of the $l_1$ loss. Please check https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html

Section 4:

• Figure 1 vs Figure 2: Both figures show the average column. We did not display the full Figure 2 in the main text because we thought it was redundant and we lacked space. However, its full version can be found in Appendix F.
• Ruan et al (2024): Please check the response "Regarding Ruan et al (2024)" above.

References

[1] Yunxiao Chen, Xiaoou Li, and Siliang Zhang. Joint maximum likelihood estimation for high-dimensional exploratory item factor analysis. Psychometrika, 84:124–146, 2019.

Dear reviewer DNEX, thank you for your prompt response! We elaborate more on some points below and hope to reach an agreement. Please let us know what you think or if you have more questions/points to discuss.

About Ruan et al (2024): We agree that the approach by Ruan et al (2024) could be adapted to make performance predictions from compute. The approach “PCA+FLOPs” represents exactly that but, as our experiments in Figures 1 and 2 show, it does not work well. Following your suggestion, we have rephrased references to Ruan et al to acknowledge that their method can be applied to predicting the performance of larger LLMs, but it was not explored in their paper (see lines 134-139 and 471-475 of the updated paper).

Intuition behind our formulation with family-specific intercepts: As discussed in the text, this type of model has long been utilized in Economics to assess the efficiency of firms and their production outputs; so our interpretation should directly connect with that. One way to make the model more intuitive is to interpret the intercept as a measure of efficiency, which directly influences how the input levels $s$ and $t$ translate into outputs. Specifically, if a particular family has a high $\alpha_i$, the inputs $s$ and $t$ will yield a bigger impact on performance compared to a model with a lower $\alpha_i$. To illustrate this, let us consider the performance of LLMs on a specific benchmark (let us forget about $\Lambda$ for now) and assume that $\gamma_j$ and the interaction term are omitted from the model. In this simplified scenario, the model can be expressed as:

$

\sigma(\alpha_i + \beta^\top x(s, t)) = \frac{e^{\alpha_i + \beta_1 \log s + \beta_2 \log t}}{1 + e^{\alpha_i + \beta_1 \log s + \beta_2 \log t}} = \frac{e^{\alpha_i}s^{\beta_1}t^{\beta_2}}{1 + e^{\alpha_i}s^{\beta_1}t^{\beta_2}} = \frac{A_i s^{\beta_1}t^{\beta_2}}{1 + A_i s^{\beta_1}t^{\beta_2}},

$

where $A_i = e^{\alpha_i}$ represents the efficiency of family $i$. This formulation highlights how the impact of inputs $s$ and $t$ on performance depends on the value of $\alpha_i$.

Including family-dependent slopes: We present a comparison of our method to prior scaling laws (Ruan et al and Owen) that utilize family-specific slopes in Figure 2 (see also lines 366-375 for a discussion). Our results demonstrate that family-specific slopes are detrimental to the performance. In the case of our scaling law, to be able to fit a good model with family-specific slopes, the number of models of different sizes and training tokens per family needs to be high (at least 4 in our case, which is the case for only a single family). As one of our goals is to allow practitioners to make decisions regarding investing resources into training large LLMs based on their results with a smaller LLM, a scaling law requiring 4 models per family would be impractical. We emphasize that an important advantage of our scaling law is the ability to utilize evaluation data across both LLM families and benchmarks, which benefits from parameter sharing. It is possible to fit our scaling law with family-specific slopes and some regularization for identifiability, but we do not expect it to perform well. However, if such experiment would help to address your question, we are happy to try it. Please let us know.

With the discussion period concluding in a few days, we want to ensure we address any remaining questions or concerns you might have about our paper. During the rebuttal process, we have updated the manuscript to (i) clarify how our work differs from Ruan et al.'s and highlight that they do not explore performance prediction from compute (see lines 134-139 and 471-475 in the revised paper), (ii) include a list of the models and families used (Appendix F), and (iii) emphasize that the intercept $\alpha_i$ accounts for hidden factors such as post-training adjustments (lines 184-187).

Thank you for your dedication to our paper! We have responded to your concerns below and have updated the paper to accommodate suggestions by reviewers. Please let us know if you have any questions.

Post-training factors: We agree that post-training factors are not captured by our measures of compute $x(s,t)$. However, our formulation guarantees that such hidden factors are taken into account by the family-specific intercepts, i.e., $\alpha_i$’s. For example, if a certain family went through a post-training procedure that made these models strong in reasoning, their values for $\alpha_i$ in reasoning will be relatively large (recall that we consider base and instruct models to be from different families). Your observations are observable in Figures 5 and 6. For example, from our text “Contrasting with Reasoning, Figure 5 shows that Knowledge is highly influenced by both model size and number of training tokens. Moreover, we can see that the range of standard deviations in the middle plot is much greater than in the other two plots, giving us evidence that this skill might be more sensitive to increases in compute resources and less dependent on the LLM families themselves” (verbatim). In Figure 6, we show that instruction following is highly influenced by instruction tuning when compared to reasoning and knowledge. Thank you for your observation! We have included some discussion regarding post-training in Section 3.1.

Contamination: You are right that some LLMs can have their training contaminated with benchmark data. Fortunately, the Open LLM Leaderboards (v1/v2) have an indicator of contamination, which can be used to filter contaminated models. Like all papers in the scaling law literature for LLM benchmarks, we are assuming no contamination.

Practical applications of our model: Our scaling law can guide practitioners in many instances. For example: (i) choosing the right size of the next (to be released) pre-trained models for a desired level of performance on benchmarks of interest, (ii) allocating resources between training tokens and number of parameters (e.g., from Fig 5, if the practitioner wants a model with better reasoning, it might be better to invest on bigger models with less training data vs smaller models with more training data), and (iii) predicting performance of hypothetical LLMs on complex downstream tasks, such as coding. We added some extra sentences at the beginning of Section 4 regarding applications explored in experiments.

With the discussion period concluding in a few days, we want to ensure we address any remaining questions or concerns you might have about our paper. During the rebuttal process, we have updated the manuscript to (i) clarify how our work differs from Ruan et al.'s and highlight that they do not explore performance prediction from compute (see lines 134-139 and 471-475 in the revised paper), (ii) include a list of the models and families used (Appendix F), and (iii) emphasize that the intercept $\alpha_i$ accounts for hidden factors such as post-training adjustments (lines 184-187). For a detailed explanation of the rationale behind our family-specific intercept, please refer to our response to reviewer DNEX.

Thank you for your work on our paper! We have responded to your concerns below and have updated the paper with some of your suggestions. Please let us know if you have any questions.

Comments about your summary of our paper: We believe two statements in the summary of our work are inaccurate and would like to clarify them below:

1. "The proposed scaling law is as performant as previously proposed approaches": this sentence understates the performance prediction of our method; as we see in Figure 1, our methods offer much better prediction results when compared with baselines.
2. "(...) to improve a previously proposed scaling law": we do not only improve previous scaling laws but propose a model that can model the scaling of abstract LLM skills using data from multiple benchmarks and LLMs, which is something no one proposed before.''

Concerns about presentation: We worked (and will work) hard to make our paper as polished and readable as possible, which seems to be the impression that other reviewers got from our work. Please let us know which points you think are not polished enough and we will update the paper with your suggestions. We note that "interaction term", for example, is an expression frequently used in regression analysis within machine learning and statistics; in spite of that, we included more explanation in the text.

Concerns about model families: We have included Table 1 in Appendix F detailing all families and models used in our paper. We consider two models to be in the same family if the (main) difference between them is the number of parameters; please check Table 1 for precise categorization. Model sizes and training token information were taken from the original papers that released the models, their Hugging Face model cards, and data collected by Ruan et al (2024); our data collection effort resulted in a more comprehensive study when compared to previous works.

Concerns about overfitting and generalization: In our paper, we show that training the activation function does not lead to poor generalization. In fact, from Figure 1 we see that the neural net model ("Sloth") performs better than any other option in terms of lower test error, including the sigmoid model ("Sloth basic"). In this experiment, the test error is computed using a cross-validation approach; that is, for each test LLM family (e.g., Qwen 2) we: (i) only include only the smallest model in the training set (e.g., Qwen 2 0.5B), (ii) fit the scaling laws using all the LLMs in the training set, (iii) predict the performance of larger models from that family (e.g., Qwen 2 1.5/7/72B) in all benchmarks, and (iv) compute the prediction error by comparing the predictions with the ground truth. We repeat these steps for all test families and average the prediction errors, which are the numbers reported in Figure 1. To make our assessment even more realistic, we do not test older versions of recent families if they are available in the training set, e.g., we do not test on LLaMa 2 models when LLaMa 3 is present in the training set.

Concerns regarding plots and table: In the main text, we include two plots like the ones you are suggesting in Figure 7. For the other results, we do not think this type of plot is the most suitable; the reason is that we run a higher number of experiments when compared to other papers and the heatmaps help us summarize our results better. For example, to show the predictive power of our method (8 variations) compared to the other 7 baselines across 12 benchmarks, a heatmap is more suitable, and if we choose to go with the "extrapolation" plots, at least 15x12 = 180 plots would be needed, which is not informative for the reader. If we consider that each one of the test families needs a different plot (due to cross-validation), 19x180=3420 plots are needed just for Open LLM Leaderboard v1.

"(Ruan et al) is not well-suited for performance prediction from compute": The observational scaling law proposed by Ruan et al (2024) is used to predict the performance of LLMs in complex downstream tasks from their PC scores (extracted from observed benchmark data); please check their Section 3.4 for more details. This implies that, for example, to predict the performance of LLaMa-3-70B in a certain downstream task, they first have to observe the scores of LLaMa-3-70B in a bunch of benchmarks. That is, it is not possible to predict the performance of LLaMa-3-70B in downstream tasks using their approach before the LLM is released (i.e., making predictions from compute); this is one of the gaps we attack in our paper. This limitation was confirmed by Tatsunori Hashimoto in a conversation we had at ICML 2024 (T. Hashimoto, personal communication, July 2024). Moreover, based on Hashimoto's suggestion, we tried to extend their method to make performance prediction from computing possible; as we show in Figure 1, that approach ("PCA+FLOPs") did not work well.

Concerns regarding Figure 7: We use the logistic link for downstream performance prediction because the intersection of our main dataset and the code completion/emotional intelligence datasets is quite small, i.e., there is not enough data (model performances) to fit link/activation functions for these benchmarks (recall that our scaling law allows each benchmark to use a different link function for predicting performance from the latent abilities).

Level curves: For each one of the $k$ skills, the level curves are obtained from the function $g_k(s,t)=\hat{\beta}_k^\top x(s,t)$ from Eq 3.2. Here, $g_k(s,t)$ represents the level of skill $k$ of a certain LLM with covariates $x(s,t)$ discounted by the family-specific intercept term.

With the discussion period concluding in a few days, we want to ensure we address any remaining questions or concerns you might have about our paper. During the rebuttal process, we have updated the manuscript to (i) clarify how our work differs from Ruan et al.'s and highlight that they do not explore performance prediction from compute (see lines 134-139 and 471-475 in the revised paper), (ii) include a list of the models and families used (Appendix F), and (iii) emphasize that the intercept $\alpha_i$ accounts for hidden factors such as post-training adjustments (lines 184–187). For a detailed explanation of the rationale behind our family-specific intercept, please refer to our response to reviewer DNEX.