Zero-Shot Performance Prediction for Probabilistic Scaling Laws

Thank your for your supportive feedback! We address your questions individually in the following:

[W1] - Model complexity and runtime:

Algorithmic Complexity:
MaGP is implemented using the inducing points method. MaGP is originally referred to as HMOGP-LV (Ma et. al., 2023). HMOGP-LV is derived from LVMOGP (Dai et al., 2017). The computational complexity is

with $N$ being the available data points per learning curve, $R$ being the number of learning curves per task, $D$ being the number of tasks, $M_X=M_r\times R$ for $M_r$ being the number of inducing input points in the r-th learning curve of a task and $M_H$ being the number of inducing output points.

Runtimes:
MaGP was trained and evaluated on an Intel(R) Core(TM) i7-8565U CPU.

$\rightarrow$ nanoGPT-Large experiments: The wallclock time for MaGP in this scenario (w/o active learning) is between 17 min and 31 min, which includes logging and saving files. It is worth noting that MaGP is not (yet) optimized in terms of speed.

$\rightarrow$ The 'active learning' component only adds negligible overhead (in comparison to the other query strategies), as the only additional operation is the computation of the variance across the already-predicted curves (which uses an optimized library).

[W2] - Patterns of Query Selection in Active Learning:
We thank you for raising this question. This was indeed of great interest to us as well.
$\rightarrow$ The paper has already a corresponding figure included in Appendix F, and we will ensure that a proper reference to this is added in the main paper. (Or we might move this to the main paper to provide further insights if space permits.)

[W3] - Partially dense writing, esp. around hier. GPs:
Thank you for drawing our attention to this! We have already expanded the relevant sections and now provide more detailed explanations to improve clarity and completeness, as well as some underlying intuitions to cater for a wider audience.

We would like to thank you again for the support of our work, and hope our answers have clarified your questions.
If any further queries remain, please let us know and we will do our best to promptly address them.

We thank you for your feedback, and address your points in the following.

[W1] - Comparison to recent scaling law estimation methods:
Recent scaling law experiments typically predict the scaling law displayed in 'blue' in Figure 2 by utilizing the full available dataset (i.e. expensive training of all models), without employing zero-shot prediction to reduce computational complexity/cost.
$\rightarrow$ In this sense, these experiments are effectively represented by our baseline 'ground-truth' scaling law, to which we compare our results.

[W2] - Limited Models and Datasets:
Thank you for raising this concern. Due to computational constraints, we focused our setup on training models for one epoch using the FineWeb-Edu dataset. Our model sizes range from 51M to 1.5B parameters, aligning with the scale used in Kaplan et al., who explored models from 768M to 1.5B parameters. We therefore consider this range representative for our investigation.

To support our hypothesis that hierarchical structures exist and can be exploited in learning curve datasets, we provide extensive experiments across multiple domains, including bilingual and multilingual translation tasks. To the best of our knowledge, our work is the first to investigate bi-level hierarchies for zero-shot learning curve prediction and extrapolation.

[W3] - Introduction of Datasets:
While the datasets we are using are introduced in Section 4, we agree that the detailed composition and origin is somewhat unclear;
$\rightarrow$ We have added an extended Section describing their creation as well as inherent properties to the appendix of the paper (including references in the main text). Note that all these datasets were created by us.

Re: Minor Notes -- We thank you for bringing these to our attention; textual and grammatical errors have been corrected.

[Q1] - Subsampling & computational demand:

Line 232: Why do authors subsample to 11 datapoints? How much more computational demand does it require to use the full dataset?

Due to computational constraints, we were limited to using 11 datapoints across 36 learning curves, which was the maximum we could accommodate on our available CPU resources (RAM-limited).

Regarding model complexity:

• MaGP is implemented using the inducing points method.
• MaGP is originally referred to as HMOGP-LV (Ma et. al., 2023). HMOGP-LV is derived from LVMOGP (Dai et al., 2017).

The computational complexity is

with $N$ being the available data points per learning curve, $R$ being the number of learning curves per task ($d$), $D$ being the number of tasks $t$, $M_X=M_r\times R$ for $M_r$ being the number of inducing input points in the r-th learning curve of a task and $M_H$ being the number of inducing output points.

$\Rightarrow$ Importantly: While additional compute would certainly enable larger-scale experiments, our current setup demonstrates that even with a limited number of datapoints and resources -- corresponding to reduced evaluation frequency and faster training -- the MaGP model performs effectively!

[Q2] - Comparison to prior work:

Why is there no direct comparison to prior scaling law estimation methods (e.g., those used by Kaplan et al., Hoffmann et al., Henighan et al.)?

As stated above in [W1], recent scaling law experiments typically predict the scaling law displayed in 'blue' in Figure 2 by utilizing the full available dataset (i.e. expensive training of all models), without employing zero-shot prediction to reduce computational complexity/cost.
$\rightarrow$ In this sense, these experiments are effectively represented by our baseline 'ground-truth' scaling law, to which we compare our results.

We hope our answers addressed all your questions.
If you have any further queries, please do not hesitate to reach out.

Dear Reviewer FUMJ,

As the discussion period ends in one day, we’d be glad to hear whether our responses in the rebuttal have addressed your concerns.

If our clarifications have helped resolve your concerns, we’d of course appreciate your reconsideration of the rating, if you feel that’s appropriate.

In case you have any remaining questions or comments, please don’t hesitate to let us know — we’d be happy to clarify them while time allows.

I thank the authors for their thorough response. I believe my main concerns have been addressed with the authors' promise to more thoroughly introduce datasets, along with their explanations of how the experiments in this work compare to prior works. I would encourage authors to explicitly clarify [W1] in the paper. I have raised my score accordingly.

We thank you for your helpful suggestions, and address your points in the following.

[W1] - Scaling to many hyperparameters:
At present, our model considers only the number of embedding parameters and the number of layers as structural inputs/hyperparameters.
Due to computational constraints, we were limited to using 11 datapoints across 36 learning curves, which was the maximum we could accommodate on our available CPU resources (RAM-limited). (see [Q1] below for model complexity)

$\Rightarrow$ Importantly: While additional compute would certainly enable larger-scale experiments, our current setup demonstrates that even with a limited number of datapoints and resources -- corresponding to reduced evaluation frequency and faster training -- the MaGP model performs effectively!

[W2] - Vertical offset and irreducible loss:
Vertical offset is not explicitly modeled, as all learning curves considered have the same number of layers vertically and an identical number of datapoints. Additionally, we do not incorporate the irreducible loss term in our scaling law analysis, following the frameworks established by Kaplan et al. and Hoffmann et al.. While the irreducible loss is addressed in Hennighan’s work, incorporating it represents a natural and compelling extension for future research.

$\rightarrow$ Please feel free to reach out if you require further clarification or additional details.

[W3] - Variance-based Active Learning:
Thank you for raising this point. You are correct that the variance-based active learning strategy explored in our work focuses on reducing the uncertainty, independent of associated compute costs. In the revised version of the paper, we therefore additionally provide AbC over cumulative compute cost to reflect the strategies' performances when computational resources are constrained.

In brief: E.g. 17 queries for 'Smallest First' are roughly equivalent to only 3 for 'Largest', 7 'Active' & 10 Random;
$\rightarrow$ This changes the ranking based on AbC metric to prefer the 'Smallest First', followed by 'Active' and 'Random', while 'Largest' stays least competitive.

Note that there exist many different ways how the cost-function in the active learning process could be structured, but this is outside the scope of this paper. We see the design of cost-terms that consider trade-offs like compute-vs-uncertainty-reduction as an interesting direction of future work.

[W4] - Query budget:
We thank you for raising this concern. In total, we have 25 learning curves available for querying -- however, this is the 'upper limit' that is considered in our experiments.
Note that Figure 7 illustrates the performance when using fewer queries, demonstrating that after querying for example just 14 curves, the AbC value already falls below 0.25.

[W5] - Hierarchical Grouping:
We are slightly unsure what you mean with 'mis-specified' in the context of model hyperparameters.
However, we have further validated our hypothesis that hierarchies are indeed exchangeable (and therefore indicating robustness against potential mis-specification).
$\Rightarrow$ For the new experimental results with exchanged (swapped) hierarchy, please see the tables provided in the response to reviewer QuFj - [W1].

[Q1] - Wallclock times and memory:

• MaGP and DHGP were trained and evaluated on an Intel(R) Core(TM) i7-8565U CPU.
• MaGP follows the same inducing point strategy as suggested in the original work in Ma et al., by inducing every second point. (Using every point increased the runtime but did not improve on the negative log likelihood optimization criterion.)
• The wallclock time for MaGP is between 17 min and 31 min, and DHGP around 4 min (including logging, file-saving, etc.); Note that DHGP uses optimized libraries, while MaGP in its current form is not yet optimized.

[Q2] - Cost-aware fits:
We are slightly unsure if we correctly understand your question. If your question refers to an active learning approach that jointly considers e.g. variance-reduction and compute-cost, this is discussed in [W3] -- and we consider this a natural and promising extension of our work for future consideration.

$\rightarrow$ Please feel free to reach out if you require further clarification or additional details, or if your question has not yet been fully addressed.

[Q3] - Irreducible loss:
As the analyses in our paper are based on Kaplan et al.'s and Hoffmann et al.'s works, we have not incorporated the irreducible loss term in our scaling law analysis. While the irreducible loss is addressed in Hennighan’s work, we see this as a natural and compelling extension for future research -- but outside the scope of this current work.

[Q4] - Improved plot quality - BLUE/ChrF:
Thank you for pointing this out. We included appropriate legends and present separate performance figures for each metric in the revised version of the paper.

We hope our answers have clarified all your questions.
If you have any further queries, please let us know and we are happy to answer them.

Dear Reviewer bTJp,

As the discussion period ends in one day, we’d be glad to hear whether our responses in the rebuttal have addressed your concerns.

If our clarifications have helped resolve your concerns, we’d of course appreciate your reconsideration of the rating, if you feel that’s appropriate.

In case you have any remaining questions or comments, please don’t hesitate to let us know — we’d be happy to clarify them while time allows.

We thank you for your constructive review, and address your points individually in the following:

[W1] - Assumptions and Modeling decisions:
We thank you for bringing this to our attention!

I think extra attention should be paid to explaining a) the hypothesis about the specific types of hierarchical relationships that DHGP and MaGP can each express

$\rightarrow$ MaGP introduces an additional latent variable $\mathbf{h}$ to model correlations among tasks $t$, allowing it to capture cross-task dependencies.

$\rightarrow$ In contrast, DHGP does not rely on this shared latent representation. Instead, it leverages a hierarchical modeling approach defined as:

Following and inspired by the approach of Hensman et al. [27], we interpret $f$ as a general function capturing global learning curve trends across tasks. The next level, $g$, captures variations due to a coarser factor, i.e., the number of embedding dimensions. At the lowest level, $l$ models individual learning curves, which vary with the number of layers.

$\Rightarrow$ DHGP is therefore particularly suited for discovering clusters in the data, where each cluster can be thought of as a group of tasks sharing similar characteristics (e.g., number of embedding parameters).
$\Rightarrow$ This is in contrast to MaGP, which assumes that inter-task correlations exist and can be captured through a latent variable $\mathbf{h}$.
$\Rightarrow$ If such correlations are weak or non-existent, we expect the hierarchical structure of DHGP to provide a more appropriate inductive bias than MaGP, allowing it to learn within-cluster trends effectively.

[...] b) why all of the data relationships in the paper more closely mimic MaGP and not DHGP.

The data relationships in the paper mimic more closely MaGP, because they not only have a hierarchical structure, but also exhibit correlations among tasks. As discussed above, we expect this to be particularly well-suited for MaGP -- a hypothesis that is strongly supported by the obtained experimental results.

[...] "hierarchies are indeed exchangeable", which I interpret to mean that modeling one parameter axis as the higher level and the other as the lower, or vice versa, leads to no difference in effect. [...] Are hierarchies exchangeable for all examined problems?

To provide experimental evidence towards the hypothesis of exchangeable hierarchies (everything else kept identical), we provide the following additional results:

1. Scaling law experiments (layer-over-embedding): | Method | TT | MSE | MAE | MNLPD | |-|-|-|-|-| | MaGP | Quad | 0.04 ± 0.04 | 0.11 ± 0.07 | 1.82 ± 2.23 | | DHGP | Quad | 0.11 ± 0.003 | 0.26 ± 0.01| 0.34 ± 0.02 | |-|-|-|-|-| | MaGP | Tri | 0.02 ± 0.01 | 0.09 ± 0.03 | 0.37 ± 0.79 | | DHGP | Tri | 0.13 ± 0.05 | 0.29 ± 0.06 | 0.48 ± 0.16 | |-|-|-|-|-| | MaGP | T1 | 0.01 ± 0.02 | 0.08 ± 0.04 | -0.88 ± 0.58 | | DHGP | T1 | 0.16 ± 0.02 | 0.34 ± 0.02 | 1.18 ± 0.43 |

2. Bilingual experiments (target-to-source) | Method | TT | MSE | MAE | MNLPD | |-|-|-|-|-| | MaGP | mBart chrF | 43.24 ± 13.72 | 5.297 ± 0.87 | 3.698 ± 0.43 | | DHGP | mBart chrF | 127.53 ± 0.10 | 8.3 ± 0.00 | 7.44 ± 0.07 | | MaGP | mBart BLEU | 13.287 ± 4.41 | 2.625 ± 0.45 | 5.15 ± 2.43 | | DHGP | mBart BLEU | 33.17 ± 0.22 | 3.835 ± 0.01 | 4.26 ± 0.14 | |-|-|-|-|-| | MaGP | Transf chrF | 18.801 ± 3.32 | 3.08 ± 0.22 | 2.866 ± 0.14 | | DHGP | Transf chrF | 35.45 ± 0.02 | 4.15 ± 0.00 | 4.34 ± 0.04 | | MaGP | Transf BLEU | 1.018 ± 0.23 | 0.53 ± 0.86 | 2.06 ± 0.48 | | DHGP | Transf BLEU | 4.577 ± 0.00 | 1.08 ± 0.0 | 4.62 ± 0.16 |

3. Architecture design experiments:
   Already provided in Section 5.3, Figure 9.

$\Rightarrow$ All results provide strong evidence that the hierarchies across all tasks examined in our work are indeed exchangeable without sacrificing performance.

In essence I am requesting additional focus on the core hypothesis, a categorization of what evidence would contribute toward or against it [...].

Core Hypothesis:
We hypothesize that learning curve (LC) datasets exhibit hierarchical structure, specifically bi-level hierarchies.
$\rightarrow$ We perform a range of experiments to provide insights and evidence towards this core hypothesis by analyzing the respective performance of MaGP vs DHGP.

Additional background info: As previously discussed, if we assume that clusters existed in the dataset, DHGP would be the preferred model, as it is capable of discovering and modeling such latent structure. On the other hand, if the dataset's underlying structure exhibits strong inter-task correlations and a hierarchical structure within each task, then MaGP are expected to outperform.

Additional hypothesis: The exact ordering of bilevel hierarchies present in this data is exchangeable.
$\rightarrow$ As stated above, our newly-included experimental results provide evidence that supports this hypothesis.

Final hypothesis: The model which best captures the structure of learning curve datasets can be successfully used for scaling law extrapolation, due its good zero-shot performance.
$\rightarrow$ This hypothesis is empirically supported in Subsection 5.1.

[W2.1&2] - Experiments and different baselines:
We have reworked the experiment section to provide more explanations for our choice of baselines across all experiments. In brief:

$\rightarrow$ The necessity for different baselines arises primarily from the different nature of the datasets and the way learning curves are structured in each setting.

• For the nanoGPT dataset, we have access to a sufficient number of learning curves and data points, making a Bayesian Neural Network (BNN) a valid and effective baseline. This choice is further motivated by prior work, such as Klein et al., where BNNs were successfully applied for learning curve extrapolation and zero-shot prediction.

• In contrast, applying a BNN in the bilingual translation setting is more challenging. While we explored modeling the learning curves as a function of source and target language pairs, the BNN struggled to generalize and capture the upward trend in learning curves. Also note, we only have $25$ learning curves for training, and each source and target language is only $5$ times represented, which given the noisy nature of this dataset (see Figure 4 left) is not enough training data to perform well.
  $\rightarrow$ Given the assumption that source and target languages may share structural similarities, we instead opted for a simpler and more robust baseline: averaging available learning curves across source and target languages. This approach offers a language-agnostic approximation of performance trends.

• The final experiment involves an extrapolation task, where we predict learning curves from a single source (or target) language into five different target (or source) languages. Unlike the bilingual translation setup, the training grid in this scenario is defined by three distinct model sizes (see Figure 4, right). The dataset for this experiment contains only 15 LCs, which is (again) insufficient for effectively training a BNN to generalize. Hence, a regression-based baseline, such as polynomial or power-law fitting, is a more suitable and reliable choice.

[W2.3] - Monte Carlo:
In Section 5.1 and Table 2, we present results illustrating how closely the scaling laws predicted via our Monte Carlo-based approach approximate the original scaling laws obtained through conventional methods -- namely, by fitting the $L(C)$ scaling function to the full dataset. For our evaluation, we use the AbC measure.
$\rightarrow$ The key advantage of our approach lies in significantly reduced computational cost, while still achieving close alignment with the original scaling law. (Note that our test sets contain the most expensive learning curves.)
$\rightarrow$ This outperforms simple fitting to training data as it provides more support to fit the scaling law in the compute region of interest.

[W3-4] - Textual quality:
We have rectified all formatting and textual issues -- thank you for bringing these to our attention!

[Q1]
Addressed in [W1&2]

[Q2] - Compute on x-axis:
Thank you for this great suggestion! We have added the curves -- and this indeed provides very interesting insights!
In brief: E.g. 17 queries for 'Smallest First' are roughly equivalent to only 3 for 'Largest', 7 'Active' & 10 Random;
$\rightarrow$ This changes the ranking based on AbC metric to prefer the 'Smallest First', followed by 'Active' and 'Random', while 'Largest' stays least competitive.

Limitations:
We acknowledge a current limitation of MaGP regarding interpolation across the architectural grid. Specifically, our dataset is structured on a fixed grid defined by the number of layers and embedding parameters. As a result, the model in its current form cannot interpolate to unseen values of embedding parameters. Interpolation across unseen numbers of layers is technically feasible, but it would yield an averaged estimate between the nearest known configurations, which may not reflect true model behavior. We see this as a promising direction for future work and plan to investigate more flexible representations to address this limitation (e.g. w/ meta parameters).

We thank you again for your detailed feedback, and acknowledge that framing and textual quality did require some polishing. We'd like to highlight that a large part of your concerns were related to these issues -- all of which have been taken care of in the revised version of our manuscript thanks to your input!

We hope we have addressed your questions; if any remain, please let us know.

Thank you again for your constructive feedback and for supporting our work!

I'm thoroughly grateful to the authors for receiving my content and presentation suggestions.

All of the answers make sense to me and I find my issues addressed.

I applaud the authors for their rebuttal and I now raise my score to a 4. I look forward to discussing with other reviewers any outstanding issues.