Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks

Thank you for your time and thoughtful comments. We address each of your comments below. If you have further questions, we’d be happy to discuss them during the author-reviewer interaction phase.

---

[Q1] The MCMC models appear to be specified without noise... This makes the Bayesian inference problem extremely misspecified.

• Thank you for pointing this out, and sorry for the confusion. Actually, we did account for the observational noise in the MCMC baselines. Otherwise, we would not be able to compute the log-likelihood.
• For example, $\mathcal{M}_1: y=ax^{-b}$ model has two parameters $a$ and $b$, and we define the likelihood as $\mathcal{N}(y; ax^{-b}, \sigma^2)$, where $\sigma^2$ is the observational noise.
• MCMC is performed over the joint posterior: $p(a, b, \sigma \mid \lbrace x_i, y_i\rbrace_{i=1}^M) \propto \prod_{i=1}^M \mathcal{N}(y_i; ax_i^{-b}, \sigma^2) \cdot p(a) \cdot p(b) \cdot p(\sigma)$, where $p(a)$, $p(b)$, and $p(\sigma)$ are predefined priors.
• We will clarify this in the revision to avoid further confusion. Thank you again for raising this point.

---

[Q2] The number of samples drawn (150) is tiny. Typical numbers for running MCMC chains are in the thousands.

• Thank you for pointing this out. We acknowledge that 150 MCMC samples are small compared to the typical choice. We chose such a small number because there are too many scaling curves to evaluate and each MCMC inference takes too much time. For instance, applying 3000 MCMC samples to all those curves takes around 4 weeks in total!
• In this external link, we compare our method against MCMC baselines with varying numbers of samples: 150, 300, 750, 1500, and 3000. To make the experiments feasible, we evaluated only on a subset of 30 scaling laws randomly sampled from the collection of all neural scaling law datasets we considered.
• As shown in the results, increasing the number of MCMC samples sometimes leads to slightly improved performance as expected, but the differences are marginal (actually, we had already verified it before submission). Also, the increased performances still fall short of our method, while the inference time increases substantially. This supports one of our key claims: NSL-PFN achieves strong performance with significantly better efficiency. After meta-training, PFN-based methods provide near-instantaneous inference, which is the power of amortized inference.
• We will discuss this further and provide the full evaluations in the revision.

---

[Q3] Eq (10): The expectation is over T which doesn't appear in the expression at all. Do you mean that (Y*, X*) is one instance from T? I have no idea what (X,Y) and "the autoregressive objective" are.

• As defined in L123–124 (the right column) in the main paper, the context and target set are denoted as $\mathcal{C}=(X,Y)$ and $\mathcal{T} = (X^*, Y^*)$, respectively.
• In Eq. (10), $\mathcal{C}, \mathcal{T}\sim p(\mathcal{D})$ is thus equivalent to $(X,Y),(X^*, Y^*)\sim p(\mathcal{D})$. We call the training objective $\mathbb{E}_{\mathcal{C}, \mathcal{T} \sim p(\mathcal{D})} [\log q (Y \mid X, \mathcal{C})]$ "autoregressive" because it is regressing $\mathcal{C}=(X,Y)$ conditioned on the same context $\mathcal{C}$.
• To improve clarity, we will revise the terminology to "context regression objective", or any terminology you have in mind. Thank you for bringing this to our attention.

---

[Q4] The priors in Table 1 seem a bit arbitrary. How were they chosen? How sensitive are the results to their choice?

• Thank you for pointing it out. For the design of our functional prior, we followed the same procedure used in LC-PFN. Specifically, we collected some of the well-known scaling law curves from various domains, and tried to match by eye the shape of our functional prior to the actual curves by manually controlling the parameters of the prior.
• Importantly, when controlling the prior parameters, we assumed that the actual values of the scaling law curves are not accessible. Also, for the ColPret dataset (whose size exceeds the sum of all the other datasets), we did not access the shape of the scaling law curves, but just tested our model on the dataset directly. Nevertheless, the model still achieved strong performance on ColPret, indicating that our prior generalizes well across tasks.
• Further, during the rebuttal period, we conducted a simple Bayesian optimization (BO) on the prior parameters to minimize the average RMSLE over 60 BO steps. As shown in this external link, the BO led to slight improvements on the performance, indicating that our prior tuning was not aggressive and that there remains room for further improvements.
• We will discuss the above in the revision.

Thank you for your time and thoughtful comments. We address each of your comments below. If you have any further questions or comments, we would be happy to discuss them during the author-reviewer interaction phase.

---

[Q1] Missing related work - Neural Processes.

• Thank you for the suggestion. We agree that the Neural Process (NP) family, originally introduced by Garnelo et al. [1], is closely related to our approach, as both aim to meta-learn an amortized posterior inference machine over a distribution of tasks. We will include the following discussion in the revision:
• Neural Processes (NPs) [1] typically learn a latent variable model where context data are summarized into a global latent representation, and target predictions are made by conditioning on this latent variable. This structure allows for uncertainty modeling and task generalization. The Conditional Neural Process (CNP) [2] simplifies this by removing the latent variable and directly conditioning target predictions on the aggregated context set. The Attentive Neural Process (ANP) [3] improves upon this by incorporating attention mechanisms to better capture context-target relationships. These models have been widely applied to few-shot learning, time series modeling, spatial inference, and so on. Further extensions include Transformer Neural Processes (TNPs) [4], which replace context aggregation with attention-based sequence modeling using Transformer architectures. While conceptually related, PFNs differ from NPs in an important way: PFNs do not require real dataset to train the inference machine. Instead, PFNs directly meta-learn the posterior predictive distribution with synthetic data generated from a manually-defined prior distribution, which allows Bayesian inference instead of maximum likelihood estimation. On the other hand, NPs are basically MLE and can be seen as implicitly learning the data-driven prior.

---

[Q2] The reference section needs to be updated, as several references point to arxiv preprints despite the respective papers being published work.

• Thank you for the suggestion. We will clarify the publication status of all references and replace arXiv citations with their corresponding published versions wherever applicable. We appreciate your attention to detail and will ensure the revision reflects the accurate and up-to-date citations as follows:

• Exploring the Limits of Large Scale Pre-Training → ICLR 2022

• An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale → ICLR 2021

• Language Models Are Few-Shot Learners → NeurIPS 2020

• Broken Neural Scaling Laws → ICLR 2023

• BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding → NAACL 2019

• A Survey on In-Context Learning → EMNLP 2024

• Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings → ICML Workshop 2019

• Scaling Laws for Neural Machine Translation → ICLR 2022

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces → COLM 2024

• Training Compute-Optimal Large Language Models → NeurIPS 2022

• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second → ICLR 2023

• Bayesian Active Learning for Classification and Preference Learning → CoRR 2011

• Adam: A Method for Stochastic Optimization → ICLR 2015

• Transformers Can Do Bayesian Inference → ICLR 2022

• In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization → ICML 2024

• ...

---

[Q3] The PFN has been trained with a rather large set of synthetic examples (1.6 million). How sensitive is the performance of the proposed approach to this size?

• Thank you for the question. First of all, it is well-known that PFNs benefit from training on a large amount of synthetic data, as shown in LC-PFN and other related works. Since PFNs are trained to approximate the posterior predictive over a distribution of tasks, a large number of sampled functions is typically required for effective generalization.
• In this external link, we provide the training losses and test performances over training iterations. We find that they keep converging as the model spends more synthetic data. Note that for each iteration, we randomly sample a batch of synthetic functions (which can be generated cheaply on-the-fly from our functional prior), hence the number of synthetic training data used is proportional to the training iterations.

---

Reference

[1] Garnelo et al. Neural Processes. ICML workshop, 2018.

[2] Garnelo et al. Conditional Neural Processes, ICML 2018.

[3] Kim et al. Attentive Neural Processes, ICML 2019.

[4] Nguyen et al. Transformer Neural Processes: Uncertainty-Aware Meta Learning via Sequence Modeling. ICML 2022.

Thank you for your time and thoughtful comments. We address each of your comments below. If you have any further questions or comments, we would be happy to discuss them during the author-reviewer interaction phase.

---

[Q1] Additional experiments - calibration of the predictive uncertainties.

• Thank you for the suggestion. In the following, we use mean square calibration error (MSCE), the calibration metric used in [1] to assess predictive uncertainty. As shown in this external link, our method demonstrates notably better calibration performances than the baselines.
• The gap is more prominent especially on the Double Descent (DD) which exhibits highly chaotic behavior. In such regimes, well-calibrated uncertainty is particularly important for reliable extrapolation. Furthermore, our method performs especially well on ColPret, the most recent and largest collection of realistic neural scaling laws, including models like GPT-3.
• These results show our model’s strength in handling challenging and real-world scaling behaviors. We will include those findings in the revision.

---

[Q2] Additional baselines - Bayesian linear regression with basis functions.

• Thank you for the suggestion. We agree that Bayesian linear regression (BLR) [2] with appropriate basis functions is a natural and lightweight approach for 1D regression and extrapolation.
• To evaluate, we conducted additional experiments with the following baselines: 1) BLR with neural network basis functions, 2) BLR with polynomial basis, 3) BLR with RBF basis, 4) BLR with Fourier basis, 5) BLR with sigmoid basis, 6) BLR with spline basis, and 7) Deep Kernel GP (DKGP) [3]. We use log marginal likelihood for tuning the parameters, such as the parameters of the NN as a BLR basis function. We provide qualitative examples of the BLR with NN baseline and DKGP at this external link and this externel link, respectively.
• As shown in this external link, those baselines are not competitive in the context of neural scaling laws. This is because, while they offer interpretability and closed-form inference, to our knowledge it is not straightforward to encode a set of inductive biases about neural scaling laws into the basis or kernel functions. This is why we originally did not consider those baselines.
• In contrast, our PFN-based method provides a flexible framework to encode our prior belief on the shape of the functions. We can use any functional prior as long as we can efficiently sample from it, which is one of the biggest advantages of using PFNs. This is precisely why we propose to use PFNs for the problem of extrapolating neural scaling laws.
• We appreciate your further suggestions as to how to effectively encode the prior belief on the scaling law behaviors into the corresponding basis or kernel functions. As long as time allows, we will try our best to do the additional experiments following your suggestions.

---

[Q3] I would have liked to see a much more comprehensive validation.

• We emphasize that, to the best of our knowledge, we have already collected all publicly available datasets of neural scaling laws, including recent and diverse settings such as ColPret (large-scale LLMs), DD (chaotic double descent behavior), and other benchmarks from $\mathcal{M}_4$ and BNSL. These datasets span a wide range of domains and scaling patterns, including smooth, non-smooth, and chaotic regimes.
• Across all of these settings, our method consistently shows strong performance, demonstrating its general applicability.
• Also, during the rebuttal period, we conducted the two additional experiments following your suggestion. 1) We additionally consider Bayesian linear regression and deep kernel GP baselines and find that they largely underperform our method. 2) We demonstrate that our method shows superior calibration performance than the baselines.
• Our paper also includes Bayesian active learning experiments to demonstrate the usefulness of uncertainty, which has not been discussed in the LC-PFN paper. If you find it insufficient, please feel free to suggest any additional experiments in detail, and we would be happy to try our best to do the experiments as long as time allows.

---

References

[1] Kuleshov et al., Accurate Uncertainties for Deep Learning Using Calibrated Regression. ICML 2018.

[2] Bishop et al., Pattern Recognition and Machine Learning, 2006.

[3] Wilson et al., Deep Kernel Learning, AISTATS 2016

Thank you for your time and thoughtful comments. We address each of your comments below. If you have any further questions or comments, we would be happy to discuss them during the author-reviewer interaction phase.

---

[Q1] Main novelty of this work compared to LC-PFN at both high- and low-level?

Thank you for the question. The main novelty of our work compared to LC-PFN [1] lies in both the high-level approach and low-level modeling choices:

High-level differences:

• Our work is the first to apply a Bayesian framework to the problem of neural scaling law extrapolation, whose functional distribution and characteristics are much different from those of learning curves the LC-PFN focused on.
• We further leverage uncertainty estimates to actively guide data acquisition, demonstrating the practical utility of uncertainty in reducing the cost of collecting high-quality scaling data — something not explored in LC-PFN.

Low-level differences:

• As you mentioned, we carefully design a novel functional prior tailored to neural scaling laws, informed by limitations of previous empirical approaches. Our prior can capture complex and chaotic behaviors such as double descent, enabling accurate extrapolation and reliable uncertainty estimates even under challenging real-world scenarios. In contrast, the prior of LC-PFN does not cover such complex patterns.
• We further propose a simple context-regressive learning objective that not only improves extrapolation performance but also enables Bayesian active learning — something that LC-PFN lacks.

We will make these distinctions more explicit in the revision.

---

[Q2] In Table 1, can the authors provide a reference that demonstrates the priors used for the coefficients? How do these choices impact the overall performance?

• Thank you for pointing it out. For the design of our functional prior, we followed the same procedure used in LC-PFN [1]. Specifically, we collected some of the well-known scaling law curves from various domains, and tried to match by the eye the shape of our functional prior to the actual curves by manually controlling the parameters of the prior.
• Importantly, when controlling the prior parameters, we have assumed that the actual values of the scaling law curves are not accessible. Also, for the ColPret dataset (whose size exceeds the sum of all the other datasets), we did not even access the shape of the scaling law curves, but just test our model on the dataset directly. Nevertheless, the model still achieved strong performance on ColPret, indicating that our prior generalizes well across tasks.
• Further, during the rebuttal period, we conducted a simple Bayesian optimization (BO) on the prior parameters to minimize the average RMSLE over 60 BO steps. As shown in this external link, the BO led to slight improvements on the performance, which indicates that our procedure of tuning prior parameters was not aggressive and there remain some rooms for further improvements of our prior.
• We will discuss the above in the revision.

---

[Q3] While the authors follow Muller et al. for implementing the PFN, can they clarify why positional encoding has been omitted?

• The original PFN paper (Muller et al. [2]) also omits the positional encoding for the following two reasons.
• In PFNs, when encoding the context points $\mathcal{C}=\lbrace(x_i,y_i)\rbrace_{i=1}^M$, the input feature $x_i$ itself already contains the "positional information" in the input space.
• Further, PFNs approximate the posterior predictive distribution $p(y^*,x^*|\mathcal{C})$, which is permutation-invariant w.r.t. the ordering of the context points $\mathcal{C}=\lbrace(x_i,y_i)\rbrace_{i=1}^M$, by definition. The same applies to our model as well as LC-PFN [1].
• We will clarify this in the revision.

---

[Q4] Update Figure 4 to better highlight the uncertainties for each method?

• Thank you for the suggestion. To improve clarity, we have updated those figures by separating the uncertainty plots for each method. This external link includes all those visualizations. We will include them in the appendix of the revision.

---

References

[1] Adriaensen et al., Efficient Learning Curve Extrapolation via Prior-Data Fitted Networks, NeurIPS 2023.

[2] Müller et al., Transformers Can Do Bayesian Inference, ICLR 2022