A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules

Thank you for taking the time to review our paper and providing extensive feedback, which has been invaluable in improving the quality of our work. You may have noticed that we skipped some of your questions in our response; this is because these questions have been addressed collectively in General Response.

W1: The scaling law expression in (2) is quite complicated, making it hard to follow the intuition.

A:

1. We note that the impact of LR decay on loss is inherently subtle and complex for its influence on training speed and stability is complex and not yet well-quantified. So the somewhat intricate form of the multi-power law is to be expected. Qualitatively, a larger LR can accelerate the reduction of training loss initially but may lead to overshooting and oscillations along sharp directions in the loss landscape over time. Conversely, a smaller LR provides a more stable training process but slows down convergence, requiring more time to achieve similar results.
2. As clearly claimed in Section 2, the main intuition in expression (2) is that we can decompose the loss into two terms which are constant LR loss in a power law form and loss reduction term.
3. Regarding the derivation and intuition behind the expression in Equation (2), we have detailed the derivation process in Section 2. Through analyzing the impact of LR decay on loss across multiple cases, we derived the general formula that loss follows. The derived formula is extensively validated in Section 3,4,5.

W2: To apply the result, the initial learning rate still needs to be specified

A:

1. In most experiments, we fixed the initial learning rate in the paper for the sake of derivation simplicity. However, when studying the loss under two-stage LR schedules, we conducted experiments without fixing the initial learning rate (Section 2.2). The results demonstrate that the multi-power law holds for different initial learning rates. Furthermore, we provided a detailed analysis of how the parameters of the multi-power law vary with the initial learning rate, as shown in Figure 4.
2. Additionally, We add the ablation experiments on the peak learning rate in the revised version. The results show that the multi-power law are applicable across various initial learning rate. See Figure 16 & Section 5, forth paragraph.

W5;Q3 Some figures with two 2 y axes is hard to read.

A: Thanks for pointing that out. We try to fix these concerns in the revision. The main improvements are as follows.

1. Separate the twin axes in the Figure 2 (Still indexed as Figure 2). Similarly, Figure 8 & Figure 16 are fixed in revision;
2. Adjust the zoom-in/out presentation, to facilitate the readers. Figure 1, 6, 7, 8, 11, 15, 17, 19, 20.
3. Add color bars, legends or captions for color-labeled/color-sorted curves. Figure 8, 10, 11, 13, 14.

Dear Reviewer ic14,

As the deadline approaches, we would like to kindly follow up to check if our responses have adequately addressed your concerns. Your feedback would be invaluable to ensure we have fully resolved any remaining issues.

In this revision, we have specifically added experiments scaling the model size, varying the model type, and incorporating downstream tasks to further support and validate our multi-power law. Additionally, we have made detailed adjustments to the presentation of figures similar to Figure 2.

Thank you for your time and thoughtful feedback.

Dear Reviewer ic14,

We sincerely thank you for your timely response and for providing valuable feedback that has greatly contributed to improving our work. We have revisited your comments and believe we have now addressed your outstanding concerns more directly.

In our supplemented response, we have reorganized our explanations within the General Response section to provide greater clarity. Additionally, we conducted an extensive sensitivity analysis of the different terms in our multi-power law. The results demonstrate the robustness and superiority of the complete multi-power law for loss curve prediction.

We deeply appreciate your professionalism and thoughtful comments. We look forward to your further feedback and hope to confirm that all your concerns have been fully resolved.

Sincerely,

The Authors

We would like to thank the reviewer for appreciating our multi-power law and noting its application in optimizing learning rate schedules to accelerate LLM pretraining. We also appreciate the acknowledgment that our work provides valuable insights into pretraining dynamics.

Through the review, we noticed that the reviewer may have misunderstood the scope of application we claimed for our multi-power law. Below we first clarify on this key point, then address the other concerns.

Major Concern: The Scope of Application of Our Law

W2, W3, & W5: In Weaknesses 2, 3 and 5, the reviewer raises concerns on the scope of application for our multi-power law: (W2) Is it model-independent, batch-size-independent, or optimizer-independent? (W3) Is it applicable to cyclic or increasing schedules? (W5) The scope of application appears restrictive because the model, batch size, and optimizer must remain unchanged.

Response:

To clarify, as mentioned in the introduction, our primary focus is on the LR schedules that decay the LR over time. For these LR schedules, we believe that the form of our multi-power law can be readily applied to a wide range of practical pretraining settings, but the parameters of the laws do depend on training details, such as the model architecture, batch size, optimizer, etc.

The Form is Correct Across Various Settings.

1. For a specific pretraining setup, after fitting the parameters in the multi-power law with a few loss curves, the law can generalize to other LR schedules with different shapes and horizons. Our law is even accurate enough to be used to optimize learning rate schedules for minimizing the final pretraining loss. See Section 3, 4.
2. Our experiments showed that the generalizability of MPL and superiority of optimized schedules are true for various model sizes (e.g., 1B models) and architectures (e.g., Llama-2, GPT-2 and OLMo [1]), though for each setting we need to fit the parameters separately. See Section 5.

The Parameters Do Depend on Training Details. We acknowledge that the parameters in the proposed multi-power law indeed depend on other training details, such as the batch size.

1. Compared to the previous Chinchilla scaling law, our law is already less restrictive because it can transfer across LR schedules. Our MPL unlocks two additional dimensions: our fitted law can predict (1) across schedules and (2) throughout the whole loss curve. We believe the LR-dependent data scaling law is an important step towards general scaling laws.
2. Our law is more sample-efficient than previous methods. In fact, with additional experiments, our multi-power law can uses only two observed loss curves from constant LR schedules and cosine LR schedules to accurately predict the loss curves of various unseen LR schedules, as well as to forecast the loss for longer training horizons with high precision. As shown in Figure 7, our MPL can achieve better prediction accuracy on 128,000-step loss of the Cosine schedule, with only $1/4$ compute resource. See Section 3.2 in revision.

Response to W2&W5: As clarified above, the form is independent while the parameters dependent on these factors. We have extended the LR-dependency and full loss curve prediction from previous work, while the previous work like Chinchilla Law fix these factors with other parameters also fixed.

Response to W3: Additional results on non-decreasing LR schedules. As mentioned in our introduction (the third paragraph,line 53), we focus on the schedule that non-increases the learning rate over time.

1. The main rationale is as follows. In large model training, using a cosine LR schedule or the recently proposed WSD LR schedule has become common practice. The LR schedules mentioned above, which are used in practical applications, typically start with a warm-up phase followed by a non-increasing decay pattern.
2. Nevertheless, we have added loss prediction experiments for a cyclic LR schedule [2] and an irregular non-decreasing LR schedule as shown in Figure 6. These experiments demonstrate that the multi-power law can effectively predict loss under these two non-decreasing LR schedules, even though our derivation is based on decay LR schedules. This highlights the strong generalization ability of the multi-power law. See Section 3.1, third paragraph.

While there is much more to uncover behind the proposed law, we would like to emphasize that we are the first to systematically propose a scaling law adaptive to learning rate schedules. As for other training details you mentioned, we believe they represent promising directions for future work.

[1] OLMo: Accelerating the Science of Language Models

[2] Cyclical Learning Rates for Training Neural Networks

W1: The theoretical justification for the proposed multi-power law is limited. The formulation of the law in Equation 2 lacks rigorous derivation or justification，doubting its generalizability.

A: We appreciate the reviewer's interest in scaling law theory and concerns regarding the theoretical justification of our paper.

1. Even though we don't have rigorous justification, we do extensive experiments, including ablations over peak LR, diffenrent LR schedule types, differnet data size and different model sizes to support our proposed law to support our proposed law. Futher, more discussions including larger model model size of 1B, as well as the non-monotomic schedules are ablated. See Section 3, Section 5.

2. It is generally very hard to prove theorems for the optimization process in deep learning under realistic assumptions. Even for linear regression, purely proving the Chinchilla scaling law in theory is already very non-trivial and needs 40+ pages [3] Nevertheless, in the rebuttal revision, we present a theoretical analysis for quadratic loss functions. We demonstrate that the multi-power law can indeed arise when the Hessian and noise covariance matrices have a power-law decay in their eigenvalues (Section 5, last paragraph & Appendix K & Appendix L)

W4: The paper demonstrates reduced training loss at specific iterations with the proposed scheduler but does not clarify if this advantage holds at convergence.

A:

1. Clarify the setting: due to the high cost of training, practitioners typically focus on minimizing the final loss within a fixed compute budget, which corresponds to fixing the total number of steps ($T$). Notably, even for models like LLaMA-2, the loss does not plateau at the end of training.
2. Learning rate schedules are always defined for a given $T$. It is generally impossible to extend a schedule beyond its predefined duration—for example, cosine and WSD schedules are explicitly tied to a specific $T$. In particular, when the ending LR is $0$, the training can not go on with non-positive LR.
3. Regarding our experiments, the data-to-model ratio in our setting is even larger than that of LLaMA-2. For instance, our 0.025B model is trained on more than 36B data tokens, compared to 2T tokens for 70B LLaMA-2.

W6: There is a typo on line 491, and Figure 8 lacks clarity.

A:

1. Thanks for pointing this out. In line 491 we mean we set the $\eta_{0}=\eta_{\max}$, in which $\eta_{\max}=3 \times 10^{-4}$.
2. The fixed Figure 8 is indexed as Figure 20 (right) in revision. In Figure 20 (right), the pink dash-dotted line at the top represents the WSD schedule with linear-decay (WSDLD) and the green solid line represents the corresponding loss curve. The blue dashed lines below represents the predictions of the momentum law. The orange dashed line represents the multi-power law prediction while the purple dashed line represents the loss prediction of the auxiliary process mentioned in Section 2.1 for comparison. As shown, the momentum law can fit the loss in the stable stage, but predicts a counterfactual concave curve in the decay stage, whereas the multi-power law accurately predicts the loss curve. We sincerely apologize for these confusions caused by our oversight.

Q2: The effectiveness of the proposed approach at long horizon is only validated on cosine and constant learning rate schedulers. Could you add experiments on other learning rate schedulers?

A: We have supplemented our study with experiments on cyclic, and an irregular piecewise linear LR schedule over a longer horizon (72,000 steps). As shown in Figure 7, the multi-power law accurately predicts long-horizon loss across these unseen LR schedules. See Section 3.1, third paragraph.

[3] Scaling Laws in Linear Regression: Compute, Parameters, and Data

Dear Reviewer Uu5m,

We sincerely thank you for your insightful comments, which has greatly contributed to the improvement of our work. As the rebuttal deadline approaches, we kindly follow up to ensure that our responses have adequately addressed your concerns. Your thoughtful comments have provided invaluable guidance, and we deeply appreciate your time and effort.

In the revised version, we have clarified our experimental settings and extended our evaluation. Specifically:

• We have tested non-monotonic schedule types, such as cyclic and polyline, over longer horizons.
• Ablation studies have been added, including scaling the model size, varying the model type, and evaluating on downstream tasks.
• We have varied additional hyperparameters, such as batch sizes and peak learning rates, to demonstrate that the form of our multi-power law holds across diverse settings, while the parameters of the law depend on specific configurations.
• A supplemental theoretical analysis has also been included, providing insights into how the multi-power law might arise, directly addressing the points you raised.

Thank you again for your time and thoughtful input. We look forward to any additional feedback you may have and would be grateful for your further comments.

Sincerely,

The Authors

Thank you for your response. It definitely clears some of my concerns. However, the lack of theoretical proof for the proposed scaling law is still concerning. Comprehensively considering all these factors, I will increase my rating to 5.

Thank you for your appreciation of the multi-power law. We have added evaluations on larger model sizes, different model types, and the performance of optimized LR schedules on downstream datasets to validate the flexibility of the multi-power law. You can find these details in General Response.

Q1： How do you treat warm-up stage in your derivation and experiments?

A: We incorporate the effect of warm-up stage into the constant loss term ($\mathcal{L}_{const}=L_0+AS_1^{-\alpha}$) in our results. We add the sum of the learning rate during the warm-up phase to $S_1(t)$ to represent the learning rate accumulation at time $t$ in the formula. The loss reduction term in the formula are unchanged, except that they are computed on the learning rates after the warm-up phase. Using this adjusted formula, we fit the loss curves observed in the experiments (Appendix E.3).

Q2: Tuning WSD to 1/10 peak learning rate is not fair. Better way is to tune WSD to 0.

A:

1. Mutil-power law outperforms WSD even when we tune the WSD ending learning rate to 0. The optimized schedules outperform WSD variants with "zero" ending learning rates. As shown in Figure 21, we compare WSD(LD) variants with near-zero ending learning rates, the optimized schedules, and the original WSD(LD) schedules. In this experiment, the ending learning rate is set to $3 \times 10^{-7}$, which is $1/100$ of the previous setting. Notably, a lower ending learning rate does not consistently lead to improved final loss. For example, the final loss of the WSD schedule increases with a near-zero ending learning rate. This suggests a complex interaction between the ending learning rate and the decay function, highlighting the challenges of jointly optimizing these hyperparameters in WSD schedules. In this context, the optimized schedule demonstrates its advantage by reducing the need for extensive hyperparameter tuning in WSD variants. Results details are shown in Figure 21 (Appendix H).
2. Actually, we follows the conventions used in previous literatures[1, 2, 3] on cosine learning rate and WSD, we tune WSD to LR / 10 as WSD paper also didn't decay the LR to 0.
3. Tuning the final LR introduces an extra hyperparameter and more training cost. For using our law, we just need to run at most 3 runs to fit the parameters, requiring less tuning.

Other Comments

Thanks for your careful reading, and all typos in your comments have been fixed. Specifically, in line 512, as you said, we mean "optimizing the WSD schedule is difficult" since this learning rate schedule includes many hyper-parameters.

[1] Scaling Laws for Neural Language Models

[2] Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

[3] Minicpm: Unveiling the potential of small language models with scalable training strategies

We greatly appreciate the efforts you put in, and the thoughtful feedback on our work. Your comments have been invaluable in improving our paper. We skip some weakness and questions you proposed, since we have answered in the General Response, including W1, W3, asking about verification experiments on larger model sizes and extra model types.

W2: Take other hyperparameters into consideration, e.g. model size, batch size, initial seeds, etc. to make this law more practical.

A:

1. When considering the impact of LR schedule only, we have conducted very extensive experiments, including ablations over peak LR, diffenrent LR schedule types, different data size and different model sizes to support our proposed law. More discussions including larger model model size of 1B, as well as the non-monotomic schedules are ablated. Details of ablation see Section 5.

2. Due to the time and resource constraints, we are not able to thoroughly verify our law in every single possible setting. But we believe our current set of experiments have already provided strong empirical evidence for justifying our proposed law. We believe the LR-dependent data scaling law like our MPL is an important step towards general scaling laws.

W4: The tuning process needs to collect data from different learning rates. Each optimization takes over $5 \times 10^4$ steps. The tuning cost is a little high.

A:

1. The sample efficiency is indeed our advantage over the previous works. We can use $2$-$3$ loss curves to predict the whole loss curves while the previous scaling laws, like Chinchilla [1], require to collect more than $10$ curves to fit the final loss given fixed model and fixed hyperparameter setting. As shown in the left of Figure 7, MPL fitted on two 24,000-step loss curves accurately predicts the final loss at 128,000 steps (over $5$ times length) trained with a cosine schedule. MPL achieved lower extrapolation error while using only $1/4$ computational resources required for Chinchilla fitting.
2. We set optimization step as $5\times 10^{4}$ to ensure the convergence of the fitted parameters. The computational cost of our law fitting is much lower than anticipated. The parameter space for the MPL ($L_0$, $A$, $B$, $C$, $\alpha$, $\beta$, $\gamma$) is only 7-dimensional, and the loss computation involves predictions over training steps, typically fewer than 100k. As a result, tuning the multi-power law (MPL) is lightweight compared to standard deep learning training.
   The optimization of the learning rate (LR) schedule may seem more computationally intensive, since its parameter dimensionality corresponds to the length of the loss curve. However, this process only requires final-step loss predictions, making it highly efficient. Consequently, MPL fitting and schedule optimization rarely require GPU acceleration.

[1] Training Compute-Optimal Large Language Models

Dear Reviewer H6ae,

We sincerely thank you for your valuable feedback, which has greatly contributed to improving our work. As the rebuttal deadline approaches, we kindly follow up to check if our responses have adequately addressed your concerns. Your feedback would be greatly appreciated to ensure all issues have been resolved to your satisfaction.

In the revised version, we have incorporated additional experiments to address your concerns more thoroughly:

• We conducted experiments with various model types, including both open-source and self-run results, and larger model sizes (e.g., a 1B model and 144B tokens) to validate the multi-power law's applicability in practice.
• We included evaluations on downstream tasks to examine the practical benefits of law-inducing schedules.
• We comprehensively explored hyperparameters, including peak learning rates, batch sizes, and random seeds, through extensive ablation studies.

We believe these additional experiments and discussions effectively demonstrate the capacity and generalizability of the multi-power law.

Thank you again for your time and thoughtful comments. We sincerely look forward to any further feedback you may have on our responses.

Sincerely,

The Authors

Dear Reviewer H6ae,

We greatly appreciate your thoughtful comments. As the deadline approaches, we sincerely look forward to your response. We fully understand how busy you may be and have summarized the supplemented experiments addressing your major concerns as follows:

1. Test our multi-power law across different model types, including both open-source models (e.g., Olmo 7B) and self-run experiments (e.g., GPT-2);
2. Scaling up experiments to a larger scale (1B-parameter models and 144B tokens);
3. Assessing large-scale results using the induced schedule on downstream tasks;
4. Ablation studies on hyperparameters, including peak learning rates, batch sizes, and random seeds.

We believe these comprehensive results address your main concerns. We hope they meet your expectations and contribute to a constructive discussion. We kindly request that you consider our detailed response in your further discussions with the AC, even if your current schedule does not allow for additional feedback.

Thank you once again for your time and valuable insights.

Sincerely,

The Authors

We sincerely thank for your effort and thoughtful feedback on our work. Your comments and suggestions have greatly contributed to improving the quality and clarity of our paper.

W1, W4 & Question: It is important to understand how the gain in training loss quality effects the validation loss and downstream performance.

A: We would like to clarify that we are considering the typical pretraining setting where each training step is taking a fresh data sample from a data stream (i.e., one-pass training).

1. The training is essentially done on the population distribution, so there is no generalization gap between the training and test loss.
2. We held out a validation set sampled from the same distribution as the training set and use the validation loss instead of the training loss in all our plots except that for the OLMo experiments we directly used their released curves.
3. we have added extensive experiments in downstream tasks and the results of our optimized LR schedule present better performance than the cosine LR schedule as shown in the General Response.

W2: It would be very interesting to understand the benefits of the proposed method in the scenario when combined with techniques like MuP.

A: The reviewer is essentially asking the following question: what the difference between our approaches and the well-known MuP in the LR-tuning is, and whether the benefit of our approach can be replaced by MuP.

• Our work is orthogonal to MuP. We compare our work with MuP as follows:
  1. MuP enables learning rate (LR) transfer across model scales. The current practice usually only searches for the optimal value of the initial LR for a specific type of schedule, such as the cosine schedule.
  2. Our method attempts to leverage the derived scaling law to understand and approximate the optimal LR shape.
• Our approch of optimizing the LR schedule with multi-power law can be combined with MuP as follows: one can use our approach to search for an optimal LR schedule for small models, then may transfer the optimal schedule to a large scale without extra tuning.

We would like to thank the reviewer for raising this practical question, and we believe it would be an promising future direction even though it falls outside the scope of our paper.

W3: It is important to understand the temperature of the training in the proposed method and how the benefits would be affected once the batch size is tuned

A: The reviewer is essentially asking the following question: if the batch size in the training runs with baseline schedules is optimally tuned to $B'$, does our optimized schedule still benefit much?

1. If the batch size of baseline runs is changed $B'$, we could just apply the multi-power law over a few training runs with batch size $B'$ and fit the parameters in the law. Then newly-fit law could induce a new schedule.
2. Our experiments with different batch sizes (e.g., 128 sequence batch size is used in the experiments in Section 3, and the 1B experiments use 512 sequence batch size in Section 5) suggest that the applicability of our law does not depend on the choice of batch size.
3. It is noteworthy that we are optimizing the schedule while fixing the peak learning rate, and our optimized schedule decays from the peak value only in the late phase. In this case, it is believable that the optimal batch size strongly depends on the peak learning rate and very weakly depends the shape of LR decay. In the paper that the reviewer pointed out[1], they also only studied the relationship between peak LR and batch size. In this perspective, the tuning of batch size is orthogonal to our discussion. Nevertheless, we agree that scaling the batch size and LR schedules simultaneously is a promising future direction.

[1] An Empirical Model of Large-Batch Training

W5: I would recommend authors testing their methodology against the existing scaling laws that allow us to understand benefits of this method when scaling the pretraining FLOP budget.

A: Details see Figure 7 & Section 3.2, first paragraph in revision.

1. When scaling the pretaining FLOP buget and predicting the final loss, MPL shows higher sample efficiency and higher prediction accuracy compared with Chinchilla scaling law. As shown in the left panel of Figure 7, MPL is fitted to two 24,000-step loss curves and used to predict the final loss of a 128,000-step loss curve trained with a cosine schedule. MPL achieved lower extrapolation error while using only $1/4$ of the computational resources required for Chinchilla fitting. Second, we evaluate MPL and Chinchilla fits on the 7B OLMo curve trained with a linear schedule. As shown in the right panel of Figure 7, the MPL aligned closely with the OLMo training loss, whereas the Chinchilla fit showed significant deviations.
2. Our MPL extends the Chinchilla law in two dimensions: after fitting, our law can be applied to (1) different schedules and (2) the whole loss curve prediction. To our best knowledge, we are the first to systematically propose a scaling law for the full training time loss prediction.

Q: How marginal the improvements are by looking at the stddev values for the validation loss due to the randomness.

A: The variance of final loss due to the random seed is minor compared to the improvements of both prediction error and the optimized schedule.

• To address the potential impact of randomness, we added an ablation study on randomness (See Section 5, last second paragraph). Specifically, we selected three random seeds and trained a 25M-parameter model for 24,000 steps using cosine learning rate schedules. As illustrated in Figure 20, the standard deviation of the resulting loss values is less than $0.001$, establishing a lower bound for prediction errors. As shown in Figure 7, the improvement of prediction accuracy from Chinchilla Laws much exceeds the randomness margin. The final losses trained with optimized schedules also surpass the baselines multiple times of this margin.
• The improvement of such scale matters due to the following reasons:
  1. The reduction in validation loss indicates the improvement in the downstream tasks. In our 1B experiment, validation loss improvement of the optimized schedule is about $0.013$ and deduces a $2.17$% accuracy progress in the LAMBADA task. See Section 5, third paragraph in revision.
  2. Although the largest model size in the experiments we ran is 1B, for models of much larger scale, reducing the validation loss by a tiny amount could even induce an emergent behavior on downstream tasks[2].

[2] Understanding Emergent Abilities of Language Models from the Loss Perspective

Dear reviewer g7Pt, As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your feedback to ensure we have fully resolved any outstanding issues.

Notably, this revision and response include the clarification of using validation in our paper and more convincing experiments including scaling the model size, varying the model type, and adding downstream tasks to address your specific concerns. Thank you for your time and consideration.

Dear Reviewer g7Pt,

We sincerely thank you for your detailed and constructive feedback, which has greatly contributed to improving our work. As the rebuttal deadline approaches, we kindly seek your further input to ensure our responses have addressed your concerns. Your insights are invaluable in helping us clarify our contributions and refine our settings.

In this revision, we have:

• Clarified the use of validation loss in our methodology,
• Conducted additional experiments to address your specific suggestions, including scaling the model size, varying the model type, and adding downstream task evaluations to verify the improvement of our method.

Additionally,to directly address your query:

W3: How might the benefits of your approach be affected by batch size tuning?

we performed batch size ablation experiments to validate that the form of our multi-power law holds consistently across different batch sizes. We compared the validation loss of our optimized schedule against commonly used schedules. As shown in the table below, our optimized schedule consistently outperforms the Cosine schedule by a significant margin and exceeds the WSD and WSDLD schedules, even without elaborate tuning. This demonstrates that the benefits of the scheduling dimension are robust across various batch sizes. (Section 5, fifth paragraph & Appendix I, last paragraph)

We hope that these additional experiments and clarifications have addressed your concerns. Thank you again for your time and comments. We look forward to any further feedback you may have.

Sincerely,

The Authors

Dear Reviewers,

Only 1 out of 5 reviewers have responded to the author rebuttal by the original deadline of Nov 26th. There is a new deadline which is Dec 2nd. Please respond timely as reviewer response and acknowledgement is a critical part of the ICLR review process.