We appreciate the reviewer's insightful feedback. Please allow us to clarify our methodologies point to point.

1. Theoretical justification for Hypothesis 1

1. The convexity of loss descent velocity w.r.t. LR

   Assume $L(\theta)$ is the loss function to be minimized, and we use SGD with the update rule:

   $$\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$$

   where $\eta$ is the learning rate (LR).

   Intuitively, if $\eta$ is too small, updates are small, and the loss decreases slowly; if $\eta$ is too large, the updates may overshoot, causing oscillation or even divergence.

   The expected loss descent velocity per step (using Taylor expansion) is:

   $$\mathbb{E}\left[ L(\theta_{t+1}) - L(\theta_t) \right] \approx -\eta \| \nabla L(\theta_t) \|^2 + \frac{L}{2} \eta^2 \| \nabla L(\theta_t) \|^2$$

   where $L$ is the Lipschitz constant of the gradient. When the learning rate is small, the first term dominates the expectation, and a smaller $\eta$ leads to a smaller decrease in loss. When the learning rate is large, on the other hand, the second term cannot be neglected. It contributes to suppress the expected loss descent value.

   Differentiating with respect to $\eta$ and setting to zero for the extremum:

   $$\frac{\partial}{\partial \eta} \left( -\eta + \frac{L}{2} \eta^2 \right) = -1 + L \eta = 0 \implies \eta^* = \frac{1}{L}$$

   This shows that the loss descent velocity with respect to $\eta$ is a convex function (has only one maximum).

2. Shared optimum across loss and loss slope optimization

   In Figure 1, we show that the optimum for both training loss and its slope remain the same across most of the training process. Based on this observation, we summarize the reason why the optimal learning rate for fastest loss decrease and lowest final loss generally coincide as follows:

   • Optimial loss slope -> optimal loss: the best learning rate discovered via the largest loss slope leads to the largest loss descent per step, resulting in the lowest attainable final loss.
   • Optimial loss -> optimal loss slope: if a learning rate cannot achieve the largest loss slope, the resulted final training loss must be suboptimal, too, because the LR with the largest loss slope will always produce lower training loss. As a result, the optimal training loss must be achieved by a learning rate with the largest loss slope.

   The conclusions above can be generalized to any learning rate values: learning rate with faster loss descent velocity always achieve lower training loss. Since we have justified that the loss slope optimization is convex, it can be easily dedued that the training loss optimization is also a convex problem.

2. Details about Figure 1

We thank the reviewer for the careful reading and useful suggestions, and we will supplement enlarged figures as well as detailed caption texts in the revised version of the paper.

We also would like to further clarify the content and main conclusions drawn from Figure 1:

1. Subplot a,b,c show the training loss curves and LR trajectories for different pretraining scenarios.
2. In Subplot e,f,g, each curve represents the training loss dynamics across different LR settings at a certain training step.
3. In Subplot h,i,j, each curve indicates the loss slope dynamics across different LR setting at a certain loss value.
4. The curve groups shown in Subplot e,f,g and h,i,j demonstrate consistent optima across the training process, and the optimal LR for the fastest loss descent and lowest final loss generally coincide.

3. Comparison with Hypergradient Descent (HD)

We thank the reviewer for the careful reading. In fact, the Hypergradient Descent (HD) and AdaLRS algorithms serve different purposes, and therefore they are not comparable. We list the differences between the two methods below:

1. Main idea
   • HD: dynamically adjusts learning rate across the entire training process.
   • AdaLRS: searches for the optimal learning rate scale at the beginning steps of pretraining.
2. Workflow
   • HD: adjusts LR dynamically across the training process, serving as a learning rate scheduler itself.
   • AdaLRS: adjusts only the learning rate scale, which is multiplied with base LRs from an existing scheduler.
3. Compatibility with modern learning rate schedulers
   • HD: cannot be used with any learning rate scheduler.
   • AdaLRS: compatible with a series of modern learning rate schedulers, such as cosine scheduler, WSD scheduler, and potentially more alternatives.

4. Additional costs in checkpoint restoration and rollback

1. Memory and wall-time costs of checkpoints restoration

   To the best of our knowledge, modern foundation model pretraining frameworks, such as deepspeed, support checkpoint restoration in parallel with model training. In our experiments, we find out that the memory and wall-time costs of checkpoint restoration are negligible, and do not block the training process.

2. Memory and wall-time costs of roll-backs

   Assuming a extreme case where all the LR upscaling attempts are failed, every training step will be performed twice (once in the look-ahead attempt, and another in alternative LR downscaling process). As discussed in Section 2.1 workflow, the LR adjustment process is restricted to certain training steps, which is the 10%-40% part of the training process. Therefore, the maximum time consumption of an entire training process will be 130% of the baseline training time. It is still much faster than traditional LR searching methods.

3. Feasibility at 70B parameter scale

   The design of AdaLRS is based solely on the training loss dynamics, which unveils promising scalibilities. In our early experiments in 72B LLM and VLM training, AdaLRS exhibits expected LR adjustment effectiveness. We will validate the applicability of AdaLRS at 70B models at scale in our future work, and supplement corresponding details in the revised version of our paper.

Thank you to the authors for their response.

1. The theoretical justification is valid. However, there appears to be a minor error in the phrasing: "This shows that the loss descent velocity with respect to $\eta$ is a convex function (has only one maximum)." Given that the objective is to minimize $L(\theta_{t+1}) - L(\theta_t)$ (i.e., maximize the descent), and the quadratic function has a positive leading coefficient (indicating convexity and a single minimum), the term "maximum" should be corrected to "minimum."

2. I appreciate the added discussion as suggested, which adequately addresses my concerns. Please incorporate this discussion into the revised paper.

3. I raised my score from 4 to 5. Happy to see the paper published.

We appreciate the reviewer's insightful feedback. Please allow us to clarify our methodologies point to point.

1. Paper readability

We thank the reviewer for the careful reading. The potentially confusing notations are introduced to prove the convergence of AdaLRS. We consider potential statistical error in loss slope estimation and several other details for broader generalizability of the proof. For better readibility, we provided a simplified workflow description of the proposed AdaLRS algorithm in Section 2.1 workflow. We admit certain algorithmic details can be further clarified, and will adjust corresponding descriptions accordingly in the revised version.

Below is a simplified workflow description of the proposed AdaLRS algorithm, and we hope it can help address potential confusion:

1. AdaLRS attempts to upscale the LR when the loss descent velocity decreases, which is triggered by a threshold $\theta$ shown in Algorithm 1 of the paper.
2. k attemping training steps are performed to estimate the loss slope after LR upscaling.
3. If the loss descent velocity increases after LR upscaling attemption, this adjustment is retained; otherwise it is discarded and replaced by a LR downscaling adjustment.
4. Either a LR upscaling or downscaling adjustment is performed, a decaying factor $\gamma$ is applied to the LR scaling factors, ensuring the convergence of LR adjustments.

2. Algorithmic representations

As shown in Equation 1 and Algorithm 1, the definition of $e$ remains consistent, which is the loss slope estimation error introduced to ensure the generalizability of our proof for AdaLRS's convergence. The threshold $\theta$ introduced in Algorithm 1 is set to automatically trigger LR upscaling attemptions for various pretraining tasks. We elaborate the necessity and importance of this threshold in point 4.

3. Additional computation cost

We mention that the attempted LR upscaling will be reverted if the resulted loss descent velocity decreases in Section 2.1 workflow. In Section 3.2, we further discuss the acceleration ratio of AdaLRS in different pretraining scenarios, and we observe that the k-step lookahead does not undermine the convergence speed and loss improvement of AdaLRS. Even for large LR settings, where a series of roll-backs are performed, models trained with AdaLRS still achieve an over 30% acceleration ratio to reach the same loss level as baselines. It is also worth noticing that the roll-backs do not undermine the efficiency of AdaLRS in optimal LR searching. Since we only conduct LR search in 30% of the training steps, the maximum computational cost of AdaLRS is 130% of the baseline training cost, which still outperforms the traditional LR searching methods markedly.

4. Hyperparamter

Besides the LR scaling factors $\alpha$, $\beta$, and $\gamma$, we introduce only two additional hyperparameters $\theta$ and $e$ in our algorithm. The threshold $\theta$ provides an automatical way to trigger LR upscaling attemptions, regardless of the training scenario or scale. We set $\theta$ as 0.9 in our experiments, i.e., an attempted LR upscaling is triggered whenever the loss descent velocity decreases by at least 10%. This simple strategy ensures steady loss descent with high velocity, as well as timely LR upscaling when the velocity decays. More importantly, this method achieves adaptive LR upscaling attempts across various pretraining tasks, regardless of the training scenario. The loss slope estimation error $e$, on the other hand, is incorporated to ensure the generalizability of our proof for AdaLRS's convergence, which is shown in Section 2.2.

For LR scaling factors $\alpha$, $\beta$, and $\gamma$, the first two scaling factors are necassary for AdaLRS, and $\gamma$ ensures the convergence of AdaLRS to the small enough neighborhood of the optimal LR. We further conduct experiments to demonstrate the importance of the scaling factors and AdaLRS's robustness under various hyperparameter settings. Starting from a 2e-5 LR for 2B LLM pretraining with WSD scheduler, we show the LR dynamics in the 64M-sample pretraining experimets:

In this table, Exp Baseline, A correspond to the "Small LR" setting for 2B LLM pretrain in our paper, and Exp B, C are conducted with different LR scaling factor values. Exp D, E adopt smaller decaying factor, which suppresses the "overshooting" behavior observed Exp B. Overall, depite the varying hyperparameter settings, learning rates converge to the neighborhood of the optimum 2e-4 robustly.

We also choose the model checkpoints at 60,000 step (150,000 in total) to evaluate their test PPL, and the results show that AdaLRS achieves improved model performance over the baseline under various hyperparameter settings. We will include the complete loss dynamic figures and evaluation results in the revised version of our paper.

5. Convexity assumptions

The convexity assumptions is indeed intuitive, but they have not been validated systematically across different foundation model pretraining setting. In this work, we not only validated this convexity in various pretraining scenarios, but also unveil that the loss and loss slope optimization share the same optimum. The shared optimum builds up the keystone of the convergence of AdaLRS algorithm, which is discussed in Section 2.2.2.

6. Logics of Equation 1

As described in Section 2.1 workflow, the proposed AdaLRS algorithm downscales the LR if the attempted LR upscaling results in decreased loss descent velocity. The condition in Equation 1 Line 2 is indeed a typo, and it should be $v(\alpha'η_t) < v(η_t) − 2e$. We thank the reviewer for the careful reading and will correct it in the revised version.

7. Loss spikes in pretraining

Loss spikes are indeed inevitable in pretraining. Actually, we also encounter them in our experiments, but these loss spikes do not undermine the applicability of AdaLRS. Firstly, the gradual LR scaling strategy shown in Algorithm 1 suppresses a number of loss spikes. Secondly, a simple loss smoothing strategy is found to be sufficient to mitigate the influence of loss spikes. As a result, despite the presence of loss spikes, AdaLRS can still achieve expected performance improvement in relatively large scale pretraining.

8. Application scope

This issue is discussed as one of AdaLRS's limitation in Appendix C. We argue that it does not undermine the effectiveness of AdaLRS in optimal LR searching. As shown in Figure 2 and Table 2, 3, AdaLRS can achieve significant performance improvement in pretraining under both small LR and large LR settings. We also discussed in Section 3.2 about the effectiveness of AdaLRS in large LR settings, presenting that even if AdaLRS cannot achieve optimal model performance in a single run, it still finds the optimal LR efficiently, outperforming traditional LR searching methods markedly.

9. Chicken-and-egg problem

1. As discussed in the point above, AdaLRS finds optimal LR efficiently and effectively, regardless of the initial LR setting. Therefore, although the chicken-and-egg problem may hinder us from reaching the optimal model performance in a single run, it does not undermine the applicability of AdaLRS in optimal LR searching.
2. In practice, researchers can refer to existing opensource works for LR setting estimation. For example, 1e-3 and 2e-4 are often used as initial LRs for VLM and LLM pretraining [1,2], respectively. Since AdaLRS approximates the optimal LR with geometric error decay, which is proved in Section 2.3 of the paper, it allows a wide range for small LR choices. As a result, the influence of the chicken-and-egg problem is limited.

[1] Li B, Zhang Y, Guo D, et al. Llava-onevision: Easy visual task transfer[J]. arXiv preprint arXiv:2408.03326, 2024.

[2] Dey N, Gosal G, Khachane H, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster[J]. arXiv preprint arXiv:2304.03208, 2023.

We appreciate the reviewer's useful suggestions. Please allow us to clarify our methodologies and conclusions point to point.

1. Statistical confidence analysis

Since large scale foundation model pretraining is resource-consuming, we opt to conduct repetive experiments at 8M-sample scale for 2B LLM pretraining for statistical confidence analysis. We use 5 different random seeds for model training, and report the final pretraining losses.

The mean value and standard deviation of the pretraining losses are 4.723 and 0.009, respectively. As shown in Table 2 of our paper, the training loss improvement of AdaLRS experiments all exceed the 3σ range, except for the Fit LR experiment for 7B LLM pretraining. These results indicate that the AdaLRS pretraining loss improvement is statistically significant.

2. Disruptive parameter updates in large LR experiments

The disruptive parameter updates is introduced by the large LR setting rather than the AdaLRS algorihhm, and therefore the large LR baseline models indeed suffer from similar performance degradations. As shown in Figure 2 and Table 2, 3 of our paper, large LR baseline experiments converge to relatively high pretraining losses and suffer from performance degradation in evaluation results. Despite the degraded model performance, AdaLRS still reaches the optimal learning rates and improves pretraining convergence significantly, demonstrating its effectiveness in optimal LR searching.

Dear Reviewer 5CYH:

We would like to thank you for the careful reviewing of our paper and for the comments. Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us.

Best regards, Authors

We appreciate the reviewer's insightful feedback. Please allow us to address the reviewer's concerns point to point.

1. The choice of hyperparameters

The default setting $\alpha$=3, $\beta$=2, and $\gamma$=0.99 in the paper is chosen for simplicity, and we did not introduce human labor in hyperparameter tuning. To validate the robustness of AdaLRS, we further conduct experiments under various hyperparameter settings. Starting from a 2e-5 LR for 2B LLM pretraining with WSD scheduler, we show the LR dynamics in the 64M-sample pretraining experimets:

In this table, Exp Baseline, A correspond to the "Small LR" setting for 2B LLM pretrain in our paper, and Exp B, C are conducted with different LR scaling factor values. Exp D, E adopt smaller decaying factor, which suppresses the "overshooting" behavior observed Exp B. Overall, depite the varying hyperparameter settings, learning rates converge to the neighborhood of the optimum 2e-4 robustly.

We also choose the model checkpoints at 60,000 step (150,000 in total) to evaluate their test PPL, and the results show that AdaLRS achieves improved model performance over the baseline under various hyperparameter settings. We will include the complete loss dynamic figures and evaluation results in the revised version of our paper.

2. Downstream task performance

To investigate the generalization of AdaLRS's improved pretraining loss, we further conduct a lightweight 6M-sample SFT with the Infinity-Instrcut dataset [1] for the 2B LLM experiments. We evaluate the SFT model performance on open-ended question-answering tasks, such as Alpaca-Gen and KNIGHT-Gen [2], which aligns with real-world LLM applications. We follow previous work [2] to use Rouge-1, Rouge-2, and Rouge-L as evaluation metrics.

As shown in the table, SFT models from AdaLRS pretrained checkpoints achieve significantly higher benchmark scores than baseline models. For fit LRs, AdaLRS pretrained models suffer from slight performance degradation, which coincides with the pretraining loss shown in Table 2 of our paper.

These results demonstrate the performance generalizability of LLMs pretrained with AdaLRS. For VLM experiments on downstream benchmarks, we refer to Table 3 in our paper for more details.

[1] Li J, Du L, Zhao H, et al. Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models[J]. arXiv preprint arXiv:2506.11116, 2025.

[2] Yang D, Xiao D, Wei J, et al. Improving factuality in large language models via decoding-time hallucinatory and truthful comparators[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(24): 25606-25614.