Hyperparameter Loss Surfaces Are Simple Near their Optima

We respectfully disagree with several characterizations in this review.

"When does the theory hold?"

Our paper discusses in-depth when the theory holds, and provides extensive empirical evidence for its generality. Formally, Theorem E.2 provides sufficient conditions and a rigorous proof of when the theory holds. Empirically, we show it holds for:

• Diverse architectures (Llama, DeBERTaV3, AlexNet, ResNet, and ConvNext),
• Multiple tasks (language model pretraining, supervised finetuning, ImageNet pretraining),
• Both continuous and discrete hyperparameters (batch size, epochs, and blurpool are all discrete),
• When models are fully trained vs. before convergence (Appendix D).

The asymptotic regime, the region where our theory applies, covers 18-58% of each search space across our experiments. Together, these results show our theory’s generality across architectures, tasks, modalities, training regimes, and different compositions of hyperparameter spaces.

"mainly discussed about continuous hyperparameters"

This is incorrect. Our experiments include multiple discrete hyperparameters:

• Llama: warmup_steps (discrete),
• DeBERTaV3: batch_size, num_epochs (both discrete),
• AlexNet and ConvNext: batch_size, epochs, lr_peak_epoch (all discrete),
• ResNet18: batch_size, epochs, lr_peak_epoch, use_blurpool (all discrete).

In our experiments, discrete hyperparameters did not pose an obstacle to applying our theory. For our proofs, discrete hyperparameters create multiple minima in the loss surface: one for each combination. We discuss this case in Appendix E: Proofs and Theorems. At a high level, you would analyze discrete hyperparameters by doing a separate Taylor expansion for each combination. While this makes the analysis more technically involved, it does not pose an issue in practice—as our empirical results show.

While these results are in the paper, perhaps they’re easy to miss. In the next revision, we’ll highlight them in a specific discussion of how our theory works for discrete hyperparameters.

Comparison with Bayesian optimization

Our work is not proposing an optimization algorithm; we provide tools to understand and quantify uncertainty in hyperparameter experiments. To develop these tools, we demonstrate that loss surfaces have quadratic structure with additive Gaussian noise near their optima (validated in Figures 3-5). This fundamental characterization could inform the design of priors for Bayesian optimization, but that's orthogonal to our contributions. Instead, we use this insight to develop statistical tools for analyzing language models and designing better experiments.

"theory does not seem to be general"

All the evidence contradicts this claim:

• The theory is validated on 5 architectures across 3 tasks and 2 modalities.
• We prove the theory under general regularity conditions in Appendix E.
• It works even when you use the same search space across different architectures.

On that last point, you state your main concern as: “maybe the noisy quadratic case only holds for the parameters they pick on the task they chose, not for other more general cases”. Appendix D presents an experiment showing precisely that this is not the case. In it, we use the same hyperparameter search space across 3 different architectures and find the noisy quadratic emerges for each one. Because we use the same search space for all three models, it cannot be particularly well-suited to each.

Substantial evidence and carefully designed experiments all demonstrate that our theory holds across many common scenarios.

Thank you for noting our "principled model" and clear presentation.

"experiments should be conducted to cover more diverse search spaces"

Our experiments already include substantial diversity:

• Language models: Llama 33M (1,024 runs) and DeBERTaV3 (1,024 runs),
• Vision models: ResNet18 (1,024 runs), AlexNet (512 runs), and ConvNext (512 runs),
• Tasks: Language model pretraining, ImageNet pretraining, and NLI finetuning,
• Datasets: SlimPajama-6B, MultiNLI, and ImageNet.

The main text covers scenarios from language and vision, including one of the most popular LLM architectures. In addition, Appendix D shows how results remain stable as you vary the architecture while keeping the hyperparameter search space constant. The theory consistently holds across all settings, with the asymptotic regime covering 18-58% of each search space.

While more experiments are always helpful, we trained over 3,000 models for this work and the current experiments demonstrate that our theory generalizes across architectures, tasks, and modalities.

"other common HPO methods… cannot benefit from it."

The goal of our work is not to benefit HPO methods, but to study the hyperparameter loss surface and develop tools for language modeling experiments. In development, researchers run many experiments where training the best model is not the goal. Instead, these experiments have goals like characterizing scaling laws or determining which data mix is better. HPO methods are not designed for these sorts of experiments.

Our theory and the tools we develop characterize the loss surface itself. Random search provides a powerful tool for analyzing the loss surface because of its clean mathematical properties; however, the structure it discovers exists regardless of the search algorithm used. This structure could benefit adaptive HPO methods by informing their design or configuration, but that is future work.

Thank you for recognizing the value in our work’s ability to "quantify the variation introduced by hyperparameter search." We appreciate your constructive feedback.

"scaling law" terminology

We refer to our result as a scaling law because it extrapolates how performance improves as you scale the iterations of random search, similarly to existing scaling laws which predict how performance improves as you scale data or compute. However, if you still object to this terminology, we’d be happy to consider alternatives.

Related work on hyperparameter optimization

Thank you for the suggestion. We'll add discussion of HPO methods to clarify how our tools for understanding the loss surface might inform their design. However, we emphasize our focus is on understanding loss surfaces, not optimizing them. Work on Bayesian optimization and other HPO methods is orthogonal to our own—typically they make no assumptions about the hyperparameter loss surface, while we reveal its mathematical structure.

As for work on the hyperparameter loss surface, our discovery of quadratic structure differs fundamentally from prior findings of unimodal/convex (Pushak & Hoos, 2022) or fractal (Sohl-Dickstein, 2024) surfaces, as already mentioned in the related work.

Thank you for your detailed feedback. We apologize for the clarity issues and will address them in the next revision.

"unclear regarding not only what is done, but even what the actual goal is"

Our goal is stated in the abstract: to "discover novel structure in the hyperparameter loss surface and propose a new theory describing it". We accomplish this, empirically showing that near the optimum the loss surface is approximately quadratic with additive Gaussian noise. With this insight, we develop a theory that yields better statistical tools for understanding hyperparameter loss surfaces and the language models they come from. These tools include:

• Confidence intervals for the best achievable loss,
• Estimates for the hyperparameters’ effective dimensionality,
• Extrapolation of random search’s convergence.

Note that we do not try to build a better hyperparameter optimization algorithm. Instead, we seek to discover structure and develop better tools for language modeling experiments.

"(one-dimensional) distribution... rather than the high-dimensional loss surface"

The noisy quadratic distribution, $\mathcal{Q}(\alpha, \beta, \gamma, \sigma)$, directly encodes the high-dimensional surface’s properties:

• $\gamma$: the intrinsic dimension (the effective number of hyperparameters),
• $\beta$: the surface’s maximum (the best achievable score),
• $\sigma$: variability around the surface (the noise due to random seeds),
• $\alpha$: the size of the asymptotic regime (how tight the search space is).

It is impossible to directly visualize the loss surface in high dimensions. That’s why our theory is so helpful: it makes it possible to study the loss surface by creating a direct link between it and the noisy quadratic distribution.

For example, we find $\gamma = 2$ across three different architectures when we use the same search space (Figure 7). Despite this search space having 7 nominal hyperparameters, only a 2 dimensional subspace matters near optimum. This dimension holds constant across all three architectures, suggesting that it’s a robust, intrinsic property of that hyperparameter loss surface.

“random search… does not seem a very realistic setting for h.p. tuning”

Our interest is not in hyperparameter tuning, but designing better tools for language modeling experiments. As you point out, random search provides a convenient theoretical framework for understanding the loss surface. We do not seek the most efficient hyperparameter tuning algorithm, and random search certainly isn’t it. Instead, it’s reasonably effective, easy to implement, and offers an important standard when comparing models across papers (where researchers need a single standard to enable fair comparisons). Since we are not interested in tuning hyperparameters, but rather comparing models, random search is the best starting point.

Of course, when practitioners tune these models for production, they’ll want to use the most efficient algorithms. In that case, our theory reveals properties of the hyperparameter loss surface that are independent of the search algorithm used. Adaptive methods can be designed to exploit these properties, once discovered.

"censored maximum spacing estimation"

This is a standard statistical method for fitting distributions (Cheng & Amin, 1983; Ranneby, 1984), and describing it would be out of scope for our paper. Since it is less widely known than maximum likelihood, we do in fact provide citations for it and the other methods we use when we describe them in Section 3: Experimental Setup (e.g., lines 161 or 169).

"novel and original work, but the significance is entirely unclear"

Our theory offers new statistical tools for studying language models. These include:

• Confidence intervals for the best achievable performance,
• An estimator for the effective number of hyperparameters,
• Extrapolation of hyperparameter tuning curves as a way to compare models.

Section 4.3 demonstrates this practical significance by showing dramatic improvements in confidence bands for tuning curves—a tool for comparing models while accounting for hyperparameter tuning difficulty. The current nonparametric bands become trivial after 8 iterations (Figure 6), while our parametric bands extrapolate past all 48 iterations used to construct them, enabling uncertainty quantification when interpreting experimental results.