CAMO: Convergence-Aware Multi-Fidelity Bayesian Optimization

We thank the reviewer for a thoughtful and insightful review from an AutoML perspective. We address the reviewers concerns systematically below.

Main Concern: Scope Clarification and Recent Literature. The reviewer is absolutely correct regrading the literature gap on HPO. We fully agree with the reviewer on the importance of HPO as an application of MFBO, especially in recent years.

We had not considered HPO and neural architecture search (NAS) as core applications of our method, but were instead focused on the vast array of applications related to expensive blackbox solvers in CFD, molecular dynamics, electronic structure calculations, mesoscopic simulations and so on. In such applications, continuous fidelity indices arise naturally as controllable, continuous parameters of the numerical solver (e.g., time step, maximal element/volume size, convergence tolerance, simulation-box dimension in MD). The fidelity parameter is intrinsic to the underlying physical or computational model. It directly controls the numerical approximation error of the forward model, and its effect on the objective function is smooth, deterministic, and model-based. In these applications there is an unambiguously defined set of design variables, in addition to which there is one or more unambiguously defined fidelity indices (with fixed ranges not dependent on the data).

In this setting, the LiFiDE formulation serves as an inductive bias on the GP surrogate, defining a convergence-aware, non-stationary covariance structure that encodes the expected monotone and asymptotic approach of low-fidelity simulations towards a high-fidelity target solution.

By contrast, HPO/NAS operate in a setting in which fidelity is stochastic and algorithm-dependent; e.g., epoch number or budget are algorithmic artefacts with no deterministic convergence law or trajectory -- and with high variance across runs. This makes 'fidelity' less straightforward to define, and certainly to control. The data generation is non-deterministic, e.g., observations for hyperparameters are subject to optimiser noise, random seeds and mini-batch options. The cross-fidelity structure is non-smooth and data-dependent, which requires empirical (ideally discrete-fidelity) MFBO methods such as BOHB, DyHPO, or Hyper-Tune.

Adapting CAMO to HPO/NAS would potentially require reformulating the underlying convergence model as a stochastic process or discrete-fidelity GP prior, which is conceptually orthogonal to the contributions of this paper. We therefore regard HPO-specific MFBO approaches (e.g., BOHB, DyHPO, FastBO, CQR) as complementary, operating under a distinct set of assumptions and data-generation mechanisms.

Given the importance of HPO, and the level of interest, we fully appreciate that it was an oversight not to at least mention it, and we thank the reviewer for pointing out this omission. In the revision, we will clearly distinguish between continuous-fidelity optimisation (our focus) and discrete fidelity HPO settings; and we will expand the related work section to include the recent HPO methods the reviewer mentioned (BOHB, FastBO, DPL, CQR, DyHPO, Hyper-Tune), acknowledging the broader MFBO landscape. This will better position our work within the entire field.

Q1: CAMO and BOCA acquisition function. We thank the reviewer for this insightful observation. CAMO does indeed leverage the BOCA acquisition function, primarily to isolate and evaluate the contribution of the LiFiDE-based surrogate kernel relative to standard stationary kernels. Theoretical guarantees for BOCA (Kandasamy et al., 2017) are agnostic to the exact kernel choice, and hence BOCA remains a valid acquisition strategy for CAMO. That said, we agree that jointly designing fidelity-aware acquisition rules that explicitly exploit the non-stationary structure induced by the LiFiDE could further improve information gain. This would require adapting the cost-normalised UCB criterion to account for directional convergence rates predicted by LiFiDE. This is a nontrivial but promising research direction. We will include acquisition function co-design as future work in the Conclusions.

Q2: CAMO-DKL-BOCA and CAMO-DKL-cfKG Variants. We apologise for this oversight. These variants should have been removed from the figures in the Appendix. They use deep learning to implicitly model the LiFiDE. Of the variants, CAMO-DKL-BOCA was on a par with the baseline, while CAMO-DKL-cfKG was inferior. Since we could not provide a theoretical analysis for the variants and wanted to focus on the main idea we had intended to leave out these results. They will be removed from the Appendix and we thank the reviewer for noting their appearance.

Q3: Scope clarification.

We have added clarification early on that our method targets continuous fidelity settings common in simulation-based optimisation, while acknowledging the parallel BO/MFBO HPO literature raised by the reviewer.

Q4: Related work. The most recent direct comparison methods for the continuous case are BOCA and cfKG. As the reviewer points out, these methods are from 2017/8. It appears that there is a greater focus on discrete fidelity problems, perhaps motivated by HPO/NAS as discussed above. Nevertheless, continuous-fidelity problems are of great interest in general design optimisation involving expensive blackbox solvers.

Q5: Presentation. We have tried to keep the analysis to a minimum in the main text, only stating the key results (as Lemmas or Theorems) with brief introductions and discussions on the relevance, applicability and implications. We will make this section even more compact by placing more details in the Appendix, while retaining essential information in the main text.

We hope that the more complete literature discussion, clearer scope and other changes will address the reviewer's concerns about positioning within the broader MFBO landscape. We welcome further discussion and suggestions from the reviewer.

We thank the reviewer for the detailed evaluation and very helpful suggestions. We address each of the questions systematically below:

Q1: Lines 78-80 - Basis for excessive exploration in costly regions. The reviewer is correct that acquisition functions such as improvement per unit cost should naturally avoid expensive regions. However, existing MFBO methods suffer from poor uncertainty calibration across fidelities. Standard product kernels treat all fidelities independently, leading to overconfident uncertainty estimates at low fidelities. This causes acquisition functions to underestimate the informativeness of low-fidelity queries, resulting in premature exploration of expensive high-fidelity regions.

Our claim is supported by the experimental observation that baseline methods spend only 40% of their budget on low-fidelity evaluations compared to CAMO's 70%, despite using cost-aware acquisition functions like BOCA.

Q2: Problem setting clarifications. Observation noise: Yes, our formulation includes observation noise. In Appendix A, we specify $y_i = f(x_i) + \epsilon$ where $\epsilon\sim \mathcal{N}(0,\sigma^2)$ is additive noise. We will update the main text to make this clear.

Highest fidelity observations: When $T$ is finite, the highest fidelity (indeed and indeed any fidelity) observations are accessible (we always have access to the cost function at any fidelity and design variable in the specified ranges). Our theoretical results hold for $T \le \infty$, but practically we must work with a finite (but arbitrarily large) $T$. The goal is optimising the highest available fidelity function. We will add a sentence in the first paragraph of section 3.1 to make this clear.

True function vs surrogate model: The true function does not need to follow the same GP as our surrogate model. Like all Bayesian optimisation methods, we assume the GP provides a reasonable approximation for optimisation purposes, not that it generates the true function.

Q3: Strength of linearity assumption. The reviewer raises an important point that we did not justify well enough in the original manuscript.

The linearity assumption $\phi(x,t,y)=\beta(x,t)y+u(x,t)$ is less restrictive than it appears. It captures many practical scenarios including mesh refinement, iterative solver convergence, and truncation-based approximations. Our extensive experiments across 11 diverse problems show the assumption provides useful inductive bias even when not perfectly satisfied. We will add a brief note to this effect at the beginning of section 4.2

Moreover, this assumption enables the first rigorous theoretical analysis of FiDE-based methods, addressing the limitation that the reviewer correctly identified - prior FiDE work lacks convergence guarantees.

Q4: Advantage without improved regret bounds. While we do not improve worst-case regret bounds, we provide some key advantages. First, we give the first rigorous convergence analysis for FiDE-based surrogate models, filling a critical theoretical gap. Second, we establish novel connections between MFBO and convex optimisation theory, opening new research directions. The consistent 2-4x empirical improvements demonstrate practical value even without improved worst-case bounds.

We would like to point out that a more explicit regret bound is not possible in this case (only the general bound from BOCA), which is why provided a qualitative analysis. The issue lies in finding a closed-form expression for the fidelity gap region, which is possible for the SE but not for our integral kernel.

Q5: Performance patterns where CAMO is overtaken. The reviewer has made an astute observation about the trade-off between early low-fidelity exploration and eventual high-fidelity exploitation. In the cases mentioned, CAMO's strategy of extensive low-fidelity exploration provides faster initial progress but can delay final high-fidelity queries needed for ultimate convergence.

This represents a fundamental trade-off in multi-fidelity optimisation. CAMO is designed for scenarios where computational budget is the primary constraint and early progress is valuable. For unlimited budget scenarios, more aggressive high-fidelity exploration might be preferable.

We will note this in the revised version when presenting these results.

Q6: Line 161 typo correction. We apologise for this error and thank the reviewer for pointing it out.

Q7: Citation errors [??]. Again, we apologise for this error and have fixed the broken citations.

Limitations addition. The reviewer is absolutely correct that we should have mentioned the limitation about not improving upon the regret bound. We will add this to an expanded discussion on limitations in the conclusions.

Clarity and self-containment. We acknowledge the clarity concerns. In response to the specific questions we have added clarifications on the issue of noise and access to fidelities. Due to a lack of space we were unable to include some of the main theorems related to general FiDEs. Since our subsequent focus was the LiFIDE, we describe the main theoretical results, provided intuitive explanations and discuss their significance. A greater focus is placed on the theoretical results for the LiFiDE case. We will further edit the theoretical analysis section to improve clarity as suggested by the reviewer.

We thank the reviewer for raising the important point regarding the interpretation of convergence in the fidelity variable. In the current formulation of the LiFiDE, once the surrogate construction and low-/high-fidelity data are fixed, the evolution of the surrogate $y(\mathbf{x},t)$ is described by a deterministic differential inclusion in the fidelity coordinate $t$. As such, the convergence is to be interpreted in a pathwise (trajectory-level) sense, i.e., for each fixed input $\mathbf{x}$, the solution trajectory $t \mapsto y(\mathbf{x},t)$ converges deterministically to the high-fidelity target $y_\infty(\mathbf{x})$ with an explicit exponential rate.

While the LiFiDE can naturally be extended to a stochastic setting where the fidelity flow is driven by data noise or randomized surrogate updates, in which case one would consider convergence in various stochastic process topologies (e.g., almost sure convergence, convergence in probability, or weak convergence in $C([0,T])$ endowed with the Skorokhod topology), such probabilistic considerations are not invoked in the present analysis. Here, the dynamics are entirely deterministic once the training data are fixed, and the established result should thus be interpreted as strong pathwise convergence of the solution trajectories in the fidelity variable. We will add a note to this effect and thank the reviewer for raising this question.

We thank you for the positive assessment and insightful questions. Below we address the main concerns.

Weaknesses: GP-LinearFIDE justification vs other modelling approaches. The reviewer raises an excellent point about directly demonstrating the benefit of a GP-LiFIDE surrogate. We will provided a synthetic experiment comparing prediction accuracy in the revision. We created a true convergent multi-fidelity function $f(x,t) = \sin(2\pi x) \cdot (1 - \exp(-3t))$, in which values converge to $\sin(2\pi x)$ as $t$ increases, and compared our LiFiDE kernel against the standard product kernels (SE × SE) and autoregressive models. The experimental results show that the GP-LiFIDE method achieves more accurate predictions. For example (fidelity range from 0 to 3 and 100 randomly generated training points)

At fidelity= 1.0 RMSE: CAMO: 0.0652 GP-SExSE: 0.00647

At fidelity= 2.0: RMSE: CAMO: 0.0380 GP-SExSE: 0.0384

At fidelity=3: RMSE: CAMO: 0.0361 GP-SExSE 0.0744

The LiFiDE kernel structurally encodes the assumption that low-fidelity evaluations become increasingly informative about high-fidelity values as the fidelity increases. We believe that this inductive bias improves both prediction accuracy and uncertainty quantification compared to unstructured approaches, as demonstrated by the results and an analysis of the asymptotic behaviour of the kernel (although this is heuristic since a full analysis is prevented by the complex nature of the kernel).

Q1: "Convergence-aware" terminology and synthetic functions. We thank the reviewer for raising this: to avoid confusion we should make clear what we mean by "convergence-aware". The term "convergence-aware" refers to our modelling approach being aware of how fidelity affects function values, not that the target function itself converges.

The synthetic functions mentioned by the reviewer are designed as weighted combinations: $f(x,t) = (1-w(t))f_{low}(x) + w(t)f_{high}(x)$ where $w(t) = \ln(9t+1)$. As $t$ increases, the function transitions from low-fidelity to high-fidelity behaviour. This represents the common scenario in which higher fidelity provides increasingly accurate approximations of some true underlying function.

The "convergence" we model is this systematic relationship between fidelity level and function accuracy, which is present even when the target function doesn't converge to a limit. Our kernel captures how function values at different fidelities relate to each other through this systematic progression.

We will add a brief note regarding the terminology "convergence-aware" to the beginning of section 5.

Q2: Performance dependency on cost function and theoretical explanation. The relationship between CAMO performance and cost function can be explained through information-theoretic arguments. When costs increase super-linearly with fidelity, the value of accurately modelling low-fidelity behaviour becomes critical for efficient budget allocation.

Theoretical insight: Our convergence-aware kernel provides better uncertainty estimates at low fidelities, leading to more informed decisions about when to query expensive high-fidelity evaluations. When costs are logarithmic, this advantage diminishes because all fidelities are relatively cheap. When costs are exponential, the advantage becomes pronounced because poor low-fidelity modelling leads to inefficient or even wasteful high-fidelity queries.

Sample efficiency connection: CAMO inherits the overall BOCA guarantee, which is kernel dependent. A standard SE kernel implies all fidelities are equally smooth and informative, which is not realistic: low-fidelity evaluations are often noisy or unstable. In contrast, the LiFiDE kernel models a convergent fidelity process in that evaluations become more stable and informative as the fidelity increases. The LiFiDE kernel leads to looser regret guarantees (although this is not straightforward to model) compared to the SE kernel but the convergence-aware structure leads to better empirical performance - it does not concentrate as sharply as the highest fidelity is approached, discarding potentially useful results at low fidelity. We discuss this after presenting Theorem 3.

Empirical validation: Our experiments show minimal advantage with logarithmic costs but substantial gains with exponential costs, confirming this theoretical relationship.

Additional clarification on impact. The significance of this work extends beyond the immediate MFBO improvements. We establish the first rigorous connection between MF surrogate modelling and non-smooth optimisation theory, potentially opening new theoretical directions. The closed-form kernel derivation enables practical implementation while maintaining theoretical guarantees, making this a valuable contribution to both the optimisation and Gaussian process communities.

The consistent 2-4x improvements across diverse applications demonstrate that convergence-aware modelling addresses a fundamental limitation in existing multi-fidelity methods, not just a narrow technical improvement.

We thank the reviewer for the positive comments and very helpful suggestions. We address the specific concerns below.

Q1: We appreciate the comment on worklflow clarity and will add a brief description to the beginning of section 5.

The CAMO algorithm follows standard MFBO practice with one key modification.

1. We initialize a Gaussian Process using our LiFiDE kernel (Equation 10) rather than a standard product kernel. This kernel explicitly models how function values converge as fidelity increases.

2. At each iteration, we compute the BOCA acquisition function to identify the most promising input $x$, exactly as in existing methods.

3. We then select the fidelity level $t$ using BOCA's cost-aware filtering mechanism, which chooses the cheapest informative fidelity.

4. We query the function at the selected point $(x,t)$, measure the cost, and update our GP model with the new observation.

5. We repeat until the computational budget is exhausted.

The key difference from existing MFBO methods is step 1 where we use our convergence-aware kernel. Everything else follows standard practice, making CAMO a drop-in replacement for existing approaches.

Q2: Cost and compute time relationship. The cost is directly measured by computational time in our experiments. For the real-world applications, we measure the actual wall-clock time required to run finite-element simulations at different mesh resolutions. The cost scales according to the inverse cubic of the maximal element size, which is the standard computational complexity for FEM problems. For the synthetic benchmarks, we use cost equals $10\cdot t$ as a simple proxy, representing increasing computational demands. Figure 5 shows the actual measured times on our hardware setup.

We agree that it is helpful to highlight the above in the manuscript so we will add brief notes to this effect

Q3: Cost computation in actual cases. For the mechanical plate vibration problem, we vary the maximum mesh element size from 0.2 to 1.2 meters, and the computational cost scales with the inverse cube of this mesh size. Similarly, for the thermal conductor design, mesh sizes range from 0.1 to 2 meters with the same cubic scaling. This cubic scaling is well-established in finite-element literature and represents the fundamental trade-off between accuracy and computational cost in these numerical simulations.

Q4: Linear approximation discussion. The linear differential equation approximation works well when convergence toward the highest fidelity is monotonic, which occurs in most common scenarios for blackbox solvers (our focus). It may be less accurate for highly nonlinear dynamics or non-monotonic convergence patterns. Notably, in hyperparameter optimisation (HPO) and neural architecture search (NAS), where MFBO has also been employed in recent years. For example, when tuning a network, 'fidelity' indices might be epoch, number of dense/convolutional layers, filter size or hidden space dimension in RNNs. Here, one would need to take care to restrict the fidelity index in order to avoid overfitting.

Our experiments across 11 diverse problems show that even when the true dynamics deviate from linearity, the linear approximation provides a beneficial inductive bias that improves performance. The consistent improvements suggest the method is robust to model misspecification.

Nevertheless, the reviewer raises an important point that requires some discussion. We will added such as discussion on the linear FiDE assumption (strengths and potential limitations) to the beginning of section 4.2.

Q5: Compute budget evaluation. We provide extensive budget-aware analysis showing CAMO achieves 2 to 4 times better regret per unit of computational cost compared to baselines. CAMO reaches target accuracy levels approximately 3 times faster than competing methods. Critically, CAMO intelligently allocates 70% of its computational budget to informative low-fidelity evaluations, compared to only 40% for baseline methods, leading to more efficient exploration.

Q6: Expanded limitations. Our method has several limitations that are worth noting. Computationally, the kernel optimisation adds roughly 20% overhead compared to standard GP methods, and the approach requires sufficient low-fidelity data to properly model convergence behaviour. Methodologically, the linear differential equation assumption may not capture all convergence patterns, and performance depends on the existence of a good cost model. Practically, users need domain knowledge to set the beta parameter in our kernel, and the method is most effective when computational cost increases super-linearly with fidelity.

The reviewer is right to raise the brevity of the discussion on limitations. We will provide a more comprehensive list and discussion of the limitations in the Conclusions section.

Significance response. We believe the significance deserves higher recognition given that this is the first convergence-aware MFBO framework with rigorous theoretical foundations. The method consistently delivers 2 to 4 fold improvements across diverse applications and establishes novel connections between MFBO and differential equations theory. Most importantly, it addresses a fundamental limitation in existing multi-fidelity methods by explicitly modelling how surrogate functions evolve with fidelity, opening new research directions in convergence-aware optimisation.