Simulating, Fast and Slow: Learning Policies for Black-Box Optimization

If I understood correctly, I see two ways in which the method could be beneficial:

1. either the total number of simulator calls over the 2 phases ((1) training the policy network and (2) optimizing the function) is smaller than the number of simulator calls in other methods.
2. or, as training the policy network can be amortized over multiple input distributions q_i, if one needs to solve multiple such optimization problems using different input distributions, the cost of training ends up being negligible compared to the gains at inference (compared to other methods) that the training policy network provides.

However, both of these benefits are neither clearly described nor demonstrated in the main body:

• regarding point 1: when reporting the metrics, it is not mentioned whether training is accounted for
• regarding point 2: the use of multiple input distributions is mentioned, but once (l 326), and without stating out clearly why it is an important use case for the method at hand. If point 2 ends up being the main advantage of the method, I would expect the introduction to provide more motivation as of why parameterized input distributions matter in practice. Moreover, I could not find a mention of parameterized distributions in real-world experiments.

In my opinion, the paper needs to demonstrate clearly that the additional training step is justified. Due to the reasons described above, this demonstration remains limited, and thus I will recommend rejection until these issues are addressed. I may have missed additional benefits, in which case I will change my recommendation.

Some important training and evaluation details are missing from the main body, see questions.

• The main weakness of the paper seems to be highlighted in the appendix, where algorithm 1 shows that the policy is required to be trained on simulated data prior to the actual black-box optimization in algorithm 2. However, this is not made clear in the main paper. This means that the results in the paper do not include the large number of simulations required to train a policy and seems to defeat the purpose of reducing the cost of simulation. Specifically, to train the policy, the simulator is queried in a triple nested for-loop, which implies the requirement of many queries before performing the black-box optimization.
• The section “Policy-Based Black-Box Optimization” does not highlight the technical approach of the paper. It continuously refers to the appendix instead of introducing the technical content. It should be the case that this is the most important section of the paper and therefore information regarding training and policy implementation (when it is key to the algorithm) needs to be made explicit in this section.
• The uncertainty feature is introduced and then not mentioned again. Is it necessary? Especially since the appendix mentions using an ensemble of 100 models. Therefore there is the potential for large overhead.
• It is not clear from the paper what the definition of a “parameterized input distribution” is, and why it might even be needed. Therefore, it is hard to interpret the difference between columns (a) and (c) in Figure 3.
• The results in Figure 3 do not match Shirobokov et al. (2020) which is a concern. For example at 100 function calls in Figure 2a of the prior paper, no algorithm has reached -0.8, but all algorithms have reached -0.8 in this paper except numerical differentiation. The difference can also be seen for the 10D Rosenbrock and the nonlinear submanifold hump problem. This needs to be addressed in the paper as to why there is this discrepancy. There is also a similar discrepancy for the Physics experiment where L-GSO gets to an objective value of about 70 after 20 calls in Shirobokov et al. (2020).

Minor:

• Is “learning active learning” an accepted phrase in the literature? Maybe meta active learning as in Ravi and Larochelle (2018)?

The fact that a single "simulator call" means calling $f_\text{sim}$ thousands of times should be prominently stated early on, and not hidden in the appendix. The figures give a false impression that the algorithms used here can work with dozens of $f_\text{sim}$ evaluations, while in fact they appear to require thousands, similar to SBI methods that do not rely on local gradient estimates. The total number of simulation calls required to train the policy network should also be reported. If we simply trained a network to map from observations to parameters using a least squares loss and this number of samples from the parameter prior, then evaluated the loss, would it actually be worse than the reported results for inference (while using zero simulator calls at inference)?

More generally because the simulation costs are comparable, this method should be directly compared to SBI methods relying on neural density estimations, for which there are mature implementations (Tejero-Cantero 2020) and benchmarks (Lueckmann 2021). Can the proposed method solve these problems, and how sample efficient is it compare to SNPE, SNLE etc.? Because the software is readily available, this should be straightforward to verify. Total simulation counts, including those required to train the policy network, should be included. It is also worth discussing the fact that an amortized, trained SBI method can carry out inference on the full posterior without any additional simulator calls. In principle there must be some important scenarios where using local information is more efficient, but this does not seem to have been demonstrated here.

The strength of using local gradients in $\psi$ space over other approaches such as neural density estimation or Bayesian optimization is that this should scale to higher-dimensional parameter spaces. But the highest we see here is 42, well within the reach of existing SBI methods. Furthermore, we never get to see the joint distribution of true and estimated parameters, nor do we get to compare simulation results from true vs. estimated parameters. Therefore, it's hard to say whether the tasks (especially the real world problems) have actually been solved, or whether the current method is simply failing to solve it in a more sample-efficient way than baselines.

It's not so clear why the latent variable $z$ is included, when only the mean output of each surrogate is used to estimate sigma. Do the distributions from individual surrogates show variation, and does this capture uncertainty? Or does including $z$ somehow regularize the surrogates? Do they actually use $z$ or discard it after training?

The section on metrics isn't very clear. Does "objective function" refer to the reward function for RL or the loss for local training? It's not clear in either case how closely either of these correspond to accurate estimation of $\psi$, which is the real aim here. Why not show accuracy of $\psi$ vs. total simulation count? While it's generally true that lowering the objective function means we've gotten closer to the distribution of observables under the ground truth parameters, exactly how close is difficult to determine from the reported metrics. Also, it's not clear to me whether AMO/ANC are calculated over the whole training process, or for a new episode when the policy has already been learned.

Overall the precise sequence of steps is not clearly stated in the main text, and inclusion of a simplified algorithm in the main text would be welcome. Giving a concrete example of what the surrogate training loss might be early on in the manuscript could help readability.

Some relevant citations are missing. Papamakarios and Murray should be cited for SBI. Nonnenmacher (2021) should be cited for emulator-based gradient estimation. Lueckmann (2019) should be cited for simulation-based inference with active learning and ensemble predictions.

References

Lueckmann, Jan-Matthis, et al. "Likelihood-free inference with emulator networks." Symposium on Advances in Approximate Bayesian Inference. PMLR, 2019.

Lueckmann JM, Boelts J, Greenberg D, Goncalves P, Macke J. Benchmarking simulation-based inference. InInternational conference on artificial intelligence and statistics 2021 Mar 18 (pp. 343-351). PMLR.

Nonnenmacher M, Greenberg DS. Deep emulators for differentiation, forecasting, and parametrization in Earth science simulators. Journal of Advances in Modeling Earth Systems. 2021 Jun;13(7):e2021MS002554.

Tejero-Cantero A, Boelts J, Deistler M, Lueckmann JM, Durkan C, Gonçalves PJ, Greenberg DS, Macke JH. SBI--A toolkit for simulation-based inference. arXiv preprint arXiv:2007.09114. 2020 Jul 17.

Papamakarios G, Murray I. Fast ε-free inference of simulation models with bayesian conditional density estimation. Advances in neural information processing systems. 2016;29.