Multi-Fidelity Active Learning with GFlowNets

Dear Reviewer 5Rpa, we appreciate your review of our manuscript. We are glad to read that you have appreciated the novelty of our work, the breadth of the evaluation, the clarity of the manuscript and our efforts regarding the reproducibility of our work. We are happy to also address your comments and questions below.

One potential weakness of this paper is that the evaluation is limited to molecular discovery tasks, and it is unclear how well the proposed algorithm would perform on other types of scientific and engineering applications.

We would like to note that besides the tasks on molecular discovery, we have included experiments with DNA aptamers and antimicrobial peptides, whose data representation and target functions are substantially different than the small molecules tasks. Furthermore, we have also included results on two well-studied synthetic functions (Appendix C.4).

[T]he authors do not provide a detailed discussion of the limitations of their approach or potential future directions for research.

Section 5 Conclusions, Limitations and Future Work includes a discussion of the future directions for research and the limitations of our work.

Questions

What is the main challenge in scientific discovery that current machine learning methods cannot efficiently tackle?

This question is addressed explicitly in the second paragraph of the introduction (”[…] Such scenarios present serious challenges even for the most advanced current machine learning methods”) as well as in the the subsequent paragraphs.

How does the proposed algorithm with GFlowNets address the challenge of querying a high fidelity, black-box objective function?

To address the challenge of querying a high fidelity, black-box function efficiently, in this work we have proposed a multi-fidelity active learning algorithm which leverages the availability of additional black-box functions with lower fidelity but much lower costs. This is the central element of our work.

What are the advantages of multi-fidelity active learning with GFlowNets compared to RL-based alternatives?

As discussed in the introduction and reflected in the results of our experiments, RL-based approaches are effective at optimisation but lack diversity in the batch of discovered candidates. The multi-fidelity active learning algorithm we propose learns to sample from the acquisition function instead of optimising it, by means of the use of GFlowNets. This approach achieves both diversity and high scores in the batch of candidates.

Dear Reviewer TJ42, thank you for the review of our submission. We are happy to read that you positively valued the results presented in our manuscript, taking into account both metrics, the average scores and diversity. In your review, you also mention a couple of weaknesses and a question that we are happy to address in what follows.

The baselines are quite simple, just compare between multi-fidelity active learning and single-fidelity active learning.

First, we would like to note that besides the (GFlowNet-based) single-fidelity active learning baseline, we also include a multi-fidelity algorithm which optimises the BO acquisition function via the widely used reinforcement learning method PPO. This method can therefore be regarded as a multi-fidelity Bayesian optimisation method that optimises the acquisition function via reinforcement learning. This a strong baseline that is actually effective at discovering samples with high value of the target black-box function, as is reflected by the results of our experiments. However, we also observe that it exhibits low diversity in the discovered samples. The algorithm we propose, which uses a multi-fidelity GFlowNet, addresses this limitation, as you acknowledge in your review.

The remaining baselines are designed to help us gain understanding about the novel aspects of our proposed method. Specifically, the contribution of multi-fidelity versus single-fidelity active learning and the advantages of a GFlowNet sampler that selects the fidelity alongside the sample. Moreover, we include a baseline with random samples ranked by the acquisition function, which is known to be a strong baseline in Bayesian optimisation.

The baselines are described in Section 4.2

The author could consider comparing it with other typical query synthesis active learning methods like [r1].

Unfortunately, it would not be straightforward to adapt the methods in [r1] for our tasks, since the active learning algorithm in [r1] does not consider multiple oracles and the task at hand is classification of sentences, while our tasks are regression problems with substantially different data. Nonetheless, we appreciate the reference as we may be able to derive a suitable baseline from it in future work.

The performance of the proposed method on tasks and domains beyond those covered in this study remains uncertain due to the limitations of the tested benchmarking datasets.

There will always be uncertainty about the performance of any method on tasks and domains on which it has not been tested. Nonetheless, in the case of our experimental setup, we would like to argue that we have offered results in four tasks, using three distinct scientific discovery domains (DNA, antimicrobial peptides and small molecules), plus two additional sets of experiments on well-studied synthetic functions (Appendix C.4). Furthermore, we have performed robustness analyses of the impact of the oracle costs (Appendix E.2), the acquisition size (Appendix E.3) and the size of the final batch (Appendix E.4). In all the experiments, we have found highly consistent results, where the proposed algorithm MF-GFN is able to find high-scoring candidates with less budget than the baselines, while keeping high diversity, unlike the multi-fidelity PPO baseline.

Would the data samples selected be out-of-distribution samples? Since the evaluation considers mean top-K score and top-K diversity, it is still possible to select out-of-distributions samples.

We are not sure of having understood this question correctly, so we would appreciate a clarification. In particular, do you mean out-of-distribution with respect to what distribution? In general, the search space in our tasks is vast, therefore the chances are that the selected samples will be out of distribution, with respect to the initial data data set on which the surrogate is trained.

Dear Reviewer Cu6t, thank you for reviewing our submission. The review mentions a number of concerns that we are happy to address below.

The novelty of this paper is very limited in terms of the model development.

The review does not provide many details explaining the assessment of the novelty of our submission, so we will here provide a general overview of the novel contributions of our paper. Our paper is, to our knowledge, the first work to propose a multi-fidelity active learning algorithm using GFlowNets. Single-fidelity active learning with GFlowNets has been explored before at least by Bengio et al. (2021), Jain et al. (2022) and Jain et al. (2023a). The multi-fidelity aspect of the algorithm is far from trivial as it implied, among other things, proposing a novel extension of GFlowNets to sample the fidelity alongside the candidate; training a multi-fidelity Bayesian surrogate with multi-fidelity data; incorporate a multi-fidelity acquisition function; regarding the evaluation, assessing the contribution of the multi-fidelity aspect on average scores, diversity, the impact of costs, etc.

Moreover, our paper is, also to the best of our knowledge, the first work presenting multi-fidelity active learning results for DNA aptamers, antimicrobial peptides and molecular design. As we discuss in our paper, the reason is that these problems present a number of challenges for existing methods, such as traditional Bayesian optimisation and reinforcement learning.

In the review, you state that “the experiments […] are primarily simulation-based.” We would like to recall that the experiments with small molecules (Section 4.3.3) are not simulations. In these tasks, we use the semi-empirical quantum chemistry method XTB with various levels of geometry optimisation as oracles. These oracles are used in practically relevant problems by the scientific community. Furthermore, the costs of the oracles are set such that they are proportional to their computational running time, as it would be done in practice. The results in these two tasks (Figure 3) demonstrate the effectiveness of MF-GFN at finding high-scoring, diverse candidates, outperforming the studied baselines, including a multi-fidelity Bayesian optimisation method using PPO as the optimiser of the acquisition function (MF-PPO). These results are consistent with the rest of the results in the paper (DNA, AMP and synthetic functions).

The plots in the experiments are densely packed

While the plots could admittedly be less packed, removing information would likely harm the completeness in the presentation of our results. Do you have a suggestion for an alternative form of presentation?

[The paper] lacks a proper description of the multi-fidelity problem

The multi-fidelity problem is described in Section 3.2 Multi-fidelity Active Learning. The desccription builds upon the preceding section, which describes the single-fidelity scenario. The multi-fidelity problem and the algorithm are also illustrated in Figure 1, and detailed formally in Algorithm 1. What details are missing, in your opinion, to make the description more proper?

GFlowNet and Active Learning in the method section are not closely related, and it would be better to refer to GFlowNet-AL as a preliminary section.

Section 3.1 Background introduces first the necessary background on GFlownets and then describes the (single-fidelity) active learning problem, explaining the specific case where a GFlowNet is used as a sampler, as done by Jain et al. (2022) with GFlowNet-AL. We have added a sentence referring explicitly to GFlowNet-AL and the corresponding citation. We have also included the name GFlowNet-AL in the description of the single-fidelity baseline, for further clarification. Do these changes address your comment?

The tasks used in the experiments are not based on real-world scenarios, which raises questions about the problem's practical importance

We refer to our discussion above, answering your comment about novelty and the experiments being “simulation-based”. Furthermore, we would like to note that the kind of experiments included in this paper are similar to those found in the literature (Angermueller et al., 2020, Zhang et al., 2022). Please let us know if this aspect needs further clarification.

References

• Bengio et al. Flow network based generative models for non-iterative diverse candidate generation. NeurIPS, 2021
• Jain et al. Biological sequence design with GFlowNet. ICML, 2022
• Jain et al. Multi-objective GFlowNets. ICML, 2023a
• Angermueller et al. Model-based reinforcement learning for biological sequence design. ICLR 2020.
• Zhang et al. Unifying Likelihood-free Inference with Black-box Optimization

Dear Reviewer EtrP,

Thank you for your thorough review. We appreciate that you have found our proposed algorithm interesting and sensible, and the overall article well-written and comprehensive. We understand that you disagree with several aspects of our paper, therefore in what follows we will gladly address your comments and concerns one by one.

Related work

A major concern in your review seems to be that our paper did not “contextualized its contributions well”. We would like to first note that out of the 10 citations that you provide as examples of the well-researched topics that our paper touches upon, 7 of them are already cited in our manuscript. Of the remaining 3, citation [3] is a review of multi-fidelity models but it doesn’t cover deep learning, Bayesian optimisation and active learning; citation [10] is a Bayesian optimisation method but it does not seem to tackle the diversity problem; citation [9], in contrast, seems very relevant, as it is a Bayesian optimisation method for finding diverse solutions. Accordingly, we have added a sentence at the end of the second paragraph of Section 2 acknowledging this work.

Bayesian optimisation

BO was dismissed too hastily by claiming that BO only cares about finding a single optimum. First, this is not true: see [9,10].

Our paper states that “Bayesian optimisation and reinforcement learning are designed to find the optimum of the target function” (Section 1). In Section 2, we state that “[t]he main difference between BO and the problem we tackle in this paper is that we are interested in finding multiple, diverse samples with high value of $f$ and not only the optimum”. In our opinion, this is not dismissing Bayesian optimisation but describing its goal and explaining why it is not directly suited for the problems we tackle in our paper. Rather than an opinion or a controversial claim, this is simply the problem statement found in the Bayesian optimisation literature (Frazier, 2018; Garnett, 2023). As a matter of fact, this aspect of BO is mentioned as the motivation in [9], which you cite: “the fact that BO traditionally seeks a single best optimizer may be a significant limitation”. From this standpoint, [9] (AISTATS, 2023) proposes a variant of Bayesian optimisation to discover diverse solutions.

Our paper could similarly be regarded as a BO variant: Not only do we not dismiss Bayesian optimisation, but rather our method strongly builds upon it. Specifically, our algorithm relies on a Bayesian surrogate (DKL) to model the data and a BO acquisition function (multi-fidelity max-value entropy search) for the exploration of the search space—the two key ingredients of BO. Note that the manuscript makes this connection explicitly:

Optionally, we can instead train a probabilistic surrogate $p(f | \mathcal{D})$ and use as reward the output of an acquisition function $\alpha(x, p(f | \mathcal{D}))$ that considers the epistemic uncertainty of the surrogate model, as typically done in Bayesian optimisation (Jain et al., 2022).

A crucial difference between our algorithm and traditional BO is that instead of optimising the acquisition function, GFlowNets learn to sample from it, which results in enhanced diversity (see below). We have added a sentence in the Active Learning section of 3.1 to further clarify this idea.

You mention that BO may also find diverse solutions “incidentally”. While we agree that this may be the case, our goal is to find diverse solutions rather systematically, since diversity is an important objective in certain scientific discovery applications, as it is also discussed in [9].

the method proposed in this paper also appears to have no built-in diversity objective: it just relies on GFlowNets incidentally generating a diverse set of points.

We respectfully disagree with this view. GFlowNets do have built-in diversity in their objective and diversity is at the core of the method, as is discussed in most “GFlowNets papers”, including the article where the method was introduced (Bengio et al., 2021). GFlowNets achieve diversity by learning to sample proportionally to the reward distribution. Therefore, diversity is not incidental, but systematic. The importance of diversity and its connection with GFlowNets is discussed throughout our paper and in particular the technical mechanisms are in described in Section 3.1. We also note that the ability of GFlowNets to systematically sample diverse solutions has been established in a number of prior works including but not limited to the following: Malkin et al., 2022, Zhang et al., 2023, Jain et al., 2023a,b.