SynFlowNet: Design of Diverse and Novel Molecules with Synthesis Constraints

Issue 5: Restricted scalability to a large building block set.

I suggest including structural diversity metrics for generated samples, such as the number of unique Bemis-Murcko scaffolds or the average pairwise Tanimoto distance

The purpose of Figure 7 ([edit]: now Figure 6) is two-fold: to show how SynFlowNet scales with increasing number of building blocks and to show the effect the utilisation of Morgan fingerprints for action embeddings. We show that the effect is positive and that the model with fingerprints exploits more within the same clusters, meaning that it is able to leverage the known structure of the chemical space embedded by Morgan fingerprints. Validating that the number of unique utilised building blocks increases with larger action space is an important check that the model does not collapse and has the capacity to scale and explore increasingly larger action spaces. We also clarify that the results in Figure 7 ([edit]: now Figure 6) are for runs following the same number of training steps, therefore the decrease in reward for larger action spaces is expected. In fact, we managed to reach the same rewards as for the smaller action spaces when increasing the computational budget.

Following the reviewer's suggestion, we include the average pairwise Tanimoto dissimilarity as a structural diversity metric in Figure A.2, and see that the model consistently samples diverse molecules.

Question 1: I was wondering if dot product operation between Morgan fingerprint (binary vector) and state embedding can satisfy the GFlowNet Objective

That is an interesting question to consider. Indeed, classical binary fingerprints could induce representational limitations on the model. In the example that the reviewer is mentioning, involving fingerprints [1,0,0,0], [0,1,0,0] and [1,1,0,0], if the first two features are deemed beneficial according to the current state, the third molecule, represented by [1,1,0,0] would always be preferred over the first two and its sampling likelihood would be a function of the other two.

We investigated a bit further how frequent this case might be. In general, it is a combinatorial problem to verify whether any of the building blocks can be obtained by simply summing (logical OR) the binary representation of some combination of other building blocks, and the computation time grows exponentially with the building block size and the group size (pairs, triples, etc.). We ran this test for the set of 10,000 building blocks which used in the paper, and found that using 1024-dimensional Morgan Fingerprints, there was not any occurrence where a building block's representation could be obtained by summing those of a pair of other building blocks. For these reasons, we believe that this potential representational limitation does not have a practical effect on our models.

Questions 2-4:

Thank you for pointing out the typos and confusing figure title, as well as the misspecified max_nodes. These have been corrected in the updated manuscript.

References

[1] Tang, Shidi et al., "Vina-GPU 2.1: towards further optimizing docking speed and precision of AutoDock Vina and its derivatives", 2023: https://www.biorxiv.org/content/10.1101/2023.11.04.565429v1

We thank the reviewer for providing valuable feedback and insights. We are addressing their questions one by one in our response below.

Issue 1: Lack of novelty / appropriate citations for the major contribution

the manuscript does not contain appropriate citations for this major contribution [action embeddings]

We provided references to the existing methods that use similar action space embeddings in Section 3.3.

Issue 2: Lack of experimental details.

what is the rewarding strategy used for multi-objective optimization (FragGFN SA, SynFlowNet SA, SynFlowNet QED)?

The strategy that we used what the multiplication of rewards, which has now been noted in the manuscipt, in Section A.3.

In Section 4.5, I couldn't find the experimental details. I suggest to include experimental details and mention it like Section 4.6.

We have added a detailed description of scaling experiments in Section A.5, including what reward function was used, clarification with regards to average time, and building blocks set used.

Issue 3: Fairness of evaluation metric

it is possible that the improvement in novelty is due to the use of the Enamine building blocks rather than the effectiveness of well-structured generative model architecture

We thank the reviewer for the suggestion to train SynFlowNet with ChEMBL-derived building blocks. We have conducted this experiment by decomposing randomly-sampled ChEMBL molecules into building blocks using AiZynthFinder retrosynthesis. The results (presented in Section B.1, Table A.2 and Figure A.5) show that SynFlowNet designs preserved their lower similarity to ChEMBL, while still performing well on reward and diversity. This suggests that although relying on a ChEMBL vocabulary, the model has the capability to extrapolate to new regions of the chemical space, as opposed to models relying on likelihood-based priors for optimisation like REINVENT. Indeed, the scope of the original experiment was to show that REINVENT, although it generates highly synthesizable and high-reward molecules, it will not manage to extrapolate to novel chemical space, and that SynFlowNet represents a more unbiased solution on the sampling problem.

Issue 4: Limited comparative study

I have concerns about the appropriateness of evaluating and comparing the performance of discovering novel molecules based on proxy models trained on a restricted and known chemical space.

Following the reviewer's comment, we have added comparisons between models trained to optimise for a KRAS binding affinity reward, calculated using a GPU-accelerated version of AutoDock Vina [1]. Such an oracle enables exploration beyond the restricted and known chemical space associated with other proxy models. Results are presented in Table A.3 in Appendix and support the claims that SynFlowNet is generating molecules with the best reward/diversity/novelty trade-off.

Thank you for immediate updates for my comments.

Now I think the paper is sufficient to be accepted, so I'll increase the score from 5 to 8.

Regardless of the score, I also have an additional question, which would be the last. A lot of time has already passed in the review period, and I've been able to find enough meaning in all experiments. This is just a question from my curiosity, so you do not have to make a rebuttal or response to this question by spending your valuable time. If you have time, please feel free to answer them.

In Section 4.6, the authors construct a target-specific building block set using fragment screening results from the COVID Moonshot project. However, in my best knowledge, the Moonshot project includes large-scale fragment screening results (common fragments), so it is unclear whether the selected building blocks are truly target-specific. Is there any additional progress to select target-specific fragments among the common fragments? If not, what factor do you think caused the improvement?

Thank you for pointing these out, and for taking the time to review our modifications. The additional formatting issues / comments have now been addressed in the updated manuscript.

We look forward to hearing from the reviewer and welcome further suggestions!

We thank the reviewer for the provided feedback and we address the points raised below.

Incorporating PMO benchmark tasks and directly comparing results with other generative baselines, such as those in Gao et al. (2022) [1], could provide a more comprehensive evaluation of SynFlowNet’s optimization performance.

We have now incorporated a study of SynFlowNet's sample efficiency, based on the suggested benchmark by Gao et al [1]. The results and discussion are presented in Section B.1.1 in Appendix and show that SynFlowNet is more sample efficient than its fragments analogue. We would also like to emphasize that generative models based exclusively on sampling from a reward function, such as SynFlowNet, are mostly designed for relatively inexpensive oracle functions, such as medium-accuracy docking simulations, or high-throughput screening platforms which can produce hundreds of thousands of readouts each weeks or months, therefore be believe that the reported sample efficiency of SynFlowNet is satisfactory.

Please let us know if there are any further questions/suggestions!

References

[1] Gao, Wenhao, et al. "Sample efficiency matters: a benchmark for practical molecular optimization." Advances in neural information processing systems 35 (2022): 21342-21357.

We thank the reviewer for valuable feedback. We will respond to each of the reviewer's questions individually.

Increasing the number of BBs slows down training linearly that may cause a computational burden when dealing with a large amount of BBs.

While the linear slow down of training with increasing number of BBs is worrying, an action space of >200k BBs (which is the current size of the Enamine BBs) can be easily handled by the current implementation of SynFlowNet. With multi-processing, we were able to train a model with 200k BBs and 10k steps, with a batch size of 64 in ~6 hours. Additionally, by looking at the history of Enamine's building blocks set, it has never increased by a lot more than 25k BBs at any given year (Enamine Real Compounds -- Youtube). Projecting this 10 years forward, the size of the set would reach ~600k in 10 years, which can be managed by today's SynFlowNet without any change. If the set of BBs was to grow faster at any point, we could consider hierarchical action spaces to handle that more efficiently.

1. the authors need to discuss the effect of the proposed backward policies on the performance of the forward policy in more detail.

The last column of Table 2 shows that backward policies trained using the Maximum Likelihood criterion or using the REINFORCE algorithm lead to the discovery of a higher number of high-reward modes compared to a free backward policy only trained to match the trajectory balance criterion, and perform similarly to a uniform backward policy, two strategies often employed in the literature [1]. Figure A.6 in Appendix further supports the claims and shows how the number of discovered high-reward modes progresses with training under the different backward policies, which looks favorable for a REINFORCE policy with a positive entropy coefficient. Crucially, as explained in Section 4.4, training the backward policy with the proposed objectives does not primarily attempt to improve the performance of the forward policy, but aims at insuring the forward-backward flow consistency in the MDP. In a synthesis DAG we cannot guarantee that a backward-constructed trajectory ends in $s_0$ (i.e. that leads back to a building block -- the retrosynthesis problem). This shortcoming would cause the model to sample from a different, biased distribution than from the Boltzmann distribution defined by the reward function $R$. In Table 2, we thus show that training the backward policy according to any of our proposed strategies dramatically increases the ability of the model to return to $s_0$ when traversing the MDP in the backward direction, thus ensuring flow-consistency and correcting that bias. We have made this point clearer in the updated manuscript to emphasize that the real benefit of our proposed training scheme for the backward policy is preserving the correct flow in the MDP while retaining the ability to discover diverse and high-reward modes. We believe that this opens up interesting opportunities for future work by showing an example where a data-driven, ML-based solution is employed as a solution to an MDP design challenge.

2. Has the poor performance of FragGFN originated from insufficient training? Or is it because of the use of SeH proxy as a reward, which may have a severe generalization problem for unseen molecules?

Indeed, we were also expecting better performance for the fragment models. We hypothesise that the observed performance can be due to the following reason: to offer a fair comparison between the models in terms of chemical space covered, FragGFN models were trained using the fragments obtained from decomposing the Enamine building blocks used by SynFlowNet. The space of obtained fragments is large (~250 fragments) and leads to a very large state space (as estimated in Figure 2), therefore finding high reward modes in such a large space can be difficult. We use a similar training budget to SynFlowNet and the experiment shows the advantage of SynFlowNet in finding high-reward modes faster. We have now clarified this in the manuscript.

3. what is the rewarding strategy used for multi-objective optimization (FragGFN SA, SynFlowNet SA, SynFlowNet QED)?

The strategy that we used was the multiplication of rewards, which has now been clarified in the manuscript.

4. it is possible that improving the novelty for SynFlowNet may be simply due to the use of the Enamine building blocks, while REINVENT has been trained with ChEMBL?

To fully answer this question and the suggestion of reviewer AADH, we have retrained SynFlowNet with building blocks derived from ChEMBL molecules and performed the same analysis as in Figure 6 ([edit]: now Figure 5). We decomposed 70k randomly sampled ChEMBL molecules into building blocks using a retrosynthesis software and used these to train SynFlowNet. We observe that although SynFlowNet was trained with a ChEMBL-dervied chemical space, it manages to preserve the novelty reported with Enamine building blocks, with an increase in average similarity from 0.43 to 0.49. This experiment thus strengthens the argument that SynFlowNet discovers more novel regions of the chemical space, as opposed to models relying on likelihood-based priors such as REINVENT, which show difficulty in departing from their training distribution. We report these results in Section B.1, Table A.2 and Figure A.5, and in the table below.

References

[1] Malkin, Nikolay et al., “Trajectory Balance: Improved credit assignment in GFlowNets.” CoRR, 2201.13259, 2022

Dear Reviewer f18i,

Thank you for reviewing our manuscript and for the insightful comments. We have now updated our manuscript based on your suggestions. Below we address each question individually.

could not find any code to accompany the paper

We have now added a link in our manuscript to the anonymised Github repository for this project: https://anonymous.4open.science/r/synflownet-01A3.

I would like to know that SynFlowNet can generate diverse and known synthesizable molecules, not just diverse predicted synthesizable molecules

We have added results showing SynFlowNet's performance in rediscovery tasks (celecoxib and aripiprazole), which we also show in the table below. Being synthesis-constrained, the expressivity of the model is reduced compared to models constructing molecules at an atom or fragment level (such as those in [1]). The performance of SynFlowNet in such rediscovery tasks is confined by the particular set of building blocks and reactions made available to the model. If a particular target molecule cannot be synthesized from these components, it will remain inaccessible for SynFlowNet. We verified whether the two molecules tested for this task have analogues in the REAL space 2024-02 using SpaceLight [2] and found that only aripiprazole is present, explaining the low rediscovery score of celecoxib (closest analogue found has similarity 0.68. Note that the REAL space is assembled from different reactions to the ones used in SynFlowNet, therefore the 0.68 analogue might not be reachable).

I would have also liked to know how SynFlowNet compares to other synthesizability-constrained molecular generation models which do have publicly available codebases

Our goal in the paper was to focus on comparisons to four families of models:

• Different algorithm using the same MDP: This analysis is aimed at RL models such as those of Gottipati et al. and Horwood & Noutahi et al.. However, the code to reproduce their method using our own MDP was not available. To offer comparison to the RL counterpart of SynFlowNet, we thus implemented Soft Q-learning, a well known, strong RL baseline similar to the algorithms employed in Gottipati et al. and Horwood & Noutahi et al.
• Same algorithm (GFlowNet) but using a different MDP: For this comparison, we focused on Fragment GFlowNet, which uses the same training procedure but samples molecules according to a different action space. To maximise fairness, we also re-generated the fragment library used by this model to closely match SynFlowNet's chemical space.
• A likelihood-based model and a strong baseline from the literature: REINVENT
• Search-based synthesisability-constrained model: SyntheMol

We believe that these represent a strong and diverse set of empirical comparisons which allow to analyse the strengths and limitations of the presented approach from diverse angles.

We also have edited the manuscript to incorporate the suggested comparison to SynNet. In Table A.4. we show new results for SynFlowNet trained on all optimization tasks performed in SynNet, and we report top-k performance. We compare these results with that of SynNet extracted from the original paper [3] and find that the two models are comparable: SynFlowNet achieves better performance for GSK3$\beta$ (0.862 vs. 0.815) and SynNet scores higher with JNK3 (0.719 vs. 0.710).

are 300K oracle calls simply what is needed for good performance in SynFlowNet?

SynFlowNet can learn to generate molecules of high reward with less oracle calls (see the newly-added Table A.6 ([edit]: now Table A.5) in Appendix), however we continue to train the model with 300k oracle calls to let it discover more modes of the distribution captured by the reward function. As shown in the bottom of Figure 5 ([edit]: now Figure 4) in the paper, the most important gains in performance happen in the first 20% of the training run, but SynFlowNet then continues to discover new high reward modes for many additional steps with diminishing returns. How long the model should be trained thus depend on how expensive the oracle function is to query (see next question) and the level of accuracy that is targetted w.r.t the sampling distribution of the model.

How sample efficient is SynFlowNet?

We now introduced a study of SynFlowNet’s performance on the PMO benchmark, which shows improvement over the Fragment GFlowNet (Table A.6 [edit]: now Table A.5). Notably, SynFlowNet does not rely on a pretrained prior, which is the case of the high-performing models in the PMO benchmark, and learns to generate molecules purely from the distribution of the reward function. Although requiring more oracle calls, SynFlowNet offers a more unbiased solution on the sampling problem, while (as shown in the last panel of Figure 7 ([edit]: now Figure 6)), it is hard for REINVENT, which uses a prior, to leave the distribution of that prior. Generative models based exclusively on sampling from a reward function, such as SynFlowNet, are mostly designed for relatively inexpensive oracle functions, such as medium-accuracy docking simulations, or high-throughput screening platforms which can produce hundreds of thousands of readouts each weeks or months.

I could not find how $s_0$ is initialized, nor how important this is

On lines 205/206 we state “In more detail, each trajectory starts from an empty molecular graph which is followed by a building block sampled from AddFirstReactant”. This explains that $s_0$ is an empty molecular graph.

Which experiment was it exactly that demonstrated the new backward policy led to improved sample quality/diversity?

The last column of Table 1 ([edit]: now Table 2) shows that backward policies trained using the Maximum Likelihood criterion or using the REINFORCE algorithm lead to the discovery of a higher number of high-reward modes compared to a free backward policy only trained to match the trajectory balance criterion, and perform similarly to a uniform backward policy, two strategies often employed in the literature [4]. Crucially, as explained in Section 4.4, training the backward policy with the proposed objectives does not primarily attempt to improve the performance of the forward policy, but aims at insuring the forward-backward flow consistency in the MDP. In a synthesis DAG we cannot guarantee that a backward-constructed trajectory ends in $s_0$ (leads to a building block, i.e. the retrosynthesis problem), which will cause the model to sample from a slightly different, biased distribution than from the Boltzmann distribution defined by the reward function $R$. In Table 1 ([edit]: now Table 2), we also show that training the backward policy according to any of our proposed strategies dramatically increases the ability of the model to return to $s_0$ when traversing the MDP in the backward direction, thus ensuring flow-consistency and correcting that bias. We have made this point clearer in the updated manuscript to emphasize that the real benefit of our proposed training scheme for the backward policy is preserving the correct flow in the MDP while retaining the ability to discover diverse and high-reward modes. We believe that this opens up interesting opportunities for future work by showing an example where a data-driven, ML-based solution is employed as a solution to an MDP design challenge.

How does SynFlowNet perform when optimizing for targets that are highly symmetric, e.g., molecules with a Cn rotation axis, where n$\geq$2?

We have tested this against a symmetrical target "CC(N)CCc1cccc(C(=O)CCC(=O)CCC(=O)c2cccc(CCC(C)N)c2)c1" and found that SynFlowNet was able to discover it (top-1 reward=1.0).

What are the failure modes of SynFlowNet?

We have now added Section B.2 "Outlook" explaining the failure modes of SynFlowNet and potential ideas for improvement left for future work.

How challenging would it be to expand SynFlowNet beyond bi-molecular reactions? I am not suggesting this should have been done here, but asking more from curiousity and wondering if this is a future direction the authors are aiming to explore in future work.

Expanding SynFlowNet beyond bi-molecular reactions is an interesting direction which we have actually considered. Tri-molecular reactions can be accounted for by additional masking based on the location of substructure match. For now, we have successfully extended SynFlowNet to work with scaffold decoration, which vouched for the flexibility of the codebase and way in which the action space is structured.

Are the points in Figure 6 (last panel) shuffled for plotting?

No, the points are not shuffled. The points have increased transparencies from top to bottom.

The references are poorly formatted, this should be fixed. gsk3$\beta$ should be GSK3$\beta$

Thanks for pointing this out, we have revised these.

Thank you to the authors for the thoughtful revisions.

I see that a codebase has been added and many of my suggested revisions addressed (at least in the comments and through additions to the appendix), and increased my score from a 5 to a 6. I will have a deeper look this Saturday to see if the added experiments are also discussed in the main text, as they are quite interesting.

Thank you for also adding the sample efficiency results in addition to the rediscovery results. I think that it is okay that SynFlowNet does not outperform REINVENT in sample efficiency, as years of engineering have gone into making REINVENT efficient and it is also not especially designed for synthesizability-constrained generation; but it is always good to have things to aim for in future work.

Finally, I would possibly suggest to replace Figure 3 currently in the main text to make room for Section B.1.1 in the main text, as that is more interesting; in other words, I recommend moving the example synthesis pathway to the appendix as IMO it does not add as much as a discussion on molecular rediscovery (though, Figure 3 is a very nice figure).