A SIMILARITY-AGNOSTIC REINFORCEMENT LEARNING APPROACH FOR LEAD OPTIMIZATION

Thank you for the review. To address common concerns by the reviewers, we have added a Common Response (https://openreview.net/forum?id=rjLgCkJH79&noteId=AeabWUzjwD). We respond to the reviewer's comments below along with reference to the Common Response where needed.

“Lead optimization in drug discovery is an important and difficult task. I have difficulty accepting the method as an invention or improvement for lead optimization. For my understanding lead optimization is a much more complicated process than purely looking on QED score or a similarity score, which the authors didn’t investigated.”

We refer the reviewer to Common Response (1).

As mentioned, our method is a generalized approach for generating alternate candidates to the lead molecule. Therefore, we have not explored the subsequent step of docking or binding affinity in the context of our work.

“In general, the paper would gain strength if the authors would compare their method against more recent methods in generative design and more properties other than QED.“

We refer the reviewer to Common Responses (2) and (3). Results for the added baseline and metrics can be found in the Results Table in the Common Response.

“The second contribution of their paper as stated on page 2, says: “we propose a search strategy…” Could the authors elaborate more on the search strategy? In case it is just generating thousands of molecules and sorting them based on a score, this seems to me not like a novel strategy.”

The reviewer is referring to the following statement in the paper:

“We propose a search strategy that separates the property optimization from the training and offloads it as a post-curation process, thereby simplifying the task of learning.”

We acknowledge the comment made by the reviewer. The novelty comes from the idea of using the search strategy to offload the property optimization from the training procedure due to the problems associated with multi-objective optimization.

To explain further, we take the example of the reward function used in MolDQN [1] for multi-objective optimization:

$r_t = w \times SIM(m_t, m_0) + (1-\omega) \times QED(m_t)$

The optimal value of ‘w’ in this multi-objective case is hard to determine quantitatively during hyperparameter tuning. In fact, the notion of optimality may be task-dependent and may very likely not be well understood for most tasks. If the user is not satisfied with the generated molecules, a new model would need to be trained with a different ‘w’. As opposed to this, in our work, the model would not need to be re-trained. Only the much cheaper search strategy needs to be employed to generate a large number of molecules that can be filtered to retain desired properties. Other multi-objective optimization challenges, such as Pareto dominance, conflicting objectives, etc are also avoided in our method.

“I very much like the idea of using reaction rules, although not completely novel, e.g. [2]. I think a more detailed description how exactly they mine the reaction rules and a better description of the reaction dataset in general would help the reader to better understand the topic. It doesn’t have to be in the main text.”

We thank the reviewer for the acknowledgment and suggestion. We have added the details to the supplementary material.

“I had trouble understanding the last paragraph of section 4.6., “we found that under the condition G(a_t leads to g)…”. Could the authors elaborate a little bit more on the issue they observed?”

Thank you for the question. We had noticed that based on different values of the negative rewards that we chose, the policy would either converge or diverge.

To recap,

• In our method, we select a fixed number L of negative return trajectories per high return trajectory.

• The policy gradient loss moves the function away from low-return areas and closer to high-return areas.

Therefore, if the magnitude of the negative reward were high, the cumulative magnitude of gradient from the negative samples in a batch would overwhelm the magnitude of positive samples, causing it to diverge. The condition we mentioned works as a normalizing factor for the negative component of the loss and was essential for reward design in our case to ensure convergence.

Questions:

Q1. Did I understand it correctly that the offline dataset just contains molecules randomly put together using the reaction rules, so potentially not chemically plausible at all?

We are unsure if we understood the question correctly. The trajectories in the offline dataset are indeed put together using a random policy given the reaction rules. This still makes them chemically feasible though.

We elaborate for further clarification: Say we are given a start molecule. To generate a trajectory for the offline dataset, first, a reaction rule will be selected randomly among those that are applicable to it. This will result in a product which is chemically feasible. even though it is the result of a random sampling. By repeating this step multiple times, we generate a trajectory for our dataset using a random policy where each molecule is chemically feasible.

Q2: My understanding of actor-critic reinforcement learning is to use the output of the critic for the loss of the actor. From eq. (1) and (2) this seems not the be the case, could the authors elaborate a little bit?

The reviewer has noticed correctly. Due to our unique setup, we were able to use calculated returns to update the actor, thus not requiring the higher variance estimation by the critic.

The critic is still a crucial part of our method. Firstly, it provides additional gradients to update the embedding module, resulting in higher-quality representations. Secondly, it is used in the generation procedure to sort the actions suggested by the policy.

Thank you for the review. To address common concerns by the reviewers, we have added a Common Response (https://openreview.net/forum?id=rjLgCkJH79&noteId=AeabWUzjwD). We respond to the reviewer's comments below along with reference to the Common Response where needed.

“The authors compare their proposed off-line Reinforcement Learning (RL) policy with on-line RL baselines. This comparison between on-line and off-line RL algorithms seems somewhat unconventional.”

We agree with the reviewer that the comparison of offline RL with online RL baseline might be unconventional. However, we would like to clarify that our intended comparison is not between online and offline RL methods but instead between our binary reward method and prior distance-based reward methods (that use molecular similarity as rewards). Due to binary rewards leading to severe sparsity in our problem setup, we have adopted the use of binary rewards in an offline setting.

We have added another baseline for Random Search and refer the reviewer to the Results Table in our Common Response.

“Moreover, it's unclear whether the on-line RL baselines, such as the S model and Q+S model, employ an expert dataset similar to LOGRL. If they do not utilize expert data, this could introduce an unfair advantage to LOGRL, as it relies on additional expert data.”

The baselines are completely online and, hence do not employ any expert data. Since the offline dataset was not collected using an expert policy or a human, we disagree with the reviewer to call it “expert data”. The data available to the offline algorithm is a result of running a random policy in the same underlying environment as the online RL baseline and relabelling the rewards. The procedure is very trivial and should not provide any extra information to the offline RL algorithm.

“Additionally, I suggest exploring the possibility of supervised learning in this context. The authors assume access to a substantial amount of expert dataset containing high-reward samples. In such a scenario, imitation learning often outperforms offline RL.”

We thank the reviewer for the suggestion. We had explored behavior cloning(BC) during our early experiments and found offline RL to significantly outperform BC. Results from those experiments are given below. We evaluated BC vs our model on trajectories of different lengths (steps) - 1, 2 and 5 and calculated the top-k accuracy of the model. Top-k accuracy here indicates the percentage of test samples for which the model ranked the positive action within the top k of its predictions. The top k predictions are the actions suggested in a single step of the generation procedure described in Algorithm 2 in the manuscript. The results for those experiments are given below.

“Finally, it would be interesting to know if the proposed method is capable of generating diverse outputs. One potential concern is whether the method might collapse and generate a single output, as there doesn't appear to be a regularizer that can control all possible outputs directed toward the target molecule. Exploring the diversity of outputs and addressing this potential issue would strengthen the paper.“

We thank the reviewer for pointing it out. We have added an evaluation of the uniqueness of generation along with metrics for validity and novelty. Uniqueness is a measure of how unique are the molecules generated; 1 indicates all molecules are unique (or perfect diversity) and 0 indices all the generated molecules are the same (no diversity). All models, including our baselines, achieve near-perfect uniqueness scores. However, a major contribution to this metric would be a result of our design of the problem; due to the incredibly large search space and the generation procedure promoting uniqueness. That said, there do exist multiple paths for generating the same molecule, therefore the models are also effective in not reproducing the same paths repeatedly. We refer the reviewer to the Results Table in the Common Response for the metric scores.

“Typo in Section 4.5 line3: in the training batch, batchwe”

Thank you for pointing it out. We have corrected it.

Thank you for the review. To address common concerns by the reviewers, we have added a Common Response (https://openreview.net/forum?id=rjLgCkJH79&noteId=AeabWUzjwD). We respond to the reviewer's comments below along with reference to the Common Response where needed.

"The significance of the work is not clear. The method is trained to optimize molecules towards the target structures, but I am unsure if I understand how this model could be used in practice. Usually, the goal of lead optimization is to improve a set of molecular properties without impacting binding affinity."

We refer the reviewer to Common Response (1).
Having not considered a target receptor, we did not discuss binding affinity. We thank the reviewer for pointing it out. We acknowledge that it would be an important step after our method for further curation.

"Only two RL baselines were trained, and there is no comparison with other state-of-the-art methods in molecular optimization."

We refer the reviewer to Common Response (2).

"The evaluation metrics used in the experiments are very simple."

We refer the reviewer to Common Response (3).

A table with additional baseline and metrics is present in the "Results Table" section of Common Response.

Questions:

Q1 (A): What is the success rate of molecular optimization using LOGRL?

Without docking, the notion of success rate depends on the user. For instance, for our chosen examples of trypsin inhibitors, among 10000 molecules generated by LOGRL, 73 had QED > 0.7.

Q1 (B): Can you find many well-optimized molecules in the post-training filtering step?

It is possible to find many optimized molecules in the post-training filtering step. Towards this effect, we can simply increase the breadth or depth of the search, which results in a larger number of molecules.

Q1 (C): Do you think additional RL objectives could improve these properties significantly?

Additional RL objectives would surely improve the properties of generated molecules. Unfortunately, this has several drawbacks, some are mentioned in our paper. Firstly, for each combination of properties, there would need to be a separate model trained. Secondly, problems related to multi-objective optimization: additional cost of training due to extra hyperparameters, Pareto dominance, conflicting objectives, etc.

Our method avoids these problems by separating molecular search and property search into two different steps.

Q2: What are the real-life applications of this optimization algorithm? Can it be used for other optimization problems besides lead optimization (see the problems I mentioned in the "Weaknesses" section)?

The proposed algorithm can be used in offline RL settings with a large discrete action space, such as recommender systems and language generation.

Q3: In Section 3.2, two GCRL methods are described. Did you try the other method and if so, could you provide the comparison results?

The two methods described are binary rewards and distance-based rewards. Our proposed method uses binary rewards, while the online RL baselines use distance-based rewards. To our understanding, the comparison requested is present in the paper.

We would also like to clarify that similarity-based rewards are a type of distance-based reward.

Table contains three baselines: Random Search, S model (similarity rewards), Q+S model (QED+sim rewards) and our model LOGRL. Evaluation is performed for the top $10^n$ molecules based on highest similarity ("Molecules" column). The rest of the columns are the metrics used for evaluation and are self-explanatory.

[1] Gottipati, Sai Krishna, et al. "Learning to navigate the synthetically accessible chemical space using reinforcement learning." International conference on machine learning. PMLR, 2020.
[2] Hoksza, David, et al. "Molpher: a software framework for systematic chemical space exploration." Journal of cheminformatics 6.1 (2014): 1-13.
[3] Brown, Nathan, et al. "GuacaMol: benchmarking models for de novo molecular design." Journal of chemical information and modeling 59.3 (2019): 1096-1108.