Biological Sequence Editing with Generative Flow Networks

We sincerely appreciate your thoughtful review of our manuscript. In response to your feedback, we meticulously revised the introduction, conducted additional experiments, and added new experimental results to strengthen our paper. Please find below a detailed point-by-point response addressing your comments.

Related Works

Existing machine learning methods within the domain of biological sequences including the ones discussed in the related works have predominantly concentrated on generating novel de novo sequences with desired properties from scratch. Therefore, there is an inherent risk of deviating significantly from naturally occurring sequences compromising safety and predictability. In contrast, this paper explores the generation of edited sequences that enhance the properties of existing sequences, ensuring the edited counterparts maintain similarity to their original counterparts. Here, we take an input seed sequence and modify a few elements to improve its properties.

To underscore the limitations of existing biological sequence design methods, we conducted experiments on AMP and CRE datasets using GFlowNet to generate sequences from scratch. Our observations reveal that GFlowNet-generated sequences for the CRE dataset differ from existing sequences at 76.29% of locations, and for the AMP dataset, the difference is found in 95.24% of locations. In contrast, the proposed GFNSeqEditor demonstrates the ability to generate new edited sequences differing from existing ones in less than 35% of locations across all datasets. Moreover, employing existing biological sequence methods the difference between generated sequences and existing ones cannot be controlled while using the proposed GFNSeqEditor the amount of edits can be controlled by hyperparameters.

In order to highlight the differences between biological sequence editing and biological sequence design problems we have added a discussion to the introduction of the revised manuscript. This revision can be found on pages 1 and 2 highlighted in red. Also, we extend the related works discussion in Appendix D and the revisions are highlighted in red.

Furthermore, another aspect that distinguishes the proposed GFNSeqEditor from prior works is that existing methods are not able to combine multiple sequences to create a new sequence. In section 4.3, the proposed algorithm combines sequences to generate a new sequence similar to its parent sequences. In section 4.3, we discuss that this can also be used for sequence length reduction which has important applications in vaccine therapies.

Comparison with GFN Methods

Existing GFlowNet methods can be used for biological sequence design from scratch and as a result, they suffer from the same problems as those of other biological sequence design methods discussed above when it comes to performing sequence editing. However, since GFlowNet can generate sequences token by token sequentially, there is a naive approach to employ GFlowNet for sequence editing. In this approach, the first tokens of a sequence $x$ can be given to GFlowNet and then the GFlowNet generates the rest of the tokens. We add this as a baseline to Table 1 in the revised version and the baseline is called GFlowNet-E. For example, for the CRE dataset, the GFlowNet-E gets the first 130 tokens and generates the rest of 70 tokens. Compared to GFNSeqEditor, this approach lacks the flexibility to choose edit locations and GFNSeqEditor outperformed GFlowNet-E. This indicates the effectiveness of sub-optimal position identification of GFNSeqEditor.

Please find below our responses to your questions.

1. We used trajectory balance loss in equation (5) to train the flow function. On page 7 before section 4.1 of the revised version, we made clear that we used trajectory balance loss. The revision is highlighted in red.
2. In general the reward function $R(x)$ can be any function and it is determined by the environment. In experiments we consider the sequence property as the reward since in experiments our goal was to maximize those properties.
3. In the first paragraph of section 3 we add that “Assume that the flow function $F_\theta(\cdot)$ is trained on an available offline training data.”
4. In order to make the sequence editing procedure of GFNSeqEditor more clear, in section 3.1 we add that “GFNSeqEditor constructs edited sequences token by token and for each position $t+1$ it examines if $x_{t+1}$ should be used or not.”
5. We revised can to would.
6. We add a comment on page 5 that “Assume that $x_t \in \mathbb A$, $\forall t$ meaning that $x_t$ is always in the available actions.”
7. To clarify how to choose hyperparameters $\delta$, in this revised version after equation (6) we add that “Choosing larger $\delta$, it is more probable that the algorithm identifies $x_{t+1}$ as sub-optimal”. Furthermore, to clarify how to choose hyperparameter $\sigma$ after equation (7) we add that “The relation between $\sigma$ and the algorithm performance will be analyzed in section 3.3 and Appendix E.”
8. In order to clarify the effect of $\lambda$ in this revised version after equation (9), we add “Choosing larger $\lambda$ leads to obtaining smaller number of edits”.
9. Based on this comment, we revise the notation and in the revised version $R_{F,T}$ denotes the expected reward of a sequence with length $T$ generated using the flow function $F_\theta(\cdot)$.
10. We checked the spelling and we believe it is correct.
11. We followed the prior studies such as Trabucco et al., (2021) and Jain et al., (2022) in which the goal in the TFbinding dataset is to increase the binding activity.
12. We would like to clarify that diversity itself is an important measure in generating biological sequences. Higher diversity as it is defined in section 4 among edited sequences increases the chance that at least one of the edited sequences successfully works in real experiments since it is expected that similar sequences show somewhat similar behavior in real experiments. Furthermore, diversity defined on page 7 is used before to evaluate the performance of biological sequence design methods (see e.g. Jain et al., (2022)).
13. We made14.34 bold in Table 1.

We would like to thank you for taking time to review our paper and let us know your valuable comments. Based on your comments, we have added more discussions on related works to the introduction and enriched the experiments section with additional experimental results. Kindly find below a detailed point-by-point response addressing each of your comments.

Pretraining

We would like to clarify that other baselines also use a pretrained model to perform editing. While we trained GFlowNet for GFNSeqEditor, we trained an MLP as the proxy model for Ledidi and Directed Evolution (DE) baselines. Ledidi and DE are highly dependent on the proxy model to evaluate their candidate edits and use these evaluations to proceed to the next iteration for editing. Also, we trained a Transformer for the Seq2Seq baseline and Seq2Seq employs the trained transformer model to perform sequence editing. Please note that:

1. We used exactly the same training data to pretrain GFlowNet and models used by the other baselines.
2. Due to the space limit, we put the experimental details in Appendix C.1.

Therefore, we believe that comparisons in the paper are fair since all baselines use a trained model and all models are trained on the same training data. Thanks to this comment to improve the clarity of the experiment section, we expand implementation details explanations in Appendix C.1 and also in section 4 we refer the readers to Appendix C.1.

Baselines

We would like to clarify that biological sequence design methods such as offline model-based optimization models including the ones discussed in [1] do not tackle the problem of sequence editing and as a result, they are not appropriate to be directly used for biological sequence editing for two main reasons:

1. Biological sequence design models usually generate entirely new sequences from scratch. Therefore, sequences generated by these methods are expected to be greatly different from existing sequences that need editing. Therefore, it might be infeasible to generate diverse sequences using biological sequence design models that are similar to the existing sequences.
2. The difference between generated sequences and existing ones cannot be controlled by employing biological sequence design methods while using the proposed algorithm by setting hyperparameters one can control the number of edits.

To underscore the limitations of biological sequence design methods, we generate new sequences from scratch using GFlowNet and we observe that on average for CRE dataset the generated sequences are different from existing sequences (i.e. sequences in the datset) in 76.29% of locations and it is not possible to control the amount of differences between generated sequences and existing ones. Also, for AMP dataset the generated sequences are different from existing ones in 95.24% of locations. In contrast, the proposed GFNSeqEditor demonstrates the ability to generate new edited sequences differing from existing ones in less than 35% of locations across all datasets. To highlight the differences between biological sequence editing and biological sequence design problems we have added a discussion to the introduction of the revised manuscript. The revision can be found on pages 1 and 2 highlighted in red.

Evolutionary Algorithms

Thanks for drawing our attention to the evolutionary algorithm in [2]. Note that [2] implements a standard directed evolutionary algorithm for sequence design. Our directed evolutionary baseline is the same as the evolutionary algorithm presented in Algorithm 1 of [2]. We cite the reference [2] in the revised version.

Comparison with GFN Baselines

Existing GFN baselines can be used for biological sequence design from scratch and as a result, they suffer from the same problems as those of other biological sequence design methods discussed above when it comes to performing sequence editing. However, since GFlowNet can generate sequences token by token sequentially, there is a naive approach to employ GFlowNet for sequence editing. In this approach, the first tokens of a sequence $x$ can be given to GFlowNet and then the GFlowNet generates the rest of the tokens. We add this as a baseline to Table 1 in the revised version and the baseline is called GFlowNet-E. For example, for the CRE dataset, the GFlowNet-E gets the first 130 tokens and generates the rest of 70 tokens. Compared to GFNSeqEditor, this approach lacks the flexibility to choose edit locations. GFNSeqEditor outperforms GFlowNet-E and this indicates the effectiveness of sub-optimal position identification of GFNSeqEditor in equation (7).

Please find below our responses to you questions.

Response to Q1. We trained an active learning based GFlowNet model following the setting in Jain et al. (2022). In the active learning setting, at each round of active learning $t \times K$ candidates generated by GFlowNet are sampled and then top $K$ samples based on scores given by a proxy are chosen to be added to the offline dataset. Here offline dataset refers to an initial labeled dataset. To train the GFlowNet, we employed the same proxy models as those used by other baseline methods. For all datasets, we set the number of active learning rounds to $1$, with $t$ equal to $5$ and $K$ equal to $100$. We parameterize the flow using a MLP comprising two hidden layers, each with a dimension of $2048$, and $|\mathbb A|$ outputs corresponding to individual actions. Throughout our experiments, we employ the trajectory balance objective for training. Adam optimizer with $(\beta_0, \beta_1) = (0.9, 0.999)$ is utilized during the training process. The learning rate for $\log Z$ in trajectory balance loss is set to $10^{-3}$ for all the experiments. The number of training steps for TFbinding, AMP and CRE are $5000$, $10^6$ and $10^4$, respectively. The remaining hyperparameters were configured in accordance with the settings established in Jain et al. (2022).

Due to space limitations, we provided the training details of employed GFlowNet models in Appendix C.1. In this revised version, we provided more training details in the main text of the paper. This revision can be found on page 7 before section 4.1 highlighted in red.

Response to Q2. Our baselines in the initial submission do not use the trained GFN since they do not construct sequences token by token sequentially. However, all baselines perform editing using a trained model. All models including GFN and the transformer of Seq2Seq are trained on the same training data.

Response to Q3. The proposed algorithm needs to query for the flow of partially constructed sequences $|\mathbb A|T$ times where $|\mathbb A|$ is the size of the action set (which is $4$ for TFbinding and CRE datasets and is $20$ for AMP dataset) and $T$ is the length of the sequence. Querying the evaluation of trained flow function models only needs inference and does not involve training. As a result, querying the evaluation of trained flow function is not computationally complex. Therefore, the complexity of the algorithm linearly scales with the size of the action set and the length of the input sequence. Therefore, the algorithm is scalable.

Response to Q4. Generating a new sequence from scratch by GFlowNet can be viewed as a search algorithm based on a pretrained policy. The benefit of using the learned policy by GFlowNet compared to other policies is that the probability of constructing a sequence by GFlowNet is proportional to the sequence reward. Moreover, naive search algorithms such as beam-search and MCTS usually need to observe the intermediate reward. However, when it comes to constructing a new object sequentially, the intermediate reward is not available and the reward is given when the object is fully constructed. Therefore, naive search algorithms may not work well in generating new objects. GFlowNet is able to learn to generate a new object with probability proportional to its final reward when there is not any intermediate reward. Leveraging GFlowNet by the proposed algorithm is its benefit compared to naive search algorithms.

Response to Q5. We performed experiments related to Theorems 1 and 2. Figure 3 studies the effect of hyperparameters $\delta$ and $\lambda$ on the property improvement and edit percentage. Furthermore, in Figure 4, we illustrate the impact of changing $\sigma$ on property improvement and edit diversity for GFNSeqEditor. These results corroborate the theoretical analyses outlined in Theorems 1 and 2 in section 3.3 as well as Theorem 3 in Appendix E.

The rebuttal and the improved manuscript have indeed addressed some of my concerns.

However, I hold a different perspective on the statement, "Moreover, naive search algorithms such as beam-search and MCTS usually need to observe the intermediate reward." I believe that we can leverage a Value estimator to facilitate beam-search or MCTS without requiring access to the true reward, similar to the approach used in AlphaGo.

It's important to note that there are numerous ways to generate edit sequences. To bolster your method's credibility, I recommend exploring a wide array of scenarios to showcase its superiority over various alternatives.

Furthermore, while I do partially agree with your assertion that editing methods can outperform de novo methods in terms of novelty (deviation from current sequences), I believe that this comparison should be made more explicit in the paper. Specifically, it would be beneficial to include (1) the actual metric used for comparison and (2) a comparison with baseline de novo methods (I appreciate you providing some experiments on this rebuttal). Doing so will enhance the paper's motivation for using the editing method.

Thank you very much for taking time to review our paper and letting us know your feedback. In response to your feedback, we have extended the literature review within the introduction and incorporated additional details regarding the training process and the innovative aspects of the proposed algorithm to the experiments section. Please find below our point-by-point responses to your comments.

Training Details

Due to the space limit, we put the training details in Appendix C.1. To address this comment, we provide more information about the training process of the flow function in the main text of the revised paper. Please see page 7 where we add that ``To train models associated with baselines and the proposed GFNSeqEditor, we partition each dataset into a 72% training set and an 18% validation set. The remaining 10% constitutes the test set, employed to evaluate the performance of methods in sequence editing tasks. The trained flow function $F_\theta(\cdot)$ employed by GFlowNet-E and the proposed GFNSeqEditor, is an MLP comprising two hidden layers, each with a dimension of $2048$, and $|\mathbb A|$ outputs corresponding to actions. Throughout our experiments, we employ the trajectory balance objective for training the flow function. Detailed information about training the flow function can be found in Appendix C.1.'' This revision is highlighted in red in the revised version. Also, we expand the implementation details in Appendix C.1.

Key Innovation:

We would like to emphasize that the primary goal of GFNSeqEditor is not to produce novel sequences and the focus is different from biological sequence design. While GFNSeqEditor can indeed be utilized in conjunction with biological-sequence generation models to create novel sequences with improved properties, as empirically examined in section 4.2, the primary intent of GFNSeqEditor is not centered around generating entirely novel sequences. Instead, the core purpose of the proposed GFNSeqEditor lies in generating a diverse array of edits for a given seed sequence $x$, ensuring that the edited sequences exhibit similarity to $x$ while simultaneously displaying improved properties. To highlight the key innovations that contributed to the performance gain of GFNSeqEditor compared to other baselines, in this revised version we expand the discussion about the results in Table 1 and Figure 3. We explain that the proposed sub-optimal identification method outlined in equation 7 and the proposed editing policy in equation 9 are the key innovations to achieve improved performance compared to other baselines. This revision is highlighted in red on page 7 in section 4.1.

Literature Review

To address this comment, we expand the literature review in the introduction and we discuss our contribution relative to the biological sequence design methods including GFlowNet. This revision can be found on pages 1 and 2 highlighted in red. Also we extend the detailed related work discussion in Appendix D. Please find below the discussion about the contributions that distinguish this paper from prior works.

Biological Sequence Design

Existing machine learning methods within the domain of biological sequences have predominantly concentrated on generating novel sequences with desired properties from scratch. Therefore, there is an inherent risk of deviating significantly from naturally occurring sequences compromising safety and predictability. In contrast, this paper addresses the generation of new edited sequences that enhance the properties of existing sequences, ensuring the edited ones maintain similarity to their existing counterparts. Here, we take an input seed sequence and modify a few elements to improve its property. To underscore the limitations of existing biological sequence design methods, we conducted experiments on AMP and CRE datasets using GFlowNet to generate sequences from scratch. Our observations reveal that on average GFlowNet-generated sequences for the CRE dataset differ from existing sequences at 76.29% of locations, and for the AMP dataset, the difference is found in 95.24% of locations. In contrast, the proposed GFNSeqEditor demonstrates the ability to generate new edited sequences differing from existing ones in less than 35% of locations across all datasets. Moreover, employing existing biological sequence methods the difference between generated sequences and existing ones cannot be controlled while using the proposed GFNSeqEditor the amount of edits can be controlled by hyperparameters.

Sequence Combination

Also another aspect that distinguishes the proposed GFNSeqEditor from prior works is that existing methods are not able to combine multiple sequences to create a new sequence. In section 4.3, the proposed algorithm combines sequences to generate a new sequence similar to its parent sequences. In section 4.3, we discuss that this can be used for sequence length reduction which has important applications such as vaccine therapies.

GFlowNet

Existing GFlowNet methods can be used for biological sequence design from scratch and as a result, they suffer from the same problems as those of other biological sequence design methods discussed above when it comes to performing sequence editing. However, since GFlowNet can generate sequences token by token sequentially, there is a naive approach to employ GFlowNet for sequence editing. In this approach, the first tokens of a sequence $x$ can be given to GFlowNet and then the GFlowNet generates the rest of the tokens. We add this as a baseline to Table 1 in the revised version and the baseline is called GFlowNet-E. For example, for the CRE dataset, the GFlowNet-E gets the first 130 tokens and generates the rest of 70 tokens. Compared to GFNSeqEditor, this approach lacks the flexibility to choose edit locations and GFNSeqEditor outperformed GFlowNet-E. This indicates the effectiveness of sub-optimal position identification of GFNSeqEditor.