Improved Off-policy Reinforcement Learning in Biological Sequence Design

Active learning setting

We follow the existing generative active learning settings (GFN-AL [1], FLEXS [2]), which are standard in this field. As noted in our main text and appendix, we perform active learning as follows:

Starting with an initial dataset $D_0$, each active learning round t consists of three steps:

• (Step A) train the proxy model on the current dataset $D_{t-1}$;
• (Step B) train the generative policy using the proxy and proposing a batch of B new sequences;
• (Step C) evaluate those sequences using Oracle and add them to the dataset.

We run 10 active learning rounds with a batch size of 128. We have the same limited oracle calls for all methods. At each round, we assess performance using the top-K sequences (with K=128). We adopt the initial dataset of DNA and RNA from [1] and [3], respectively. For GFP and AAV, we generated the initial dataset using wild-type, but as discussed in the following questions, we have re-conducted experiments using the dataset from [REF1, 2].

Experimental setting for GFP

First of all, our GFP benchmark is quite standard in this field; we use TAPE [4], one of the widely adopted benchmarks in representative studies (e.g., [2], [7]). The benchmarking methods for the GFP task are diverse (e.g., TAPE [4], Design-Bench [5]; see [6] for detailed comparisons among them). Additionally, scoring normalization varies across studies, with some methods employing normalized scores [REF1, 2] and others using unnormalized scores [1, 2, 7].

As suggested, we have conducted experiments by following the experimental setting of LatProtRL [REF2]: we evaluate 256 sequences in 15 active learning rounds by adopting the oracle function with the given initial dataset that is split following [REF1]. As with [REF2], we report the median value of the top-128 sequences over 5 independent runs. The results for AdaLead and LatProtRL are directly obtained from [REF2]. Our approach achieved better performance than LatProtRL and substantially improved GFN-AL.

In the revised manuscript, we will discuss the challenges in GFP with additional experimental results.

Distance to the wild-type and restricted exploration in GFP

We assume the oracle to be perfect, as with previous works. The performance of proxies typically degrades when predicting samples far from the offline data distribution, which is particularly evident in our constrained search.

We agree that incorporating distance from wild-type sequences leads to more realistic benchmarking. However, we focus on proposing a broadly applicable off-policy search method, not benchmarking itself. Importantly, our approach is compatible with distance-based constraints, as they are orthogonal to our method. To demonstrate this, we applied δ-CS to Proximal Exploration (PEX) [7], which explicitly encourages proximity to the wild type—see results below.

Validation with other off-policy RL algorithms

As suggested, we applied our method to other RL methods: soft Q learning. Please refer to the answer to reviewer 6J1t (W3).

Other baselines

As suggested, we compare with random mutation and LS-GFN. LS-GFN is our special case where the masking positions are fixed as the last 𝛿L tokens.

The table shows that fixing masking positions can overly restrict the search space. However, specifying more significant positions in sequences based on domain knowledge can be more efficient than randomly choosing the positions.

Related works

As suggested, we'll include the missing references and re-organize Sec 5.

Other comments

DyNa PPO in GFP

In Fig 4, the mean score of Dyna PPO remains unchanged over active rounds, which means it fails to generate sequences with higher scores than the initial dataset. Note that GFP scores are not normalized.

"Large-scale"

We use "large-scale" to denote longer sequences, especially with large action space, like protein design tasks.

line 212

We'll revise it.

[1] Jain et al. "Biological sequence design with GFlowNets." ICML (2022)

[2] Sinai et al. "Adalead" (2020)

[3] Kim et al. "Bootstrapped training of score-conditioned generator for offline design of biological sequences." NIPS (2023)

[4] Rao et al. "Evaluating protein transfer learning with TAPE." NIPS (2019)

[5] Trabucco et al. "Design-bench" ICML, 2022.

[6] Surana et al. "Overconfident oracles." (2025).

[7] Ren et al. "Proximal exploration for model-guided protein sequence design." ICML, 2022.

W1, Q2: The term "mask" is confusing

Because our infilling sites are uniformly distributed across the entire sequence, we refer to them as “masks,” similar to BERT. However, we recognize that this term can be confusing. Alternatively, we could describe the algorithm as random teacher forcing: some tokens are predicted while others remain identical to the original sequence. This approach also removes the need for $\tilde{x}$ in the equation, since it is not directly used as input by the neural network.

W2: 𝛿CS with other algorithms beyond GFlowNet

𝛿-CS can be applied to any off-policy RL method; we chose GFlowNets because they are representative method in this domain. To demonstrate $\delta$-CS as a plug-and-play component, we applied 𝛿-CS to Soft Q-Learning (SQL), which is a representative off-policy RL method in the general domain:

W3: Conservative proxy

We acknowledge conservative model-based optimization (COMs) methods, which aim to construct robust proxy models [1, 2]. However, mere conservativeness is insufficient for active learning scenarios, where active querying for information gain from the oracle is essential. Although COMs perform well in offline model-based optimization (MBO), our results show they underperform in active learning settings. Specifically, COMs that impose conservativeness through adversarial smoothness in proxy training (Trabucco et al., 2021) achieve poorer outcomes than alternative methods.

Our method shares the philosophy that conservatism is beneficial but differs significantly in implementation. Instead of embedding conservativeness directly into proxy training, we integrate it into the search process itself. This approach preserves high information gain through explicit uncertainty modeling while ensuring conservatively robust optimization, effectively balancing uncertainty-driven exploration with robustness. We will put this discussion at the main paper. Thanks.

Trabucco et al. "Conservative objective models for effective offline model-based optimization." ICML, 2021.

W4: Validation in complex biological sequence task

We have already included longer sequence benchmarks (such as GFN and AAV), which are widely recognized and commonly used in the literature. While TFB tasks are indeed simpler and considered toy benchmarks, they remain popular for initial evaluations. We agree that incorporating additional real-world DNA benchmarks could further strengthen our evaluation.

Regarding the suggested benchmarks, we noticed that the majority of existing literature [1,2,4] focuses on diffusion-based algorithms, which fall outside the scope of our research (therefore we focused to benchamrk [3]). Although we aimed to replicate results from TACO [3], which employs autoregressive models, their dataset and pre-trained checkpoints are unfortunately not publicly available. Consequently, extending our method to include these benchmarks within the rebuttal period is challenging. However, we commit to evaluating our method on longer DNA benchmarks by the camera-ready deadline, and we anticipate similar performance trends to those observed in GFP and AAV tasks.

W5: $\delta$-CS vs. conservative reward

As discussed in W3, our approach still can utilize acquisition function like UCB while introducing conservatism to avoiding overoptimization. Though we have different approach, we can utilize the uncertainty oracle as proposed in [1] since we do not any restriction for the way of measuring uncertainty; this will be an interesting future work.

We'll include this discussion into our revised manuscript

Q1. Do we actually know the distribution $P_{D_{t-1}}$?

We define a discrete distribution by normalizing the weights in line 132. Specifically, $P_{D_{t-1}}(x) \propto w(x; D_{t-1}, k),$ where $k$ is a hyperparameter controlling the distribution’s peakiness.

Q3. Couldn't DyNA PPO directly leverage offline data through teacher-forcing methods?

We agree that teacher-forcing offers a way to incorporate offline data into DyNA-PPO. However, because PPO is inherently on-policy, directly leveraging offline data remains challenging without methods like importance sampling, which can introduce high variance. While teacher-forcing also can be used to pretrain the policy pretraining from offline sequences, it may lead to overfitting to the offline distribution since the reward information is not used during pretraining. To further verify this point, we conducted DyNA PPO experiments where we first pretrain the policy via teacher-forcing on offline data and then fine-tune it with PPO on TFBind8:

(Theoretical Claims) No theoretical guarantees on convergence or stability with increasing rounds, would be interesting to see some theoretical or experimental insights.

Thanks for highlighting the importance of theoretical guarantees and analysis. We acknowledge that rigorous theoretical guarantees, such as formal convergence rates, are indeed valuable. However, deriving such guarantees for deep-learning-based active learning is exceptionally challenging—moreover, the proxy model and the generative policy interplay to explore the black-box landscape. Our primary goal is to provide a methodological and empirical contribution. We demonstrate the effectiveness of $\delta$-CS by presenting performance gains on DNA, RNA, and protein design tasks. These results clearly illustrate the practical benefits of our proposed approach across various domains. We will include this in our limitations of Section 7.

W1. Does training on $D_{t-1}$ (containing more synthetic sequences) drift proxy model accuracy?

In the active learning setting, we assume that $D_{t-1}$ consists of annotated data with true oracle functions with limited accessiblity; e.g., we can query sequences with batch size of B = 128 at each round. Note that the predicted data $(x, f_{\phi}(x))$ is added to $D_{t-1}$ during the policy training.

W2. Any intuition on how extreme $\delta$ values (near 0 or 1) might affect performance? No major experiments needed, just some textual discussion would suffice.

When $\delta$=0, no noise is injected into the offline data, so the generative policy is trained purely on the existing data without any additional exploration. In contrast, when $\delta$=1 the original offline data is completely replaced with noise, and the model fully relies on its own policy to denoise, meaning fully on-policy learning with no conservatism.

Q1. Unclear if growing synthetic data proportion (from $\delta$-CS) leads to accuracy drift in later rounds (potential model bias).

At the start of each round, we reinitialize the generator model. During training, the model uses a proxy (acquisition function) to evaluate sequences, but we do not add them to our dataset. After training, we propose new sequences to be evaluated with oracle functions and then add those annotated results to the dataset, so the dataset always contains only real (annotated) data.

Thanks for your valuable comments

W1. The initial value and adjustment strategy of $\delta$ may depend on task experience, and there is a lack of general guidelines.

In this study, we simply set delta=0.5 for DNA/RNA (masking 4 to 7 tokens on average) and delta=0.05 for protein design (masking approximately 4 to 12 tokens). Our intention is to show that even with this simple setting without task-specific tunning, $\delta$-CS gives consistent improvements. Moreover, the results in Fig 12-14 in the appendix show that setting delta to mask approximately 1~12 tokens (i.e., 0.1 ≤ $\delta$ ≤ 0.5 in DNA/RNA, 0.01 ≤ $\delta$ ≤ 0.05 for protein design) is consistently beneficial compared to exploration without $\delta$-CS.

As the reviewer mentioned, carefully setting delta based on experience and domain knowledge can bring even more improvements.

W2. The effectiveness of adaptive $\delta$ is highly dependent on the uncertainty estimation accuracy. If the estimation deviation is large, it may affect the performance.

We measure the uncertainty as the inconsistent proxy predictions using MC dropout or Ensemble. The key insight of the adaptive $\delta$ is to search the space more conservatively when the proxy gives inconsistent predictions on a certain datapoint compared to other datapoints. In addition, thanks to the scale parameter $\lambda$, we can adjust $\delta$ according to each point’s relative uncertainty. So, even if the estimation deviation (i.e., the discrepancy between predicted uncertainty and true uncertainty) is large, it does not overly degrade performance.

W3. While $\delta$-CS is designed for GFlowNets, its applicability to other types of generative models or RL algorithms in biological sequence design is not extensively explored.

Thanks for the valuable feedback. As suggested, we conducted additional experiments using Soft Q-Learning (SQL) on the Hard TFBind-8 benchmark to demonstrate the broader applicability of our approach. As shown in the table below, $\delta$-CS significantly improves both exploration and performance in SQL, confirming that our method extends beyond the GFlowNet setting.

The rationale for the experimental focus: $\delta$-CS introduces a new off-policy exploration strategy under the unreliable proxy by conservatively leveraging known high-reward sequences. We initially focused on applying $\delta$-CS within GFlowNets because they offer a natural framework for structured sequence generation in biological domains. Moreover, there is a theoretical equivalence between GFlowNet training objectives (e.g., detailed balance, trajectory balance) and established off-policy max-entropy RL methods (e.g., Soft Q-Learning, Path Consistency Learning) when reward corrections are applied (Tiapkin et al., 2024; Deleu et al., 2024). This connection supports that the insights from our GFlowNet experiments can be extended to improvements in general off-policy RL settings.

Q1. Does the oracle $f$ in the paper come from ground truth or a proxy evaluation model? If it comes from ground truth, how to evaluate the noised sequence that does not have ground truth?

In the case of TFBind-8, we use ground truth since all possible sequences are experimentally evaluated (Barrera et al., 2016) as the search space is relatively small ($4^8$). For other tasks, we assume a simulator or trained model with a larger dataset as our ground truth oracle, like various previous works (Sinai et al., 2020; Kirjner et al., 2023), and evaluate perturbed sequences using these oracle functions.

Specifically, we use ViennaRNA (Lorenz et al., 2011) to evaluate the newly proposed RNA sequences. For GFP, the TAPE transformer model trained with 52,000, which is much larger than our initial dataset, is used as an oracle (Rao et al., 2019). In AAV, the oracle is built using comprehensive single-mutation data from AAV2 capsids, modeled additively (summing individual mutation effects with some noise), and applied to multiple target tissues across varying design lengths (Ogden et al., 2019).

• Tiapkin et al. (2024) “Generative flow networks as entropy-regularized RL.”
• Deleu et al. (2024) “Discrete probabilistic inference as control in multi-path environments.”
• Sinai et al. (2020) “Adalead: A simple and robust adaptive greedy search algorithm for sequence design.”
• Kirjner et al. (2023) “Improving protein optimization with smoothed fitness landscapes.”
• Barrera et al. (2016) “Survey of variation in human transcription factors reveals prevalent DNA binding changes.”
• Lorenz et al. (2011) “ViennaRNA Package 2.0.”
• Rao et al. (2019) “Evaluating protein transfer learning with TAPE.”
• Ogden et al. (2019) “Comprehensive AAV capsid fitness landscape reveals a viral gene and enables machine-guided design.”

Thanks for your valuable comments