GLID$^2$E: A Gradient-Free Lightweight Fine-tune Approach for Discrete Biological Sequence Design

Thank you for your time and effort reading and reviewing our paper. Also, thank you for the valuable questions. We provide detailed answers below, hope it may clarify ourselves as well as enhance your understanding.

1. The authors position GLID2E ... metrics (reward convergence, sequence diversity)?

Thank you for your suggestion. We acknowledge that the current work indeed has limitations in multi-objective optimization evaluation. However, we claim that GLID2E can explore space through reward shaping and likelihood constraints from pretrained diffusion models. Compared to MTCS, it can achieve higher computational efficiency. We will subsequently update PepTune to the current benchmark.

2. Hyperparameter choices, ... methodological robustness and transparency.

We will supplement ablation experiments regarding k. According to experimental results, we can see that k=1 achieves the best trade-off. When k is less than 1.0, it will lead to conservative exploration, generating samples closer to the original distribution, but functionality will be limited. When k is greater than 1.0, the model will have more aggressive exploration, and lower quality samples will also lead to decreased model functionality.

K	Pred-Activity (Median)	ATAC-Acc	3-mer Corr	Log-Lik (Median)
0.5	7.12	0.99	0.53	-252.9
1.0	7.45	0.99	0.49	-244.7
2.0	7.06	0.97	0.48	-232.5
3.0	7.21	0.83	0.50	-262.1

3. The absence of rigorous statistical ... robustness and reproducibility of performance claims.

Thank you for the suggestion. We will conduct Wilcoxon signed-rank tests, and the results are as follows: In Table 2, comparing GLID2E and DRAKES, except for 3-mer, all other p-values are 0.03125 (<0.05), meeting statistical significance characteristics. In Table 3, comparing GLID2E and DRAKES, the results are shown in the following table, and we can see that the performance of DRAKES and GLID2E is basically similar.

Metric	Pred-ddG (Median)	%(ddG>0)	scRMSD (Median)	%(scRMSD<2)	Success Rate
p-value	0.3125	0.2187	0.21875	0.03125	0.0625

4. While computational efficiency ... memory usage analyses, are limited. Thank you for the detailed suggestion. We will provide runtime comparisons for different lengths and memory usage comparisons: We have collected inference metrics for sequences of different length ranges. Since we set the generation site amount fixed as 25 (following the same setting as DRAKES), the differences in these metrics are not significant.

seq_length	batch_size	num_runs	mean_time	std_time	mean_gpu_memory
50	8	50	0.011207	0.002391	1048.058
100	8	50	0.009823	0.000122	1050.813
150	8	50	0.010457	0.000191	1053.742
200	8	50	0.010417	6.69E-05	1057.576
250	8	50	0.010685	8.98E-05	1055.277
300	8	50	0.010617	5.87E-05	1057.729
350	8	50	0.010712	8.75E-05	1053.25
400	8	50	0.010695	0.000147	1071.35

5. Regarding the PPO-based surrogate... would help me understand its role.

We appreciate your detialed review of our method, the derivation corresponding to eta is as follows. This ratio is r_t, which directly participates in the calculation of L_CLIP and gradient updates. The larger eta is, the more the updates are constrained by priors and the more conservative it becomes; conversely, it allows for large-scale exploration. $\frac{\pi_\theta(a|s)}{\pi_{\text{ref}}(a|s)} = \frac{\pi_\theta(a|s)}{\pi_{\text{prior}}(a|s)^\eta \pi_\theta(a|s)^{1-\eta}} = \left(\frac{\pi_\theta(a|s)}{\pi_{\text{prior}}(a|s)}\right)^\eta.$

6. The authors utilize reward shaping ... (especially given the stochastic nature of replacing MASK tokens)?

Thank you for pointing out. Regarding the questions on both aspects, our claims are as follows:

Before claims, we would like to build a mathematical formulation for the sampling process to demonstrate the unbiased property. To obtain the reward $\Phi$, we need to score along a complete sequence, so every MASK token in $s_t$ is independently replaced by a sample from the current policy $p_\theta(\cdot | s_{t-1}, t)$ with the MASK token set to zero probability. Let $B$ denote the number of completions drawn. Then we have

$\hat{\Phi}(s_t) = \frac{1}{B} \sum_{b=1}^{B} r(x_t^{(b)}), \quad \mathbb{E}[\hat{\Phi}(s_t) | s_t] = \Phi(s_t),$

where x_t is the sampled sequence at time t. Therefore, $\hat{\Phi}$ is an unbiased estimator of Φ. Its variance is

$\text{Var}[\hat{\Phi}(s_t) | s_t] = \frac{\sigma_r^2(s_t)}{B},$

where $\sigma_r^2(s_t)$ is the conditional variance of the reward. During the diffusion process, the number of MASKs decreases, reducing both $\sigma_r^2$ and gradient noise, making the estimator progressively stable.

Then, for the two perspective being mentioned:

1. Since only one token is sampled at each step, the incremental rewards at each step will cancel each other out across the entire sequence, so that the cumulative reward and original reward are always on the same order of magnitude, and gradient explosion will not occur. Meanwhile, in the latter half of generation, the number of MASKs gradually decreases, and the corresponding variance of sigma also gradually decreases, making the values more stable.

2. Lines 205-206 state that the expected return and original return are independent of the trajectory. Meanwhile, the potential function of Monte Carlo estimation is unbiased for the true potential function, so the corresponding gradient is also unbiased and will not introduce systematic bias.

7. The work extensively rely on ... policy optimization outcomes.

We appreciate your insight to the effect of reward oracle.

1. We supplemented Oracle noise experiments on protein systems: We added three scales of noise to the ddG predicted by the diffusion model. Compared to before adding noise, the success rate was generally affected, mainly impacting Pred-ddG, and the larger the noise scale, the greater the impact.

2. From the reward model performance of DRAKES paper, we can conclude that protein oracle has higher oracle noise than DNA, which leads to the unstable training and high sensitivity to oracle noise. However, we would like to claim that the proposal of reward shaping and mixture is to address Oracle bias, and the ablation results w/o M1 have demonstrated it.

Noise Scale	Pred-ddG (Median)	%(ddG>0)	scRMSD (Median)	%(scRMSD<2)	Success Rate
0.01	0.81	0.73	0.94	0.91	0.66
0.1	0.71	0.74	1.00	0.84	0.60
0.5	0.21	0.53	0.88	0.95	0.49

We appreciate your time and effort for reviewing our submission. Your key insight and valuable question help us boost our paper's quality. Regarding the questions, we provide the following clarification:

1. Statistical Significance. Could you report ... strengthen the claim of superiority.

Thank you for the suggestion. We will conduct Wilcoxon signed-rank tests, and the results are as follows: In Table 2, comparing GLID²E and DRAKES, except for 3-mer, all other p-values are 0.03125 (<0.05), meeting statistical significance characteristics. In Table 3, comparing GLID2E and DRAKES, the results are shown in the following table, and we can see that the performance of DRAKES and GLID²E is basically similar.

Metric	Pred-ddG (Median)	%(ddG>0)	scRMSD (Median)	%(scRMSD<2)	Success Rate
p-value	0.3125	0.2187	0.21875	0.03125	0.0625

2. Reward‑Model Robustness. How ... CLC prevent reward hacking?

We added three scales of noise to the ddG predicted by the diffusion model. Compared to before adding noise, the success rate was generally affected, mainly impacting Pred-ddG, and the larger the noise scale, the greater the impact.

Noise Scale	Pred-ddG (Median)	%(ddG>0)	scRMSD (Median)	%(scRMSD<2)	Success Rate
0.01	0.81	0.73	0.94	0.91	0.66
0.1	0.71	0.74	1.00	0.84	0.60
0.5	0.21	0.53	0.88	0.95	0.49

3. Hyper‑parameter Burden. The method ... is most critical?

Very practical perspective. The PPO algorithm involves many hyperparameters, but most of our model's related parameters adopt commonly used ones. For specifics, please refer to Table 6-Table 8. The two hyperparameters that have significant impact on results have already been proposed in the ablation study. The clip ratio directly controls the exploration-exploitation tradeoff - too strict will prevent beneficial exploration, while too loose will cause excessive distribution shift; the mixture ratio also directly controls the model's exploration of modes, similar to the principle of KL penalty. Being overly conservative will prevent effective exploration, while maintaining smaller values can provide training stability while serving as regularization.

4. 3‑mer Correlation Drop. In DNA ... adversarial samples undetected by CLC?

Thank you fory your very insightful observation. Regarding the question about the decrease in 3-mer correlation, we believe this precisely demonstrates GLID2E's core advantage. We need to distinguish between two types of "naturalness" measures: 3-mer correlation reflects traditional statistical patterns, while Log-Likelihood embodies the learned distribution of pretrained models. GLID2E performs best on Log-Likelihood (-239.889) and achieves ATAC-Acc as high as 90.6%, indicating that generated sequences both conform to the model's perceived naturalness and maintain key biological properties, effectively ruling out the possibility of adversarial samples. In fact, this pattern demonstrates GLID2E's unique value: traditional methods can only mimic known enhancer patterns, while we discovered functionally equivalent but structurally different new sequence variants, expanding the feasible design space. The Clipped Likelihood Constraint mechanism fundamentally prevents adversarial attacks. Therefore, the decrease in 3-mer correlation combined with improved activity and good biological metrics indicates that GLID2E achieved principled exploration of new functional sequence spaces, which is precisely our method's core contribution.

5. Generalisation Beyond Training Domains. ... vs. functional generality?

Good point! However, different cell types have different scales and biological experimental conditions compared to protein families, which leads to rewards not being able to accurately express the functionality of samples. Therefore, we believe they cannot be put together for fine-tuning.

6. Missing baselines. From the RL ... is very limited (only diffusion models).

Thank you for pointing out the CTRL method. We believe adding this baseline is necessary and will subsequently include it in the evaluation. However, we need to claim that our paper aims to propose a reinforcement learning paradigm based on Discrete Diffusion Models, targeting discrete guided sampling. Therefore, we believe it is not necessary to include other non-diffusion baselines. Hope for your understanding.

7. In the beginning you mention ... statements like this one.

Thank you for the correction. This statement is indeed not accurate enough. Our method's advantage lies in data efficiency during the fine-tuning stage, not the pre-training stage. We will address this in the paper.

8. Why didn't you use standard benchmarks and splits like AAV and GFP?

We are not particularly clear what you refer to as the "Standard" benchmark. We are following DRAKES's setting in both experimental systems, and we believe this is a reasonable split with relevant baselines. We look forward to your insights on the current benchmark.

Dear Reviewer 8abn,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the discussion period is approaching its conclusion, we would greatly value hearing your feedback on our rebuttal and learning whether it has adequately addressed your concerns. Should there be any follow-up questions or issues that require further clarification, we hope to have sufficient time to address your concerns and provide any necessary clarifications.

Once again, we are deeply grateful for your time and constructive feedback throughout this process.

Best regards,

Authors

Thank you for the detailed response and explanations. The authors have addressed most of my concerns, and I have no further comments.

We greatly appreciate the time and effort you put into the review. The concerns you raised are very targeted and constructive. In response, we attempt to clarify the issues one by one and clarify the advantages and contributions of our method through addressing these questions.

1. In the DNA design task, ... more measured conclusions.

We appreciate the insightful question. We believe it is necessary here to distinguish between two different "naturalness" measurement standards: 3-mer correlation reflects the similarity of traditional statistical patterns, while Log-Likelihood embodies the distributional cognition learned by pretrained diffusion models from large-scale real data. Our experimental results show that GLID2E achieved optimal performance on the Log-Likelihood metric (-239.889), while maintaining a high level of 90.6% on ATAC-Acc, comparable to the current best method DRAKES (92.5%). This combination of "excellent Log-Likelihood + good biological metrics" effectively rules out the possibility of adversarial samples, proving that the generated sequences both conform to the model's cognitive naturalness standards and maintain key biological properties.

In fact, the decrease in 3-mer correlation combined with significant improvement in activity precisely demonstrates GLID2E's unique value: traditional methods can only mimic the n-gram patterns of known enhancers, limiting them to existing sequence feature spaces, while our Clipped Likelihood Constraint mechanism, while fundamentally preventing adversarial attacks, successfully discovered functionally equivalent but structurally different new sequence variants, achieving principled exploration of new functional sequence spaces. However, the current results are only based on model predictions, and we apologize for being unable to provide biological experimental validation.

2. This work also does not address... generated sequences, metrics to measure k-mer diversity, etc.

Thanks for pointing out. We acknowledge that the paper indeed lacks quantitative analysis of the diversity of generated sequences, which is an important omission. We will supplement comprehensive diversity analysis in the revised version, including key metrics such as average pairwise edit distance between generated sequences, k-mer diversity statistics, sequence entropy, and unique sequence ratios, and compare with baseline methods to verify whether GLID2E maintains sufficient sequence diversity while optimizing functionality.

3. On line 166, the authors mention ... number. Is this a possible typo?

Yes, this is a typo. We didn't count the samples within the batch. It should be 4*16(batch size)*10(scaling coefficient)*4(gradient accumulation step)=1,280.

4. I think some sections of ... explain the overall framework being used.

Thank you very much for pointing this out. We will add the reference to Figure 1 in the original text and redesign Figure 1 to highlight our advantage of not needing to store entire trajectories.

5. Another major area for improving clarity ... data from 5 different cell lines.

Thank you very much for pointing this out.

First, regarding details, we completely followed all evaluation settings of DRAKES, including generating 640 sequences for each method (batch size of 64, with 10 total batches). We also conducted experiments with three random seeds and reported means and standard deviations.

Then, there are two reasons for choosing HepG2: 1) HepG2 is a liver cancer cell line that is widely used in enhancer activity research and has been used by multiple baselines; 2) Although the dataset contains five cell lines, we believe focusing on a single cell line can avoid noise that might come from mixing multiple cell lines and can more precisely evaluate the effectiveness of the method.

6. There is also a large body ... authors do not cite that body of literature.

This is a very key suggestion. Our initial idea was to focus on diffusion+biological work, but offline RL-based biological sequence design still has high relevance. We promise to supplement this in subsequent versions.

7. How is the GLID2E method gradient-free?... confused with the terminology.

"Gradient-free" refers to avoiding backpropagating gradients through the entire diffusion trajectory, rather than not using gradients at all. PPO still requires policy gradients, but the computational complexity is reduced from O(T·L·d²) to O(L·d). We are considering changing it to "lightweight gradient." Thank you for the suggestion!