Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL

We thank the reviewer for their thoughtful feedback and voting to accept our paper. We address each point individually below. “W/Q” numbers the weakness or question, followed by our response.

[w1]

We follow the training protocols described in [1] for our reward models. For the human enhancer task, we directly adopted the pretrained model weights from [1]. For human promoters, we trained a separate reward model using the data split from [2] (70% train, 20% validation, 10% test). Training was conducted using the AdamW optimizer with a learning rate of 1e-4 and MSE loss, over 20 epochs. The checkpoint with the best validation performance was used for evaluation.

All reward models are based on an Enformer architecture [3], combining convolutional and Transformer layers, which has shown strong performance in DNA regulatory prediction tasks. Both the enhancer and promoter models share the same architecture: 11 layers with a hidden dimension of 1536. We will add these details in the revised appendix.

[w2]

Our method optimized and evaluated sequences using the same pretrained reward model from regLM [4], trained on promoter data excluding chromosomes 7, 13, 21, and 22. To assess potential reward hacking, we introduced two additional settings:

Setting 1: Optimization used the regLM reward model; evaluation used a separate full reward model trained on the full dataset (including all chromosomes).

Setting 2: Optimization used a reward model trained on a random 80% subset of the full data; evaluation used the same full reward model from Setting 1.

In both settings, we conducted experiments on the human enhancer dataset, targeting HepG2 cell-specific activity. We observed consistent performance trends across all metrics (target expression, off-target suppression, motif enrichment, and diversity), comparable to our main results.

These findings suggest that regCon generalizes well, and that the observed performance gains are not an artifact of overfitting to the reward model used during optimization.

[W3]

We clarify that our experiments use pretrained HyenaDNA models without task-specific fine-tuning prior to reinforcement learning. To assess the role of pretraining, we compared pretrained and randomly initialized models on HepG2 enhancer optimization. We did not use fine-tuned models in our main experiments, as prior work [6] and our own early results both showed minimal performance differences between pretrained and fine-tuned models.

We found that the randomly initialized model underperformed the pretrained model. Notably, as shown in Figure 2 in our manuscript, TACO also leverages a pretrained HyenaDNA model but still underperforms our method, indicating that while pretraining offers improvements, our framework provides more effective optimization.

[W4]

Due to the rebuttal space constraints, we are unable to include full baseline implementation details here, but we will provide them in the revised appendix.

[Q1]:

Thank you for pointing this out. In our implementation, we use the term "epoch" to refer to one full pass over the collected batch of trajectories. We agree that "iteration" would be a more appropriate term and will revise it in the manuscript.

[Q2 & 3]:

To investigate scalarized objectives, we ran additional experiments comparing regCon and the top three baselines (TACO, AdaLead, and PEX), each adapted to optimize differential expression (DE). We also tested a variant of regCon (regCon-DE) trained with DE as the reward.

Our method offers two main advantages over DE objectives:

• Explicit constraint control: As shown in Figure 2 in our paper, regCon supports variable off-target thresholds (e.g., 0.3, 0.5, 0.6), allowing users to tailor optimization to specific biological requirements. Once the constraint is satisfied, the model focuses on maximizing on-target activity within the feasible region. In contrast, DE optimization can lead the model to suppress off-target activity, even at the cost of reducing on-target activation. For example, when optimizing for SK-N-SH (Table 2), regCon-DE tends to suppress off-target expression at the expense of decreasing SK-N-SH cell activity.

• Improved DE optimization: Most of the baselines are single-objective optimizers. Although they can be adapted to use DE as a reward, our results (Tables 1 and 2) show that these DE-based variants consistently perform worse than regCon, even in terms of DE values. This is because using a DE reward imposes a fixed trade-off between target and off-target expression, which often results in suboptimal solutions when the objectives conflict. This fixed weighting lacks the flexibility to adjust the balance between objectives throughout training [7]. In contrast, our Lagrangian approach allows for a more effective coordination of competing goals, resulting in better performance.

Table 1

Table 2

[Q4]

Yes, a core difference lies in the advantage estimation method. regCon uses a GRPO-inspired approach based on batch-normalized rewards, eliminating the need for a value network. In contrast, PPO-Lagrangian relies on Generalized Advantage Estimation (GAE), which depends on training a value network.

As noted in Section 4.2 (Lines 246), training a value network from sparse, sequence-level rewards is inherently unstable, since it must infer accurate token-level estimates. regCon avoids this issue by directly using normalized rewards, leading to more stable policy updates and potentially explaining the observed performance gap.

[Q5]:

For each target cell type, we initialize three penalty coefficients (λ): one for each off-target constraint (λ = 0.5 for JURKAT, K562, SK-N-SH, HepG2; 0.7/0.3 for THP1, reflecting the [0,1] range of λ) and one for the TFBS constraint (initialized at 0.0 with an upper bound of 0.1). These values were selected by grid search to balance expressions in different cell types and motif enrichment. Although Lagrangian methods can be sensitive to initializations[8], we observed stable performance across all cell types with the same initializations. As noted in the line 332, we plan to incorporate PID controllers [6] to improve λ adaptation in future work.

Paper Formatting Concerns

We will revise Figure 2 to use more distinguishable colors and marker shapes for improved clarity.

Thank you again for your valuable suggestions, which have helped improve the clarity and contribution of our work. Please let us know if you have any more questions.

[1] Lal, Anusha, et al. "Designing realistic regulatory DNA with autoregressive language models." Genome Research, vol. 34, no. 9, 2024, pp. 1411–1420.

[2] Reddy, A. J., et al. "Strategies for effectively modelling promoter-driven gene expression using transfer learning." bioRxiv, 2024, doi:10.1101/2023.02.01.526753.

[3] Avsec, Žiga, et al. "Effective gene expression prediction from sequence by integrating long-range interactions." Nature Methods, vol. 18, no. 10, 2021, pp. 1196–1203.

[4] Yang, Zhen, et al. "Regulatory DNA sequence design with reinforcement learning." The Thirteenth International Conference on Learning Representations, 2025.

[5] Schulman, John, et al. "High-dimensional continuous control using generalized advantage estimation." arXiv preprint arXiv:1506.02438, 2015.

[6] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. "Responsive safety in reinforcement learning by pid lagrangian methods." International Conference on Machine Learning. PMLR, 2020.

[7] Liu, Enbo, et al. "Pareto set learning for multi-objective reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, 2025, pp. 18789–18797.

[8] Ji, Jiaming, et al. "Safety Gymnasium: A unified safe reinforcement learning benchmark." Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 18964–18993.

We appreciate the reviewer’s recognition of our method’s originality, comprehensive evaluation, and potential for designing highly cell-type-specific regulatory elements. We also thank the reviewer for the constructive feedback. Please find our reponse below regarding the added diffusion baseline experiemnts and clarification on the TFBS frequency reward.

[Q1]

Thank you for this valuable suggestion. To address your comment, we performed an additional benchmark comparison with DRAKES[1], a recently proposed diffusion based method optimized using RL that has demonstrated success in human enhancer generation for the HepG2 cell line. Specifically, we used their publicly available checkpoint fine tuned on HepG2 and followed their experimental setup to further fine tune the model separately for K562 and SK-N-SH cell lines. It is important to note that DRAKES optimizes only a single objective, which is maximizing the expression in one target cell type. As shown in the newly added benchmark results, regCon consistently outperforms DRAKES by achieving higher expression levels in the target cell lines (0.77 versus 0.68 for HepG2, 0.93 versus 0.83 for K562, and 0.86 versus 0.79 for SK-N-SH) and significantly lower off target expressions. These results highlight the advantage of regCon in handling multiple constraints compared to diffusion based methods optimized for single objectives.

[Q2]

Thank you for raising this important question. We acknowledge the potential limitation arising from the use of motif frequency correlation as both a regularization term and an evaluation metric. To address concerns of circularity, we conducted an additional analysis using distinct reference sets for evaluation, created by splitting sequences at the 90th and 50th percentiles based on their on and off target activities. Although this filtering significantly reduced the dataset size (leaving only 18 sequences for JURKAT, 4 for K562, and 7 for THP1 out of the original 12,335), regCon continued to demonstrate strong performance, substantially outperforming baseline methods across all tested cell lines with correlations of 0.37 for JURKAT, 0.52 for K562, and 0.60 for THP1.

We also want to emphasize that using motif frequency correlation as a reward promotes generalizability, as shown in Figure 3a of our manuscript. Specifically, the TFBS reward is computed by first deriving a motif frequency vector from a reference set of real DNA sequences, then calculating the Pearson correlation between this reference vector and the motif frequencies in generated sequences. This encourages the model to capture global frequency patterns rather than explicitly match individual motifs, in contrast to prior methods like TACO. Despite this global formulation, regCon still successfully generates cell type–specific motifs such as HNF4A for HepG2 and GATA1 for K562, demonstrating that it acquires this ability purely through optimization on a coarse-grained reward. This highlights the robustness and generalizability of regCon’s optimization strategy.

[Q3]

Thank you for the insightful comment. In our experiments, we selected the sequences with activity above the 50th percentile in the target cell type, and off-target sequences as those with activity below the 50th percentile in non-target cell types. We then computed each motif’s occurrence frequency from these selected sequences. The motifs were selected following the protocol in [2].

This percentile-based thresholding, as adopted from prior work [2,3], helps mitigate differences in expression scaling across cell types and avoids reliance on absolute activity values, which may vary due to experimental noise or normalization schemes.

While we agree that using a more stringent threshold, such as the 90th percentile for on-target activity, could better capture highly specific sequences, doing so significantly reduces the number of selected sequences. For instance, in the human promoter dataset, using a 90th/50th percentile split for on/off-target activity results in only 18 sequences for JURKAT, 4 for K562, and 7 for THP1, out of a total of 12,335. In the human enhancer dataset, no sequences satisfy such criteria across all three cell types among around 660,000 candidates. These sample sizes are insufficient to support reliable transcription factor binding site (TFBS) motif enrichment analysis. Therefore, we opted to use a relaxed threshold to ensure sufficient statistical power and robustness in estimating TFBS frequencies.

Importantly, to mitigate potential overfitting to spurious motif correlations, we use a learnable weight for each TFBS regularization term and enforce an upper bound on these weights, as described in Line 193.

That said, we agree with the reviewer that using a 90th percentile cutoff would be preferable in scenarios where sufficient data is available, and we consider this a promising direction for future work.

Thank you again for your valuable suggestions, which have helped improve the clarity and contribution of our work. Please let us know if you have any more questions. Reference:

[1] Wang, C., Uehara, M., He, Y., Wang, A., Biancalani, T., Lal, A., ... & Regev, A. (2024). Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design. arXiv preprint arXiv:2410.13643.

[2] Lal, A., Garfield, D., Biancalani, T., & Eraslan, G. (2024). Designing realistic regulatory DNA with autoregressive language models. Genome Research, 34(9), 1411-1420.

[3] Yang, Z., Su, B., Cao, C., & Wen, J. R. Regulatory DNA Sequence Design with Reinforcement Learning. In The Thirteenth International Conference on Learning Representations.

I am satisfied with the authors' response. The comparison to DRAKES is a useful addition to the paper. I agree that the amount of data available limits the percentile cutoff that can be applied. It would be good to mention in the paper that this is the reason for choosing 50th percentile as the cutoff.

I think this method is especially useful to the field because of the explicit constraints enforced on the off-target cell activity. Current methods based on simple DE optimization are difficult to use for therapeutics design since they can easily result in excessive off-target activity (leading to potential side effects) or insufficient on-target activity (lack of efficacy). The evaluation convincingly shows that regCon provides a practical advantage in this area.

We thank the reviewer for the positive feedback and the insightful question.

How to choose the best hyperparameters

We performed a grid search over key hyperparameters such as the KL coefficient, learning rate, and multipliers. We select hyperparameters by running experiments across different hyperparameters and monitoring reward-based metrics (e.g., target activity, constraint satisfaction) on the generated sequences.

How to choose the hyperparameters for a new cell type

When designing for a new cell type, we use the corresponding reward model to guide optimization. In practice, we initialize with hyperparameters that worked well on similar cell types and find that our method is relatively robust across a wide range of settings. As shown in Table 3 (line 477 in the manuscript and also down below), many cell types share the same hyperparameters, suggesting generalizability across targets.

We acknowledge that our method involves a number of hyperparameters. This is in part due to its foundation in Lagrangian-based reinforcement learning [1]. However, many of these hyperparameters can be reused across cell types. For the reviewer’s convenience, we have included the hyperparameter table from the appendix in the rebuttal below. Users can start from the configurations we provide and refine them through a lightweight grid search based on training dynamics.

Experiment Hyperparameters

Thank you again for your constructive review and positive comments. Please let us know if you have any further questions!

Reference

[1] Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1), 2.

We appreciate the reviewer’s recognition of our method’s novelty, the concreteness of our experiments, and the framework’s potential for broader biological design tasks. We also thank the reviewer for the constructive feedback. Please find our reponse below regarding the added baselines experiemnts and clarification on dataset processing.

Comparison with simple DE-based reward functions and other baselines

Summary:

We added the requested baselines and differential expression (DE) as a scalar reward. Across enhancer and promoter settings, regCon yields higher on target expression while meeting off target constraints. Scalar DE optimization underperforms and often increases DE by suppressing off target at the cost of on target signal.

Detail response:

We have added experiments involving the baselines mentioned: Reddy et al. [1], Linder et al. [2], Brookes et al. [3], and Gosai et al. [4], as well as additional experiments using DE as a scalar reward function (denoted with a "-DE" suffix in Table 1-3).

Specifically, Gosai et al. [4] proposed three generation strategies: AdaLead, Fast SeqProp (FSP), and Simulated Annealing (SA). Since AdaLead was already included in our main experiments, we additionally implemented and benchmarked FSP and SA. Due to time and resource constraints, we evaluated:

• Brookes et al. [3] and Gosai et al. [4] (FSP and SA) on the human enhancer dataset, alongside the DE-based variants of our baselines (Table 1-3).
• Reddy et al. [1] and Linder et al. [2] on the human promoter dataset (Table 4-6).

As shown across all tables, DE-based optimization consistently underperforms compared to regCon. Scalar DE optimization applies a fixed linear trade-off (Target - Off), which tend to converge to suboptimal optimization space[5]. In contrast, Lagrangian methods dynamically adjust penalty coefficients through learned multipliers, enabling the model to escape poor local optima and achieve more effective optimization.

Furthermore, maximizing DE as a scalar reward can lead the model to aggressively suppress off-target expression, even at the expense of reducing on-target activation. This behavior is evident in the performance of regCon-DE in Table 3, where the method achieves higher DE but at the expense of significantly lower on-target expression compared to regCon. In contrast, regCon enforces explicit off-target constraints and directs optimization toward increasing on-target activity within the feasible region, leading to higher on-target cell activity and more biologically meaningful solutions.

Table 1.

Table 2.

Table 3.

Table 4.

Table 5.

Table 6.

Clarity: add dataset processing details

Thank you for the suggestion. We will add an Appendix section that details data processing steps to enhance reproducibility.

Thank you again for your helpful feedback. We’re happy to address any further suggestions or questions.

References:

[1] Reddy AJ, Geng X, Herschl M, Kolli S, Kumar A, Hsu P, Levine S, Ioannidis N. Designing cell-type-specific promoter sequences using conservative model-based optimization. Advances in Neural Information Processing Systems. 2024 Dec 16;37:93033-59.

[2] Linder J, Bogard N, Rosenberg AB, Seelig G. A generative neural network for maximizing fitness and diversity of synthetic DNA and protein sequences. Cell systems. 2020 Jul 22;11(1):49-62.

[3] Brookes D, Park H, Listgarten J. Conditioning by adaptive sampling for robust design. InInternational conference on machine learning 2019 May 24 (pp. 773-782). PMLR.

[4] Gosai SJ, Castro RI, Fuentes N, Butts JC, Mouri K, Alasoadura M, Kales S, Nguyen TT, Noche RR, Rao AS, Joy MT. Machine-guided design of cell-type-targeting cis-regulatory elements. Nature. 2024 Oct 23:1-0.

[5] Liu, E., Wu, Y. C., Huang, X., Gao, C., Wang, R. J., Xue, K., & Qian, C. (2025, April). Pareto set learning for multi-objective reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 18, pp. 18789-18797).