Reinforced Active Learning for Large-Scale Virtual Screening with Learnable Policy Model

We sincerely thank the reviewer for the detailed feedback and constructive suggestions. Below, we summarize the reviewer's questions and provide point-by-point responses.

W1: The Contribution of the Proposed Methodology

R: Thank you for this insightful question. We would like to clarify that, while our approach leverages existing reinforcement learning (RL) techniques, our contribution goes well beyond simply combining known components. Our study specifically addresses the unique challenges of active learning in virtual screening, an area that differs substantially from standard RL tasks.

To our knowledge, this is the first work to integrate GRPO within an active learning framework and apply it to the context of virtual screening. We innovatively formulate the virtual screening process as a Markov Decision Process (MDP) and embed active learning principles, allowing GLARE to adaptively select the most informative molecules for labeling. By coupling the RL agent's selection strategy with docking feedback in an active learning loop, our method, GLARE, efficiently prioritizes the most informative docking computations, offering a efficient and practical solution for real-world virtual screening workflows.

In addition, based on GRPO, we have substantially redesigned the framework. Specifically, we introduced a new reward formulation specifically tailored for virtual screening, which incorporates an exploration enhancement based on gradient uncertainty, a feature absent from prior methods.

We also experimented with Direct Policy Optimization (DPO), another effective RL algorithm, for optimizing the policy network. As shown in the table below, GRPO consistently outperforms DPO in our setting. We attribute this to GRPO's use of group-wise advantage estimation, which better captures the relative quality of selected molecules within a batch (a property that aligns well with the objectives of virtual screening). This provides a direct motivation for our choice of GRPO.

We hope this clarifies the novelty and significance of our contribution.

W2: Incorporating Protein or Pocket Features in the Policy Model

R: Thank you for this valuable suggestion.

We would like to clarify that our previous experiments have utilized protein features by incorporating DrugCLIP pretrained representations. However, the policy model did not include a protein encoder to fully exploit these protein or pocket-level features during the selection process. To address this, we have conducted additional experiments by explicitly integrating protein or pocket-level features into the policy model.

These results also highlight both the strong extensibility and superior performance of our approach in virtual screening scenarios. We plan to further explore and design more effective protein feature fusion modules in future work.

W3: Scalability for Applications

R: Thank you for raising this important point.

We would like to clarify that our method does not require training a separate policy model for each target. In our experiments, we utilized DrugCLIP-pretrained representations to encode protein or pocket features, enabling the model to generalize across multiple targets. For example, on the LIT-PCBA benchmark, our model simultaneously handles 15 protein targets and their associated binding pockets within a single framework, demonstrating our superior performance across all targets (Table3 in manuscript).

To better exploit protein or pocket-level information, we also have conducted additional experiments incorporating these features directly into the policy model via a protein encoder. During training, the policy model takes both the molecular features and the corresponding protein or pocket features as input. This design enables efficient many-to-many matching between molecules and multiple targets or pockets within a single model, significantly improving scalability and practical applicability in large-scale virtual screening scenarios.

W4: Baseline Methods in Experiments

R: Thank you for your comments.

In active learning research, comparisons are typically organized around both acquisition functions (strategies) and surrogate models. Accordingly, TcsAL and PtAL each represent a series of baselines in our experiments. TcsAL includes baselines with various strategies and has been evaluated with both MLP and GNN surrogate models. PtAL, on the other hand, emphasizes comparisons using different surrogate models, including pretrained models such as MolCLR and MoLFormer, while employing two widely validated strategies (Greedy and UCB).

To further compare active learning methods with traditional virtual screening baselines, and to demonstrate the added value of GLARE, Table 3 presents results of GLARE and several virtual screening baselines (including DrugCLIP) across all 15 LIT-PCBA subsets.

Additionally, we have included comparisons with a classic baseline (ε-Greedy strategy) and a recently proposed baseline (Rank strategy)[1], as shown in the table below. While the Rank strategy performs better than the methods reported in Table 1, it still underperforms compared to GLARE. This is primarily because Rank remains reliant on manually defined selection strategies.

We hope these extensive comparisons address the reviewer's concerns and further validate the robustness and effectiveness of GLARE.

[1] Deng, Xun, et al. Improving the Hit Rates of Virtual Screening by Active Learning from Bioactivity Feedback. Journal of Chemical Theory and Computation, 4640-4651, 2025.

We sincerely thank the reviewer for the detailed feedback and constructive suggestions to improve our manuscript. Below, we summarize the reviewer's questions and provide point-by-point responses.

W1: Clarification on Novelty and GRPO

R: Thank you for your valuable comments.

We respectfully disagree with the view that our work is merely a straightforward application of Group Relative Policy Optimization (GRPO). In fact, our study goes significantly beyond simply adopting GRPO. We specifically address the unique and underexplored challenges of active learning in virtual screening, challenges that are fundamentally different from those in standard RL domains. These include extreme class imbalance, label scarcity, and the need for efficient prioritization of candidates from ultra-large chemical spaces.

To our knowledge, this is the first work to integrate GRPO within an active learning framework and apply it to large-scale virtual screening. We not only reformulate the virtual screening process as a Markov Decision Process (MDP) but also introduce active learning-driven reward structures, enabling GLARE to adaptively and efficiently select the most informative molecules. By coupling the RL agent's selection strategy with docking feedback in an active learning loop, our method, GLARE, efficiently prioritizes the most informative docking computations, offering a efficient and practical solution for real-world virtual screening workflows.

In addition, based on GRPO, we have substantially redesigned the framework. Specifically, we introduced a new reward formulation that incorporates gradient-based uncertainty, specifically tailored to the active learning context, an aspect not addressed in prior GRPO works.

We also compared GRPO with Direct Policy Optimization (DPO), another effective RL algorithm, for optimizing the policy network. As shown in the table below, GRPO consistently outperforms DPO in our setting. We attribute this to GRPO's use of group-wise advantage estimation, which better captures the relative quality of selected molecules within a batch (a property that aligns well with the objectives of virtual screening). This provides a direct motivation for our choice of GRPO.

We hope this clarifies the novelty and significance of our contribution.

W2: The Computational Cost

R: Thank you for raising this important question. To provide a comprehensive comparison, we evaluated both training and inference runtimes on the ALDH1 dataset, using the same hyperparameters and hardware as TcsAL (see Appendix C.4).

Active learning typically involves training on a small number of labeled samples while making predictions on a much larger pool of unlabeled data. As a result, inference time becomes the dominant computational cost, especially in large-scale virtual screening scenarios. As shown in the table below, while our method requires additional training time due to the computational cost of gradient-based uncertainty in reward calculation, this overhead is confined to the training phase. During inference, reward computation is not required, enabling our method to achieve inference speeds comparable to TcsAL.

The characteristic is particularly important for large-scale dataset. To further demonstrate the scalability of our approach, we tested it on the AmpC dataset (~$10^8$ compounds) using the same hyperparameters as PtAL. As dataset size increases, inference time becomes even more significant. Our method maintains inference speed similar to baseline methods while delivering much higher accuracy, making it especially well-suited for ultra-large virtual screening tasks.

W3: The choice of Pretrained GNN.

R: Thank you for highlighting this important point.

For GLARE (with GNN), we use a Graph Isomorphism Network (GIN) as the backbone. For GLARE (with Pre. GNN), we also employ a GIN architecture, but it is initialized with weights from the pretrained GraphMVP or MolCLR model.

From the table below, GLARE (with Pre. GNN) substantially outperforms PtAL (with MolCLR), even though both use pretrained GINs (with different pretraining protocols). Notably, GLARE (with GNN, no pretraining) also surpasses PtAL (with MolCLR) in the final phase (Iter 6), confirming the strength of our learned policy.

To further isolate the effect of the learning strategy, we additionally compared GLARE (with MolCLR) and PtAL (with MolCLR) using the same pretrained GINs. As shown in the the table below, GLARE consistently outperforms PtAL under the same backbone, demonstrating that the performance gain is primarily attributable to GLARE's superior learning strategy, rather than the encoder itself.

Q1: Extension to Multi-Objective Optimization.

R: Thank you for this excellent question. We appreciate the suggestion and will discuss this direction in the Discussion section as future work.

GLARE can indeed be extended to support multi-objective optimization, which is highly relevant for practical drug discovery.

Reward Extension: A straightforward way is to incorporate additional objectives as regularization terms in the reward function. For example, the reward can be reformulated as $r_i = (1-(y_i-a_{i,y_i})^2 \cdot u_i) \cdot c_i,$ where $c_i$ represents a constraint or score for another property, such as druglikeness or similarity to a reference compound. By tuning $c_i$, the model can learn to favor molecules optimizing multiple objectives.

Policy Extension: Alternatively, the framework can support multiple policy networks, each focusing on a different objective (such as activity, druglikeness, solubility, etc.). Each policy provides a confidence score for its respective objective, and decisions can be made by aggregating these scores. This can be formulated as a multi-agent optimization framework: $M = \lbrace \pi^{c_k}(a_i^{c_k}|s_i) | k \in 1, \cdots,n \rbrace,$ $a_i = a_i^{c_1} \otimes \cdots \otimes a_i^{c_n},$ where actions from different agents (objectives) are combined to determine the final selection.

Thanks to our flexible reward design and reinforcement learning-based policy, GLARE naturally accommodates such extensions. This makes it well-suited for practical drug discovery scenarios that require simultaneous optimization of multiple molecular properties.

We sincerely thank the reviewer for their positive comments, especially recognizing the novelty of our work and the extensiveness of our experimental section. We also appreciate the constructive suggestions, which have helped us to further improve the quality of our manuscript. Below we provide detailed point-by-point responses to each of the reviewer's concerns.

W1 & Q1: Computational Cost in Large Chemical Space

R: We appreciate your valuable comment on the computational scalability of GRPO-based reinforcement learning, particularly as the screening library size increases to $10^8$–$10^9$ compounds.

To address this concern, we conducted large-scale experiments on the AmpC dataset (~$10^8$ compounds) using the same hyperparameters as PtAL. As shown in the table below, our method (GLARE) achieves inference speeds comparable to the baseline PtAL across both medium (EnamineHTS-0.1, 2 million compounds) and ultra-large (AmpC, 99.5 million compounds) datasets. Notably, while our method introduces additional computational overhead during training due to gradient-based reward optimization, this overhead does not affect inference. During inference, reward computation is not required, and the runtime is dominated by model forward passes, as in the baselines.

We also evaluated both training and inference runtimes on the ALDH1 dataset (PKM2 and VDR have similar scales), using the same hyperparameters and hardware as TcsAL.

W2: The Design of the Exploration Discount Factor

R: Thank you for this insightful comment.

We would like to clarify that the exploration discount factor $u_i$ in our reward function is not solely based on heuristic or empirical design, but is fundamentally data-driven. The motivation for introducing $u_i$ is to ensure that even if a molecule is predicted as inactive, it can still be selected if its uncertainty is high (i.e., it may be informative for model training). Thus, we penalize the reward by a factor that captures uncertainty, rather than using a fixed or manually tuned value.

Specifically, inspired by [1], we incorporate Exploration Modification in the reward, where uncertainty is quantified by the L2 norm of the model's gradient. This approach makes the discount factor adaptive to the actual characteristics of each dataset, instead of being a static or hand-crafted parameter. As a result, our method can flexibly adapt to different tasks, targets, and hit rate distributions, thereby enhancing both its generalizability and effectiveness.

We will revise this part in the manuscript to improve its clarity. Thank you again for your valuable feedback.

[1] Jordan T. Ash et al., "Deep batch active learning by diverse, uncertain gradient lower bounds." arXiv preprint arXiv:1906.03671, 2019.

W3: The Contributions of the Pretrained Encoder and Learned Policy

R: Thank you for highlighting this important point.

To clarify the respective contributions of the pretrained encoder and the learned policy in GLARE, we provide a condensed analysis of the results from Table 1 and Table 2.

As shown in the table below, GLARE (with GNN) significantly outperforms TcsAL (with GNN), regardless of the strategy used. Both employ the same non-pretrained GIN architecture. This demonstrates that our Learnable Policy strategy in GLARE provides a substantial advantage over prior approaches. GLARE (with Pre. GNN) further outperforms GLARE (with GNN), indicating that pretraining effectively addresses the cold-start problem and enhances early-phase retrieval performance.

From the table below, GLARE (with Pre. GNN) also substantially surpasses PtAL (with MolCLR), even though both use pretrained GINs (with different pretraining protocols). GLARE (with GNN, no pretraining) even outperforms PtAL (with MolCLR) at the final phase (Iter 6), confirming the strength of our learned policy.

To further isolate the effect of the learning strategy, we compared GLARE (with MolCLR) and PtAL (with MolCLR) under same pretrained GINs, also shown in the table below. GLARE consistently outperforms PtAL, showing that the performance gain is primarily due to GLARE's superior learning strategy rather than the encoder alone.

These comparisons clearly demonstrate that GLARE's superior performance arises mainly from its learned policy, while the use of a pretrained encoder provides an additional warm-up effect, further enhancing screening efficiency.

Q2: Sensitivity Analysis of GLARE's Policy Performance

R: To evaluate the sensitivity of GLARE's policy to the reward formulation and exploration heuristics, we conducted a series of ablation studies on the ALDH1, PKM2, and VDR datasets using GLARE (with GNN). The results are summarized in the table below.

• Removing exploration modification entirely ($\nu_i=0$, i.e., $u_i=1$) results in the lowest performance among GLARE variants, as the model loses the ability to identify uncertain but informative molecules.

• Using a constant exploration factor ($\nu_i=0.5$) yields better results than no modification, but is still less effective than dynamically adjusting $u_i$ based on gradient uncertainty.

• When the reward in GLARE is modified as $r_i=2-(y_i-a_{i,y_i})^2 \cdot u_i$, performance drops slightly compared to the original GLARE, but still remains well above all baselines, indicating that GLARE is not highly sensitive to the specific reward formulation.

Overall, these results highlight the robustness of GLARE and the added value of our adaptive exploration strategy. Importantly, all GLARE variants consistently outperform the existing baselines, regardless of how the reward or exploration modification is set. This demonstrates the robustness of the GLARE framework.

L1: Discussion

R: We appreciate the reviewer's valuable feedback and acknowledge several limitations of our current framework. We will incorporate these discussions and additional analyses in the revised manuscript.

While GLARE demonstrates strong performance and reasonable computational efficiency in our experiments, scaling to ultra-large chemical libraries (e.g., $10^{9}$ compounds) remains challenging. To address this, we have included additional experiments on a $10^8$-scale compound database, which demonstrate the method's scalability in large-scale settings.

Although our sensitivity analysis shows that GLARE is robust to changes in reward formulation and exploration heuristics, there may still be cases where performance is affected by unusual or out-of-distribution targets.

Finally, our current framework does not explicitly consider biological interpretability, which is important for real-world drug discovery applications. We recognize these as important directions for future research to further enhance the reliability and practical impact of our approach.

We appreciate the reviewer's careful review of our paper, positive feedback, and recognition of our work, particularly the effectiveness of our proposed method. Below, we address the concerns point by point:

W1: The Incremental Modification of Existing Reinforcement Learning Frameworks

R: Thank you for your valuable comments. We would like to clarify that our work is not a simple incremental modification of existing RL frameworks. Our study specifically addresses active learning in virtual screening, which poses unique challenges compared to standard RL tasks.

To our knowledge, we are the first to integrate RL (GRPO) with active learning and apply it to virtual screening. We innovatively model the virtual screening scenario as a Markov Decision Process (MDP) and combine it with active learning principles. This enables our method, GLARE, to learn how to select the most informative molecules adaptively. By coupling the RL agent's selection strategy with docking feedback in an active learning loop, our method, GLARE, efficiently prioritizes the most informative docking computations, offering a efficient and practical solution for real-world virtual screening workflows.

In addition, based on GRPO, we have substantially redesigned the framework. Specifically, we introduced a new reward function based on gradient uncertainty (Exploration Modification), specifically crafted to balance exploration and exploitation in this context, an aspect not addressed by existing methods. We hope this clarifies the novelty and significance of our contribution.

W2 & Q1: The Runtime of Training or Inference

R: Thank you for raising this important question. To provide a comprehensive comparison, we evaluated both training and inference runtimes on the ALDH1 dataset, using the same hyperparameters and hardware as TcsAL (see Appendix C.4).

Active learning typically involves training on a small number of labeled samples while making predictions on a much larger pool of unlabeled data. As a result, inference time becomes the dominant computational cost, especially in large-scale virtual screening scenarios. As shown in the table below, while our method requires additional training time due to the computational cost of gradient-based uncertainty in reward calculation, this overhead is confined to the training phase. During inference, reward computation is not required, enabling our method to achieve inference speeds comparable to TcsAL.

The characteristic is particularly important for large-scale dataset. To further demonstrate the scalability of our approach, we tested it on the AmpC dataset (~$10^8$ compounds) using the same hyperparameters as PtAL. As dataset size increases, inference time becomes even more significant. Our method maintains inference speed similar to baseline methods while delivering much higher accuracy, making it especially well-suited for ultra-large virtual screening tasks.

Q2. The Sufficiency of Iterations and Training Convergence in Table 2

R: Thank you for pointing out this question.

We closely follow the setting of the baseline PtAL, which adopts four and six active learning iterations with a fixed total annotation budget. Specifically, the total budget = (number of iterations) × (budget per iteration), and this total budget is kept the same across all methods for fair comparison. This practice is standard in the active learning literature and is widely considered sufficient for benchmarking enrichment factors. In our experiments, we observed that the model performance stabilizes after several iterations, and additional rounds yield diminishing returns. Thus, using four and six iterations is both representative and adequate for this benchmark.

Regarding training convergence, we ensured sufficient training within each active learning round. Following the TcsAL baseline, we trained each model for 50 epochs per iteration and monitored the training loss. As shown in the table below, the loss curves for MLP, GNN, and Pre. GNN all reach convergence within each iteration, confirming that our models are well-trained.

Q3: Active Learning Strategies in Table 2

R: Thank you for your question.

In the experiments summarized in Table 2, we adopted several active learning acquisition strategies, as detailed in Appendix C.2. Specifically, we used Greedy and Upper Confidence Bound (UCB) methods. The corresponding acquisition functions are as follows:

• Greedy: Selects samples with the highest predicted value: $\psi=\arg\mathrm{max}_{n}\left(\mathbb{E}\left(y|m\right)\right)$

• Uncertainty: Selects the most uncertain samples: $\psi=\arg\mathrm{max}_{n}(\mathbb{H}(y|m))$

• Mutual Information (MI): Selects samples with the highest mutual information: $\psi=\arg\mathrm{max}_{n}\left(\mathbb{H}(y|m)-\mathbb{E}_M\left[\mathbb{H}\left(y|m,\theta\right)\right]\right)$

• Upper Confidence Bound (UCB): Selects samples with the highest upper confidence bound: $\psi=\arg\mathrm{max}_{n}\left(\mathbb{E}\left(y|m\right)+\beta\cdot\mathbb{V}(y|m)\right)$

These strategies are standard in active learning and allow for a fair and comprehensive benchmark as presented in Table 2.