Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors

Responses to Reviewer nYSV

We thank the reviewer for your constructive feedback. Please find detailed responses below.

W1 The novelty of our MP-Adapter for parameter efficient tunning.

Based on your valuable feedback, we have provided a detailed description in the Global Response to clarify the specific design and considerations of our MP-Adapter for molecular representation fine-tuning. Additionally, we conducted experiments to empirically demonstrate the advantage of our method. We kindly suggest referring to our Global Response regarding this issue.

Q1 The reasons for adopting different strategies for embedding layers and message passing layers, as well as the results of the opposite strategies.

Our goal is to perform parameter-efficient tuning on pre-trained molecular encoders to address the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules.

• For message passing layers, the number of parameters is disproportionately large compared to the training samples. To mitigate this imbalance, we design a lightweight adapter targeted at the message passing layers, called MP-Adapter.
• Unlike message-passing layers, embedding layers have a very small number of parameters. Therefore, we treat embedding layers in a different way than message passing layers by not using adapters. Instead, we directly fine-tune the parameters of the embedding layers, but impose a constraint called Emb-BWC to limit the magnitude of parameter updates, preventing aggressive optimization and catastrophic forgetting.

Based on your constructive suggestions, we conducted experiments on the opposite design to further verify that the two strategies are suitable for their respective target components. Specifically, on the basis of our Pin-Tuning, we (Variant 1) changed the tuning strategy for the embedding layers to MP-Adapter, and (Variant 2) changed the tuning strategy for the message passing layers to Emb-BWC constraint.

Changing the tuning strategy of any component to the opposite on the basis of Pin-Tuning results in degraded performance, which indicates the suitability of the two tuning strategies for their respective target components. Furthermore, selecting the appropriate tuning strategy for the message passing layers has a greater impact on performance. Particularly on the MUV dataset, adapter-based parameter-efficient fine-tuning of the message passing layers is key to achieving success on this dataset. For the reasons behind the degraded performance of the variants, we provide the following insights:

• Reasons why the Emb-BWC is not suitable for message passing layers: First, the parameters of message passing layers are numerous and diverse in type. Therefore, applying the same weight constraint to numerous and diverse parameters cannot account for the differences in importance between parameters, making it less flexible than adapters. Second, in our method, adapters serve as carriers for introducing molecular context during fine-tuning, which cannot be achieved by directly updating the parameters. Overall, the Emb-BWC constraint is not flexible enough for tuning modules with a large number of parameters and cannot introduce additional molecular context information.
• Reasons why the MP-Adapter is not suitable for embedding layers: The embedding layers function as lookup tables that map raw features into vectorized features, and they inherently have a small number of parameters. The advantage of adapters lies in their ability to perform parameter-efficient tuning. However, for the embedding layers, the number of parameters to be tuned is comparable whether directly updating the embedding layers or updating them through an adapter. Overall, adapters are more suitable for modules with a large number of parameters, rather than for modules like the embedding layers with relatively few parameters.

Q2 Given that the simplest weight consolidation method (i.e., IM) is the best, does the BWC theory really required as it is just L2 penalty on the changed parameters?

Although the simplest identity matrix approximation of Emb-BWC achieved the best results, we also obtained the following important observations from this set of experiments. These observations reveal the intrinsic nature of the importance of parameters in pre-training and fine-tuning in molecular property prediction tasks, and can provide guidance on how to effectively fine-tune these parameters.

• Since the empirical results with three types of Emb-BWC regularizers are better than those without any regularizer, it indicates that imposing importance constraints on the parameters is beneficial for fine-tuning molecular representations.
• The principle of the BWC theory is to measure the importance of parameters during pre-training and use this importance to constrain the fine-tuning process. This approach helps in preserving the knowledge acquired during pre-training by preventing significant changes to the important parameters during fine-tuning. It is observed that the more relaxed the Emb-BWC constraint, the better the fine-tuning performance. The explanation is that the important parameters in pre-training and the parameters that need to be retained during fine-tuning do not completely overlap, and some mechanisms of message passing require considerable updates during the fine-tuning process.

Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors

Responses to Reviewer u7hY

We thank the reviewer for your constructive feedback. Please find detailed responses below.

W1 Q1 Add details on pilot experiment in Figure 1.

Thank you very much for your reply. We apologize for the lack of details about the pilot experiment due to the length limit of the paper. We further supplement the details of the comparison of GIN-Mol [1], CMPNN [2], and Graphormer [3] as follows.

• Featurization: To ensure a fair comparison with existing methods for few-shot molecular property prediction (FSMPP), we employed the same featurization approach as them. Following prior works of molecular representation learning [1] and few-shot molecular property prediction [4], we used two atomic features (atomic number and chiral tag) and two bond features (bond type and bond direction).
• Architecture Choice: GIN-Mol is a GIN tailored to molecules, proposed in Pre-GNN [1]. This molecular GIN has become a representative encoder in molecular pretraining, which is equipped with multiple bond embedding layers and introduces bond features into message passing. The standard 5-layer Mol-GIN is adopted. To facilitate interaction between node and directed edge information, CMPNN introduces a node-edge message communication module and offers multiple architectural options. We selected the multilayer perception (MLP) to implement the message communication module, since experiments with CMPNN have demonstrated that this architecture performs best. For Graphormer, we adopted the standard architecture based on multi-head attention, using node degrees, shortest distances between nodes, and edge types to construct the position encoding.
• Hyperparameters: We adopted the hyperparameters provided by Mol-GIN, CMPNN and Graphormer. Primarily, the size of the hidden dimension is 300 and the number of encoding layers is 5. For Graphormer, the number of heads is set to 8, and the dropout rate for attention is 0.1.
• Pre-training and Adaptation Strategy: To ensure a fair comparison, we followed prior FSMPP methods that use a pre-trained GIN as the molecular encoder. We pre-trained GIN-Mol, CMPNN, and Graphormer with both the supervised task and the Context Prediction self-supervised task [1], then adapted them to the FSMPP task through fine-tuning.

W2 Make a comparison in terms of tunable parameter size in Table 4.

Thank you for your valuable feedback. We appreciate your suggestion to provide additional details in Table 4 regarding the size of the tunable part of the model. In response to your comment, we have added a new row to Table 4 that specifies the size of the parameters tuned in our method, thereby better illustrating the advantages of our approach. The revised Table 4 is shown below, and is also provided in the Global Response PDF.

Q2 Inconsistency in the notion of the Emb--BWC regularizer between Figure 2 and Section 4.1.2.

Yes, $\mathcal{L}\_{update}$ in Figure 2 refers to $\mathcal{L}\_{Emb-BWC}$. Thank you for pointing out this mistake and helping us improve the clarity of the paper. We have revised this figure in the Global Response PDF.

References

[1] Strategies for Pre-training Graph Neural Networks. ICLR 2020

[2] Communicative Representation Learning on Attributed Molecular Graphs. IJCAI 2020

[3] Do Transformers Really Perform Bad for Graph Representation? NeurIPS 2021

[4] Property-Aware Relation Networks for Few-Shot Molecular Property Prediction. NeurIPS 2021

Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors

Responses to Reviewer 1DKF

We thank the reviewer for your constructive feedback. Please find detailed responses below.

W1 Q2 Compared to adapters in NLP, the specific design and advantages of the proposed MP-Adapter.

Based on your valuable feedback, we have provided a detailed description in the Global Response to clarify the specific design and considerations of our MP-Adapter for molecular representation fine-tuning. Additionally, we conducted experiments to compare the performance of fine-tuning molecular encoders using NLP adapters and our MP-Adapter, empirically demonstrating the advantages of our method. We kindly suggest referring to our Global Response regarding this issue.

W2 Discussion on the experimental observations of Emb-BWC.

Thank you for your insightful comments. This is a very meaningful question that deserves in-depth exploration. As we stated in Section 5.3, the results indicate that keeping pre-trained parameters to some extent can better utilize pre-trained knowledge, but the parameters worth keeping in fine-tuning and the important parameters in pre-training revealed by the Fisher information matrix are not completely consistent. Here, based on our experimental results, we provide a more in-depth discussion and our insights from the following two aspects:

1. Importance of each parameter: This involves two questions: (a) Is it necessary to impose constraints on the importance of parameters? (b) What kind of importance assignment strategy is optimal for fine-tuning molecular representations? For the first question, since the empirical results with three types of regularizers are better than those without any regularizer, it indicates that imposing importance constraints on the parameters is beneficial for fine-tuning molecular representations. For the second question, the observation is that the more relaxed the Emb-BWC constraint, the better the fine-tuning performance. The Emb-BWC constraint measures the importance of parameters in pre-training and uses this importance to constrain the fine-tuning process. The explanation of this observation is that the important parameters in pre-training and the parameters that need to be retained during fine-tuning do not completely overlap, and some mechanisms of message passing require considerable updates during the fine-tuning process.
2. Correlation between different parameters: Since the computation of the Hessian matrix in the original form of the Emb-BWC regularizer is intractable, we provide three diagonal approximation methods. Since all three approximation methods result in diagonal matrices, with non-zero values only on the main diagonal, they all assume that the contribution of each parameter update to the model performance is independent. In other words, these three diagonal approximation methods imply that importance is assigned to each parameter independently, streamlining the correlations between parameters. The off-diagonal values of the Hessian matrix can constrain the joint updates of parameters, but calculating these off-diagonal elements, whether through the original form or the approximated Fisher information matrix, is intractable due to the high-dimensional nature of the parameters. Therefore, the better performance of the identity matrix approximation does not imply that the correlations between parameters are unimportant in molecular property prediction. Instead, it reflects that parameters which are not important during pre-training may have high importance during fine-tuning.

Q1 Is $a$ a typo for the weight $\lambda$ of the Emb-BWC regularizer in Section 5.4?

Yes, the $a$ in the sensitivity analysis should be the regularization coefficient $\lambda$. This is a typo, and we greatly appreciate you pointing it out.

Q3 How to ensure the quality and reliability of the context information?

Since the molecular context is encoded based on the labels of the molecules in the target task and auxiliary tasks, the accuracy and adequacy of these labels determine the quality and reliability of the contextual information. When the required contextual labels are complete and accurately measured, the context is sufficiently reliable. However, the real situation is often not ideal, with some missing and noisy labels.

The GNN-based context encoder we adopted has already mitigated the issues of missing and noisy labels to some extent by propagating and smoothing information on the molecule-property bipartite graph, thereby obtaining context representations that are robust to missing and noisy labels.

To further improve the reliability of the context, our potential solution is to rigorously measure and calibrate the uncertainty of the contextual labels, such as measuring the entropy of the contextual label matrix. This could potentially further enhance the robustness of our method.

Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors

Responses to Reviewer W6tN (3/3)

W-S2 W-C1 The contribution of parameter-efficient tuning and context encoding.

In the above response, we have clarified the novelty of our parameter-efficient tuning and in-context tuning. Here, we further clarify their contributions to performance improvement. Both parameter-efficient tuning and context encoding can contribute to performance improvement. Specifically, on the MUV dataset, fine-tuning the message passing layers with an adapter plays a decisive role, having a greater impact than context, as observed from the experimental results in our Global Response. Therefore, the influence of parameter-efficient tuning and context encoding on performance improvement depends on the data distribution, and each may significantly contribute on different datasets.

W-C2 W-C3 W-C4 Suggestions for improving the quality and clarity of the presentation.

Thank you for your suggestions to improve the presentation of the paper. We have made the following efforts:

1. We have added a description of the context encoder in Figure 2. The revised Figure 2 is provided in the Global Response PDF.
2. We have performed language editing and proofreading to enhance the clarity and readability of the paper.
3. We have standardized the use of mathematical notation and formulas. We replaced notation like "$ContextEncoder(\cdot)$" with letter-based notation to make the usage more standard and consistent.

References

[1] Cross-Domain Few-Shot Learning by Representation Fusion. 2020

[2] Low Data Drug Discovery with One-Shot Learning. ACS Central Science 2017

[3] Property-Aware Relation Networks for Few-Shot Molecular Property Prediction. NeurIPS 2021

[4] Context-Enriched Molecule Representations Improve Few-Shot Drug Discovery. ICLR 2023

Responses to Reviewer W6tN (2/3)

W-Q1 W-S1 Experiments on the FS-Mol benchmark.

Thank you for your constructive suggestions. We have added experimental results on the FS-Mol benchmark. Note that the data distribution of FS-Mol is significantly different from that of another import benchmark MoleculeNet:

1. FS-Mol only includes properties related to the biological activity of small molecules against protein targets, unlike MoleculeNet, which encompasses a variety of property types such as toxicity, solubility, blood-brain barrier penetration, and side effects.
2. FS-Mol contains over 5,000 tasks with 233,786 unique compounds, and 489,133 measurements have been conducted. Although FS-Mol covers a wide range of molecules and properties, the number of molecules measured for each property is relatively small, and there is little overlap between the molecules covered by different properties. Therefore, it is difficult to provide effective context between different properties on the original FS-Mol dataset.

To support our claim regarding the sparse context in FS-Mol, we conducted a statistical analysis of the distribution in the original FS-Mol dataset:

• The number of properties measured for each molecule

• The number of molecules commonly measured between two properties

Since the original FS-Mol is not suitable for encoding and perceiving molecule context based on labels, we designed two evaluation settings.

Evaluation Setting 1: We evaluated our parameter-efficient tuning method under the standard FS-Mol evaluation setting, without considering molecular context. Based on PAR [3], which is the state-of-the-art baseline model irrelevant to the context from auxiliary tasks, we tested the results of introducing MP-Adapter and Emb-BWC. We tested different support set sizes of 16, 32, and 64. We run 10 times with different seeds and report the average ΔAUPRC.

Evaluation Setting 2: To better evaluate the effectiveness of our context encoding and in-context tuning on the FS-Mol dataset, we selected two subsets of data from FS-Mol with dense context information to construct two sub-datasets and evaluated them using the standard N-way K-Shot setting. The FS-Mol-6K dataset contains 15 properties and over 6,000 molecules, while the FS-Mol-24K dataset contains 128 properties and over 24,000 molecules. Experiments were conducted under 10-shot and 5-shot settings with 10 different seeds, using the average AUC-ROC score as the evaluation metric. These two constructed sub-datasets have been uploaded to our anonymous code repository.

From the evaluation results of the two settings, our parameter-efficient tuning method outperforms the baseline methods on the FS-Mol benchmark, regardless of whether molecular context is considered.

Responses to Reviewer W6tN (1/3)

We thank the reviewer for your constructive feedback. Please find detailed responses below.

W-O1 W-Q2 The novelty of our adapter-based parameter-efficient tuning.

We greatly appreciate your comments. Indeed, there are several adapter-like methods used to fine-tune pre-trained models in various fields. Unlike other adapters, our MP-Adapter is specifically designed for molecular encoders based on graph neural networks (GNNs) that follow the message passing mechanism. Here, we compare our MP-Adapter with two potentially similar methods to highlight the novelty of our approach: (1) adapters used in natural language processing (NLP) for transformer-based language models, and (2) the representation fuser CHEF proposed in previous few-shot drug discovery work [1] designed for fully-connected deep neural networks (FCNs).

1. The comparison between our MP-Adapter and adapters used in NLP: This comparison has been provided in our Global Response to clarify the specific design and considerations of our MP-Adapter for molecular representation fine-tuning. Additionally, we conducted experiments to compare the performance of fine-tuning molecular encoders using NLP adapters and our MP-Adapter, empirically demonstrating the advantages of our method. We kindly suggest referring to our Global Response regarding this issue.

2. The comparison between our MP-Adapter and the representation fuser CHEF [1]: CHEF fuses representations from different layers by attaching a learnable learner after each frozen pre-trained layer and ensembling the outputs of all learners. A critical difference between CHEF and our MP-Adapter is that in CHEF, the output of each learner is fed into the final fuser for ensembling rather than being fed to the next layer of the encoder. Therefore, in the CHEF framework, updating the output of the previous layer through the added learner does not affect the output of subsequent layers. This approach is only suitable for global encoding models, such as the FCNs used in the experiments evaluating CHEF, which take molecular fingerprints (ECFP6) as input [1]. However, the current state-of-the-art molecular encoders are based on GNNs and take molecular graphs as input. GNN-based molecular encoders follow a message passing mechanism to achieve localized neighborhood information aggregation. In this case, the interaction between different layers of the encoder is more important, and changes in the output of the previous layer will significantly affect the message passing of the next layer. Therefore, passing the adapter's output to the next encoding layer as input, as our MP-Adapter does, is more appropriate for GNN-based molecular encoders.

Overall, neither NLP adapters nor the representation fuser CHEF can be trivially used to fine-tune GNN-based molecular encoders. We have also summarized the comparison of them in the table below to clearly illustrate the advantages and novelty of our MP-Adapter:

W-O2 W-Q2 Novelty of in-context fine-tuning.

Although some previous works have also mentioned the concept of molecular context, our work differs in the definition, encoding method, and perceiving method of molecular context.

• Definition of molecular context: In IterRefLSTM [2] and PAR [3], the context refers to the structural and property labels of support molecules concerning the target property, while in MHNfs [4], the context refers to molecules that are structurally similar to the given molecule among a large number of unlabeled molecules. Unlike these methods, the molecular context in our approach refers to the label information of seen and unseen molecules in auxiliary properties, which are informative and more relevant to the target task.
• Encoding of molecular context: Previous works [2,3,4] typically use LSTM or attention mechanisms to learn the correlations between molecules, often neglecting the interactive relationship between molecules and properties. We represent the molecular context as a molecule-property bipartite graph and encode it using a GNN-based context encoder, which allows for more effective and robust learning of the relationships between molecules and properties.
• Perceiving of molecular context: Previous works [2,3,4] combine non-contextual molecular representations with context representations to predict molecular properties. In this paradigm, the resulting molecular representations cannot perceive the molecular context information. In our method, the molecular context is introduced into the fine-tuning process through the MP-Adapter, allowing the molecular encoder to be fine-tuned under the guidance of the context, thereby obtaining contextual molecular representations.

Adapter / CHEF discussion:

• GNNs / MLPs:

However, the current state-of-the-art molecular encoders are based on GNNs and take molecular graphs as input.

This is a minor point but still worth mentioning: I'd disagree here if it was meant in the way that GNNs are SOTA and MLPs are not. As far as I know, generally and merged across multiple bioactivity datasets, both can be considered SOTA. Also see [1] which combine GNNs with MLPs to reach SOTA performance.

• CHEF for GNNs:

This approach is only suitable for global encoding models, such as the FCNs used in the experiments evaluating CHEF, which take molecular fingerprints (ECFP6) as input

I do think CHEF suits well for GNNs. The core idea of CHEF is to use both low-level and higher-level features for domain adaptation. This fits very well with GNNs, as representations retrieved from very early layers can be interpreted as very local features (i.e. a low-level feature), while representations from later layers capter larger parts of the molecular graph (i.e. a higher-level feature).

• This is why CHEF can be used to fine-tune GNN-based molecular encoders

neither NLP adapters nor the representation fuser CHEF can be trivially used to fine-tune GNN-based molecular encoders.

• Novelty: The way label information is included is novel. Also, I appreciate the authors' reasoning that the model might benefit from feeding the adapter's output into the next layer. However, the authors missed to include experiments which show that their claim really holds.

Context discussion:

I appreciate the authors' discussion here and I mostly agree. (The MHNfs' representations could be considered as contextual molecular representations also because the enrichment step is analogous to a LLM which uses information from a large context-window to update token representations). The authors' main contribution is their context-guided fine-tuning, while context means both structural and label information. This is valuable to the community. However, both the quality of the manuscript as well as the value for the community heavily depend on an extensive discussion about similarities and differences in how different approaches use context. Reading answer 3/3, I am not convinced the authors addressed this enough in their current version of the manuscript.

FS-Mol experiment:

• An experiment with support set size of 16 is not a 16-shot experiment. A support set size of 16 means that the total number of available labeled samples for tasks during inference is 16, e.g. 6 actives and 10 inactives. For the FS-Mol benchmark experiment the support set is created using a stratified split, compare [2].
• The reported $\Delta$AUPRC values are very high. Recently published SOTA methods are typically reported with performance values between 0.2 and 0.3 (*) for the support set sizes 16 and 32. Considering that a random classifier achieves AUCPR values around 0.5, the reported AUCPR values indicate close-to-perfect classifiers. Given this significant performance gap, and considering that PAR has already been evaluated on FS-Mol with $\sim\Delta$AUPRC 0.17 [3], an explanation of why the evaluated models performed so well in the included experiment would be valuable.

Summary:

I appreciate the work the authors put into the rebuttal, I see the potential of this work, and I generally would evaluate this work to be interesting for the community. However, I also think this manuscript would benefit from another review round (see adapter discussion, context discussion, and FS-Mol discussion) since some improvements still seem crucial to eventually end with a manuscript of high quality. For this reason, I believe it is too early to publish this work at this conference, and I therefore stick to my initial rating.

[1] Yang, Kevin, et al. "Analyzing learned molecular representations for property prediction." Journal of chemical information and modeling 59.8 (2019): 3370-3388.

[2] Stanley, Megan, et al. "Fs-mol: A few-shot learning dataset of molecules." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.

[3] Context-Enriched Molecule Representations Improve Few-Shot Drug Discovery. ICLR 2023

What we intended to convey is that for many molecular encoders with GNN backbones, our MP-Adapter suits their message passing mechanisms

Understood!

we have added experiments to fine-tune the GNN-based encoder using CHEF. [...] Their performance lags behind that of our MP-Adapter because they do not integrate molecular context

This improves the quality of the manuscript. The reasoning w.r.t. the molecular context makes sense.

Context modeling in few-shot molecular property prediction

Thank you for showing this passage. This resolves my concerns about the context discussion.

FS-Mol experiment:

• The performance values seem reasonable now.
• The authors are able to show that their approach helps to boost PAR
• I still think there are weaknesses in the FS-Mol experiment:
  • PAR under-performs - compare performance reported in [6] with more suitable hyperparameters (differences are not that big though)
  • All presented variants are outperformed by the Frequent Hitter model [6] which is a baseline which is not aware of any support set samples and simply learns average activity across tasks. Another backbone model - perhaps a GIN-encoder based ProtoNet or Neural Similarity Search version - might have been the better choice.

Overall, I'd evaluate this paper to be a borderline paper now. Since I think the authors' manuscript has improved a lot and since the presented way of guiding fine-tuning by including molecular context is interesting, I'd adapt my score and slightly vote for acceptance.