MIPT: Multilevel Informed Prompt Tuning for Robust Molecular Property Prediction

Dear reviewer dDZn:

Thank you for your valuable and constructive comments. We have revised the paper according to your suggestions.

Experimental Designs Or Analyses

1. Why choose LoRA: To learn the features of both node-level and graph-level, we used multi-level fine-tuning, but in order not to increase the training cost, we chose lightweight LoRA. Although there are alternative methods such as Adapter tuning [1] and BitFit [2], previous study [3] have shown that LoRA achieves the best balance between efficiency and performance.

[1] Parameter-Efficient Transfer Learning for NLP.

[2] Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.

[3] LoRA: Low-Rank Adaptation of Large Language Models.

2. Not all datasets were evaluated: We apologized for the mistake in title. Due to space, we did not include all the data, which we will add to the appendix. Here we supplement the performance results of other datasets.

Table. Ablation analysis of different configurations on Tox21, Toxcast, HIV, and MUV datasets based on RUC-AOC(%).

3. The ablation experiment does not include the random noise mask. Two parts of our approach mention noise. The random node mask discussed in the method is used to augment the original features, which are included in the GIP and the ablation experiment (MGIP). The noise penalty mechanism, aimed at reducing the uncertainty of the method and increasing confidence, has been applied in the ablation analysis (NPP).

We appreciate the reviewer for spending time reviewing our paper and offering valuable suggestions. If you have any further questions, please tell us and we are willing to address your concerns.

Dear reviewer Cxne:

Thank you for your valuable and constructive comments. We have revised the paper according to your suggestions.

W1: LoRA, as a parameter-efficient fine-tuning method, has been widely adopted in the LLM domain. The manuscript should more clearly clarify LoRA's unique advantages in molecular property prediction tasks and how it has been specifically optimized for molecular graph data.

Our work is the first to adopt LoRA for fine-tuning GNNs in molecular property prediction. In NLP, LoRA is typically employed to cut down training costs. However, our goal is to construct graph prompts that capture the relationships between graph structure and node features, thereby optimizing LoRA specifically for molecular graph data while maintaining parameter efficiency.

W2: Different from Pin-tuning. Pin-tuning is designed to be based on adaptor fine-tuning strategy for few-shot learning. Unlike Pin-Tuning, our method bridges pre-training and fine-tuning via multilevel graph prompts that align node/graph structures with task requirements. Additionally, we incorporate a noise penalty to mitigate mismatches between pretrained representations and downstream tasks.

W3: What is the role of $\epsilon^{(k)}$ in Eq. (5)? In Eq. (5), $\epsilon^{(k)}$ is a learnable parameter that modulates the contribution of the node's own features relative to its neighbors during the update at the $k$-th layer. Essentially, it scales the self-feature term, allowing the network to adaptively balance the importance of a node's current representation with the aggregated information from its neighbors.

W4: In Table 2, the performance improvement on some datasets is not significant. Additional analysis should be provided.

We acknowledge that the performance improvement of our method is modest on some datasets. We are aware that public molecular datasets are limited, and different SOTA methods tackle these challenges in various ways. For instance, Uni-Mol incorporates 3D information, while InstructMol uses pseudo-labels to enhance model confidence. In contrast, our approach mainly aims to bridge the gap between pre-training and fine-tuning. Although our method may not achieve SOTA performance on every molecular property, the enhancements it brings in transferability and robustness are still of great significance.

W5: Algorithm proofreading. We have thoroughly proofread and revised Algorithm 1 to ensure its clarity and correctness.

W6: The manuscript does not provide the code. We will make the code publicly available after the submission is published.

Minor issues. For typos, we’ve corrected error and updated the description in the paper.

We appreciate the reviewer for spending time reviewing our paper and offering valuable suggestions. If you have any further questions, please tell us and we are willing to address your concerns.

Dear reviewer kVKq:

Thank you for your valuable and constructive comments. We have revised the paper according to your suggestions.

Weakness

W1: In this paper, only the benchmark of classification tasks is studied, and it is necessary to further increase the generality of other types of tasks to illustrate tasks, such as regression tasks.

In our paper, while the primary focus was on benchmark classification tasks, we recognize the importance of demonstrating the generality of our approach across other types of tasks, such as regression. We extended our experiments to regression tasks on Table I. The supplementary results indicate that our method maintains its superiority even in the regression setting.

Table I. Comparison of the performance of different tuning strategies on regression tasks based on MAE.

W2: The SOTA performance in Table 2 is not available on all datasets. Thank you for pointing this out. Indeed, in molecular representation learning, the scope and design of different models can vary significantly, often incorporating specialized features or data. For example, UniMol leverages 3D information, and InstructMol adopts pseudo-labeling strategies to enhance confidence. While these SOTA methods focus on addressing certain limitations in molecular representation, our primary goal is to bridge the gap between pre-training and fine-tuning. We believe that even if not all molecular properties achieve SOTA performance, they still hold significant value.

W3: The author does not have open source code. We will make the code publicly available after the submission is published.

W4: The two LoRA module in Sec.4.1 and Sec.4.2 should provide more details on how they differ. We emphasis that the node-level LoRA is used for the node features of the adaptive learning molecule, and the LoRA on the graph-level features is a variant of LoRA, which is used as a prompt for adaptive learning of graph-level features.

W5: More benchmarks about Graph Prompt-based Methods. We have conducted a comprehensive comparison with existing graph prompt benchmarks. As shown in the Table II, our method outperforms both GPPT and GraphPrompt on several key datasets, particularly on BBBP, Toxcast, SIDER, HIV, and BACE.

Table II. Comparison of benchmarks about graph prompt for the models pre-trained by Edge Prediction based on RUC-AOC(%).

W6: The LoRA have already been used in many tasks, such as NLP tasks. The technical novelty of this paper is limited.

We agree that LoRA is an excellent and widely-used technique in many domains, including NLP. However, our paper does not claim novelty for the LoRA component itself. Instead, our main contribution lies in multilevel graph informed prompt and noise penalization mechanisms. This combination uniquely bridges the gap between pre-training and fine-tuning in molecular graph learning.

Questions

Q1: What is the difference between $L_{NPP}$ and $L_{cls}$ in Figure 1?

$L_{cls}$ (Eq. 14) is the standard classification loss (binary cross-entropy loss) used to train the model on the downstream task. In contrast, $L_{NPP}$ (Eq. 18) adds a Noise Penalty Mechanism to $L_{cls}$.

Q2: The author emphasizes the multi-level graph prompt, essentially using the LoRA module twice, so the increase in performance could be due to an increase in the amount of training parameters?

Table III. The number of tunable parameters for different tuning strategies.

While it is true that our method employs the LoRA module at multiple levels, the increase in performance cannot be solely attributed to a higher number of tunable parameters. We had provided the parametric analysis in tab. 5 in the appendix. Although our approach uses more parameters than GPF/GPF-plus, it is still far below FT, which utilizes 1.92M-2.14M parameters yet performs worse. This clearly indicates that our performance gains stem from the unique integration of the multilevel graph-informed prompt and the noise penalization mechanism, which effectively bridges the gap between pre-training and fine-tuning in molecular graph learning.

Dear reviewer Swha:

Thanks for your positive review of our paper and for your thoughtful comments.

For W1 model versatility.

• Thank you for your positive feedback on our approach. We appreciate your recognition of the potential our MIPT framework for other graph-level applications.

Below is a comparison of the performance based on RMSE of different tuning strategies on regression tasks.

• In our initial research, we focus on molecular property prediction, where MIPT demonstrated exceptional performance. To explore its broader applicability, we extended our method to regression tasks according to your constructive comments. The experimental results indicate that our approach maintains superior performance in these tasks. We remain committed to further investigating and validating the framework's performance across a wider range of graph-level tasks.

For W2 model complexity. Since fine-tuning time is often difficult to compare fairly due to differences in hardware and system environment, we analyze the training costs based on the number of tunable parameters.

Table I. The number of tunable parameters for different tuning strategies. * is the frozen parameters.

• FT requires the highest number of tunable parameters (1.92M-2.14M), leading to the highest training cost.
• GPF and GPF-plus significantly reduce the number of tunable parameters by freezing the GNN encoder. This makes them more parameter-efficient compared to FT.
• Our method also freezes the GNN encoder but introduces a larger prompt size (0.115M). Despite this, the total number of tunable parameters (0.175M-0.295M) remains significantly lower than FT, suggesting a balance between efficiency and effectiveness.

In summary, ​GPF and GPF-plus are the most parameter-efficient​, while ​our approach achieves a middle ground​, tuning more parameters than GPF but significantly fewer than FT, likely leading to a moderate training cost.

For Q3. the datasize of fine-tune: In our study, we use the entire training set during the fine-tuning phase. This approach, which fine-tunes only a subset of model parameters rather than the entire network (see our discussion in ​Q2​), is a common practice in this field. Prior works [1,2,3,4] also employ the full training dataset for model fine-tuning, ensuring that the model benefits from as much labeled information as possible.

We appreciate the reviewer for spending time reviewing our paper and offering valuable suggestions. If you have any further questions, please tell us and we are willing to address your concerns.