PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction

Q1. Lack of comparison to SOTA. In Tab. 1 and Tab. 2, the only previous works on few-shot MMP included are ADKF-IFT and PAR. The authors should compare the proposed methods with more existing SOTAs such as [a].

Reply. We compare PACIA with ADKF-IFT and PAR, which are recent baselines obtaining the best performance on MoleculeNet and FS-Mol respectively. We also mention MHNfs (Schimunek et al., 2023), which is not included as it uses additional reference molecules from external datasets. For reference [a], could you please provide the paper title?

Q2. Lack of novelty. The core component is HyperNet. Based on HyperNet, the proposed PACIA makes no significant technical contribution.

Reply. Hypernetworks are a general concept involving neural networks that are trained to generate parameters for a main network. Designing how a hypernetwork modulates the main network is a problem-specific and non-trivial challenge. One cannot directly apply hypernetworks designed for other problems to a new problem. Considering the few-shot MPP problem, existing works can be summarized within an encoder-predictor framework, where GNNs perform effectively, acting as both encoders and predictors. Correspondingly, we have designed a unified GNN adapter that generates a few adaptive parameters. These parameters modulate the message passing process of the GNN in two key aspects: node embedding and propagation depth. As a result, the encoder adapts at the property level, while the predictor adapts at the molecule level. Our proposed PACIA effectively addresses the few-shot MPP problem, achieving state-of-the-art performance on benchmark datasets. In conclusion, PACIA contributes significantly to both the technological and empirical advancements in the field of few-shot MPP.

Q3. Poor writing. Many sentences are grammatically wrong.

Reply. We sincerely apologize for the grammatical issues. We have now carefully proofread the manuscript and marked the grammar corrections in the revised version.

Q4. The design in modulating propagation depth seems not fully reasonable. The design in modulating propagation depth seems not fully reasonable. According to Eqn. (8), $[p]_l$ measures the probability of the event that "the $l$-th layer is in the model". However, from Eqn. (7), the only constraint on $[p]_l$ is $\sum_l [p]_l=1$. So it is likely that for some $1<i<j<L$, $[p]_i<[p]_j$. That is to say, a hidden layer (i.e., layer $j$) is more likely to be in the model than a layer before it (i.e. layer $i$), which is unreasonable.

Reply. This is a misunderstanding: the relative magnitude between $[p]_i$ and $[p]_j$ is independent of the order between i and j.

For any input sample $\mathcal{X}_{\tau,q}$, it will always go through a complete message passing procedure in a $L$-layer GNN. Afterwards, for each layer $l\in1,2\cdots,L$, hypernetwork takes the node embeddings after $l$-th layer (i.e., {$**h**^l$}) as input, and generates a scalar $[p]_l$ representing how likely it can get good performance by using {$**h**^l$} as the final node embeddings.
During meta-training, $\{[p]_l\}$ act as differentiable combination weights in Equation (8). In this way, the hypernetwork is trained to learn the optimal mapping from {$**h**^l$} to $[p]_l$, through gradient flow among Equation (12)-(3)-(5)-(8)-(10) or Equation (12)-(3)-(5)-(8)-(11). During meta-testing, we directly select one optimal layer $l'$ by Equation (13). Then, only {$**h**^{l'}$} of $l'$th layer are fed to next module.

Q1. Comprehensive Comparison with Baselines: Could the authors provide additional comparisons with a broader range of existing methods, especially those that have shown promising results in related domains, to strengthen the validation of PACIA’s performance?

Reply. Please check Q2 in global response.

Q2. Analysis of Hypernetworks: How do different configurations or architectures of hypernetworks affect the performance of PACIA? Are there specific settings that are more optimal for this application?

Reply. For specific settings, please refer to Table 4 and Table 5 for hyperparameters chosen on MoleculeNet and FS-Mol, where the configuration of hypernetwork is included.

For ablation studies, we now add results concerning three aspects in hypernetwork in Appendix C.4, which are reproduced below.

1.Effect of concatenating label $**y**_{{\tau},s}$ in Equation (9). Table below shows the testing ROC-AUC obtained on SIDER. As shown, "w/ Label" helps keep the label information in support set, which improves the performance.

2.Different ways of combining active prototype $**r**^l_{\tau,+}$ and inactive prototype $**r**_{{\tau},-}^l$ in Equation (10) and Equation (11). As "Concatenating" active and inactive prototypes allows MLP to capture more complex patterns, it obtains better performance on SIDER as shown in the table below.

3.Effect of layers of MLP in hypernetwork. Table below shows the testing ROC-AUC obtained on SIDER. Here, we constrain that the MLPs in Equation (9-11) have the same layer number. As shown, using 3 layers reaches the best performance. Please note that although we can set different layer numbers for MLPs used in Equation (9-11) which further improves performance, setting the same layer number already obtains the state-of-the-art performance. Hence, we set layer number as 3 consistently.

Q1. The approaches of modulating node embedding and GNN depth do not have sufficient theoretical support. It is more like experimental attempts. Can the authors provide more details about why the implementation is designed as such? How does such implementation ensure the adaptor learns sufficient information?

Reply. Our PACIA is designed upon hypernetworks. It is proved that hypernetworks can be more expressive and can reach the same error with lower complexity (number of trainable parameters) than embedding-based methods in general [a1]. A line of amortization-based approaches (Requeima et al., 2019; Lin et al., 2021; Przewiezlikowski et al., 2022) have shown the effectiveness of designing hypernetworks to handle few-shot image classification and cold-start recommendation. We share the same spirit. While theoretically analyzing a particular hypernetwork is still an open question, we empirically analyze the contribution of each component in PACIA in Section 5.3, and provide evidence of property-level adaptation and molecule-level adaptation in Section 5.4.

[a1] Galanti, T. and Wolf, L. (2020). On the modularity of hypernetworks. Advances in Neural Information Processing Systems, 33, 10409-10419.

Q2. The main framework of PACIA is based on PAR, making the technical novelty incremental.

Reply. Please note that we propose PACIA to enhance the encoder-predictor framework, which is adopted by existing works, with a parameter-efficient GNN adapter, rather than merely aiming to improve PAR. As discussed in Section 3.2, existing works take GNN as molecular encoder consistently, and predict by pair-wise similarity (Altae-Tran et al., 2017), multi-layer perceptron (Guo et al., 2021), Mahalanobis distance (Stanley et al., 2021), and GNN operates on relation graph (Wang et al., 2021). In particular, PACIA learns to generate a few adaptive parameters to modulate the message passing process of GNN. If it is only applied on GNN encoder, i.e., w/o M variant in Section 5.3, it obtains 83.62 and 82.01 for 10-shot and 1-shot tasks of Tox21. These results already outperform the best results obtained by existing methods, i.e., 82.43 obtained by ADKF-IFT and 80.02 obtained by PAR for 10-shot and 1-shot tasks respectively. Further, as the recent PAR obtains the state-of-the-art performance by learning a GNN on relation graphs, we can apply our adapter to modulate this GNN predictor on a molecular level. The benefit is that molecular-level difference can be captured. Empirical results in Figure 2(a) and case study in Section 5.4 validates that applying molecule-level adaptation is useful.

Q3. The figure font is small and hard to recognize. Fig.1 (b) is too abstract. The authors may consider plotting a more detailed overall framework to help understand their method.

Reply. We are sorry for the inconvenience. We have revised Fig.1 (b) and added a more detailed one in Appendix D.

Q4. In Table 2, why the baseline methods are different? PAR is the most similar baseline model, and should be compared.

Reply. Please check Q1 in global response.

Q5. I am also curious that will it works on general MPP problems? Did the authors try to see the performance not under the few-shot setting?

Reply. Please check Q2 in global response.

Q1. The exposition on the methodology appears somewhat nebulous, which hampers a clear comprehension of the distinct contributions of each module. Specifically, the average representation at the l-th GNN layer, as depicted in Equation (9), seems disconnected from subsequent steps. The property adaptation and molecule adaptation are not clear in the algorithm.

Reply. We apologize for the confusion. We have revised Generating Adaptive Parameters by Hypernetwork paragraph in Section 4.1 to make it clearer. Property-level adaptation is generated by Equation (9) and Equation (10), and molecule-level adaptation is generated by Equation (9) and Equation (11). In Equation (9), we obtain class prototypes $**r**\_{{\tau},+}^l$ and $**r**\_{{\tau},-}^l$ by mean-pooling over the embeddings of samples in active class (+) and inactive class (-) from $\mathcal{S}\_{\tau}$. Consequently, the generated adaptive parameter are permutation-invariant to the order of input samples in $\mathcal{S}_{\tau}$.

Q2. There is a noticeable discrepancy in the baselines used for comparison in Tables 1 and 2. The rationale behind this difference remains unexplained. For instance, while PAR[1] is conspicuously absent from Table 2, it seems like a plausible candidate for few-shot molecular property prediction in FS-Mol.

Reply. Please check Q1 in global response.

Q3. The presentation of results in Tables 1 and 2 would benefit from a consistent format, ensuring ease of interpretation for readers.

Reply. Thanks for the advice. We now use a consistent format for Table 1 and Table 2 in the revised manuscript.