Extracting Training Data from Molecular Pre-trained Models

Response to Reviewer qKJK.

We greatly appreciate reviewer qKJK’s insightful feedback and critical comments to help us refine our work.

W1&4: Although the authors made a scenario to clarify the problem, it is not realistic ...; The proposed method should be evaluated in the regression tasks ...

Thanks for your comment. We first clarify that our research is centered on model-sharing collaboration for molecular data. In this context, sharing a graph representation learning model is more realistic than sharing classification or regression models due to several reasons. On one hand, from the perspective of data owner, sharing a classification/regression model poses a higher privacy risk due to the potential leakage of confidential label information, such as properties of molecules. These labels are highly valuable and sensitive, as highlighted by your insightful comment, making data owners hesitant to share models trained on such confidential data. Under such consideration, research efforts have proposed collaborative approaches based on sharing graph representation learning models [1,7]. On the other hand, from the model user's standpoint, there is often a need for a general pre-trained model to facilitate various downstream tasks, e.g., property prediction, drug-target interactions and etc. Under this consideration, since it is impractical for data owner to share different classification/regression models for different tasks, sharing a general representation learning model presents more beneficial. Previous efforts also highlight the necessity for such a general pre-trained representation learning model [2, 3].

We also agree with you that in some target-specific collaboration cases, classification/regression model is shared. Therefore, as suggested, we further explored data extraction attacks on regression models. When integrating regression tasks, we adapted our model by replacing the final output of the regression with the representation in Eq (2). In our implementation, we chose the real-world chemical dataset FreeSolv [4] regressed the hydration free energy, utilizing 5% of the molecules from FreeSolv as an auxiliary dataset. The detailed results are as follows. We discovered that, our model still performs well with regression model.

W2: If the authors claim the importance of the training data privacy in the molecular representation learning tasks, I cannot agree on ...

We would like to clarify that privacy protection of molecular structures is also significant. Take a real-world example, consider the MELLODDY (MachinE Learning Ledger Orchestration for Drug Discovery) project, where 10 pharmaceutical companies collaborated to train a model for structure-activity. These companies had substantial privacy concerns regarding their molecule structure, treating them as confidential information that cannot be shared [6]. In addition, previous research has demonstrated that molecular structure often involves sensitive intellectual property information and must be controlled by the owning company at all times [4]. Moreover, information on the structural similarity between partners’ compounds is also considered sensitive [5]. These underscore the privacy issue of molecular structures.

We also thank the reviewer reminding us that the labels for molecular data, such as target physical and chemical properties, are also private. So we extend our model's application to share classification/regression model that is trained on labels (see our response to W1). We will include these discussions in the revised version and explore this aspect further in the future. Thank you for providing this important perspective again.

W3: The score function in Eq. (1) is trivial...

Thank you for your valuable and insightful feedback. Here, we illustrate the rationale behind the scoring function from a theoretical perspective.

Assume that graph $G$ is composed by $G:=G_{1} \cup G_{2}$ and graph pre-trained model is represented by $f$. Let the loss function for the pre-training task be $\mathcal{L}$, which takes graph representation as input. Further, we assume that $\mathcal{L}$ is a bijection and linear mapping. Without loss of generality, we assume that loss function belongs to the category of weighted sum, that is $\mathcal{L}(f(G)) = \alpha_1 \mathcal{L}(f(G_1)) + \alpha_2 \mathcal{L}(f(G_2))$, with $\alpha_1$ and $\alpha_2$ serve as hyper-parameters (This assumption is common among various tasks. For instance, in the most common case of cross-entropy for classification task, $\alpha_1 = |G_1|/|G|$ and $\alpha_2 = |G_2|/|G|$.). We can infer that $f(G)=\mathcal{L}^{-1}(\alpha_1 \mathcal{L}(f(G_1)) + \alpha_2 \mathcal{L}(f(G_2)))=\alpha_{1}f(G_{1})+\alpha_{2}f(G_{2})$.

Given the theoretical analysis, we can observe that the relationship between $f(G)$, $f(G_{1})$, and $f(G_{2})$ is akin to a weighted combination, which justifies our design of score function.

Reference:

[1] Xie, Han, et al. "Federated graph classification over non-iid graphs." NeurIPS, 2021.

[2] Hu, Weihua, et al. "Strategies for pre-training graph neural networks." ICLR, 2019.

[3] Xia, Jun, et al. "A systematic survey of chemical pre-trained models." IJCAI, 2023.

[4] Mobley, David L., and J. Peter Guthrie. "FreeSolv: a database of experimental and calculated hydration free energies, with input files." 2014.

[5] Simm, Jaak, et al. “Splitting chemical structure data sets for federated privacy-preserving machine learning.”, 2021.

[6] MELLODDY: Machine Learning Ledger Orchestration for Drug Discovery. https://www.melloddy.eu/.

[7] Tan, Yue, et al. "Federated learning on non-iid graphs via structural knowledge sharing." 2023.

Dear Reviewer qKJK,

Thanks again for your detailed review and questions. We have provided detailed answers to them in the rebuttal. As we are approaching the end of the discussion phase, we would like to kindly ask if you still have any unanswered questions about our paper?

Best, Authors

Thank you for the careful response.

However, as a researcher in chemical science, I still do not agree with the scenario of this work. Researchers and engineers in academic and industrial chemistry want to build a classification or regression model for their molecular datasets because they need to skip time-consuming chemical experiments to observe the physical and chemical properties of target molecules.

The graph representation of the molecule is just intermediate information for classification and regression in most real-world chemical applications. Furthermore, as I mentioned, many large molecular databases containing extensive 2D and 3D molecular structures are publicly accessible, such as PubChem, ChEMBL, and QM. For this reason, training data for molecular representation learning is not expensive and private.

The authors need to handle the validity of their problem definition before developing the method.

Response to Reviewer o6iJ.

We sincerely thank reviewer o6iJ for detailed feedback and we address the reviewer’s concerns as follows.

Q1: The experiments might not fully capture the diversity and complexity of real-world datasets, which could affect the generalizability of the findings. I recommend conducting additional experiments using a wider variety of datasets to better evaluate the model's performance in real-world scenarios.

Thank you for your valuable suggestions. We have further evaluated our approach on other real-world datasets. We utilized the PubChem dataset from [1]. We sampled 20,000 molecules from it to serve as an auxiliary dataset and employed the other molecules as the pre-training dataset. The specific results are as follows and the performance on the PubChem dataset indicates the broad applicability of our method.

Q2: The scoring function is based on G, which is a linear combination of the representations of R and M. However, considering that the target model is treated as a black-box model, it is unclear whether there is a linear relationship between the representations of R and M obtained by the black-box model and the representation of G. To clarify this point, I suggest providing a thorough analysis of the relationship between the representations, as well as discussing any assumptions made in the scoring function.

Thank you for your valuable and insightful feedback. Here, we illustrate the rationale behind the scoring function from a theoretical perspective.

Assume that graph $G$ is composed by $G:=G_{1} \cup G_{2}$ and graph pre-trained model is represented by $f$. Let the loss function for the pre-training task be $\mathcal{L}$, which takes graph representation as input. Further, we assume that $\mathcal{L}$ is a bijection and linear mapping. Without loss of generality, we assume that loss function belongs to the category of weighted sum, that is $\mathcal{L}(f(G)) = \alpha_1 \mathcal{L}(f(G_1)) + \alpha_2 \mathcal{L}(f(G_2))$, with $\alpha_1$ and $\alpha_2$ serve as hyper-parameters (This assumption is common among various tasks. For instance, in the most common case of cross-entropy for classification task, $\alpha_1 = |G_1|/|G|$ and $\alpha_2 = |G_2|/|G|$.). We can infer that $f(G)=\mathcal{L}^{-1}(\alpha_1 \mathcal{L}(f(G_1)) + \alpha_2 \mathcal{L}(f(G_2)))=\alpha_{1}f(G_{1})+\alpha_{2}f(G_{2})$.

Given the theoretical analysis, we can observe that the relationship between $f(G)$, $f(G_{1})$, and $f(G_{2})$ is akin to a weighted combination, which justifies our design of score function.

Reference:

[1] Y. Wang, J. Wang, et al. "MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks." Nat. Mach. Intell., 2022.

Dear Reviewer o6iJ,

Thanks again for your detailed review and questions. We have provided detailed answers to them in the rebuttal. As we are approaching the end of the discussion phase, we would like to kindly ask if you still have any unanswered questions about our paper?

Best, Authors

Response to Reviewer JnnP.

We greatly appreciate reviewer JnnP for the time and effort you have dedicated to reviewing our paper. We address the reviewer's concerns as follows:

Q1: Regarding the selection of template structures, the paper ultimately chooses rings as the starting structures for subsequent generation/growth. However, a proportion of molecular data does not contain rings, so this heuristic choice of starting templates might not be the best strategy. The authors could consider constructing a template bank to avoid this issue.

Thanks for your valuable suggestion. We first clarify rationality behind choosing ring structures as starting template is that the ring structures are very common in chemical datasets. In the ZINC15 dataset with 2 million unlabeled molecules, we identified 1,990,890 molecules containing ring structures, constituting the majority of the dataset. Furthermore, the diversity of ring structures offers a wide range of options for template structures (such as tetrahydrofuran and cyclobutane). Therefore, we have implemented a template bank that incorporates the 84 most frequently occurring rings from the auxiliary dataset.

We also appreciate your suggestions to explore template structures that do not contain rings. We further incorporate five ring-free scaffold templates (i.e., alkanes) into the template bank. The performance are shown as follows. It can be observed that incorporating these ring-free templates has led to slight improvement in performance.

Q2: Are the 20k molecules used in the experiments on line 269 also obtained from ZINC15? Is this a strong iid assumption? I would like to see the effect of data extraction when the auxiliary dataset is replaced with molecules having significantly different distributions, for example, selecting some molecules from the PubChem dataset.

We first clarify that 20k molecules employed for the auxiliary dataset were also sourced from ZINC15. In Appendix A.3, we have illustrated the overlap between the pre-training dataset and the auxiliary dataset. We found only 103 identical molecules, indicating a relatively small overlap ratio, which suggests it does not strongly adhere to the IID assumption.

Besides, following the reviewer's insightful suggestion, we sampled 20,000 molecules from PubChem [1] as auxiliary dataset to ensure a difference from the ZINC pre-training dataset. The results based on GraphCL pre-trained model are shown as follows. We observed that under these scenarios, performance of our method have a slight decline. However, it still demonstrates comparable efficacy and showcases the robustness, and this observation aligns with what we presented in Appendix A.4.

Q3: The meaning of "step" in Table 1 seems unclear. Is it referring to the time-step mentioned earlier? If so, why only explore the case of one and two steps? What characteristics would molecules generated with more steps have? This seems to lack discussion and analysis.

Apologize for not being clear and appreciate for providing an opportunity to clarify. The "step" mentioned here refers to the time-step as understood by the reviewer. The reason for selecting only one and two steps is that other baseline methods have difficulty managing molecular structures beyond two steps, due to the increase in complexity. Consequently, we confined our comparisons to just one and two steps.

However, our approach based on reinforcement learning can be extended to multiple steps. Here, we extend the time-step, and results are as follows (we only take "Random" as baseline considering the runtime of other baselines). It can be observed that as the time-step increases, performance may decline due to the increased difficulty of extraction caused by the complexity of the molecules. Nevertheless, our model still outperforms the baseline model in precision, indicating the effectiveness of our extraction method.

Table:Precision of molecular extraction results among different time step

Q4: The author explores few model backbones, so I suggest the authors also conduct experimental evaluation on Graph Transformer or other architectures. And the pertaining data set is also limited, which would be more convincing if another one dataset was added.

Sorry for not explaining clearly. In our experiments, we have employed various encoder architectures for molecular pre-trained models. The Grover graph pre-trained model included in Table 1 is based on the GTransformer [2], a Graph Transformer. The performance demonstrates the effectiveness of our model across various pre-trained model architectures.

As for the limitation of pre-training dataset, we further introduced a new dataset. We employed the PubChem dataset used in [1]. We sampled 20k molecules from the PubChem dataset to serve as the auxiliary dataset and employed the other molecules as the pre-training dataset. The specific results are as follows and the performance on the PubChem dataset indicates the broad applicability of our method.

Q5: A key metric in the experimental evaluation is Precision. How do you determine whether the generated molecules exist in the dataset? Since molecules lack unique identifiers and it is easy to consider two identical molecules as different mistakenly, I hope the authors can provide clearer details on how the presence of data is judged.

Sorry for not explaining it clearly. We utilized the RDKit library to determine molecular identity [3]. Specifically, it assesses whether there exists an atomic mapping such that each atom in the query molecule can be paired with a corresponding atom in the target molecule, with the same connectivity as in the query. Detailed explanation will be included in the revised version.

Reference:

[1] Y. Wang, J. Wang, et al. "MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks." Nat. Mach. Intell., 2022.

[2] Rong, Yu, et al. "Self-supervised graph transformer on large-scale molecular data." NeurIPS, 2020.

[3] Landrum, Greg. "RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling." 2013.

Dear Reviewer JnnP,

Thanks again for your detailed review and questions. We have provided detailed answers to them in the rebuttal. As we are approaching the end of the discussion phase, we would like to kindly ask if you still have any unanswered questions about our paper?

Best, Authors