Form follows Function: Text-to-Text Conditional Graph Generation based on Functional Requirements

[Reviewer] Is it possible to include additional analysis for Table 3, since sometimes baselines perform better?

[Authors] What additional analysis would you like to see in Table 3? Although sometimes baselines perform better in Table 3, our main focus is on generating examples which meet functional requirements. The results in table 2 suggest that our proposed approaches always outperform baselines in terms of generating examples which more closely meet functional requirements.

The language in the paper needs to be further polished.

Given this comment, are there are any specific areas of the paper you would like us to focus on? Or possibly a specific way we explain things that you think could be improved?

[Reviewer] The simple autoregressive model can outperform the proposed message passing by increasing its training set.

[Authors] In many cases, the more data provided to a model, the less inductive bias (i.e. adding MP into an LLM) required to improve performance. But in most applications of data driven modelling, the amount of data available is a limiting factor. So even if two methods are perform comparably with enough data, the method that is more data efficient is preferred because it can be used in more real world situations. Given that experiments suggests our proposed method with MP is more efficient, we view the proposed method with MP as a contribution even if it has comparable performance to a model without MP with more data.

[Reviewer] BLOOM is not a suitable baseline for molecular generation since it is not trained on molecular data. Galactica (Taylor et al., 2022) would be better since its training data includes smile strings.

[Authors] While the experiments for this paper focus on the application of molecule generation, the focus of the paper itself is not a new method for performing molecule generation. The focus is a new method for generating graphs through serialized graph generation that match conditional functional requirements. Why do you feel that BLOOM as a language model is not suitable for the general problem of generating graphs through serialized graph generation that match conditional functional requirements?

[Reviewer] The idea of message passing for molecular graphs is incremental. For example, Klicpera et al., show message passing is important for molecular graphs, although their message-passing function is different from this paper. The idea of fine-tuning objectives is also an engineering trick rather than a model contribution.

[Authors] It seems that we could improve the way we described our technical contributions in the paper. Our goal was to propose using an LLM to generate graphs and interleaving message passing layers in between LLM layers to improve the LLMs performance. To the best of our knowledge this is a novel proposal. We propose fine tuning as part of our method as we found it is the only way to get a model to output parsable serialized graphs, not because we view it as a technical contribution. The contribution with respect to fine-tuning in our opinion is the novel training objective we propose which is not an engineering trick by a modification of the standard autoregressive training objective that takes into account the structure of our data. How could we make our technical contributions clearer so that it does not seem like we are proposing message passing for molecule graphs?

[Reviewer] Moreover, why the results on QED and Valency are inconsistent on 400k examples?

[Authors] They are inconsistent because they are not the same data set, the two data sets have different labels (QED vs Valency) which are not equally easy to learn to conditionally generate good examples from. Given that they are different data sets, we cannot be sure whether we should expect consistency between the two of them. Would it make the paper clearer if we separate the results of the two different data sets into different tables?

[Reviewer] It seems that interleaving with MP is the only technical contribution, however, when the number of fine-tuning examples increases, the model without MP can achieve much better performance (half MAE on QED), then what will happen if we fine-tune the model on 400k examples with MP? is it possible that w/ MP and w/o MP are comparable when there are enough fine-tuning examples?

[Authors] In many cases, the more data provided to a model, the less inductive bias (i.e. adding MP into an LLM) required to improve performance. But in most applications of data driven modelling, the amount of data available is a limiting factor. So even if two methods are perform comparably with enough data, the method that is more data efficient is preferred because it can be used in more real world situations. Given that experiments suggests our proposed method with MP is more efficient, we view the proposed method with MP as a contribution even if it has comparable performance to a model without MP with more data.

[Reviewer] (1) the technical contribution is not clear to me.

[Authors] We attempted to highlight the technical contribution of the paper in the introduction with the bold-faced sentence. In more detail, we believe there are four technical contributions;

• A novel training objective that significantly improves performance of models on the proposed task
• A novel serialization method that can reversibly serialize any graph
• Interleaving MP layers in between LLM layers so that an LLM can take into account graph structure during generation
• An experimental design to evaluate a model's ability to generate graphs that meets requested functional requirements

Do you agree with these contributions? If yes, how do you think we could highlight them more clearly? If not, which do you think is not a contribution?

[Reviewer] Baselines are also inadequately explored, and the results are far from convincing.

[Authors] Can you also provide more detail about why you feel baselines are inadequately explored?

[Reviewer] The technical contribution is also limited. The primary contribution emphasized by the authors involves the use of causal masks during generation. However, the employment of causal masks for autoregressive generation is just a trick to ensure efficient training. It's weird to label this well-known technical detail as the key contribution.

[Authors] We politely disagree with the claim that causal masking is just a trick to ensure efficient training. In fact the time complexity of causal masking is the same as no masking. In addition, a model trained without causal masking will perform very poorly when generating text because without masking the model has access to the next token it is trying to predict during training, so it does not learn anything. Novel attention masking methods have been published in ICLR, ICML and NeurIPS before, so masking methods are considered a contribution. The masking method we are proposing allows the model to incorporate graph structure into the inference process which is a contribution because to the best of our knowledge it has not been proposed before.

[Reviewer] Additionally, the evaluation dataset and metrics lack soundness, as they are derived from existing ones that were not originally designed for the newly proposed task. The authors should thoroughly address the suitability of the datasets and metrics.

[Authors] The evaluation data sets were modified to fit the newly proposed task using an open source software that allows us to calculate functional properties of molecules. Could you provide us with more detail what elements of the modified data set or the metrics we employed were not well-suited to the task we were evaluating models on?

[Reviewer] The authors claim to introduce a "novel problem setting," but fail to provide a clear problem definition.

[Authors] Could you provide us with more detail about what is unclear in our problem definition?

[Reviewer] The task appears to be ill-defined. The authors claim to introduce a "novel problem setting," but fail to provide a clear problem definition. It remains unclear what the key challenges of the proposed property-to-graph generation task are. Additionally, the evaluation dataset and metrics lack soundness, as they are derived from existing ones that were not originally designed for the newly proposed task. The authors should thoroughly address the suitability of the datasets and metrics. Baselines are also inadequately explored, and the results are far from convincing.

[Authors] Your comments suggest that there is a lot of room for us to improve our paper and we would like to work with you to learn how we can improve our work.

Dear authors,

1. I agree that masking methods are worth being published definitely; can you highlight the novelty of the proposed masking method in this paper? If it's causal masking as pointed out by the reviewer, then it's hard for readers to appreciate its novelty. Can you elaborate?

2. In order for the reviewers and myself to take a more detailed look, can you point to the place where the problem is defined in the paper?

Best, AC

Hi AC,

Generating a molecular graph based on properties is not an entirely novel task. Previous works [1,2], while not converting all inputs into textual sequences, can still be applicable in this context. This paper only compares their approach with two T5-based knowledge graph generation methods. Comparing with methods not specifically designed for this domain raises concerns about the persuasiveness of the comparison.

[1] Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. "Junction tree variational autoencoder for molecular graph generation." International conference on machine learning. PMLR, 2018. [2] Popova, Mariya, et al. "MolecularRNN: Generating realistic molecular graphs with optimized properties." arXiv preprint arXiv:1905.13372 (2019).

Best, reviewer fDhs

[Reviewer] As per Figure 1, is the reason that graph masking includes a self node (node i attends to node i) the same reason that causal masking has a token attending to itself? Is there any specific reason not to exclude that self node?

[Authors] We think there is a conceptual reason which is not very strong and an experimental design reason which is strong. The conceptual reason is that if the self node is excluded, so is its representation from the new representation calculated by the attention layer. So some information about the node itself is excluded from the new representation. This argument is weakened by the fact that most message passing and attention layers contain skip connections, but still holds water. The experimental design reason is that all prior work includes the self node, so excluding it would be another element of the method we would have to justify. However the inclusion of a self node in the attention or message passing layers is not directly related to the problem this paper is addressing, so it would not strengthen the paper to include it.

[Reviewer] Baseline without fine-tuning: Under Appendix D, how was the SSG-LLM w/out fine-tuning prompted? Was there any investigation done using in-context learning / few-shot prompting to get a parsable generation?

[Authors] SGG-LLM w/out fine-tuning used the same prompts as with fine tuning, which requested a molecule with a specific functional property. Please note, models that are performant at in-context learning like GPT-4 or Llama-2 were too large to fine tune given cost and hardware constraints. However, qualitative experiments were performed to see if it was possible for GPT-4 to generate molecules with a requested number of valence electrons and it was only able to produce molecules which that consisted of a single atom. We will do our best to extend the results to include performant in-context learning LLMs for the camera ready if given the opportunity.

[Reviewer] The paper does not explain the reasoning for formulating the training objective as it is in Eqn 1. This is surprising given that experiments in Appendix D show a difference when using the differing term, but no satisfactory reason was given for why this might be the case.

[Authors] Thank you for pointing out the lack of clarity in our explanation. Upon reflection, it would have been best to introduce Eqn 2 first (the standard autoregressive training objective), then explain its deficiency with respect to the structure of the data and then introduce Eqn 1 (how we are proposing to train the model) as a objective which does taken into the structure of the data. This could make it clearer the reason for formulating Eqn 1. Instead we introduce Eqn 1 and then Eqn 2 and explain the difference between them with respect to the structure of our data. Do you think this approach would improve the clarity of the objective?

[Reviewer] With reference to Table 3, why was parsability affected when using message passing?

[Authors] Since sampling is a stochastic process and the difference in parsability is less than or equal 0.2% in all cases, it is hard to tell if message passing had any impact or if it was an artifact of the sampling process. All methods generated at least 998 /1000 parsable graphs. Even SGG-LLM w/o message passing did not have a parsability of 100% on one of the data sets, so a model without message passing also did not achieve 100% parsability.

[Reviewer] Why was BLOOM selected as the LLM backbone?

[Authors] Prior work on serialized graph generation used the LLM T5, we found this model performed very poorly on the task of molecule generation. So we tried using BLOOM in experiments and it had good performance.

[Reviewer] The feature vector for a node vector was selected as the feature vector of the last element describing that node. Were there any experiments done to vary this, for e.g. mean-pooling?

[Authors] No experiments were performed for other aggregation operations, but we chose this method because state-of-the-art sentence embedding techniques use the same approach. We leave it as future work to determine if there is a more performant aggregation method.