Text-to-graph Generation with Conditional Diffusion Models Guided by Graph-aligned LLMs

Thank you for your detailed review and thoughtful comments.

Please find our responses to your points below:

Q: The explanation of the claim (lines 17-18, 77-80).

A: Thank you for pointing out the inaccuracies in our phrasing. We have updated the wording in the revised version of the paper. By "human instructions," we refer to commands expressed in natural language, such as "Generate a graph with 5 cycles." While some methods provide conditional generation capabilities [1,2], they lack the ability to perform text-to-graph generation. These methods rely on the gradient of a separate prediction network to guide the diffusion sampling process, which does not scale well to more complex conditions, such as text descriptions. Specifically, (1) the prediction network often struggles to produce high-quality results, and (2) the gradient surface becomes highly complex, making the sampling process challenging to converge [3]. In contrast, our method employs a classifier-free guidance strategy, leveraging text features extracted through graph-aligned LLMs. This approach enables the model to accurately generate graphs based on complex textual instructions.

[1] DiGress: Discrete Denoising diffusion for graph generation. ICLR 2023.

[2] Exploring Chemical Space with Score-based Out-of-distribution Generation. ICML 2023.

[3] Classifier-Free Diffusion Guidance. NIPS 2021.

Q: Why LLMs are necessary to guide diffusion models.

A: Generating graphs from textual descriptions is a critical application with significant implications in fields such as drug discovery, code generation, and beyond. As large language models (LLMs) are currently the most effective models for processing text, their ability to understand and interpret text is central to this task. However, existing LLMs encounter challenges when it comes to understanding graph structures. To address this issue, we have designed a self-supervised text-graph alignment task to fine-tune LLMs, enabling them to extract graph-related features from text. This, in turn, allows the LLM to guide the graph diffusion model in generating graphs that are well-aligned with the provided textual descriptions. The ablation study in Table 4 further confirms that our fine-tuned LLM outperforms the original LLM in guiding the graph diffusion model for generation.

Q: Whether the proposed method uses Gaussian noise instead of the discrete noise used in DiGress.

A: We use Gaussian noise, as it is more general and applicable to both continuous and discrete features.

Q: Why would LLMs already align with graphs simply by counting edges/nodes?

A: Accurately counting the different types of nodes, edges, and subgraphs is crucial for understanding graph structures [4,5]. The ability of an LLM to perform this task suggests that it has partially reconstructed the entire graph internally (though not necessarily in a graph format). This implies that the large language model can be considered aligned with the graph. Ablation studies demonstrate that our fine-tuned LLM outperforms the original LLM in guiding the graph diffusion model for generation.

[4] Graph Self-Supervised Learning: A Survey. TKDE 2022.

[5] Self-Supervised Learning of Graph Neural Networks: A Unified Review. TPAMI 2022.

Q: Compare with diffusion models or other generative models designed for graph generation.

A: Other diffusion models and graph generation models are unable to generate graphs based on text instructions, so there is no direct baseline for comparison. While [6] claims to achieve this, it has not been evaluated on widely accepted task-specific metrics (such as FTS, FCD, etc.), and the code has not been made publicly available.

[6] 3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation

Q: There is no metric reflecting the model's ability to satisfy the requirements specified in the text.

A: In the synthetic dataset, we use accuracy as the metric, evaluating whether the generated graph aligns with the given text. For the molecular dataset, there isn't a direct equivalent metric. Instead, we assess the quality of the generated molecular graph by comparing various similarity measures (e.g., fingerprint-based molecule similarity metrics, FCD) between the generated graph and the real molecular graph to determine how well the generated result meets the textual requirements. Examples of molecular graphs generated from given text descriptions are provided in the appendix for reference.

(to be continued in the next comment)

I acknowledge that I've read the rebuttal. I also have read other reviews. The comments / responses may help the authors improve their work. However, the responses cannot address most of my concerns. I choose to keep my evaluation.

Thank you for your detailed review and valuable comments.

Please find our responses to your points below:

Q: The reason why graph-to-text generation was not included in our work.

A: In this paper, we focus on the text-to-graph generation problem, specifically leveraging a text-conditioned diffusion model for graph generation. As such, we do not address the graph-to-text generation problem, as it lies beyond the scope of our study. Diffusion-based methods are inherently one-directional, and existing diffusion-based approaches [1, 2] have exclusively explored the text-to-molecule task without addressing the molecule-to-text problem. Furthermore, it is important to highlight that text-to-graph generation is fundamentally different from the text-to-molecule generation task explored in [3]. We included molecule generation in our text-to-graph experiments because molecules can naturally be represented as graphs, thereby serving as a relevant test case for our method.

[1] Text-Guided Molecule Generation with Diffusion Language Model. AAAI 2024.

[2] 3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation. COLM 2024.

[3] Translation between Molecules and Natural Language. In 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022.

Q: Add both mean and standard deviation in the paper.

A: Thank you for your suggestion. We have added both the mean and standard deviation in the latest version of the paper that we uploaded.

The result of graph generation on the synthetic dataset.

The result of molecule generation guided by the text in the test split of CheBI-20.

Q: Computational complexity analysis

A: Our structure-aware cross-attention mechanism has the computational complexity of $O((N + L) d^2 + N^2 L d)$, while the standard cross-attention mechanism applied to both nodes and edges has the complexity of $O((N^2 + L) d^2 + N^2 L d)$, where $N$ is the number of nodes, $L$ is the number of text tokens, and $d$ is the feature dimension. The graph transformer architecture has the complexity $O(N^2 d^2)$ for each layer. Importantly, the complexity introduced by our structure-aware cross-attention module is non-dominating compared to the existing layers, ensuring computational efficiency.

Q: The typo in Equation 8.

A: Thank you for reading carefully, this was a typo and we have corrected it in the revised paper.

Thank you for your detailed review and thoughtful comments.

Please find our responses to your points below:

Q: More datasets.

A: ChEBI-20 is currently the most representative dataset for the text-to-molecule generation task, containing 33,010 molecule-description pairs. As demonstrated in works [1,3,5,6], this dataset is the most commonly used in related research. In addition to molecule generation, we have also developed a text-to-graph dataset that focuses exclusively on topological structures and conducted comprehensive experiments on it. Our method achieves superior generation performance compared to existing LLMs.

Q: The discussion of the related works about text to molecule generation method and text-to-graph diffusion models.

A: [1] and [2] are text-to-molecule methods that leverage language models for generation, where molecules are represented in the SMILES format. [3] employs a text diffusion model to generate the SMILES representation of molecules. While these methods focus on generating molecules from text representations, they do not extend to other graph generation tasks. [4] utilizes a latent space diffusion model to generate latent features, which are then decoded into molecule graphs using a graph decoder. In contrast, our work is the first to explore a general approach for text-to-graph generation. We have included these discussions and references in the related work section of the revised version of the paper.

[1] Unifying Molecular and Textual Representations via Multi-task Language Modelling. ICML 2023.

[2] Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models. ICLR 2024.

[3] Text-Guided Molecule Generation with Diffusion Language Model. AAAI 2024.

[4] 3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation. COLM 2024.

[5] MolT5: Translation between Molecules and Natural Language. EMNLP 2022.

[6] BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations. EMNLP 2023.

Q: The explanation of the experimental results.

A: The experimental results for MolT5 presented in this paper are based on MolT5-base, which has 220M parameters, rather than MolT5-small. As mentioned earlier, we use a graph diffusion model for molecule generation. Due to limitations in model architecture and data scale, there is an inherent semantic gap between graph-based models and text-based models. Nevertheless, leveraging the graph-aligned LLM and the structure-aware cross-attention mechanism we propose, our text-to-graph generation method achieves comparable performance to mainstream text-to-molecule methods in molecular semantic metrics on the molecule generation task. Furthermore, thanks to the inherent characteristics of the graph diffusion model, our method demonstrates superior validity and diversity in the generated results compared to the baseline language models, despite using a significantly smaller model size.

Q: Our novelty compared with existing text-to-graph diffusion models.

A: As noted earlier, existing works primarily focus on text-to-molecule tasks rather than the broader text-to-graph domain. Our method is the first to utilize a graph diffusion model for generating graph-structured data in a text-to-graph setting, introducing a general framework for text-to-graph generation. This framework integrates large language models (LLMs) with graph diffusion models. We propose a novel approach that includes self-supervised text-graph alignment to improve LLMs' comprehension of graph structures and a structure-aware cross-attention mechanism to guide diffusion models in generating diverse and relationally accurate graphs.

Thank you for your detailed review and valuable comments.

Please find our responses to your points below:

Q: The objectives used for pretraining the LLM.

A: As highlighted in [1,2], self-supervised tasks are known to enhance the performance of neural networks. In our work, tasks such as predicting the number of nodes, edges, or subgraphs help the LLM develop a graph-centric understanding of the input text, enabling the extracted features to effectively guide the graph generation process. The ablation study in Table 4 further confirms that our fine-tuned LLM outperforms the original LLM in guiding the graph diffusion model for generation. We appreciate your thoughtful suggestion regarding alternative pretraining objectives and will consider it for future exploration.

[1] Graph Self-Supervised Learning: A Survey. TKDE 2022.

[2] Self-Supervised Learning of Graph Neural Networks: A Unified Review. TPAMI 2022.

Q: Explanation of Equation 10 regarding the question about the matrix $A_{edge,uv}$.

A: Thank you for your careful review. We noticed a typo in Eq. (10) and have corrected it. The correct equation is $G_{2,uv} = 1 - G_{1,uv}$, which ensures that the result in Eq. (11) is properly normalized over the text tokens. We have updated this in the revised version of the paper. We appreciate your attention to detail.

Q: Providing a detailed explanation of the steps that follow the construction of these matrices

A: Our conditional graph diffusion model works on a tensor representation of graphs, where the node types $X$ and adjacency matrix $E$ are encoded into $X_{enc} \in \mathbb{R}^{N \times C_{node}}$ and $E_{enc} \in \mathbb{R}^{N \times N \times (C_{edge} + 1)}$ respectively using one-hot encoding. $X_{enc} = one\_hot(X), E_{enc} = one\_hot(E)$ To generate new graphs, we sample $X_{enc}$ and $E_{enc}$ using the trained graph diffusion model, and convert them into graphs by taking the argmax of $X$ and $E$ as the node types and adjacency matrix. $X = argmax(X_{enc}), E = argmax(E_{enc})$ As for $X_{cond}$ and $E_{cond,uv}$, they indicate the outputs of the structure-aware cross-attention mechanism, and they will be used in the score predictor of the graph diffusion model.

Q: Code or sufficient implementation details.

A: For the graph-aligned LLM, we use Llama-3-8B and fine-tune it on the graph structure prediction task using LoRA (rank=8, alpha=64) over 6 epochs. For the conditional graph diffusion model, we adopt the GDSS-Transformer implementation as the base architecture for the score predictor and integrate a structure-aware cross-attention module at every layer of the graph transformer. This modification enables text-conditional graph generation. The number of layers and feature channels remain consistent with those used in GDSS-Transformer. For molecule generation, the model is initialized with a pre-trained checkpoint on the ZINC250k dataset and subsequently fine-tuned with a learning rate of 2e-4 on the ChEBI-20 dataset for 1000 epochs. For the synthetic dataset, the model is trained with a learning rate of 1e-4 for 1000 epochs. Further details on the methodology and experimental settings, along with the associated codes and models, will be made publicly available on GitHub upon acceptance of the paper.

Q: More datasets.

A: ChEBI-20 is currently the most recognized dataset for the text-to-molecule generation task, containing 33,010 molecule-description pairs. As demonstrated in works [1,2,3,4], this is the only dataset used in similar studies. Beyond molecule generation, we also constructed a text-to-graph dataset focused solely on topological structures and conducted extensive experiments on it. Compared to current LLMs, our method achieves better generation performance.

[1] Unifying Molecular and Textual Representations via Multi-task Language Modelling. ICML 2023.

[2] Text-Guided Molecule Generation with Diffusion Language Model. AAAI 2024.

[3] MolT5: Translation between Molecules and Natural Language. EMNLP 2022.

[4] BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations. EMNLP 2023.

Q: The explanation of Equation 8.

A: The shape of $C$ is [number of tokens, text feature dim], and the shape of $X$ is [number of nodes, node feature dim]. We can select $W$ with appropriate dimensions so that $Q$ and $K$ have the same size in the second dimension, avoiding any dimension mismatch issues.

Q: The explanation of Equation 12.

A: The edge conditioning matrix is computed based on graphs generated during the diffusion process, which have Gaussian noise injected into their adjacency matrices, resulting in inherently dense graphs. This is a characteristic of score-based graph diffusion. Exploring alternative diffusion frameworks that can preserve graph sparsity is an interesting direction that we leave for future work.

Thank you authors for your rebuttal.

However, my concerns about the reproducibility of results and generalization to more real-world datasets remain. Therefore, I'd keep my score.