InstructG2I: Synthesizing Images from Multimodal Attributed Graphs

Thank you so much for your thoughtful review!

Regarding your questions:

1. The scenario the authors discuss in lines 28-30 seems like it could be well-handled by only text. We would like to answer this question from three aspects. 1) On one hand, the problem introduced in this paper can be grounded not only in the virtual artwork scenario but also in others such as the e-commerce scenario, where generating an image for a product node connected to other products equates to recommending future products. Such scenarios are hard to handle by text only (e.g., user interests are implicitly expressed in their purchase history which is hidden in the graph structure). 2) On the other hand, even in the artwork scenario, for the artists who are not that famous (e.g., my little brother), it is hard to use text to represent them; and for the artists who are new and not seen during model training, it is hard to expect the model can be generalized through text (artist names). However, using graph structure as the condition, we can well solve the not famous artist and new artist problem by discovering neighbors in the graph. We add an example of controllable generation for a node connecting to “Pablo Picasso” and my little brother on the artwork graph to illustrate this in the rebuttal PDF (in general response). 3) Another advantage of adopting graph conditions is that we can combine multiple graph conditions (e.g., different art styles) with controllable generation (with mixed ratio according to the user’s interest), which is extremely hard to express by text only.

2. Add a reference to the recent [1]. Thank you for bringing this related work. We would like to mention that the reason why we do not reference this paper [1] in our submission is that it is put on arxiv later than the neurips submission deadline. However, as the reviewer mentioned and we agreed, this is a related paper and we would like to reference it in our revision.

3. Question about the notations, especially P, D, and V. We agree with the reviewer that since this paper tackles multimodal learning on graphs, the notations will be a little different and complex compared with conventional graph literature. As mentioned in the paper, each node $v_i\in V$ is associated with a text $d_{v_i}\in D$ and an image $p_{v_i}\in D$. In other words, $d_{v_i}$ and $p_{v_i}$ can be understood as features for $v_i$ from the conventional graph learning perspective. We will make it clear in the revision.

4. Perhaps the authors could consider adopting a similar approach [2] if it makes sense in this task. Thank you for bringing up this great paper and we are glad to reference it in the revision. At the same time, we would like to answer this question from three aspects: 1) The design of Graph-QFormer is non-trivial: as we mentioned in the paper, in MMAGs, we need to tackle the graph entity dependency (including image-image dependency and text-image dependency) to conduct image generation. To extract such dependency, we design Graph-QFormer and show consistently better performance than other designs in Table 2; 2) Different task with different backbone models: we agree with the reviewer that LLaGA is a great work, but we would like to emphasize that the problem tackled in LLaGA (text generation on TAGs with LLM) is different from that in this paper (image generation on MMAGs with stable diffusion). Since the problem and the backbones are different, it is non-trivial to directly compare these two methods; 3) We have compared with simple GNN tokenization methods in the ablation study (Table 2) and demonstrate the effectiveness of Graph-QFormer compared with them.

5. Comment: the notation on pages 3-5 is quite heavy and would benefit from a symbol table. We will add a symbol table in either the main content or the appendix in the revision according to your suggestion.

6. Learnable importance of multiple edge types similar to [3]. We agree with the reviewer that learnable sampling is a promising future direction. As we are the first paper to introduce this problem, we would like to tackle this problem in a simple and effective way. As a result, we would like to leave this more complex learnable sampling for future studies.

7. How graph conditions are introduced to InstructPix2Pix and SD? We compare our method with InstructPix2Pix and SD with an advanced design for graph information in Table 2. For InstructPix2Pix, we aggregate the neighboring images on the graph with mean pooling and serve the resulting image as the input image condition. For SD, we concatenate the text information from the center nodes’ neighbors on the graph to its original text as text condition. From Table 2, our method outperforms both baselines significantly which demonstrates the effectiveness of the InstructG2I design.

8. A quantitative understanding of semantic PPR-based neighbor sampling approach. We are glad to add some quantitative results (DINOv2 score) on ART500K and Amazon datasets as below:

From the result, we can see that both PPR-based sampling and semantic-based reranking can benefit the InstructG2I. This is demonstrated in both quantitative and qualitative results (Figure 5). The reason we adopt semantic PPR-based sampling rather than attention-based selection and others is that semantic PPR-based sampling can be performed offline only once and it is computationally efficient compared with SD training and inference. This enables us to scale to large-scale graphs in the real world. However, we agree with the reviewer that attention-based selection and others are interesting directions and we would like to leave them for future research.

Dear authors -- thank you for your response to my concerns. I think adding this discussion to the work will help strengthen the positioning (especially the parts around necessity/utility of graph conditioning vs. text). Overall, I stand by my initial review that parts of the work feel a bit heuristic and incremental, but the work is generally an interesting proposal which leans towards a different application of graph structure than most are used to seeing. I will retain my score.

Thank you so much for your thoughtful review!

Regarding your questions:

1. The description in Eq.10 may be incorrect. Thank you so much for your comment. We have found the typos and will correct them in the revision.

2. Descriptions of symbols in subsection 3.4. This section mainly discusses controllable generation with classifier-free guidance (CFG) on graphs. The main philosophy of CFG is that $\epsilon_\theta$ learns the gradient of the log image distribution, and increasing the contribution of $\epsilon_\theta(c,z_t)-\epsilon_\theta(z_t)$ will enhance convergence towards the distribution conditioned on $c$. We agree with the review that the description of the symbols can be improved to make this section easier to understand. We plan to add detailed illustrations to symbols including $\varnothing$, $z_t$, $s^{(k)}_G$, and others. We would be grateful if the reviewer could provide us with inputs on what other symbols need further explanation and we are glad to make this section clearer in the revision.

3. Why do GNNs underperform the straightforward approach in Eq.7? Thank you for the comments. The philosophy of vanilla GNN is to propagate and aggregate information from neighbors and compress the neighboring information into one embedding for the center node, while the straightforward approach in Eq.7 will directly provide neighboring information separately without aggregation or compression. It is worth noting that the aggregation/compression step could lead to neighbor information loss, while the straightforward approach in Eq.7 does not. Since more information is provided to the stable diffusion model, the latter can perform better for image generation. However, we believe that future work can consider a more advanced design of GNN methods that could keep all the neighboring information and extract the common knowledge for more accurate image generation on graphs.

4. The images sampled by the semantic PPR-based method appear to have the same image as the ground truth. Does this indicate any label leakage? We are sorry for the confusion. This is a typo and we will correct it in our revision. We have written data processing codes to ensure that there is no label or data leakage in both training and testing. To demonstrate this, we have uploaded the data processing code here: https://anonymous.4open.science/r/Graph2Image-submit-607E/art500k_dp.ipynb

5. Comparison with other SOTA multimodal LLMs. Thank you for your comments. We would like to argue that the philosophy of introducing graph information into image generation can be not only adopted to the diffusion model (in our paper) but also to any multimodal LLM backbones as mentioned by the reviewer. However, we agree with the reviewer that adding the comparison can make the experiment more comprehensive. As a result, we compare with a recent advanced multimodal LLM called DreamLLM [1] and show the results below:

The image-image CLIP score:

The image-image DINOv2 score:

From the results, we can find that our method can outperform the advanced DreamLLM consistently, which demonstrates the effectiveness of our model design.

[1] DreamLLM: Synergistic Multimodal Comprehension and Creation. ICLR 2024.

Dear Reviewer nWct,

We are glad that we address your raised issues and we are grateful that you increase the score!

We will further enhance the submission according to your suggestions.

Best,

Authors

Thank you so much for your thoughtful review and support of our work!

Regarding your questions:

1. How large graphs can the method be applied?

Thank you so much for your question. Our method can be adapted to large-scale graphs with millions or even trillions of nodes. In InstructG2I, we only need to conduct offline sampling with semantic PPR-based neighbor sampling once to extract useful structure information from the large-scale graphs. Since this step is performed offline, the size of the graph will not introduce sampling bottlenecks to stable diffusion model training and inference. Empirically, the semantic PPR-based neighbor sampling only takes about 10 minutes on a graph with millions of nodes, which is quite short compared to stable diffusion training and inference time cost.

2. Why is the DINOv2 score on the Goodreads dataset lower than the others?

Thank you for your great observation. 1) Training data size: As shown in Table 3, the graph size of Goodreads is the smallest compared with ART500K and Amazon. This means that the training data on Goodreads is smaller than the other two datasets. Since a larger training data size can contribute to more sufficient training, the results on ART500K and Amazon are better than the results on Goodreads. 2) Data distribution: In ART500K, Amazon, and Goodreads datasets, the images are art pictures, product pictures, and book cover pictures respectively (some samples can be found in the rebuttal PDF in general response). We believe that compared with book cover pictures, product and art pictures are closer to the distribution of images used in stable diffusion (which is our base model) pretraining. In conclusion, training InstructG2I on ART500K and Amazon will provide better performance than training InstructG2I on the Goodreads dataset.

Thank you so much for your thoughtful review!

Regarding your questions:

1. Question about the problem setting.

Thank you for your comment. We would like to answer this from two aspects.

Why graph is important?

1. Graph structure helps discover multiple informative neighbors: We agree with the reviewer that Graph-QFormer is to transfer “neighbor images” into graph prompt tokens for center node generation. However, how to select such “neighbor images” is important and non-trivial, which needs help from graph structure. As shown in Figure 5, the “neighbor images” condition affects the generation results a lot, and our proposed semantic PPR-based sampling can discover informative neighbors based on graph information for high-quality image generation.

2. We utilize global graph structure and semantics: We would like to clarify that in our method, we do not “only use subgraph structures” but utilize global graph structure as well as text semantics for neighbor sampling. In semantic PPR-based sampling, we first adopt PPR on the global graph to discover informative neighbors based on graph structure and then adopt semantic search to find more fine-grained informative neighbors (e.g., if we would like to generate a picture of a “horse”, the neighbors related to “horse” will be more useful, as shown in Figure 5). In addition to the qualitative result, we also show the quantitative analysis of semantic PPR-based sampling, where we can find that utilizing global graph structure (Ours in the table below) outperforms utilizing only subgraph structures (- PPR in the table below), which also demonstrates the importance of graph structure information.

3. Not just one, but many and extract their similarity from graph: Graph enables discovering multiple related neighbor images and extracting their similarity rather than solely based on one image to conduct image generation (which is widely adopted in the literature). This is important for image generation in many scenarios. For example, we would like to generate a “bird” picture drawn by “Monet”. If we only conditioned on his picture of a “scenery”, we may overfit the corresponding content and fail to draw a “bird”. However, if we conditioned on a multiple of his images covering diverse content including animals, the model would better extract his style and successfully conduct image generation.

Why are image or text prompts alone not enough?

1. Text is not enough to describe everything: For example, in the artwork scenario, for the artists who are not that famous (e.g., my little brother), it is hard to use text to represent them; and for the artists who are new and not seen during model training, it is hard to expect the model can be generalized through text (artist names). However, using graph structure as the condition, we can well solve the not famous artist and new artist problem by discovering informative neighbors in the graph. We add an example of image generation for my little brother on the artwork graph to illustrate this in the rebuttal PDF (in general response).

2. Image conditions need to be discovered from the graph: As discussed above, neighboring image conditions can affect the generation result a lot, and graph structure can benefit the informative neighbor image sampling. In addition, solely based on image condition will result in unclear content (e.g., if we want to generate a “bird” image connected to the Picasso node, solely based on the sampled neighboring images does not provide the content indicator of a “bird”).

3. Graph enables flexible and diverse controllable generation: Another advantage of adopting graph as the condition is that we can combine multiple graph conditions (e.g., different art styles) with the controllable generation (with any mixed ratio according to the user’s interest), which is extremely hard to express by text only or image only. We add an example of controllable generation for a node connecting to “Pablo Picasso” and my little brother on the artwork graph to illustrate this in the rebuttal PDF (in general response).