DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning

Thank you for your valuable feedback. Below we address your concerns with further clarifications & experiments.

W1. Ablation studies: We would like to bring your attention to Appendix F.3, where we present ablation studies for different number of denoising steps, choosing which layers to fine-tuning for DiagramGLIGEN, and different LLMs for planning.

W2. Comparison to diagram generation from GT layouts: Following your suggestion, we generate diagrams from GT diagram plans and present the results here:

From this, we see that using GT diagram plans leads to higher overall performance, however, using diagram plans from our first stage is not too far behind. This is also reflected in Table 2 where we show that humans label GPT-4 diagram plans very highly (4.96 / 5 object presence and 4.72 / 5 object relations). Thanks for the suggestion, and we will add the results to the paper.

W3. Different LLMs: In Appendix F.3, we present layout generation experiment results from GPT-3.5 as well. As mentioned in the section, we also tried LLaMA2-Chat (13B) via both zero-shot and fine-tuning, but we found it does not perform well, often missing objects or failing to generate meaningful plans.

We also just tried the recently released LLaMA3 (8B) model but even with fine-tuning, it fails to produce good diagram plans, often just repeating the same object, missing objects, or sometimes failing to produce meaningful plans. We summarize the result below:

Layout recall checks whether certain objects/texts present in GT diagram plans are present in the generated diagram plan.

The LLaMA models may potentially be suffering from the limited training data.

Q1. In-context details: For each of the three domains (astronomy, biology, engineering), we use 10 in-context examples for each prompt. For open-domain prompts, we use all 30 in-context examples. The examples chosen for in-context use were randomly selected from the AI2D dataset.

Q2. Figure enlargement: We will try to increase the text label/figure size where needed to improve readability. Thanks for the suggestion.

We thank the reviewer for their time and effort in reviewing our paper.

We hope that our response has addressed all the questions and hope that the reviewer can consider revising the score based on our response. We are also happy to discuss any additional questions.

With sincere regards, The authors

Thank you for your valuable feedback. Below we address your concerns with further clarifications.

W1. In-context details: For open-domain diagram generation (Sec. 5.4 / Fig. 5 &14), we use all 30 in-context examples mentioned in Sec. 4.1. These examples do not change (i.e. static). We find that the LLM (GPT-4) can generalize these examples into other domains well (see Fig. 5, 14, & 15). We will clarify this in the paper.

Q1. GLIGEN details: The high-level intuition of the gated self-attention layer is similar to cross-attention layers, which are used to incorporate extra information (e.g., text, layouts) into the model. The cross-attention layer in the image generation backbone is used to incorporate text tokens for text-guided image generation, while the gated self-attention layer takes the grounding tokens for layout-guided image generation.

Additional details are clarified in the GLIGEN paper (Li et al., 2023b). However, following your suggestion, we can clarify the important architectural details in our paper.

Q2. AI2D-Caption details: For the test set of AI2D-Caption, we randomly choose 25 diagrams from astronomy (e.g., solar system, etc.), biology (e.g., life cycles, etc.), and engineering (e.g., circuits, etc.) for a total of 75 diagrams. We will add the selection criteria to the paper.

Q3. Human evaluation instructions: Here is how we describe the image-text alignment and object relationships to annotators:

• Object Relationships is a measure of which diagram better captures the proper relationships of the objects (i.e. spacing/positioning of them, arrows/lines between them, etc.).
• Alignment is a measure of which diagram is a better representation of the input sentence.

The annotators are then shown a diagram from DiagrammerGPT and SD v1.4 and provided 3 options: Diagram A, Diagram B, and Equal. The diagrams are randomly shuffled so there is no pattern between which model produces diagram A or B.

We will add a screenshot of the setup to our paper to make sure these details are clarified.

Q4. Vector graphic generation implementation: We take the diagram plan generated by the LLM and convert it into a format that is accepted by the related vector graphic platform (Appendix F.2; e.g., VBA for PowerPoint) via a script. Objects are represented using icons automatically pulled from the public icon library “The Noun Project” (Appendix F.2). We will release all related code for render vector graphic rendering.

We thank the reviewer for their time and effort in reviewing our paper.

We hope that our response has addressed all the questions and hope that the reviewer can consider revising the score based on our response. We are also happy to discuss any additional questions.

With sincere regards, The authors

Thank you for your valuable feedback. Below we address your concerns with further clarifications and experiments.

W1. Paper contributions: We would like to remind you of the strong contributions of our project:

1. We present a new text-to-diagram generation framework that leverages the knowledge of LLMs for planning & refining the open-domain diagrams (Sec. 3), which is more effective than existing T2I methods (Table 1 & 3; Fig. 4 & 5).
2. We show that iterative self-refinement can help create better diagram plans (Table 2; Fig. 13).
3. We show that our diagram plans can be rendered via any vector graphic software (Fig. 6, 15, & 16) as well as T2I models, allowing for more flexible real-world use (e.g., for teachers) and human-in-the-loop editing (Fig. 7).
4. We produce a new dataset (AI2D-Caption; Sec. 4.1) with evaluation suites (VPEval + LLaVA 1.5 finetuned on diagrams; Sec. 4.3) that can help the community benchmark and improve the text-to-diagram methods.
5. We train DiagramGLIGEN to have improved control of text labels/entity relations (Sec. 3.1).

W2. VPGen additional analysis: Our improvement over VPGen primarily comes from stage 1. Following your suggestion, we compare fine-tuned VPGen to our DiagrammerGPT in the first stage.

Layout recall compares GT diagram plans to the generated diagram plan.

As shown in the table, the fine-tuned VPGen struggles to produce the required entities for a diagram, while DiagrammerGPT does better.

The reason the fine-tuned VPGen does worse sometimes compared zero-shot VPGen is likely due to the diagram planning stage. After fine-tuning, VPGen generates objects in the right domain (as reflected by the increased object score 55.8 $\rightarrow$ 62.8 in the main paper Table 1). However, potentially due to limited training data, it often generates too many objects, overlaps objects, etc. which leads to lower scores in certain metrics.

Q1. Arrow/line annotations: The original AI2D dataset labels which objects are connected via a line/arrow.

Q2. Fine-tuned GLIGEN in VPGen vs. DiagramGLIGEN: We fine-tune the GLIGEN model in VPGen like we do for DiagrammerGPT to give a fair comparison. DiagrammerGPT still has a different diagram planning stage than VPGen and also explicitly renders text labels, whereas VPGen does not.

We thank the reviewer for their time and effort in reviewing our paper.

We hope that our response has addressed all the questions and hope that the reviewer can consider revising the score based on our response. We are also happy to discuss any additional questions.

With sincere regards, The authors

Thank you for your valuable feedback. Below we address your concerns with further clarifications.

W1. Paper contributions:

We would like to point out that the previous works mentioned do not tackle diagram generation, which requires fine-grained/dense layout and relationship control. They also do not use iterative refinement. Below, we summarize our strong contributions:

1. A new text-to-diagram generation framework that leverages the knowledge of LLMs for planning & refining the open-domain diagrams (Sec. 3), which is more effective than existing T2I methods (Table 1 & 3; Fig. 4 & 5).
2. Iterative self-refinement can help create better diagram plans (Table 2; Fig. 13).
3. Our diagram plans can be rendered via any vector graphic software (Fig. 6, 15, & 16) and T2I models, allowing for easier uses and human-in-the-loop editing (Fig. 7).
4. New dataset (AI2D-Caption; Sec. 4.1) with evaluation suites (VPEval + LLaVA 1.5 finetuned on diagrams; Sec. 4.3) that can help the community benchmark and improve the text-to-diagram methods.
5. Improved control of text labels/entity relations (Sec. 3.1).

W2. Iterative refinement effectiveness: In Table 2, we show that refinement can help improve object relationships (4.56 $\rightarrow$ 4.72). We find the initial diagram plans are mostly correct, and the refinement cleans minor details. Fig. 13 also shows examples of how refinement helps.

W3. Can LLMs generate diverse diagram plans?: Yes, LLM can generate meaningful variations from the same prompt, as in this example link. We also show that the LLM can generate plans in diverse domains (Fig. 4-7, 10, 12-16). The backbone T2I model is still free to generate the diagrams in any art style, allowing further diversity.

W4. Visual quality: We agree that the visual appeal of the diagrams can be improved. However, please note that we also proposed diagram rendering via vector graphic software, such as PowerPoint (Sec. 5.4; Fig. 6, 15, & 16). This can create high-quality diagrams that are more visually appealing and easy to adjust (Sec. 5.4; Fig. 7).

In addition, the current backbone is SD v1.4, and we expect that a bigger backbone (e.g., SDXL, SD3) would yield more visually appealing results. However, such scaling is expensive for our academic lab, and we focus on introducing the first LLM-based diagram generation framework. Our methods' flexibility allows adapting to bigger models with sufficient GPU resources.

We thank the reviewer for their time and effort in reviewing our paper.

We hope that our response has addressed all the questions and hope that the reviewer can consider revising the score based on our response. We are also happy to discuss any additional questions.

With sincere regards, The authors