Feynman: Knowledge-Infused Diagramming Agent for Scaling Visual Reasoning Data

Thank you for taking time to read our paper and the meaningful feedback you provided. Below we address some of the issues raised.

Unclear discussion about Penrose vs. TikZ

Thank you for pointing this out. There are some general tradeoffs between using TikZ and Penrose. TikZ offers granular control suitable for users who need detailed customization and are comfortable with manual specification. While Penrose provides a higher-level approach, automating the visualization process by interpreting the relationships between objects, which can be advantageous for quickly generating diagrams without delving into graphical details.

In our use case, Penrose helps with scalability. Penrose code decoupled the manipulation of low-level graphical details and high-level concepts, and LLM are better at the high-level reasoning and knowledge extraction, which we try to utilize. TikZ is potentially more powerful if the data size continues to increase with more TikZ diagrams on the internet. However, different from textual math or code data, TikZ code and diagrams usually are less readable and have worse captions, limiting their quality and usability.

Weak comparison with baselines.

We believe that our choice of using the Llama-2 model released by AutomaTikZ paper is a suitable choice as baseline. In the work of AutomaTikz, they compared with GPT-4 baseline and claimed that their trained LLama-2 + MCTS produce better results, which is why we base our comparison on their Llama-2 model. They also explained in their paper why the current LLM API does not support their approach of using MCTS for code generation. We took the code and model from their implementation and tested against our results, as the reproduction of their work based on Llama-3 is beyond our capability.

In the appendix, we also compared the non-agent approach that uses a much stronger coding model such as OpenAI o1-preview to generate TikZ code. We found that even though o1 is very good at writing TikZ code that can be successfully compiled into diagrams, its output diversity is rather limited, possibly due to the inherent difficulty of diversifying the visual layout in TikZ code.

Missing related work of DiagrammerGPT

Thank you for pointing out this extremely relevant work that we were not aware of. DiagrammerGPT is an agentic model that generates a text plan with LLM and uses a diffusion model for diagram generation. It shares a similar goal to our work and is pretty advanced in terms of their techniques and results. However, we realize there are some major differences between our approaches, which makes our approach potentially better in some domains.

1. The language planning part of diagram generation in DiagrammerGPT is very similar to our approach. The major distinction between our approach and theirs is that our code planning step serves to improve the following coding process (which could be done by either a fine tuned or off-the-shelf LLM), while they trained a diffusion model to read the plan as input.
2. The image generation part has similar drawbacks to the approach of using diffusion models directly, where the elements drawn by the diffusion models contain rich visual features that are beyond the conceptual relations described in the plan. We think their image generation is more suitable for illustration of scientific knowledge where natural image generation is helpful. While our approach of using programming tools is better for abstract diagramming due to the clarity of conceptual programming.
3. We believe that our approach is more suitable for scaling the complexity of abstract/geometric relationships. Here the ability of generating complex diagrams depends on the coding and reasoning ability of LLMs, which is much easier to scale and improve than the diffusion models.

Weak analysis of the correctness of the generated diagrams.

We acknowledge the importance of conducting a more rigorous evaluation on the correctness of the diagrams generated by Feynman. But we also believe that there is some misconception about how we obtain the Benchmark Diagramma, which led to the unfair accusation that Feynman’s generations are of low quality.

• Diagramma is a benchmark curated by the authors manually. We checked all the question-answer pairs, and we also filtered out the visually incoherent/unrecognizable diagrams, to improve benchmark quality. Although we cannot guarantee 100% correctness, we believe that the erroneous examples are sparse, since the majority of the image QA samples in the dataset are easy for humans (both visually and textually).

• Since we have manually curated the benchmark dataset, we believe that the inability of MLMs to answer the questions in Diagramma is not due to low data quality, but something else. In fact, we conjecture that our dataset is comparably more out-of-distribution than most of the existing visual-language benchmarks for MLLMs.

Thank you so much for your time reading our manuscript. Your suggestions on conducting a more detailed error analysis of the agent pipeline makes a lot of sense!

Although we were not able to do such analysis systematically, we provide a qualitative view based on our running logs on some notable errors.

Idea In this stage, the primary goal is to elicit knowledge of an LLM in a specific domain. For example, to draw structural chemical formulas, we ask FEYNMAN (GPT-4o backend) to enumerate n number of chemical reaction formulas. Something we observe is that when the model does not have ample knowledge in a particular domain, the given ideas have some repetition when n is large. We think that this is the reason caused discrepancies between scaling experiment shown in Fig. 8

Render/Iterate

We identify two types of errors at this stage:

1. Compile Errors: These occur when the model fails to generate correct Penrose code. We observed that providing compiler error messages as feedback significantly improved the model's code generation in subsequent rounds.
2. Diagram Errors: These arise when the code compiles successfully, but the generated diagram is flawed (e.g., overlapping shapes, illegible text). To address this, we utilized VLM judges with a critical mechanism to evaluate and refine these elements (details in Section B.5).

Overall, both error types demonstrate that multi-round refinement with targeted critic feedback improves the alignment and quality of diagrams generated based on an inquiry.

If you are interested, you are welcome to review the running log of agent pipeline yourself and the link to Diagramma, is provided here [https://anonymous.4open.science/r/feynman_anonymous-D5A7/README.md]

Thank you for your time writing the review and reading the paper. We want to address some of your major concerns and questions. And we also want to emphasize that the major goal of this paper is not to introduce a new benchmark but to demonstrate a scalable method for generating scientific diagrams with well-aligned text captions.

More details of the visual judges are needed, What are the visual judges?

Most of the details are talked about in the paper. We adopted open Visual Language Models (VLMs) as the judges, prompted to give binary scores on certain predefined criterions. Details about which model and prompts we used are provided in section B.5 in appendix. Our goal is mainly to showcase the possibility of iterative improvements from the feedback of visual judges, but there could be a significant room left for fine tuned judges in special domains. We acknowledge that concerns may arise regarding the validity of using multiple LLM models to critique work generated by another LLM. But this choice of our pipeline does have certain advantages.

1. It is more scalable (to dataset size) than human evaluation. To synthesize data at scale, common approaches such as human verification or programmatic verification fall short because of the diversity and knowledge coverage of the data generated. This is the major reason why we chose to use VLMs to provide feedback in a scalable manner.
2. It is possible to train customized VLMs to serve as judges. However, that requires VLM annotations which is also lacking currently and costly to obtain. We hope that our work could be a step towards enriching the data of scientific diagrams for various purposes.
3. We also want to emphasize that the judges' evaluation primarily focuses on structural aspects of the image. For instance, determining whether an image contains legible text—one of the evaluation criteria—should be relatively straightforward. Moreover, we expect these results to improve along with the other improvements in VLMs.

Why were there no human evaluations even at the very least for sanity checking conducted?

We would like to clarify that the benchmark Diagramma is human curated, where we have manually checked and corrected all mistakes spotted. For your reference, we have made both Diagramma and the image-text pairs publicly available here: [https://anonymous.4open.science/r/feynman_anonymous-D5A7/README.md]

If the question is referring to the whole dataset we generated, we have several reasons for our failure to do so.

1. As all synthetic data generated by LLM or any neural networks, they are not error-free and are subject to the error patterns they learned in training. It is beyond the scope of this paper to check all the possible patterns incurred by the agent in generating the data, which involves significant manual labor.
2. We believe that the quality of the data will improve similarly to how math data quality in textual form gets improved, as we mainly used off-the-shelf LLM API for our agent pipeline. This relieves us with the task of improving the custom VLMs or LLMs to catch up with the progress every several months. In fact, our agent pipeline depends mainly on the LLM’s ability to code and enumerate domain knowledge and VLM’s visual recognition capability. It can be viewed or understood as some form of “self-distillation”, since we are trying to transform LLM’s textual knowledge into visual-language data.

Why choose the QA format after all the hard work?

Thank you for this question. QA format is just a choice we made as it is the mostly used format in visual instruction-tuning. The choice of the textual format of the image-text data is actually very flexible, as long as they are about the content of the images. Our approach naturally generates both diagrams and their corresponding code pairs by creating programming language representations of diagrams. We let LLM to summarize/describe the generated code, which is somewhat close to natural language, to obtain the text captions.

In addition, how do we envision the community would benefit from this dataset or any of the proposed methods?

We believe that the primary contribution, which is the Feynman pipeline, offers substantial value by enabling a scalable method for generating training pairs, particularly with images that are out-of-distribution compared to internet-sourced datasets. It is important to note that our primary objective is not to introduce a new benchmark but to demonstrate a scalable method for generating scientific diagrams.

Thank you for your time writing the review and reading the paper. We want to address some of your major concerns and questions.

How many data were generated exactly?

It seems like there is some confusion about our approach. We clarify here but please refer to Section 2 and the appendix in the paper for comprehensive description.

As mentioned in the contribution, our pipeline generates caption and programs that compiles to image. There are ~110,000 of them. To further test the quality of our pipeline, we curated a benchmark, Diagramma, in which there are 1,058 question and answer pairs in total. Although the source data of the benchmark was created using LLMs, all 1,058 question-answer pairs in Diagramma have been manually verified for accuracy. Please refer to Table. 1 in the paper for details.

Significant of Section 4.2 and Figure. 8

It is correct of what you wrote about that token usage naturally increases as the number of samples grows. However, as we discuss in this section and highlight in the caption under Figure 8, the key observation is not the increasing trend itself but the curvature of the token usage. A linear trend suggests potential for further scaling within a specific domain as token counts continue to grow, whereas a decaying trend indicates diminishing returns in image generation as token usage increases.

The goal of this section is to demonstrate that, in certain domains, the number of unique generated images can be scaled up after deduplication, even as the number of samples rises.

How exactly were KP, CP, and S ablated? If CP is missing, how is code generation achieved?

Each component refers to a distinct aspect of the prompting process. In your example, if the Code Planning (CP) component is omitted, the agent is instructed to directly write code for generating a Penrose program without employing a chain-of-thought step to plan the code generation.

Were these evaluation metrics designed by the author (PR, LR,...) all given by VLMs? Is this accurate enough? We acknowledge that concerns may arise regarding the validity of using multiple LLM models to critique work generated by another LLM. However, we emphasize that the judges' evaluation primarily focuses on structural aspects of the image. For instance, determining whether an image contains legible text—one of the evaluation criteria—should be relatively straightforward and objective. Furthermore, it is some compromise we have to commit given the generation is at scale.

If we remove Iterative Visual-Refine based on VLMs and only reflect based on feedback by the compiler. What will be the effect? How many samples can it generate?

Thank you for this question. We have incorporated the results in the paper. As shown in Algorithm 1, VLM judges are only utilized when the compilation is successful. This means that feedback from the compiler serves as a baseline for the agent to produce correct programs. Consequently, there are two distinct forms of improvement:

1. Improvements in program correctness: Ensuring that the generated programs compile successfully.
2. Improvements in image quality: Enhancing the visual output of the generated programs. While rerunning the full scale-up experiment is resource-intensive, it is important to note that the compilation rate is not influenced by the inclusion of VLM judges. For insights into different setups, please refer to Table 3.

Please elaborate in more detail on what the innovation of the paper is? It seems that generating charts through code as a medium has been explored by some works.

Thank you for this question! We would like to highlight the following key innovations:

1. Iterative improvement through visual judges and compilation feedback: This approach enables the generation of correct and high-quality code programs by iteratively refining the outputs based on visual and compilation feedback.
2. Conceptual diagramming: Conceptual diagramming separates the visual production of layouts from the conceptual relationships of visual elements, which is a key ingredient of our approach and it help utilize LLM's knowledge capacity and improve the precision and clarity of generated contents. By leveraging Penrose as the backend for our diagramming tool, we were able to make the model enumerate its rich domain knowledge without the need to let LLM to fully control the layout process. Moreover, conceptual diagramming provides knowledge diversity/flexibility compared to fixed programmatic approach such as using matplotlib to draw flow charts.

We hope that you recognize our work as an attempt to generate scientific diagrams at scale. To further address your concern, we provide following end results for our work in link provided here [https://anonymous.4open.science/r/feynman_anonymous-D5A7/README.md] It contains: the Diagramma benchmark 2. Generated diagram-caption 3.A few example illustrating the pipeline.