Can Large Language Models Understand Symbolic Graphics Programs?

We sincerely thank the reviewer for the constructive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are happy to address them. We summarize all the concerns into the following questions:

---

Q1: L108: "highly scalable benchmark." While I agree that SGPs have this potential, the paper does not demonstrate specific contributions toward achieving scalability. For example, benchmarks like MineDojo achieve high scalability by developing tools to efficiently scrape relevant data from platforms like YouTube, Reddit, and Wikia, resulting in a dataset of around 700k Minecraft videos and 6M+ comments. In contrast, the SGP dataset in this paper only contains ~5,000 data points. This is surprising given the abundance of online SVG images (for example, SVGRepo claims to have over 500k SVGs). If SGP-Bench were a "highly scalable" benchmark, I would have expected an order of magnitude more data points or an explanation of the barriers preventing larger-scale sampling. Although this does not diminish the significance of the benchmark (Multimodal reasoning benchmark do not have to be "internet-scale" to be influential: https://arxiv.org/abs/2401.06209), the scalability claims should be adjusted accordingly.

A1: Thank you for the great question! By 'highly scalable', we are referring to both the benchmark creation process and the creation of the instruction-tuning dataset. With this scalable pipeline, we are able to label more QA pairs for larger SVG datasets, but our benchmarking results show that the current scale already suffices. In addition, directly crawling internet-scale data can be costly for academia and demands more caution to copyright issues.

Our benchmark comprises 6,740 unique questions (spanning SVG and CAD program codes), while our instruction-tuning dataset includes over 72,000 unique instruction-following pairs. During benchmark creation, we used our pipeline to generate 25,167 questions for SVG programs and 78,198 questions for CAD programs. Specifically, for the 3D subset, we selected 1,000 questions from an initial set of 32,307; for the 2D subset, 700 questions from 20,455; and for the 3D_complex subset, 700 questions from 25,536. To ensure variety and quality, we manually reviewed the generated questions to remove any that were repetitive or too visually similar, finalizing a subset of 6,740 questions for our SGP-Benchmark. Our evaluation of current LLMs indicates that this benchmark is sufficiently large and diverse to highlight performance differences across different models.

Thanks for mentioning this paper (https://arxiv.org/abs/2401.06209). We agree with the reviewer that we can adjust our scalability claim to make it more justified. We will make a more detailed discussion of our scalability claim and our benchmark scale in the revised paper.

---

Q2: The current definition of symbolic graphical programs might be overly broad. For instance, HTML could technically qualify as an SGP within the given definition since it uses a consistent markup language and rendering engines (like Chrome, Safari, or Firefox). While this isn’t inherently a weakness, it could impact the paper's novelty claims. If the SGP definition admits HTML as well, the authors might need to expand their related work and adjust their novelty claims.

A2: We appreciate the reviewer's suggestion. However, we would like to clarify that our work does not involve HTML, as our focus is on assessing LLMs' understanding of 'symbolic graphics programs.' Following the definition of 'symbolic graphics programs' from [1], we specifically focus on symbolic programs that uniquely produce visual data. Such representations serve as compact, interpretable, high-quality formats for graphics content. Beyond SVG and CAD, other symbolic graphics programs that fit this definition include constructive solid geometry (CSG), boundary representations (B-reps), L-systems, and shape grammars, with SVG and CAD representing the most prominent examples. Other symbolic graphics programs can easily adopt our pipeline to create additional benchmark tasks.

Our SGP benchmark encompasses 2D primitive shapes, 3D geometry, and complex 2D renderings, facilitating both zero-shot testing with standard, known syntax (SVG) and in-context learning with domain-specific languages (DSL). This range substantiates our claim that our benchmark is able to assess LLM’s understanding of symbolic graphical programs. We will add HTML-related studies to our related work section and further clarify the distinctions between our approach and HTML-based methods.

[1] Ritchie, Daniel, et al. "Neurosymbolic models for computer graphics." Computer graphics forum. Vol. 42. No. 2. 2023.

We sincerely thank the reviewer for the constructive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are happy to address them. We summarize all the concerns into the following questions:

---

Q1: In L76-L77, the authors claim that "The order of symbolic operations may substantially affect semantic meaning". However, this does not seem applicable to SVG, where objects are defined by coordinates and unaffected by drawing order. Could this be clarified?.

A1: We appreciate the reviewer's observation that, in SVG, the order of symbolic operations does not always affect semantic meaning. However, in many cases, the operation order is crucial for the final rendering or shape. For instance, in SVG, elements defined earlier are drawn first, with subsequent elements layered on top. Changing the order of elements alters their stacking order on the canvas, which is especially evident with overlapping shapes, where later elements may cover or partially obscure earlier ones. Similarly, in CAD program codes, altering the order of operations can significantly impact the final shape. For example, an 'extrude' operation is defined based on a preceding 'profile' operation, transforming a 2D sketch into a 3D shape.

---

Q2: In Figure 6, what qualifies as an "operation" in vector graphics? For example, does a polyline with multiple vertices count as one operation or several?

A2: For our analysis, we count the number of drawing-related operations that contribute to the visual output, including commands such as 'circle,' 'rect,' 'line,' 'polyline,' 'polygon,' 'path,' and similar. Therefore, each 'polyline' operation with multiple vertices is counted as a single operation.

---

Q3: Inconsistencies in notation between Figure 6(b) and the text (in L257-L258) need clarification.

A3: Great catch! We apologize for the mistake we made in the notations, the correct notation should be as the text 3D_complex. We will update the figure in the revised paper.

---

Q4: During fine-tuning, was there any overlap between training and test datasets? Additionally, could more detail be provided on the construction of the symbolic instruction dataset such as whether it includes both SVG and CAD formats?

A4: Thank you for raising this concern. We carefully apply a filtering step to ensure that no data from the test set is included in the fine-tuning dataset. We are happy to provide additional details regarding the construction of the SIT dataset. Currently, our proposed SIT dataset contains only SVG data. The primary purpose of SIT is to offer insights to the research community on how SIT can enhance general reasoning ability. Prior research has shown that training on general code data can improve reasoning skills [1,2]. For example, several code-focused LLMs [3,4,5] have demonstrated enhanced reasoning abilities in solving mathematics problems by incorporating more code data [6]. Symbolic graphics programs, a specialized form of code data, are used to generate 2D and 3D graphics content. They share a hierarchical structure similar to that of general code data, but unlike traditional programs that typically produce numerical outputs, symbolic graphics programs generate outputs rich in semantic information, involving complex reasoning tasks such as part/component localization, color identification, affordance prediction, semantic understanding, and geometric reasoning. For instance, consider a question about a symbolic graphics program: 'What is the color of the top part of the object?' Answering this requires the LLM to identify which part of the code corresponds to the top section and determine the color based on the color code. Similarly, a question like 'What is the object primarily used for?' demands that the LLM interpret the object’s semantic meaning and suggest a suitable application. These questions involve multiple, interdependent reasoning steps, where an error at any step would result in an incorrect final answer. In future work, the same pipeline could be applied to create instruction-following datasets for other types of symbolic graphics programs. More importantly, the data creation pipeline is quite scalable and requires minimal human interference.

[1] At Which Training Stage Does Code Data Help LLMs Reasoning? ICLR 2024

[2] To Code, or Not To Code? Exploring Impact of Code in Pre-training, https://arxiv.org/abs/2408.10914

[3] Code Llama: Open Foundation Models for Code, https://arxiv.org/abs/2308.12950

[4] DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence, https://arxiv.org/abs/2401.14196

[5] WizardCoder: Empowering Code Large Language Models with Evol-Instruct, https://arxiv.org/abs/2306.08568

[6] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, https://arxiv.org/abs/2402.03300

Thank you for your response. Most of my concerns are addressed but I'm still confused on few points.

1. You said that "Initially our benchmark consisted of 25,167 questions on SVG programs and 78,198 questions on CAD programs", but in your paper L215 and L255 you said the final benchmark contains 4,340 questions in the SVG set and 2,400 questions in the CAD set. In your human filtering process, are you only selecting 17% of your generated SVG questions and 3% of your generated CAD questions? The passing rate looks inconsistent with the passing rate of 90% in the human checking process.
2. I'm not convinced that "models tested in Table 1 have not been finetuned with SIT data and possess good instruction-following ability". Some models in table 1 are small and don't have good instruction-following ability. Therefore, I think it might be the enhanced answer extraction with LLM improving the performance (e.g. by making the output format consistent and preventing errors due to format). Could you please report at least one result of those small models (without SIT) using enhanced answer extraction with LLM to make the experiment fair? It don't need to be large scale, only a small portion of data proving that enhanced answer extraction won't affect accuracy is fine.

We sincerely thank the reviewer for the constructive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are happy to address them. We summarize all the concerns into the following questions:

---

Q1: Regarding Weakness 1, it appears the model learns from structured data without evidence of internal image rendering, likely due to lack of instruction. It would be interesting to test whether explicitly prompting for “internal rendering” as a Chain-of-Thought (CoT) step impacts performance.

A1: Thank you for the question! In fact, our question prompt does explicitly use Chain-of-Thought (CoT) reasoning, and our prompt also implicitly includes the concept of 'internal rendering' as part of the prompt itself.

Below we give our original prompt templates:

Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.

[**Question prompt**]

A) [A]
B) [B]
C) [C]
D) [D]

The [Question prompt] consists of the following instructions and the corresponding symbolic program:

Examine the following CAD code carefully to understand the 3D object it generates and answer the question based on your interpretation of the rendered image of that object.
[**CAD program code**] + [**Benchmark question**]

Examine the following SVG code carefully and answer the question based on your interpretation of the rendered image.
[**SVG program code**] + [**Benchmark question**]

To address your question, we have included an additional test that explicitly uses the prompt for 'internal rendering':

As shown in the table above, explicitly incorporating 'internal rendering' as part of the CoT slightly enhances model performance. This finding suggests that the ability of internal rendering is beneficial for solving the tasks in our benchmark. Moreover, the improvement is marginal, implying that the “visual imagery” ability is fundamental, and simply using better instruction prompts can not easily improve it.

---

Q2: I'm curious whether any experiments have been conducted using rendered images as input in multi-modal LLMs to compare performance with raw SVG code inputs, providing insight into how well models perform with visual representations.

A2: Thanks for the question! Our SGP benchmark is automatically constructed by leveraging the visual understanding capabilities of LLMs, meaning that the questions are designed to be easily answerable from visual input (see Figure 7 in our main paper). In our benchmark, GPT-4’s responses to questions using rendered images serve as the ground truth labels. To ensure both accuracy and diversity in our benchmark, we manually reviewed the generated question-answer pairs and removed the questions with incorrect answers. As a result, GPT-4 with visual input achieves nearly perfect accuracy on the benchmark questions.

---

Q3: In Appendix E.2, it’s noted that fine-tuning SIT-55k on Llama3-8B actually degrades performance. Is this due to answers not following the specified template, or are the answers correctly formatted but simply inaccurate?

A3: Thanks for the question! We appreciate the reviewer for taking the time to examine the experiments in our Appendix. For evaluation, we used an LLM-based approach to assess the responses of the SIT-finetuned models, instead of matching-based approaches. There are almost no template mismatches under this evaluation protocol. The degradation of performance is indeed caused by the incorrect answers.

---

Q4: The SGP-MNIST experiments show limited spatial reasoning in LLMs, suggesting that some claims about models’ spatial reasoning or “visual imagery” abilities may be overstated.

A4: Thanks for the comment. Most existing LLMs indeed perform poorly on the SGP-MNIST benchmark. This experiment aims to demonstrate the task of symbolic program understanding can be extremely challenging and is far from being solved at the moment. However, from Figure 4 in the main paper, we can observe that stronger LLMs (e.g. OpenAI-o1) can perform qualitatively better than less powerful LLMs (e.g. Llama-3.1). Such results suggest that the ability of symbolic program understanding/reasoning is fundamental and it can serve as a good indicator of LLM’s reasoning abilities. More importantly, the current results on our SGP-MNIST benchmark shows that there is still much room for existing LLMs to improve.