Enhancing LLM Reasoning via Vision-Augmented Prompting

We sincerely appreciate the constructive comments on our work, which have helped us to enhance the paper. The following are our detailed responses to each comments.

W1.1:Efficiency of iterative reasoning

This is a good point. We added an efficiency experiment for VAP in our revision. The results for the geometry and Sudoku tasks are shown below:

While VAP is indeed time-consuming, it achieves comparable time usage to ToT and CoT-SC (k=10) in the Sudoku task. The reason is that, although VAP's context is quite large (tool instructions, thought trajectory, image encoding), its output is concise (API calls and immediate thoughts), leading to fast inference during each step (time usage is mainly affected by the number of decoded tokens). We believe the time usage is acceptable given VAP's superior effectiveness.

W1.2:Trade-off between the number of images and model performance

This is an inspiring comment. Currently, the number of images is determined by the planner, and we don't provide access to modify it. To address the reviewer's concern, we add an experiment of controlling the number of images. The results of this experiment can be found in our response to all reviewers.

W2/Q1:More LLM models need to be added for comprehensive comparison

We follow the reviewer's advice and add GPT4 and LLaMA 3 8B as additional LLMs for more fairer comparison in our revision. The results can be found in our response to all reviewers. Based on results, we believe that the superiority of VAP over other baselines remains valid, even when utilizing different LLMs.

W3:Effect of different figure drawing tools

We sincerely thank your comment and it actually pushes us to think deeply about the relationship between tool selection and performance.

First, we analyze the distribution of drawing tools selection:

Matplotlib was used more frequently, especially for time series prediction, due to its ability to construct coordinate systems efficiently. DALLE3 is never selected by planner as the images of four tasks are all diagrams (we have explored another use case of DALLE3, detailed in Q2).

Second, we are also interested in which tool perform better for these tasks. We conduct an experiment where we forced VAP to choose a specific tool. The results are as follows:

We find that the impact of different tools depends on the task. For geometry and TSP, using Matplotlib or Turtle makes no significant difference, while in Sudoku and time series forecasting, Matplotlib is better than Turtle. Because drawing grids and coordinates are too complicated for Turtle (draws by pen movement). The results also reflect the rationale behind the LLM's tool selection, as it automatically avoids choosing tools that are not suitable for a given task.

W4: Scalability of VAP

Solving complex reasoning problems has been a well-known weakness of LLMs and this domain has attracted significant attention because once the challanges can be overcome, many applications can well benefit from the power of LLMs. So, it is not surprising to find that the accuracy drops when facing challenging reasoning problems. Compared with other LLM-based reasoning frameworks, our VAP still demonstrates the best performance. The performance gap is even widened when the problem becomes more challenging.

W5: Although VAP performs better than the baselines, its overall performance is still poor and has a noticeable gap compared to traditional methods

We totally agree with the reviewer that the current performance of LLMs is still not comparable to traditional methods in many tasks. Its strength lies in generality, with one model to support vairious decision and prediction problems. Thus, how to exploit the potential of LLMs and improve its capabilities in reasoning and prediction has become a hot research topic in recent years. Our work falls in this category and we believe with more efforts devoted in this domain, the performance gap may continue to shrink.

Q2: The potential of VAP in solving real-world reasoning problems, where the model generates more photorealistic images

We've actually explored VAP's potential in a real-world sotrytelling task. The task setting is to write a story based on the input of text prompts.

In the implementation of VAP, DALLE3 is always selected as the external image tool. Then at each iteration, VAP generates a photorealistic image to render a scene. This image is then sent to GPT-4v to enhance the model's creativity in story generation.

Our previous findings show that, based on our subjective judgement, VAP can significantly benefit the task. We did not put this task in the manuscript due to the lack of a convincing performance metric.

We sincerely appreciate the constructive comments from the reviewer. We provide detailed responses to each concern in the following.

W1&Q1: "When the LLM generates Python API calls, there is a chance that the generated code may not run successfully (bugs in the code). What is the probability of this phenomenon occurring in the experiments?"

We understand the reviewer's concern and have added an experiment to assess the code generation error rate. The following table presents the ratio of syntax error and runtime error:

The results show that the componnet of LLM-based code generation is sufficiently robust, with no syntax error. This is because the task of code generation for API call is not challenging for LLM, without involving complex logic. For example, "a circle center at (1, -1) with radius 2" is translated to plt.circle(c=(1, -1), r=2). We can observe slight runtime error in the Sudoku task. These errors occurred when translating Sudoku positions to actual coordinates in the image. For example, in a 9x9 board, the grid position "1 9 1" should be translated to plt.text("1", x=0.5, y=8.5)(placing the text in the center of the grid). However, the LLM occasionally made mistakes like plt.text("1", x=1, y=9), causing out-of-bounds placement.

W2&Q3: "Generalization: CoT and ToT are more generalized to different LLM tasks, while VAP is limited to a few geometry problems. Does it generalize to normal Visual QA tasks?"

Yes, we agree with the reviewer that CoT and ToT are more generalized, but their performances are inferior to our VAP in the four diversified reasoning tasks that can benefit from dual-modality reasoning.

In the current stage, we are unable to support normal visual QA tasks for two main reasons. First, the problem setting is different. The input of visual QA tasks consists of an image and text question. In our setting, the input is a text question. The image is automatically synthesized according to the text input. Second, VAP is designed to improve numeric reasoning, whereas normal visual QA tasks focus on semantic understanding of the input image. Nonetheless, we agree that it would be very impactful if we can extend VAP to be more general to support VQA.

W3: "There is a missing reference to relevant works on tool usage ability for LLMs, such as ViperGPT, VisProg, and LLava-Plus."

We thank the reviewer for providing these relevant works. In revision, we incorporated these works in related work and add necessary explanations and discussions.

Q2: "Does the model require customized instructional prompts for LLM on different tasks?"

In fact, VAP does not require task-specific prompts for different tasks, which we think is a desirable feature. This ows to the planning step, in which the prompt specifies the role, drawing workflow, external image tools, and output format. These elements are identical for different tasks. The prompt also contains the input problem text, which triggers LLM to understand the task and generate different plans accordingly.

The key components of the prompt template for the planning include:

Your role is to visualize a problem by creating an image... // Role play

The drawing will be executed through an iterative process... // Workflow clarification

Here are drawing tools available:... // Introduction of the drawing tools

You must strictly follow the JSON format... // Format control

Problem Description:... // Problem input

More details on this prompt can be found in Appendix A.1.1.

Q4: "Is it possible to compare with VCoT mentioned in Section 2.3 (line 76)?"

VCoT was designed for tasks like storytelling and summarization, instead of the numeric reasoning tasks presented in our work. Furthermore, we found that its source code is not available, with an empty page in the provided github link. To address the reviewer's concern, we came up with an idea to apply our VAP to the sotrytelling task as presented in the VCoT paper. The task setting is to write a story based on the input of text prompts.

In the implementation of VAP, we incorporate DALLE3 as one of the external image tools. Then at each iterative step, VAP generates a photorealistic image using DALLE3 to render a scene. This image is then sent to GPT-4v to enhance the model's creativity in story generation.

As to performace metric, VCoT adopts human evaluation to judge creativity etc, which we think are subjective. Therefore, for reviewer's reference, we simply provided an example below to demonstrate the results of two approaches.

We sincerely appreciate the constructive comments from the reviewer! We provide detailed responses to each concern in the following.

W1: "I wonder, is generating an explicit image the best practice for this process. As the generation of images could result in unwanted features, we could also keep the representation in the latent space as a prompt."

This is a really inspiring idea! In the current stage, commercial VLMs we used in this paper have not provided the API interface that allows us to use encoded image feature in the latent space as part of the input. We consider it as our future work to examine whether we can implement the idea on other open-sourced VLMs.

W2: "Is there a more formulated way to generate the plan, or do we have somewhat of tools to detect the possible problem?"

We appreciate your constructive feedback. We totally agree that a more formalized approach to generate the plan would likely improve the performance and robustness of our method.

The ideas we currently have include:

• Format control: we can utilize the open-source repository lm-format-enforcer (https://github.com/noamgat/lm-format-enforcer) to strictly formulate the output JSON schema. This tool uses a prefix tree decoder during the streaming output of the language model to enforce the output format.
• Reflection mechanism: recent works, such as Reflexion, have shown that LLMs can self-reflect on previously generated answers and produce more reasonable responses. This mechanism can be applied on the planning stage, which could help detect potential problems.
• Controllable parameters in planning prompt: we can introduce controllable parameters in the prompt of planning step, which would provide users with more control over the generated plan and would also contribute to performance (also proved in the experiment in Q1.2).
• Planning decomposition: for complex tasks, decomposition can be an effective strategy. Techniques like ToT or ReAct could be applied to generate a more reliable plan.

Q1.1: "Since the drawing steps and numbers are decided by the planner. What if the plan is quite long, and we are out of tokens?"

Token length is indeed a common issue in LLM-based reasoning tasks. So far, the maximum number of tokens required by the planner of VAP is 2841, which is safely below the 8k token context limit in the default GPT-4 setting.

Q1.2: "Do we have a control on the granularity on the intermediate drawing?"

Thank you for your insightful comment. Currently, the granularity on drawing is determined by planner and we don't provide access to modify it. Inspired by your feedback, we explored the possibility of controlling granularity to enhance drawing efficiency.

We conducted an experiment on Sudoku task by controlling the number of iterative steps (which is equivalent to controlling the granularity of drawing). Specifically, we inject the prompt "You must finish within n_iterations by drawing multiple rows in parallel" into the prompt of iterative reasoning. n_iterations is set to 8, 4, 2, and 1 to investigate performance. The results are as follows:

It is interesting to find that set n_iterations to 4 improve performance over the original version and also enhancing efficiency. This suggests that more iterations do not necessarily lead to higher accuracy. When n_iterations is set to 1, the iterative reasoning process is almost skipped, resulting in poor performance. These findings support your previous comment (W2) that a more formulated generation approach is beneficial and worth further exploration. We appreciate the reviewer's comment for pointing out this.

Q2: "How frequent the generated images will content incorrect information? And how well the self-alignment module detect some very specific errors?"

We analyzed the frequency of generated images containing incorrect information and report the results in the following table:

We define an image as "correct" if it contains the integral information described in the text. From the result, we observe that the geometry and Sudoku benefited significantly from self-alignment. In geometry task, LLM would occasionally initialize coordinates with the wrong range, causing the shapes outside of the view. In Sudoku, LLM may have difficulty translating position descriptions to image coordinates. For example, in a 9x9 board, the correct API call grid position "1 2 4" should be translated to plt.text("1", x=1.5, y=3.5)(placing the element in the center of the grid). However, the LLM occasionally make mistakes, such as:

• plt.text("1", x=2, y=4), placing at the bottom right corner
• plt.text("1", x=1, y=3), placing at the top left corner
• Missing this element

These mistakes can be detected by self-alignment because the LLM will describe the content of the image and check it against the original input, which is proved to be crucial in our ablation study.

We thank the reviewer for the in-depth review! Below, we respond to the weaknesses raised in the review.

W1.1:"For matplotlib and turtle, how can the framework guarantee the syntax correctness of generated code?"

We understand the reviewer's concern. We add an experiment for code generation assessment and find the LLM-based code generation is sufficiently robust to guarantee syntax correctness by itself. The following table presents the ratio of syntax error and runtime error:

Here, LLM produces no syntax errors. Because the code generation for API call is not challenging for LLM, without involving complex logic. For example, "a circle center at (1,3) with radius 2" is translated to plt.circle(c=(1,3),r=2). We can observe slight runtime error in the Sudoku task. These errors occurred when translating Sudoku positions to actual coordinates in the image. For example, in a 9x9 board, the grid position "1 9 1" should be translated to plt.text("1",x=0.5,y=8.5)(placed in the center of the grid). However, the LLM occasionally made mistakes like plt.text("1",x=1,y=9), causing out-of-bounds placement.

W1.2: "For dalle3, how is the prompt generated for the image drawing?"

For DALLE3, LLM also generates an API call to create the image(OpenAI offers a package for DALLE3).

As DALLE3 is not selected in the tasks presented in our experiment(detailed in W1.3), we take another task we have previously tried as the example. This task is creative writing task, where LLM is asked to write storied based on some keywords.(We did't put the task in manuscript due to the lack of a convincing performance metric)

Example of this task:

Input: "Write a story according to given key words: Mage, Warriors, Priest"

Output: "A light breeze swept the ground, ..."

In this example, VAP first make a plan like:

{
  "tool": "DALLE3",
  "initialization": "dalle3 = OpenAI().images",
  ...
}

Each iteration, VAP will generate an image using API call. For example, img = dalle3.generate(prompt="The Mage dressed in a dark cloak, holding a staff and surrounded by a magical aura.", size="1024x1024").

W1.3: "What is the distribution of calls on these three different tools?"

As responded in W1.2, DALLE3 is not selected in the four tasks. So, we report the distribution of calls on the tools of Matplotlib and Turtle in the following:

Matplotlib is used more frequently, especially for time series prediction, due to its ability to construct coordinate systems efficiently.

We've also explored the relation between selected tool and performance, which can be found in response to Reviewer gmDj.

W1.4: "In iterative reasoning, the selected tool draw a complete image, or overlay the new drawings from previous iteration?"

In each iteration, our method uses the drawing tool to overlay the new drawing on the image from the previous iterations.

For example, consider a geometry problem with two shapes, as follows:

There's a circle centered at (3, 2) with radius 1
There's a circle centered at (6, 1) with radius 4
How many intersection points are there?

In the second iteration, the LLM will use the API call plt.circle(c=(6,1),r=4) to draw an additional circle on top of the previous image incrementally.

W1.5: "Can the framework use a mixture of different tools in resolving one problem?"

Unfortunately, our method does not support using a mixture of different tools during iterative reasoning. The drawing tool is determined in the planning step by LLM and remains fixed to ensure consistency.

W2: "I wonder if the authors can include benchmarks with both text and image in question (e.g., VQA benchmarks). This is a fairer setting since the current baselines ignore the visual capability of the GPT4v model."

We understand the reviewer's concern regarding the fairness of baseline comparision. It can be challenging to create an absolutely fair setting when baselines only require text ability of VLLM. Our ablation study may provides valuable insights. When the iterative reasoning stage is removed, it downgrade VAP to standard prompting with visual ability. And the results show a clear superiority of VAP over such a baseline.

We also take the reviewer’s advice to investigate VQA benchmarks. However, we find our task setting differs from VQA. Our work focuses on enhancing text-only problems using the additional image channel, while VQA typically involves answering questions based on provided images. This represents a distinct area of research.

W3: "In ablation studies, there is one experiment that removes the planning step. I wonder if the authors can provide more details. Does it mean that a different set of iterative reasoning prompt is used?"

We apologize for the lack of details. When the planning step is removed, we lose access to meta-information like selected tool, which is used to fill into iterative reasoning prompt template. Therefore, we use an alternative prompt template for iterative reasoning. The key changes to this new prompt will be detailed in the Appendix in revision. The key changes of this prompt include:

• No specific draw content and thought content given

..., provide your thoughts according to this problem.

• No specific tool given

... update the image content using Python API calls.

W4: "Typos at Line 294"

Thank you for pointing these out. We have fixed the typo and it's indeed that Sudoku benefits most from self-alignment. We have revised the statements accordingly.