Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

Dear Reviewer,

We appreciate your time and great feedback for improving our work! We would like to address each of your concerns below.

Comment 1: "What is the drop (or increase) in performance in other tasks..."

This is a very interesting experiment! To answer this question, we have conducted experiments on the HumanEval [1], HumanEval+ [2], MBPP [3], and MBPP+ [4] tasks. These are popular program synthesis tasks. The results are shown below:

We found that fine-tuning on the SIM dataset leads to a small performance drop (3-6%) on other program synthesis benchmark tasks. Similar phenomena have also been observed in other works such as [5,6]. We hypothesize this is because the SIM dataset focuses on visual programming tasks which place more focus on visual understanding, spatial reasoning, and planning, which are not directly related to other Python program synthesis tasks. Therefore, it does not bring additional knowledge to solving other benchmark tasks in HumanEval and MBPP. Instead, such a fine-tuning process may cause the model to forget some knowledge that it has learned from the previous training stage, finally resulting in the performance drop in other benchmark tasks. I hope this could provide some insights into your question.

We have also included this interesting experiments in the updated version of our paper (See Appendix D.4).

Comment 2: "What is the role of the prompt engineering in solving these tasks?..."

We appreciate the reviewer's feedback and would like to elaborate on our efforts in prompt engineering.

We developed the prompt in the following manner:

First, we considered using natural language to describe the visual task grid, as it is the most natural and straightforward way for LLMs to understand. When creating the natural language prompts, we began by drafting an initial version and then requesting outputs from GPT-4 and Llama3-70B. We analyzed these models' outputs and adjusted the prompt to ensure that if the output code was incorrect, it wasn’t due to missing information in the prompt. If any information was missing, we incorporated it into the prompt. We repeated this process several times for multiple tasks in our evaluation dataset. Eventually, we arrived at a final version of the prompt and then used ChatGPT to refine it further to ensure clarity and eliminate ambiguity.

However, since our domain includes the visual task grid, natural language may not be the optimal representation. Therefore, we also investigated whether we could represent the visual task grid differently. Specifically, we examined an ASCII representation of the task grid, as illustrated in Figure 14 (See Appendix). Through our experiments, we found that natural language descriptions were more effective than ASCII representations for both GPT-4 and Llama3-70B models. However, when fine-tuning the models using these representations, the differences in effectiveness between ASCII-based and natural language-based prompts were minimal.

Due to our visual programming nature, we also tried to use the visual task image as an additional input. The prompt was almost identical, except we explicitly stated that the image is also provided.

These are the efforts we made in prompt engineering for our domain. I hope this clarifies your concerns.

Comment 3: "I would probably not call the REAL dataset "real-world" tasks..."

Thanks for this great feedback! We agree that the term "real-world" might be misleading. However, updating the term in the paper may confuse other reviewers at current stage. So we keep the term as it is in the current version. However, we promise to use a more appropriate term in the final version after the rebuttal phase. Thank for this great point!

Comment 4: "It would also help if you shoed an example of a SIM task"

Thank you for this suggestion! We've added a few examples of a SIM task in the Appendix (See Figure 12) due to the page limit. We hope this can provide a better understanding of the tasks in the SIM dataset.

References:

[1] https://github.com/openai/human-eval

[2] https://huggingface.co/datasets/evalplus/humanevalplus

[3] https://huggingface.co/datasets/google-research-datasets/mbpp

[4] https://huggingface.co/datasets/evalplus/mbppplus

[5] Towards Reliable Latent Knowledge Estimation in LLMs: In-Context Learning vs. Prompting Based Factual Knowledge Extraction. https://arxiv.org/pdf/2404.12957

[6] Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models. https://arxiv.org/abs/2406.11201

Dear Reviewer,

Thanks for your constructive feedback! We appreciate your feedback and would like to address each of your concerns.

Comment 1: "Lack of Task Challenge..."

Lack of Task Challenge: The task presented in this paper appears to lack sufficient challenge, as evidenced by Table 6...

To understand whether our benchmark is challenging, we would like to compare with other related benchmarks.

For example, we can compare our benchmark with MMMU [1,2], a widely used multimodal benchmark:

1. On MMMU (Val), GPT-4V achieves 56% accuracy without fine-tuning (https://mmmu-benchmark.github.io/#leaderboard).
2. On our benchmark, GPT-4V only achieves 20% accuracy without fine-tuning (See Table 6 in our paper).
3. Even our best fine-tuned model (Llama3-8B-Emu) achieves 60.23% accuracy, which is similar to GPT-4V's performance on MMMU. There is still a large room for further improvement on our benchmark.

We can also compare our benchmark with other popular program synthesis benchmarks, such as MBPP [3] and HumanEval [4]:

1. On MBPP [3], GPT-4 achieves 73.3% pass@1 accuracy. On HumanEval [4], GPT-4 achieves 86.6% pass@1 accuracy.
2. On our benchmark, GPT-4 achieves 12.94% accuracy (See Table 6 in our paper).
3. The performance of GPT-4 on our benchmark is much lower than its performance on MBPP and HumanEval, indicating that our benchmark is inherently challenging. We believe this is because our benchmark requires multiple combined skills, such as spatial reasoning, programming skills, etc.

Finally, we believe that the challenges of our benchmark come from the fact that the tasks are originally designed for K-2 students, making them quite simple for human experts. In comparison, tasks in other benchmarks (e.g., MBPP, HumanEval) are designed for beginners to entry-level programmers, which are much harder than our tasks. However, we observed that models' performance on our benchmark is still far behind their performance on MBPP and HumanEval, indicating that our benchmark is quite challenging and interesting for further study. We had also provided more insights to understand why models fail on our benchmark (See Section 5.3).

Comment 2: "Narrow Scope of Visual Programming..."

Thank you for the feedback! Regarding your concerns about the narrow scope of visual programming and the limited code space, we would like to clarify from the perspectives of both task space and code space.

First, while our benchmark focuses on the XLogoOnline environment, it maintains a broad task space. As illustrated in Figure 1, our tasks span diverse types of visual reasoning, each requiring different programming concepts and spatial understanding (e.g., math, geometry, etc.). The combination of varied task types and different code constraints creates a large task space.

Second, the simplified code space in our benchmark also creates unique challenges. While this simplified code space makes tasks easier for humans, it actually poses significant difficulties for models. Our failure analysis (See Section 5.3) shows that models often struggle to adhere to the constrained code space, frequently generating invalid commands from other domains (e.g., attempting to use turn_around(), which is not part of our command set). This indicates that models face challenges when operating within a more restricted code space.

Comment 3: "Insufficient Evaluation of Large Multimodal Models..."

Thanks for this valuable feedback! According to your suggestion, we have expanded our model evaluation since the initial submission. We've added 7 new multimodal models in our evaluation. Here is a summary of the success rates for all the multimodal models evaluated in our paper:

We have also updated our paper to reflect these changes (See Table 6).

Thank you for your valuable feedback for improving our work. We hope our responses have addressed your concerns and you would consider raising the score. If you have any further questions, please feel free to reach out and we are more than happy to answer them!

References:

[1] https://evalplus.github.io/leaderboard.html

[2] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. https://arxiv.org/abs/2311.16502.

[3] MBPP: Program Synthesis with Large Language Models. https://arxiv.org/abs/2108.07732.

[4] HumanEval: Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374.

We hope that our responses have addressed the reviewer's concerns and questions. Given that the discussion period is ending soon, we would greatly appreciate the reviewer's input. If you have more questions, we would be glad to provide further responses. Thank you for your feedback!

Dear Reviewer,

We appreciate your feedback and would like to address each of your concerns.

Comment 1: "The paper only provides results on two VLM families..."

Thanks for this valuable feedback. We've added 7 new multimodal models in our evaluation. Here is a summary of the success rates for all the multimodal models evaluated in our paper:

We have also updated our paper to reflect these changes (See Table 6).

Comment 2: "Need clarity in the data generation process..."

This is an great question! In our case, all the generated (task, code) pairs in our dataset are correct. Since our method does not involve LLMs and does not generate explanations, reasoning, or comments for the code, there are no cases where the code is correct for the wrong reasons. Additionally, we have included a more detailed explanation of our dataset generation process in Appendix B.1. We hope this clarifies your concerns!

Comment 3: "Lacks detailed explanation of the emulator..."

Thanks for the feedback. Our emulator runs the code for a given task and providing detailed execution results and messages, similar to a Python interpreter. This process is designed to be precise and unambiguous.

The emulator operates in the following way:

1. Given a (task, code) pair in our domain, the emulator runs the code for the task and then returns the execution result.
2. During execution, the emulator checks the code format, execution crashes, code constraints, and whether the code meets the task's goal. The code constraints and the task's goal are specified in JSON format for precise and unambiguous checking. When creating prompts for models to generate the code, these code constraints and goals were translated into natural language using a fixed translation template. For example, a task's code constraints and goal might be translated as "Find the strawberry using at most 8 commands."
3. After all above checks are performed, the emulator finally provides the execution result, which is either success or an error message indicating the specific reason for the failure. For example, when code execution is successful for a task, the execution result is success. If there is an error, such as hitting the wall, the emulator generates the appropriate error type and message.

We have noticed that this might be not clear in the original paper. So we've incorporated this clarification in the updated version of the paper in the Appendix (see Section C). Please let us know if you have any further suggestions or concerns. We're happy to address and improve the work.

Comment 4: "It's very interesting that fine-tuning a VLM (Llava1.5-13B-Uni) performs worse than fine-tuning an LLM (Llama2, Llama3)..."

Thank you for your insightful question. We have also been thinking about this question. We list some of our thoughts for potential reasons below:

First, the choice of base model may affect performance. Llava1.5-13B is built on the Llama1 model, which may limit its performance when compared to Llama2 and Llama3. We've noticed that fine-tuning more advanced models typically results in better performance (for example, fine-tuning Llama3 yields better performance compared to fine-tuning Llama2). Therefore, fine-tuning a state-of-the-art vision language model with an improved base model (e.g., Llama3.1, Qwen2) might also enhance its performance.

Besides, we observed significant instability in Llava1.5-13B-Uni's performance during fine-tuning. We found that, after fine-tuning Llava1.5-13B with 5 different seeds, only one seed achieved performance comparable to the fine-tuned Llama3 models (around a 54% success rate). The other 4 seeds failed to reach similar levels of performance, resulting in an almost zero success rate. This is also why the standerr for Llava1.5-13B is much higher than fine-tuning other models. This may suggest that a more complex model with both visual and language components might require more careful fine-tuning to achieve stable performance. In our experiments, the visual component of the Llava model is not being fine-tuned. If the visual encoder has inherent limitations in understanding the task images, the integrated image may also confuse the model instead of enhancing its performance.

These are our thoughts about the potential reasons. There may be other factors at play that we have not considered. We will keep exploring this question in the future. Thanks for this insightful question! We appreciate your feedback and are happy to address any further concerns.

Dear Reviewer nRED,

Thank you for recognizing our work! We hope our responses have addressed your concerns and questions. As the discussion period is coming to a close, we would greatly appreciate any further input you may have. Thanks again for your great feedback!

Dear Reviewer,

Thank you for time to provide such detailed and thoughtful feedback on our work. We would like to address your concerns one by one.

Comment 1: "The emulator-driven fine-tuning provides only binary correctness feedback..."

The emulator-driven fine-tuning provides only binary correctness feedback on the predicted code, which might not be sufficient for identifying and correcting specific errors in the generated code. Can it be in natural language?

You are absolutely right in pointing out that our emulator-driven fine-tuning technique currently provides binary correctness feedback. This emulator's feedback is used to determine the likelihood of a sample being included in the training process for the next epoch.

Regarding your suggestion about providing feedback in natural language, our emulator is indeed capable of generating natural language feedback, similar to the error messages typically seen in Python. These execution messages could be leveraged to calculate the probability of a sample being used, depending on the nature of the error. For example, if the error is minor, we could increase the likelihood of including the sample, whereas for more significant errors, the probability could be reduced.

This is a valuable suggestion, and we recognize its potential. However, given that our current focus is on datasets and benchmarking model performance, we have left the exploration of natural language feedback for error correction as a topic for future work. We appreciate your insightful feedback, and it will certainly be considered in the direction of our future research.

Comment 2: "There's a risk that models could overfit to the synthetic data..."

There's a risk that models could overfit to the synthetic data used in training, which might not perfectly mimic the complexity and variability of real-world programming challenges. I think this is the most serious disadvantage

We appreciate the concern regarding the potential for models to overfit to synthetic data.

To understand if our model was overfitting to synthetic data, we analyzed its performance across different epochs (Figure 15 in the Appendix). We observed that the model's performance on the synthetic dataset (SIM-EVAL) continued to improve after epoch 6, while its performance on the real-world dataset (REAL) plateaued. If we continue to fine-tune the model on the synthetic training dataset for more epochs, it might overfit to the synthetic data. Consequently, we limited the fine-tuning to 8 epochs to mitigate overfitting. On the other hand, we acknowledge that generating synthetic data that perfectly mimics the complexity and variability of real-world programming challenges is inherently difficult, and we will continue to address this issue in our future work.

Thank you for your valuable feedback. We hope our response has addressed your concerns and positively influenced your evaluation. We appreciate your time and effort to improve this work!

We hope that our responses have addressed the reviewer's concerns and questions. Given that the discussion period is ending soon, we would greatly appreciate the reviewer's input. If you have more questions, we would be glad to provide further responses. Thank you for your feedback!

Dear Reviewer,

Thank you for your time to provide such detailed and thoughtful feedback on our work. We are grateful for your constructive comments and suggestions. We would like to address each of your comments in detail in the following.

Comment 1: "This paper provides few new insights about identifying models' limitations..."

This paper provides few new insights about identifying models' limitations: This paper shows that existing models fail in visual programming in XLogoOnline. But this task is similar to some tasks in works like MMMU, which also combine different skills like vision and reasoning and have already shown that current models struggle in visual reasoning tasks

We appreciate this observation and would like to emphasize the key differences between our benchmark and MMMU:

1. Focus on Different Domains:

   • MMMU addresses broader challenges in multimodal understanding and reasoning, whereas our benchmark focuses on program synthesis for visual programming tasks. This fundamental difference in focus also leads to differences in task types.
   • For example, MMMU tasks primarily involve multiple-choice and open-ended questions with unambiguous correct answers. In contrast, our benchmark tasks require models to generate executable code snippets for grid navigation. These solution codes are not required to be unique but must adhere to specific coding and grid constraints to achieve the desired goals. As far as we know, elements such as writing code, navigating grids, and satisfying constraints are absent in MMMU.

2. Emphasis on different Skill Sets:

   • While both benchmarks test visual understanding and reasoning, our benchmark introduces additional challenges requiring programming skills, planning, and spatial reasoning. These aspects are not explored in MMMU.

Moreover, our analysis (see Section 5.3) reveals that spatial reasoning remains a significant weakness in large models, despite their strong performance in other domains. We believe that our benchmark provides valuable insights into the limitations of large models, particularly in this underexplored area.

Comment 2: "This paper also lacks novelty about improving models' performance..."

This paper also lacks novelty about improving models' performance:

• The data synthesis method is based on sampling and filtration, which has been widely adopted in works like STaR, RaFT and Rest-EM.

We would like to clarify the differences between our approach and existing methods such as STaR [1], RaFT [2], and Rest-EM [3]:

1. Answer-First Synthesis Approach:

   • Unlike the question-first approaches used in STaR [1] and Rest-EM [3], where questions are generated first and answers synthesized later, we adopt an answer-first approach.
   • In our method, we start by synthesizing the answer (i.e., code) and then derive the question (i.e., task) to ensure it aligns with the synthesized answer. This approach is particularly effective in our domain, as existing LLMs are not well-trained on domain-specific data. Using a question-first approach would result in poor performance, as demonstrated in Table 6, where Llama3-70B achieves only 2.35% accuracy in generating correct domain-specific answers.

2. Symbolic Execution and Constraint Solving:

   • Our synthesis technique does not rely on LLMs for generating data, as seen in STaR [1], RAFT [2], and Rest-EM [3]. Instead, we leverage symbolic execution and constraint solving.
   • Specifically, we begin with a reference (task, code) pair. By mutating the reference code, we generate variations and then synthesize tasks solvable by the mutated code. This ensures the generated data is both correct and tailored to the benchmark, a critical distinction from existing works.

We hope that our responses have addressed the reviewer's concerns and questions. Given that the discussion period is ending soon, we would greatly appreciate the reviewer's input. If you have more questions, we would be glad to provide further responses. Thank you for your feedback!