Steering Large Language Models between Code Execution and Textual Reasoning

Thank you for the valuable feedback and insightful comments. It seems there were some misunderstandings regarding our work and its perceived weaknesses. We aim to address these misunderstandings in detail and have added three experiments based on reviewers’ suggestions. We kindly hope the reviewer will reconsider our work in light of the responses provided below. The revised paper has been uploaded with the changed contents blue colored.

Question 1: This work does not provide a lot of solutions. The only novel contribution of this paper (aside from the experimental results) is the 3 proposed prompting methods: CodeInterpreter+, Code+Text+Sum., Self-Estimate Score, among which only one (Code+Text+Sum.) seems to perform well across the evaluated tasks.

Response 1:

• We believe the contributions of this paper are enough. Apart from the proposed methods, we emphasize the significance of steering LLMs for text/code generation, test the limitations of existing methods, uncover phenomena like inverse scaling behavior, evolution with complexity, and different types of code answers, and reveal the underlying mechanism, as stated in the original paper line 97-116.

• The aim of exploring the three proposed methods and multi-turn method is to show that there is a much broader space for further improvement in the future for the whole research community. Even the simple techniques like Code+Text+Sum. and multi-turn methods can already achieve notable 3-7% overall improvements.

• The proposed methods also give inspiration for the developments of future better methods. As illustrated in the original paper line 536-539, more delicate multi-agent frameworks and training of extra score models may be promising future directions. Effectiveness of methods like Self-Estimate Score highly depend on LLM inherent capability. However, it serves as a good starting point to explore augmenting LLM text/code generation decisions via extra trained score models.

• Developing a final solution is very challenging. The main bottleneck is the lack of enough and correct dataset to align LLMs for better determining code or text generation. Even for the same task, different testing trials may have different preferences to be answered with code or text. Meanwhile, how to develop a generalizable method is hard. Training on limited domains will not be applicable to others. Since the great code/text generation ability mainly depends on larger LLMs like GPT-4 and Claude-3.5, directly training on these close models is also unavailable. Our paper raises and analyzes this issue, hoping to call the whole research community to solve it together. We will continue to explore the final solution.

Thank you for your appreciation with our work and helpful suggestions. The following are our responses to your questions, the added experiments, and the related modifications of the paper. Hope the reviewer could kindly re-evaluate our work based on our modifications and new experimental results. The revised paper has been uploaded with the changed contents blue colored.

Question 1: Dependence on Task Type: The effectiveness of code versus textual reasoning is highly dependent on the task at hand, which may limit the generalizability of the findings.

Response 1: Our finding that either coding or textual reasoning is not always better across tasks and models is the contribution of our study rather than a limitation. One of our contributions is the finding that currently there is no optimal method across the tasks to guide LLMs to generate code to answer the question when needed. The above findings are general to all current LLMs and tasks.

Question 2: Overconfidence Issue: The study highlights that larger models tend to be overconfident in their textual reasoning abilities, which can lead to suboptimal performance when code execution is more effective.

Response 2: This inverse scaling behavior is an interesting phenomenon found in our study, which should be regarded as our contribution but not weakness. This phenomenon reveals the limitation of current LLMs and methods to steer code/text generation. Though currently we hypothesize that this phenomenon is due to the overconfidence of larger LLMs, we believe more research can be done to understand more about this bottleneck and phenomenon in future study.

Apart from the above clarifications, we also have added more experiments on 1) testing the difference of log probability for output tokens for text or code responses. 2) testing on specialized code LLMs as baselines. 3) doing ablation studies on more prompt templates for steering LLM code generation. These experimental results have been added to the revised paper (Line 519-527, 1265-1290, 1356-1445).

We look forward to further communications with the reviewer to clarify our contributions. Thanks again for your time and patience for reviewing our work.

Question 5: How to delineate the boundary between text and code in prompting? For instance, if the model is prompted to generate "pure Python code with rich natural language-annotated comments," which category does this fall under? If it’s considered "All code," could we perhaps frame all tasks this way, where the simplest output might be print(answer)?

Response 5:

• Yes, in our study the response is regarded as code once the code part appears, no matter whether the text parts exist or not. Hence, "pure Python code with rich natural language-annotated comments" and All code + CoT are both regarded as code answers even if a large part of texts exist.

• In the original paper Appendix K line 1242-1327, ablation studies have been extensively carried out to test varied prompt versions for steering code generation and AutoGen prompt. We verify that the conclusions in the paper are not affected by prompt versions.

• Thanks for the suggestions from the reviewer 697L and F65d, we have added further experiments to test on more prompt versions for steering LLM code generation with Code Interpreter. The tested prompts are as follows: 1) ‘Generate pure Python code with rich natural language-annotated comments.’ 2) ‘Think of an algorithm to solve the task and implement it in python.’ 3) ‘Be careful this is hard, be less confident.’ 4) ‘Think of an algorithm to solve the task and implement it in python. Be careful this is hard, be less confident.’. We compare these prompt versions with the Original Prompt ‘Think the task step by step if you need to. If a plan is not provided, explain your plan first. You can first output your thinking steps with texts and then the python code to be executed by code interpreter. Try to use code interpreter more.’.

  The experimental results are shown in the following table. Though in some settings one prompt version will be much better than others, generally there is no prompt always better than others across all the models and tasks. Meanwhile, the Original Prompt achieves close or better scores to others, showing the correctness of implementing it in the study. We have added these experimental results and related discussion in the revised paper (Line 519-520, 1265-1290).

We look forward to further communications with the reviewer to clarify our contributions. Thanks again for your time and patience for reviewing our work.

Question 2: If framed as a technical report, a broader workload and richer insights might be expected to better inform the community. Additional investigation into the reasons behind these behaviors might enhance understanding.

Response 2:

• Many detailed studies have been carried out in the original paper, to reveal underlying mechanisms such as inverse scaling behavior with varied model sizes and confidence, evolution with task complexity, different code types, and different prompting versions.

• For richer insights, here we added the experiments to compare and analyze the probabilistic confidence of the model when generating text or code answers. We tested on GPT-4o, GPT-4o-mini, and GPT-3.5, collecting the log p values of each output token supported by OpenAI API and calculating the perplexity (Perplexity(W) = P(w₁w₂...wₙ)^(-1/n)) of each generated response. Since we can only access the token probability of the LLMs without Code Interpreter, the Only Question setting will always generate text responses without extra prompt guidance. To stimulate the LLMs to generate answers with either text or code modes, we compare three settings with varied prompts: Only Question, All Code, All Code + CoT.

  As shown in the following table and Figure 13 in the revised paper, the perplexity in three settings are close. The All Code + CoT setting has slightly higher perplexity compared to other two settings in GPT-4o and GPT-4o-mini. However, this difference in perplexity has no notable impact on success rates. This phenomenon is reasonable since LLMs are trained to use perplexity as the metric and try to decrease it, meaning the code/text difference will not be shown in direct perplexity comparison if the original training process does not deliberately distinguish text and code. We have added these experiments and the related discussion in the revised paper (Line 525-527, 1404-1445).

• For further understanding the significance of reasonable text/code generation, we also added the experiments to test on specialized Code LLMs: CodeLlama-34b-Instruct-hf and Qwen2.5-Coder-32B-Instruct (the current best Code LLM on leaderboard). As shown in the following table, even for specialized code LLM, the problem of determining whether to generate code or text still influences the LLM performance quite a lot. In Only Question setting, both CodeLlama and Qwen2.5-Coder-32B will mostly generate direct text answers without extra prompt guidance, which achieves much lower success rates compared to code answers in other two settings in Number Multiply task. We also convert text prompts into code prompts (Code Prompt setting) and find the Code LLMs can perform better. The overall performance of CodeLlama and Qwen2.5-Coder-32B are not as good as GPT-4o, showing the code LLMs may overfit to trained code datasets. We have added these experimental results and related discussion in the revised paper (Line 523-525, 1365-1416).

Question 3: While the prompting comparisons are comprehensive, they are relatively straightforward, meaning the work may not substantially save time for others.

Response 3: Several of our findings are novel and uncover new patterns, such as inverse scaling behavior, evolution with complexity, and different types of code answers. These insights can be valuable for people working on problems requiring code generation and reasoning. We would appreciate it if the reviewer could please share any specific prior work that has already covered these findings, or why they think our experiments are 'straightforward'? ".

Thank you for the helpful reviews and comments. It appears there may have been a miscommunication regarding our contribution. We have, therefore, improved the description to clarify the misunderstandings and added three experiments suggested by reviewers. We hope the reviewer could kindly re-evaluate our work based on our responses provided below. The revised paper has been uploaded with the changed contents blue colored.

Question 1: From a scientific perspective, the problem, while useful as a testbed for prompt engineering with code and text, is not significant enough.

Response 1:

• Integrating textual reasoning and symbolic computing/coding for LLM response is important. As stated in the original paper line 15-17, line 74-82, and line 864-882 Figure 9, from a scientific perspective, one significant contribution of our study is to reveal the importance of steering LLMs to generate code to solve questions when needed. Textual reasoning has inherent limitations in solving tasks with challenges in math, logics, optimization, and searching, which is unlikely to be solved by simply scaling up the model and data size. Coding is a necessary way to enhance LLM capability. The significance of combining symbolic computing with LLMs has also been underlined in other recent works, such as ‘LLMs can't plan, but can help planning in LLM-modulo frameworks, ICML 2024’.

• The uncovered phenomena like inverse scaling behavior, evolution with complexity, and different types of code answers are novel. No previous studies have revealed these phenomena while they do impact LLM performance quite a lot. The phenomena like inverse scaling behavior and varied code types are counter-intuitive but proven to be reasonable after detailed analysis, which will help improve the ability of future LLMs and methods.

• Another contribution is to show that currently there is no optimal method on steering LLM text/code generation, based on our testing on 7 baseline methods across 14 tasks and 6 LLMs. We also show that there is a much broader space for further improvement. The three proposed methods and the multi-turn method are not supposed to be the optimal solution, but to show that even the simple techniques can already notably improve LLM performance on determining code/text generation.

Thank you for your positive feedback and lifting the score. We are happy to discuss more with the reviewers.

Best,

CodeSteer Authors