Once Upon an Input: Reasoning via Per-Instance Program Synthesis

We thank the reviewer for the thoughtful comments. We address the reviewer’s specific concerns in detail below.

Baselines

While the paper [1] does not present a zero-shot method for LLM reasoning, we have added a comparison with Buffer of Thoughts [2] as suggested by the reviewer. This method uses other test problems from the same task to help with determining an effective method for solving other problems of the task. Overall, we find that this is highly ineffective for BBEH problems, performing even worse than just CoT or PoT alone.

We note that we also included an agentic baseline in our original submission, the Gemini code interpreter, which we described on line 188. For another agentic baseline, we have included the smolagents implementation of a CodeAct agent, and we provide the results for all BBEH datasets using Gemini 2.0 Flash below. The complete results for all models will be included in the camera ready.

[1] When Do Program-of-Thought Works for Reasoning? AAAI 2024. [2] Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models. NeurIPS 2024 Spotlight.

LLMs Used

Following the reviewer’s suggestion, we have evaluated the Qwen3-235B-A22B-Instruct-2507 model as an open-weights model. We report preliminary results over all of the BBEH tasks for this model and will include the complete results in the camera ready version. We observe a 1.3% absolute improvement in harmonic mean accuracy.

20% Holdout Set for Calibration

We thank the reviewer for bringing this up. We agree that for a strict zero-shot setting, the use of a validation set is not very standard. We only use the validation set for the switch and do not use any of the validation samples in any of the prompts or for improving the downstream CoT or synthesis methods. It is possible to implement the switch without any validation samples by just using the value of the tenth question rather than training a classifier to combine all ten questions. This zero-shot switch correctly chooses between CoT and synthesis 60.3% of the time for the problems where the outcome correctness of CoT and synthesis differs. Since choosing the method which performs the best between CoT and synthesis on these problems results in only 51.2% accuracy, this result is much better than both CoT or synthesis alone, but slightly worse than the trained switch which achieves 65% accuracy. We also evaluate a leave-one-out evaluation setting where the switch used on one dataset is only trained on the 20% validation samples from other datasets. This method, called “PIPS w. LOO” shows that the switch generalizes to held out datasets rather than just held out samples.

We will include results for the setting with a switch implemented without any validation samples in our revised paper as well as the leave-one-out ablation. The PIPS w. LOO method still outperforms all baselines. For all models except GPT 4.1-mini (which already had the weakest performance), PIPS with a zero-shot switch still significantly outperforms the strongest baselines.

Cost Overhead of PIPS

We have calculated the average input and output tokens as well as average cost for each baseline method and PIPS for Gemini 2.0 Flash. We find that the iterative approaches (CodeAct and Buffer of Thoughts) increase cost over 10X compared to PoT or CoT, but PIPS is overall only 3-4X more expensive than CoT or PoT while achieving much greater accuracy.

Quantitative Assessment of Semantic Fidelity of Input to Symbolic Mapping

Since our method first abstracts the raw input into a symbolic form, represented as JSON, there may be issues in this raw-to-symbolic mapping, as noted by the reviewer. Our preliminary investigation revealed that the vast majority of problems occur in the code itself (shown in Figure 3) and the model is often very good at the input to symbolic form translation. Now that we have addressed some of the most common problems, we think finer-grained issues should be studied in greater depth in future work, such as carrying out annotation efforts to determine the fidelity of the raw input to symbolic form mapping.

Low Improvement for GPT 4.1-mini

Our method results in only a 0.8% absolute increase in harmonic mean accuracy for GPT 4.1-mini, but this is our worst result out of many strong results across other models. In addition, we are adding an open-weights model, Qwen3-235B-A22B-Instruct-2507 (results are included above), where we see better results.

We believe PIPS has a low effectiveness with GPT 4.1-mini because the model struggles at revising generated code based on the structural feedback we provide, due to its overall weak performance. For o4-mini, a model which performs better overall in terms of the baselines, we see that PIPS improves over PoT performance by 5.7% absolute harmonic mean accuracy. Therefore, we still see that stronger models significantly benefit from using PIPS.

We thank the reviewer for the thoughtful comments. We address the reviewer’s specific concerns in detail below.

Confidence Metric

For our confidence-based switch between CoT and code synthesis, the 65% reported accuracy is only over the problems where exactly one of synthesis or CoT gets the correct answer, so this excludes cases where both are correct or both incorrect. Out of these problems where the outcome of CoT and synthesis differ, the best performing method out of the two gets 51.2% accuracy, so 65% accuracy is a significant improvement over both CoT and synthesis. There is still room for improvement in this switch (possibly through probe-based approaches as suggested by the reviewer), but we wanted an approach that would be easy to apply to new models, which to-date often do not provide hidden state access.

Evaluation of Baselines on 80% Non-Heldout Data

The reviewer is correct that we evaluated all methods on the 80% of the data that was not included in the heldout set used for calibration.

Iterative Baselines

The reviewer is correct that the PoT baseline did not allow for regeneration and fixing of code errors. The original version of PoT and other related methods such as FCoT [1] and PAL [2] also do not reprompt the model if there are execution errors. To address this concern, we re-run PoT with retries. Addionally, in our experiments, we included the Gemini code interpreter as a baseline which can iteratively write and debug code, as mentioned on line 188. For another such baseline, we have added the smolagents library implementation of CodeAct [3], which we find performs poorly on BBEH with gemini-2.0-flash. We will include the CodeAct results for all models in the camera ready version as well as Buffer of Thoughts (a baseline suggested by another reviewer).

Finally, we perform an additional experiment where we resample a potential solution using PoT whenever the generated code results in a code related error or a null result. The following shows the harmonic mean accuracy over all BBEH tasks using Gemini 2.0 Flash as the base model. We see that resampling is minimally beneficial by itself.

[1] Lyu, Qing, et al. "Faithful chain-of-thought reasoning." IJCNLP-AACL 2023. [2] Gao, Luyu, et al. "Pal: Program-aided language models." ICML 2023. [3] Wang, Xingyao, et al. "Executable code actions elicit better llm agents." ICML 2024.

We thank the reviewer for the thoughtful comments. We address the reviewer’s specific concerns in detail below.

Method Generality

Our method results in only a 0.8% absolute increase in harmonic mean accuracy for GPT 4.1-mini, but this is our worst result out of many strong results across other models. We also emphasize that we follow the setting from the BBEH authors of using the harmonic mean accuracy over tasks which is a particularly challenging metric as it downweights easier tasks and significantly upweights the importance of the hard tasks. Even though harmonic mean accuracy improvement is low for GPT 4.1-mini, we see a 11.3% absolute improvement over PoT on Boolean Expressions and a 33.1% absolute improvement over PoT on Dyck Languages.

In addition, we are adding an open-weights model, Qwen3-235B-A22B-Instruct-2507. We report preliminary results over all of the BBEH tasks for this model and will include the complete results in the camera ready version. We observe a 1.3% absolute improvement in harmonic mean accuracy.

We believe PIPS has a low effectiveness with GPT 4.1-mini because the model struggles at revising generated code based on the structural feedback we provide, due to its overall weak performance. For o4-mini, a model which performs better overall in terms of the baselines, we see that PIPS improves over PoT performance by 5.7% absolute harmonic mean accuracy. This trend is repeated for Qwen3-235B-A22B-Instruct-2507, which is also more performant in terms of baselines.

Upfront Choice Between CoT and Synthesis

We decided to use an upfront choice between CoT and synthesis since we observed that most of the reasoning problems we were considering were solvable directly by code when the problem was “algorithmic”. For these cases, the problem was that the model could determine that the problem was solvable via program synthesis, but when prompted to generate the code, it failed to produce well-formed code. Since this was the case, even for a system that can interleave code and CoT based reasoning, the problem with code synthesis would remain.

A natural generalization of PIPS is to enable a dynamic problem-solving approach, perhaps where some subproblems are solved by synthesis and others by CoT. We believe this is an interesting future direction, but we did not find it characteristic of the BBEH problems we studied.

Agentic / Iterative Baselines

We note that the paper included an agentic baseline, the Gemini Code Interpreter, which we describe on line 188. To provide another agentic baseline, we have additionally evaluated the smolagents library implementation of a CodeAct [1] agent and we include the results below for Gemini-2.0-flash.

We find that CodeAct performs poorly due to several reasons. One issue is from multiple choice questions where the agent does not respond with a letter even when we explicitly give instructions for answer formatting. The other big problem can be seen for object counting and shuffled objects (two highly algorithmic tasks) where CodeAct performs very poorly due to approaching the problem by performing each low level computation with the Python interpreter and everything else with CoT rather than solving the high level problem with code. We will include CodeAct results for all models for the camera ready version.

We additionally include results for the Buffer of Thoughts [2] method over BBEH as suggested by another reviewer. This method uses other test problems from the same task to help with determining an effective method for solving other problems of the task. Overall, we find that this is highly ineffective for BBEH problems.

[1] Wang, Xingyao, et al. "Executable code actions elicit better llm agents, 2024." ICML 2024. [2] Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models. NeurIPS 2024.

Limited Feedback Mechanism

When we studied the failure modes of current models when generating code for solving reasoning problems, we found that the overwhelming majority of problems came from failing to generate code and the model resorting to textual reasoning. Since this was the most common issue prior to our work, our primary objective in this paper was to address this problem with a simple, but general framework. With this issue now addressed, we agree with the reviewer that more sophisticated forms of feedback will be necessary to address deeper problems such as logical errors now that the code is well-formed, and we are excited about future work in this direction.

Analysis of Computational Overhead

We include the average input and output tokens and the average cost for all methods below using Gemini 2.0 Flash with costs determined based on current Gemini API costs. We find that the iterative approaches (CodeAct and Buffer of Thoughts) increase cost over 10X compared to PoT or CoT, but PIPS is overall only 3-4X more expensive than CoT or PoT while achieving much greater accuracy.

We thank the reviewer for their thoughtful comments and for recognizing the effectiveness and originality of our work. Following the reviewer’s comment about improving the breadth of our related work section, and the pointer to two relevant papers, we have revised the related work and believe it is much improved. We will add the following to the current related work section:

Approaches which prompt an LLM to produce code to solve a problem have been used in several domains beyond math and text-based reasoning questions. ViperGPT and followup work tackle visual question answering problems [1, 2], Voyager applies to game playing [3], and Code as Policies focusses on the application of robot control [4]. Recently, general systems such as CodeAct [5] and OpenCodeInterpreter [6] have been proposed for solving problems via code generation similar to the previously mentioned solutions for each task.

[1] Surís, Dídac, Sachit Menon, and Carl Vondrick. "Vipergpt: Visual inference via python execution for reasoning." ICCV 2023.

[2] Koo, Jaywon, et al. "PropTest: Automatic Property Testing for Improved Visual Programming." EMNLP Findings 2024.

[3] Wang, Guanzhi, et al. "Voyager: An Open-Ended Embodied Agent with Large Language Models." TMLR 2024.

[4] Liang, Jacky, et al. "Code as Policies: Language Model Programs for Embodied Control." ICRA 2023.

[5] Wang, Xingyao, et al. "Executable code actions elicit better llm agents." ICML 2024.

[6] Zheng, Tianyu, et al. "OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement." ACL Findings 2024.

We then additionally include CodeAct as a baseline and results for Gemini 2.0 Flash are included in the response to reviewer ki2H. We see that CodeAct significantly underperforms even PoT. The full results will be included in the camera ready version.