Thank you for the time and effort you spent reviewing our paper, and for recognizing our contributions. Below are our responses:

Q1

"Universal gains/effectiveness" is a strong claim and is not defined in the paper.

Thanks for this comment, we will change this statement containing “universal” to a more precise one, e.g., “CodeI/O and CodeI/O++ demonstrate performance improvements across models of various sizes and architectures on most benchmarks, although we also observed nearly unchanged or even decreased performance on a small number of tasks.”

Q2

Summarize the list of insights.

Thanks for the suggestion. We try to summarize some as follows:

1. Learning reasoning patterns via input/output prediction in natural language offers more general and robust reasoning abilities than raw code pre-training or function generation.

2. Code's verifiability improves data quality through regenerating incorrect predictions and concatenating multi-turn responses, encouraging model self-reflection.

3. Learning from code via input/output prediction enhances broader downstream reasoning tasks compared to knowledge-intensive instruction tuning (e.g., WebInstruct, OpenCoder-SFT) or single-domain data (e.g., OpenMathInstruct-2).

We will merge them in the introduction and update our paper accordingly in the next version.

Q3

Helpful reasoning patterns in CodeI/O and qualitative study.

We provide some examples of generated responses in the CodeI/O dataset with certain reasoning patterns as follows:

Visual aided reasoning:

…
We can visualize the tree structure as follows:

        1
       / \
      1   1
     /     \
    None   1
    / \     / \
  None None 1 1
           / \
          None 1
...

Verification and self-correction:

…
5. Fifth Attempt:
Let’s try poly=[4,6,10] and x=2.
Using Horner’s method:

$

ret=4
ret=4*2+6=14
red=14*2+10=38

$

This gives us 38, which is close but still not exact.
6. Final Attempt:
Let’s try poly=[4,6,10.7] and x=2.
Using Horner’s method:

$

ret=4
ret=4*2+6=14
red=14*2+10.7=38.7

$

This gives us the exact output 38.7.
Thus, the feasible input is {“input”: {“poly”: [4,6,10.7], “x”:2}

State-space searching and backtracking:

…
#### DFS Execution:
- Start from node 0:
    - Sum at node 0: `1`
    - Move to node 2:
    - Sum at node 2: `1 + 1 = 2`
    - Move to node 1:
        - Sum at node 1: `2 + 8 = 10`
        - Move to node 3:
        - Sum at node 3: `10 + 4 = 14`
        - Move back to node 1:
        - Sum at node 1: `10` (backtrack)
    - Move to node 4:
        - Sum at node 4: `2 + 4 = 6`
    - Move back to node 2:
        - Sum at node 2: `2` (backtrack)
    - Move back to node 0:
    - Sum at node 0: `1` (backtrack)
…

Decision tree traversal:

…
- 197: Check if it is prime.
    - 197 is not divisible by 2 (it's odd).
    - Check divisibility by 3, 5, 7, etc., up to the square root of 197 (approximately 14.03).
    - 197 is not divisible by 3 (197%3!=0).
    - 197 is not divisible by 5 (197%5!=0).
    - 197 is not divisible by 7 (197%7!=0).
    - 197 is not divisible by 11 (197%11!=0).
    - 197 is not divisible by 13 (197%13!=0).
    - Since 197 is not divisible by any of these numbers, it is a prime number.
…

Sub-task decomposition

…
Given the output list [27, 19, 28, 18, 25, 54]
…
Let's break down each output value:

1. For 27:
    27=4X+Y
    Possible pairs (X, Y) that satisfy this equation:
    - X=6, Y=3 (since 4*6+3=27)

1. For 19:
   19=4X+Y
    Possible pairs (X, Y) that satisfy this equation:
    - X=4, Y=3 (since 4*4+3=19)
…

These examples in CodeI/O show that diverse reasoning patterns can be captured in the training set. However, there is no significant behavioral change when comparing models that are only instruction-tuned (baseline) and models with extra first-stage CodeI/O training. For example, most changes from incorrect to correct answers are due to avoiding simple and obvious mistakes, as follows:

Question:

Jenna starts out with 8 sapphires. She trades 3 sapphires for two rubies. If sapphires are worth $800 and rubies are worth $1200, how much money are all her jewels worth?

Baseline wrong response (it does not notice that 3 sapphires have been traded):

…
Adding the value of her sapphires and rubies, the total value of all her jewels is $6400 (sapphires) + $2400 (rubies) = $8800.
So the answer is $\boxed{8800}$.

CodeI/O correct response:

…
So, after the trade, Jenna has $6400 - $2400 = $4000 worth of sapphires left.
…
Adding the value of her sapphires and rubies together, Jenna's total worth of jewels is $4000 + $2400 = $6400.
So the answer is $\boxed{6400}$.

We hypothesize that 2nd stage instruction tuning on shared data leads models to converge to similar states rather than develop distinct response patterns. More in-depth analysis is needed to further understand this subtle behavior change, and we leave it as important future work.

Thank you for the time and effort you spent reviewing our paper, and for recognizing our contributions. Below are our responses:

Q1

Diverse reasoning patterns may not be fully captured by code & Compare to other methods that tackle abstract, non-linear reasoning

Thanks for this comment. We agree that not every reasoning pattern exists in code. However, as shown in case study (response to reviewer z6HH Q3), many foundational patterns can indeed be identified. We also tested on domains that are rare in code, e.g., medical reasoning (response to reviewer qybs Q7), and observed gains, indicating that the improvement is generalizable to some extent. Regarding comparison with other methods that tackle abstract, non-linear reasoning, we provide a discussion about neuro-symbolic systems (response to reviewer qybs Q4).

Q2

Code-centric logic might limit exposure to broader reasoning patterns: the selection criteria (e.g., filtering for complexity) might bias the dataset

Thanks for this comment. Actually, the selection criteria are set for mitigating bias rather than enhancing it. In the CodeMix source, most samples are pure algorithms, thus we deliberately filtered out pure algorithms in PyEdu source and did not apply complexity-based filtering. However, we also acknowledge that certain biases may still exist, as we only include executable code with proper JSON input/output formats. We leave this issue as future work to explore.

Q3

Concerns on error propagation in multi-turn revision and discussion on recent advances in related topics.

Thanks for this comment. We listed the statistics about multi-turn revision in Fig 7, Appendix D of the submission. Actually, only a small number of errors (16% in input pred and 10.7% in output pred) can be fixed. The value can be even smaller if a next round of revision is conducted.

As a result, a slight improvement when involving this revision is expected. Potential reasons may be that DeepSeek-V2.5 still lacks the ability to revise. We did not tune this part with huge effort, as we only wanted to try this as a preliminary attempt to utilize the verifiable nature of code.

On the other hand, there are also directions we could integrate into our workflow to enhance multi-turn revision effects. For example, involving multi-agent debate and discussion [1], interactive critiquing with diverse tools [2], or using models with strong self-reflection abilities [3]. We also plan to explore them in our future work to address error propagation more effectively.

[1] Improving Factuality and Reasoning in Language Models through Multiagent Debate; ICML 2024

[2] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing; ICLR 2024

[3] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning; arXiv:2501.12948

Q4

The combination of using query+code in the prompt and letting the response be both CoT+Code in Table 3

Thank you for this question. The two Code parts in prompt and response are actually identical - both refer to the reference code(see Table 8 for an example). If we include Code in both parts, the model will just learn to copy a block of contents, which would bring no benefits in learning. Therefore, we did not include this variant in the submission.

Q5

Quality concerns on the generated responses in CodeI/O: 1) if they contain the valid logic flow from the user query, 2) how to balance the complexity of the generated CoTs

Thank you for this comment. Our observations show that responses with correct predictions usually demonstrate valid logical flow. While removing incorrect responses seems intuitive for improving quality, our results (Table 2, rows 1 and 5) show this actually degrades performance. We hypothesize that such filtering, though ensuring valid logic, reduces exposure to diverse reasoning patterns, particularly difficult ones. Besides, our two-stage training approach helps mitigate these flaws, as high-quality second-stage data can remediate first-stage issues.

Regarding CoT complexity balancing, we find that CoT complexity naturally corresponds to the input/output prediction complexity. Our sampling across data sources implicitly covers different complexity levels, though we didn't explicitly balance this in a fine-grained manner, which we agree is an important direction and we leave it as future work to explore.

Q6

What is the deterministic reverse function (Line 167 to 178) and why is it important?

Reverse functions take an output and return a feasible input for the original functions. If they are deterministic, one output maps to exactly one input, and we can just use the execution trajectory (i.e., print the executed lines of codes and the intermediate variables sequentially) as the perfect responses. However, since multiple inputs often produce the same output, truly deterministic reverse functions rarely exist. This is partly why we use DeepSeek-V2.5 to generate responses directly.

Thanks for the time and effort you spent reviewing our paper, and for recognizing our contributions. Below are our responses:

Q1

Evidence to show code integrates diverse reasoning patterns

We conduct a case study on the CodeI/O dataset, and witness typical examples with certain reasoning patterns. Please refer to our response to Q3 of Reviewer z6HH for more details.

Q2

Qualitative analysis to show why the CODEI/O dataset is more effective than others

We compare samples from CodeI/O and other baseline datasets. Key differences are:

1. Structured Reasoning Process: CodeI/O shows a standard problem-solving method with clear steps, while other datasets show more varied and less ordered reasoning.

2. Algorithmic Thinking Pattern: CodeI/O shows orderly step-by-step processes, unlike the more direct shortcuts in other datasets.

3. Complete Reasoning Traces: CodeI/O gives full traces that record the whole reasoning process, making it better for training models to explain their thinking.

These differences stem from both the input/output prediction task and our DeepSeek-V2.5 response generation. We will add these analyses to our next paper version.

Q3

Data leak risks

We conduct a strict 13-gram-based leakage detection on CodeI/O data following [1], the results are as follows:

We see most of the benchmarks are not leaked. Upon manual inspection on the two ones with high ratio:

• KorBench overlaps only contain general descriptions like Sudoku rules or common letter sequences ("A B C D...") rather than specific questions – our training tasks and the benchmark tasks are completely different.

• LeetCode-O overlaps stem from sibling problems sharing common descriptions (e.g., Two Sum I & II), despite that we have removed all original problems from our training data

To further detect whether the gains on these two are due to data leakage, we calculated the sample-wise acc gains of CodeIO compared to the baseline on both the full set (F) and the non-leaked set (UN). The results are as follows:

The similar gains on both full and unleaked subsets across all models confirm that our improvements are not affected by data leakage.

These analyses will be in our next paper version.

[1] Evaluation data contamination in LLMs: how do we measure it and (when) does it matter? arXiv:2411.03923

Q4

Discussions on neuro-symbolic methods

Thanks for this comment. Our work can also be regarded as neuro-symbolic integration, specifically in the "neuro:symbolic->neuro" category per [1]. We use symbolic rules (Python execution) to guide LLM training, but rely solely on neural components in inference. This mainly differs from other categories like Symbolic[Neuro], Neuro[Symbolic], or Neuro|Symbolic, which utilizes both techniques in inference. Further discussion will be included in the next revision.

[1] Towards Cognitive AI Systems: a Survey and Prospective on Neuro-Symbolic AI, arXiv 2401.01040

Q5

Further insights into interpretability & Qualitative examples of reasoning improvements

Thanks for this advice. We conduct a case study on model behavior in response to Q3 of Reviewer z6HH. Please kindly refer to that for details. Also, as the input-output prediction accuracy is only about 50% in CodeI/O data, it’s hard to memorize them and hack the benchmarks, and the gains should mostly come from the underlying reasoning logic flow.

Q6

Training costs

Q7

Performance on unseen reasoning categories

We test on two medical reasoning tasks for complex clinical diagnosis: MedQA (US subset) [1] and MedBullets [2]. The results on Qwen 7B 2.5 Coder are as follows, indicating CodeI/O can also improve categories that may not align with codes:

[1] What disease does this patient have? a large-scale open domain question answering dataset from medical exams; arXiv:2009.13081

[2] Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions; arxiv 2402.18060

Q8

Performance on smaller models

We test Qwen 2.5 1.5B as follows:

The results show CodeIO is still effective, though with less gains than in larger models. This suggests smaller models may lack sufficient capacity to fully leverage the reasoning patterns in our datasets.

Thank you for the time and effort you spent reviewing our paper, and for recognizing our contributions. Below, we have listed our responses to your questions and comments.

Q1

The claim "CodeI/O and CodeI/O++ exhibit universal effectiveness …" is not exactly true given that the proposed method does underperform on some base models/tasks.

Thanks for this comment, we will change this statement to a more precise one, e.g., “CodeI/O and CodeI/O++ demonstrate performance improvements across models of various sizes and architectures on most benchmarks, although we also observed nearly unchanged or even decreased performance on a small number of tasks.”

Q2

Can the authors attempt to evaluate the proposed method with a larger base model?

Thanks for this suggestion. We have trained a larger model, LLaMA3 70B. The results are as follows:

The results show that CodeI/O also works well on larger models. Although we see some performance drop on a small set of benchmarks, gains are obtained on most of them, indicating an overall improvement in reasoning ability.