Chain of Execution Supervision Promotes General Reasoning in Large Language Models

We sincerely thank you for the comprehensive review and the constructive feedback shared. Below are our responses to your insightful comments.

Q1: Unfair comparison with CodeIO

Thank you for your valuable feedback regarding the baseline comparison. We understand your concern about the differing training epochs (1 for CodeI/O vs. 3 for TracePile).

Firstly, we understand your emphasis on consistent training configurations. However, it's important to note that due to significant differences in the scale and characteristics of our TraceMind dataset compared to the dataset used by CodeI/O, directly enforcing the same training epochs might not fully unleash the potential of our method. Different datasets often necessitate varying training strategies and convergence steps to achieve optimal performance.

Crucially, to ensure fairness, we also evaluated our model after just 1 epoch of training. Even in this configuration, our model significantly outperformed CodeI/O, demonstrating the inherent effectiveness of our approach and the TraceMind dataset. We will clarify these details and consider adding these comparative results to the paper to fully address your point.

In our experiments, we indeed conducted extensive explorations of model performance across different training epochs. The experimental results consistently showed that within 3 epochs, the model's performance continued to improve with more training steps, exhibiting superior average performance. Therefore, to present the best performance of TracePile in the paper, we chose to report the results obtained after 3 epochs of training.

Q2: Comparison with different supervision approaches

Thank you for your crucial feedback regarding the fairness and scope of our baseline comparisons. We entirely agree that ensuring fair training and testing conditions is paramount to accurately demonstrate the unique value of Chain of Execution (CoE) supervision against other approaches.

We agree that fair comparisons against alternative supervision methods are crucial. Our paper already includes extensive comparisons beyond base models, as detailed in our discussion (RQ1) and Tables 3, 4, and 7.

• Generic Instruction Data: Tables 3 and 7 show comparisons against TuluSFT, a widely recognized generic instruction dataset.

• Alternative Structured Code Data: Table 4 explicitly compares TracePile against OpenMath (pure code solution data) and CodeI/O (different forms of code input-output data).

To provide a clearer overview of these comparisons, we have conducted a comprehensive set of experiments using the Qwen-2.5-Coder 7B model under a two-stage fine-tuning paradigm. This allows for a more direct and equitable comparison of TracePile against various alternative supervision approaches. For OpenMathInstruct 1 and 2, we specifically selected 2.6M subsets and trained for 3 epochs to ensure comparable data scale and training duration.

As shown, TracePile consistently excels across diverse reasoning domains. While pure math solutions perform slightly better on math benchmarks due to specialization, TracePile significantly outperforms all other baselines, particularly in LiveCodeBench-Output and Zebra Logic. This highlights TracePile's ability to enhance generalizable reasoning capabilities by fostering transferable understanding of logical and procedural patterns.

Q3: Concerns about Related Work

We apologize for the related work section's placement. It is indeed included in Appendix A (lines 536-562) due to space limitations. We will explicitly reference it in the main paper.

Our work introduces fundamental and distinct contributions beyond prior literature like TRACED, SemCoder, and CodeI/O:

1. Execution Tracing Granularity:

   • TRACED: While TRACED also uses code traces, its backbone is RoBERTa, limiting it to simpler classification tasks and primarily focusing on code understanding. Crucially, its traces are not in natural language, which restricts its applicability to general reasoning tasks in LLMs.
   • SemCoder: Employs "monologue reasoning" with high-level functional descriptions and verbal reasoning about local execution effects. However, it primarily uses synthetic data with limited real-world complexity.
   • CodeI/O: Condenses reasoning patterns by predicting inputs/outputs from given functions in natural language, but its explanations focus predominantly on abstract input-output reasoning without detailed execution tracing of internal program states.
   • TracePile (Ours): Explicitly emphasizes fine-grained, structured execution traces (Chain-of-Execution, CoE), capturing detailed variable state changes and control flow at each step in natural language. This structured granularity significantly surpasses the abstract verbal reasoning of SemCoder and CodeI/O, enabling models trained on TracePile to exhibit deeper understanding and robust execution reasoning capabilities.

2. Data Sources:

   • SemCoder: Relies entirely on synthetic data (PYX).
   • CodeI/O: Derives data from generic raw code files transformed into structured reasoning tasks.
   • TracePile (Ours): Uniquely integrates diverse, real-world-oriented sources, including:
     • Classical algorithms with intermediate checks.
     • Algorithmic competition code verified with detailed execution traces.
     • Synthetic algorithmic problems designed for high complexity.
     • Mathematical code problems for multi-step decomposition.

3. Training Paradigm:

   • SemCoder and CodeI/O: Primarily rely on instruction tuning for explanations.
   • TracePile (Ours): Adopts three different multi-stage training paradigms: continue pretraining (CPT) + few-shot prompting; continue pretraining + general instruction tuning; and two-stage instruction tuning. We are the first to demonstrate that CoE-similar format data can significantly contribute to model performance in the CPT stage. Our comprehensive training settings further push the boundaries of using this type of data in the field.

We believe that these distinctions, particularly our emphasis on fine-grained, natural language CoE traces from diverse real-world sources and our exploration of multi-stage training paradigms, clearly establish the novelty and significant contribution of TracePile to the literature on enhancing LLM reasoning through execution supervision. We will ensure these points are more prominently highlighted in the main paper to address your concern directly.

Q4: Intrinsic Evaluation

Thank you for highlighting the importance of intrinsic evaluation, specifically whether our model learns to generate correct traces. We agree that this is a crucial aspect to complement existing evaluations and directly demonstrate the value of our Chain of Execution (CoE) supervision.

To address this, we conducted an additional, insightful experiment. We compared our TracePile CoE method against baseline approaches, including CodeI/O, on the LiveCodeBench-Output test set. Beyond just measuring the final output accuracy, we also assessed the** accuracy of the intermediate steps** generated by the models. For this intermediate step accuracy verification, we leveraged a powerful external model, Qwen3-235B-A22B-Instruct-2507, to check the correctness of each step in the generated traces.

Here are the results of this intrinsic evaluation:

Intermediate Step Accuracy Explanation: This metric represents the proportion of samples where the final output is correct AND all generated intermediate reasoning steps are also verified as correct. This metric more rigorously measures the end-to-end reasoning fidelity of the model.

As the table demonstrates, our TracePile CoE method not only achieves superior final output accuracy on LiveCodeBench-Output but also shows a significantly higher accuracy in its intermediate steps compared to both the generic instruction baseline and CodeI/O++.

We will adopt this more precise definition and ensure its calculation method is clearly articulated in the final version of the paper to more comprehensively demonstrate the advantages of our approach. Thank you again for your invaluable suggestion!

Q5: Average token of Tracepile

Regarding your question about the 8k-token limit and the average token count in Table 2, you are correct to notice a potential discrepancy.

Upon re-examination, we found Table 2's total token count inadvertently included tokens from few-shot examples within the data generation prompts. After removing these, the corrected average token length per training sample is approximately 7452 tokens, and the total TracePile token count is around 19.5 billion tokens. This revised statistic accurately reflects the data adhering to our 8k-token limit and resolves the apparent contradiction. We will update Table 2 accordingly.

Dear AC,

Thanks for your message. I had updated the final justification in my main review as instructed by the new policy. I don’t have further questions for the authors.

Best,

Reviewer

We are truly grateful for your thorough review and insightful feedback. Our responses to your comments are outlined below.

Q1: Effects of CoE in other reasoning benchmarks beyond math and code.

Thank you for requesting further clarification on the effect of our proposed continual pretraining on benchmarks not related to coding or mathematics. We appreciate this opportunity to highlight the broader generalizability of our method. As you noted, this information is indeed already present in our paper.

Specifically, in Table 1 of our submission, we report the performance improvements of models after the continual pretraining stage across various general reasoning benchmarks, including logical and algorithmic reasoning.

Below is a summary table, extracted from our paper, showcasing the performance of Llama-3-8B and Llama-3.1-8B models on these non-coding and non-mathematical benchmarks after continual pretraining with TracePile:

As the table illustrates, our TracePile method, during the continual pretraining stage, yields substantial performance gains on Llama 3-8B and Llama 3.1-8B models across logical and algorithmic reasoning and instruction following tasks.

Q2: Access to Data

Thank you for your diligent review and for pointing out critical issues regarding our NeurIPS checklist responses, particularly concerning the transparency of code and data access. We apologize for any ambiguity or incompleteness in our initial submission.

You are absolutely correct that our response to Checklist Question 5 ("Open access to data and code") is misleading if the code and data are not openly shared. We acknowledge this discrepancy.

Indeed, TracePile was not publicly released at submission due to company-related copyright constraints. As per NeurIPS guidelines, we are unable to introduce new data or anonymous links during the rebuttal phase. Currently, we are actively processing internal company approval for public release of TracePile in one month. Once approved, we commit to releasing the full dataset on Hugging Face, providing comprehensive instructions and scripts to ensure full reproducibility. This ongoing approval process is also why we have refrained from publicly posting our manuscript to platforms such as arXiv at this time.

Q3: Avoid test-data contamination

Thank you for your critical question regarding test-data contamination. Ensuring the validity and trustworthiness of experimental results is paramount.

How Test-Data Contamination is Avoided (General Principles):

Avoiding test-data contamination primarily involves:

1. Strict Data Splitting: Ensuring no overlap between training, validation, and test sets. Test data must remain unseen during all training and hyperparameter tuning phases.
2. Independent Data Sources: Ideally, test data should originate from sources entirely separate from training data, or represent novel scenarios the model has not encountered.
3. Leveraging Established Benchmarks: Utilizing widely accepted public benchmarks with pre-defined splits and robust de-duplication/anti-contamination measures.
4. Preventing Information Leakage: Ensuring no implicit information from the test set influences the training process (e.g., through feature engineering, data augmentation, or model selection that "peeks" at test data).

How We Avoided Test-Data Contamination in Our Paper:

In our work, we implemented several rigorous measures to prevent test-data contamination and ensure the validity of our experimental results:

1. Independent Evaluation Benchmarks: We conducted comprehensive evaluations on 20 diverse, publicly available benchmarks spanning mathematics, code, logical, and algorithmic reasoning (as detailed in Section 3.1 and Appendix C of our paper). These benchmarks (e.g., GSM8K, MATH, LiveCodeBench, CRUX, Zebra Logic) are well-established with pre-defined splits and are managed by public evaluation toolkits (OpenCompass, Qwen2.5-Math, ZeroEval), ensuring consistency and reproducibility.

2. Domain and Format Discrepancy: Our TracePile dataset, while code-execution-based, focuses on fine-grained, CoE-style natural language reasoning traces. The evaluation benchmarks, especially those targeting out-of-domain generalization (e.g., LiveCodeBench, CRUX, RuleTaker, Zebra Logic), often differ significantly in format, domain, or reasoning style from our training sources (as discussed in Section 4, "Out-of-domain (OOD) Generalization"). This distinction helps ensure that the model learns generalizable reasoning capabilities rather than simply memorizing test patterns.

3. Rigorous Data Generation and Filtering:

   • For mathematical code data, we employed a model-based filtering strategy (Section 2.1) where problems correctly answered three times by LLaMA3-8B were discarded as too simple, preventing overfitting to trivial or potentially similar test patterns.
   • We enforced intermediate result verification and consistency checks (Section 2.3) for our CoE data, ensuring logical rigor beyond just the final answer. This encourages the model to learn deep reasoning logic. These measures collectively ensure the reliability of our experimental results and the generalizability of our model's enhanced reasoning capabilities.

We sincerely thank you for the comprehensive review and the constructive feedback shared. Below are our concise responses to your insightful comments.

Q1: Concerns about the novelty and incremental nature of TracePile compared to existing works (SemCoder and CodeI/O).

We appreciate the reviewer for highlighting the critical aspect of novelty and differentiation. Although TracePile shares the general goal of leveraging natural language for code execution reasoning with SemCoder and CodeI/O, our work introduces fundamental and distinct contributions:

Execution Tracing Granularity:

• SemCoder employs "monologue reasoning," emphasizing high-level functional descriptions, constraints, and verbal reasoning about local execution effects. However, it primarily uses synthetic data generated from scratch with limited real-world complexity
• CodeI/O condenses reasoning patterns by predicting inputs/outputs from given functions in natural language, but the explanations focus predominantly on abstract input-output reasoning without detailed execution tracing of internal program states
• TracePile (ours) explicitly emphasizes fine-grained, structured execution traces (Chain-of-Execution, CoE), capturing detailed variable state changes and control flow at each step. This structured granularity significantly surpasses the abstract verbal reasoning of SemCoder and CodeI/O, enabling models trained on TracePile to exhibit deeper understanding and robust execution reasoning capabilities.

Data Sources:

While SemCoder’s data (PYX) is entirely synthetic and CodeI/O derives data from generic raw code files transformed into structured reasoning tasks, TracePile uniquely integrates diverse, real-world-oriented sources: classical algorithms with intermediate checks, algorithmic competition code verified with detailed execution traces, and synthetic algorithmic problems designed for high complexity，and mathematical code problems for multi-step decomposition.

This deliberate selection ensures broad coverage of real-world coding challenges, including tasks that demand intensive structural reasoning, dynamic tracking of complex data structures (arrays, stacks, heaps), and control-flow interpretation—scenarios less comprehensively addressed by SemCoder or CodeI/O.

Training Paradigm:

SemCoder and CodeI/O primarily rely on instruction tuning explanations. TracePile adopts three different multi-stage training paradigm: continue pretraining (cpt) + few-shot prompting; continue pretraining + general instruction tuning; two-stage instruction tuning. We are the first to prove that CoE-similar format data could contribute the model performancnes in cpt stage. Our compresive trainning settings further push the boundry of using data in the field.

In summary, TracePile distinguishes itself clearly through its structured, fine-grained trace annotation, comprehensive use of real-world coding scenarios with detailed instrumentation, and multi-stage augmented training strategy, thus substantially enhancing model performance on complex, real-world execution reasoning tasks.

We will clarify these differences explicitly in the revised manuscript.

Q2: Concerns about data availability and reproducibility due to the non-release of the TracePile dataset.

We sincerely appreciate the reviewer highlighting the critical aspect of data availability and reproducibility. Indeed, TracePile was not publicly released at submission due to company-related copyright constraints. As per NeurIPS guidelines, we are unable to introduce new data or anonymous links during the rebuttal phase. Currently, we are actively processing internal company approval for public release of TracePile in one month. Once approved, we commit to releasing the full dataset on Hugging Face, providing comprehensive instructions and scripts to ensure full reproducibility. This ongoing approval process is also why we have refrained from publicly posting our manuscript to platforms such as arXiv at this time.

Q3: Suggestion to add a discussion explicitly comparing TracePile to SemCoder and CodeI/O, clearly highlighting similarities and differences.

We thank the reviewer for this excellent suggestion, which indeed helps clarify our unique contributions. We partially addressed this comparison in our related work section (e.g., lines 552–562 for CodeI/O), highlighting some key differences. To further enhance clarity, we will expand the discussion explicitly comparing TracePile's Chain-of-Execution (CoE) to SemCoder and CodeI/O, clearly stating how our structured fine-grained execution traces, real-world-oriented datasets, and multi-stage augmentation differ from these prior works (R1 to Q1). We agree that explicitly specifying both similarities and differences will greatly benefit the reader’s understanding of our unique contributions, and we will incorporate this detailed discussion into the revised manuscript.

We sincerely thank you for your comprehensive review and the constructive feedback shared. Below are our responses for your insightful comments.

Q1: Improvements of Figure 3, including the LLama-3.1-8B line, x-axis scaling, and final segment distinction.

We appreciate the reviewer’s insightful feedback on Figure 3. We will plot the LLama-3.1-8B as a single point at Data Scale = 0, use a linear x-axis from 0 to 4.3M, and visually distinguish the final segment with a different color to indicate rejection-sampled data augmentation. These adjustments will enhance clarity and resolve the concerns raised.

Q2: Additional details about rejection sampling for constructing TracePile++.

We thank the reviewer for highlighting this crucial aspect, which directly touches on the diversity and quality of our dataset. Specifically, we construct TracePile++ by sampling five CoE rationales per problem using Qwen-2.5-72B-Instruct, and select semantically diverse yet correct outputs using Sentence-BERT for semantic differentiation. The resulting samples, clearly differing from the original TracePile outputs, are incorporated into TracePile++. We will include comprehensive details of this method in the appendix of the camera-ready version.

Q3: Clarification on the correctness validation approach mentioned in lines 159–161, including code instrumentation and symbolic tools.

The reviewer insightfully identifies a core component of our data quality assurance process. To ensure correctness, particularly for Algorithmic Competition Code, we utilize Python's snoop tool, which provides detailed execution traces by capturing variable states and changes throughout code execution. These generated traces are then systematically compared with the model-generated CoE outputs, discarding inconsistent samples. This validation method, distinct from intermediate result checks used in Classical Algorithm Code, ensures semantic fidelity and consistency. We will add detailed explanations of this process, including the use of snoop, in the appendix of the revised manuscript.

Q4: Minor corrections and clarity improvements (typos, table references, figure readability).

We greatly appreciate the reviewer's careful proofreading and thoughtful suggestions to enhance clarity. We acknowledge these minor mistakes and readability issues, and sincerely apologize for the oversight. We will correct the mentioned typos (e.g., "CoE formats significantly", "117,083,6"), replace "TraceMind" with "TracePile" consistently in the abstract, and properly reference Tables 1 and 3 instead of Tables 2 and 3. Furthermore, we will expand the caption of Figure 1, clearly indicate the three distinct data sources, and consider integrating dataset size information to improve readability. All these improvements will be incorporated into the camera-ready version.

Q5: Reasoning behind the selection of benchmarks in Table 4.

The reviewer raises an important point about benchmark selection. The benchmarks shown in Table 4 were specifically chosen to ensure a fair comparison, as the CodeI/O models based on TuluSFT are not publicly available. Thus, we selected benchmarks explicitly evaluated in both the CodeI/O paper and our own study to enable a direct and equitable comparison.

Q6: Clarification of the conclusion in lines 306–308 regarding the effects of code rewrites and query diversification.

The reviewer insightfully highlights a key point regarding our interpretation of results. In our analysis, we consider performance on logical reasoning benchmarks as a strong indicator of structural generalization. We observed that removing code rewrites resulted in a more noticeable performance drop on logical reasoning tasks, while removing query diversification had a greater negative impact on mathematical reasoning. This differential pattern guided our conclusion that code rewrites predominantly improve structural generalization, whereas query diversification mainly enhances task-specific interpretability. We will clarify this reasoning explicitly in the revised manuscript.

Q7: Clarification on the meaning of "control-flow interpretation" mentioned in lines 154-155.

We thank the reviewer for highlighting this ambiguity. Indeed, the reviewer’s interpretation aligns correctly with our intent. By "control-flow interpretation," we specifically refer to tracking changes in structures like arrays, stacks, or heaps during code execution—commonly involved in classical algorithmic tasks. While such tracking is prevalent in most algorithmic tasks involving loops or conditionals, a substantial portion of our dataset consists of mathematical code, which frequently does not require detailed control-flow tracking. We will explicitly clarify this distinction in the revised manuscript.