Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning

Thank you for your time and thoughtful review. We have carefully considered your comments and provide detailed responses below, with the hope that they clarify our intentions and strengthen the understanding of our work.

W-1: Although the dataset is useful, the core research contributions are somewhat unclear. Representing reasoning in executable code and using fine-tuned models like CodeGen for data generation have both been explored in prior work. For example, [1] uses (a similar idea) Formalizer models to generate formal proof data from informal sources. It would be great if the authors could clarify their technical contributions and how they compare with existing, well-known ideas.

In [1], the approach first converts math problems into formal theorem statements and then uses a prover model to generate formal proofs. These proofs are filtered based on verifiability, and the correct ones are added for iterative self-improvement. The primary goal of [1] is to sample the highest-quality answers possible under a fixed set of problems.

In contrast, our work focuses on generating large-scale, diverse, and verifiable problems across domains such as mathematics, algorithms, and logic—all unified under a Python-based code representation. By leveraging a code generation model, we synthesize new problems by first generating code-based reasoning traces and then translating them into natural language, enabling both diversity and scalability.

To clarify these distinctions, we will revise the manuscript to explicitly highlight the differences from [1], and we will include proper citations to both works as suggested.

For more precise definitions and contribution statements, we refer the reviewer to our responses to Reviewer oWXi-W1 and oWXi-W2.

W-2: While the results reported in Table 1 are promising, existing methods such as rStar-Math appear to achieve substantially better performance with similarly sized models (see Table 5 in [2]). The authors should clarify why this baseline was not included and, if possible, provide a fair comparison with such existing methods.

Thank you for raising this important point. Our method focuses on scalable reasoning data generation with basic SFT, while rStar-Math is a System 2-style test-time reasoning framework that uses multi-stage evolution, Monte Carlo Tree Search (MCTS), and a PPM-based reward model to select final answers.

To enable a fair comparison, we consider both greedy and sampled decoding modes:

Greedy Inference

The main results reported in rStar-Math (Table 5 in [2]) rely on sampling 64 responses per question and selecting the best one using PPM scoring (denoted rStar-Math-best@64). This is not directly comparable to our default greedy decoding setting.

To enable a fair comparison, we refer to rStar-Math’s greedy baseline (Table 10 in [2]) and compare it to our Caco-1.3M model under similar decoding.

System2 Sampling

To compare with the main results in rStar-Math, we initially considered replicating rStar’s full MCTS+PPM pipeline but found it incompatible:

• The rStar-Math model is trained to decode step-by-step outputs (### Step n) to enable MTCS which our model doesn’t generate natively.
• Instead, we adopt long-form CoT with majority voting—a simpler and widely used System 2-style approach.

We also evaluate Caco-Long, a System 2-style model trained on long CoTs, using both greedy decoding and 8× sampling with majority voting (temperature=0.8, top_p=0.9).

Due to the large evaluation cost on Math and Olympiad, we only performed sampling on AMC and AIME. All model are based on Qwen2.5-Math-7B.

Note: Unlike the results reported in the original paper, we found after submission that the model requires 3 epochs of training to fully converge. The results shown here are based on this updated 3-epoch training.

Key observations:

• Under greedy decoding, Caco-Long outperforms rStar-Math’s greedy policy by a large margin.
• With just 8 sampled outputs, Caco-Long surpasses rStar-Math-best@64, which uses 64 samples and a reward model.

While rStar-Math introduces a sophisticated multi-round framework, our work focuses on improving cold-start performance via better data generation. These two directions are complementary: Caco can provide stronger initial models and data, which rStar-style pipelines can further evolve.

We appreciate the reviewer’s suggestion and will include this comparison in the revised version to better contextualize our contributions.

We’re grateful for your valuable feedback and the careful attention you’ve given to our submission. Below we provide responses to your questions, and we hope this helps clarify our methodology and resolve your concerns.

W-1: Clarity: I found the paper difficult to read at times. W-2: Clarity: I find the term “Code CoT” to be misleading.

Response to W-1: Clarity of Presentation

Thank you for the feedback. As suggested, we have revised the abstract and introduction to provide a clearer and more concise overview of the core methodology, ensuring readers can grasp the main idea without needing to read the full methods section.

In addition, we will redesign Figure 2 to make the workflow more intuitive and self-explanatory. We also plan to include a dedicated “Problem Formulation” section, as recommended by reviewer oWXi-W2, to help organize key concepts.

Furthermore, in response to your point in W-3, we have consolidated scattered technical details from the appendix into the main text to improve clarity and cohesion.

Response to W-2: On the Term “Code CoT”

We appreciate your observation. Our original intent behind the term “Code CoT” was to describe reasoning traces expressed in code (i.e., programmatic) form, as a structured and executable variant of chain-of-thought.

We used the term “CoT” because such programs encode sequential reasoning steps implicitly—for example, variable initialization reflects problem setup, and each computation (e.g., via SymPy) corresponds to an intermediate logical operation.

This framing is consistent with prior terminology:

• PAL [1] refers to this as “programmatic steps,”
• PoT [2] describes it as “Programs of Thought,” suggesting that “LMs can express reasoning steps as Python programs.”

We acknowledge that the term “Code CoT” may be misleading, as such traces do not contain explicit step-by-step natural language explanations. In the revised version, we are considering replacing it with a more precise term such as “Program of Thoughts” for improved clarity.

[1] Gao, Luyu, et al. "Pal: Program-aided language models." International Conference on Machine Learning. PMLR, 2023

[2]Chen, Wenhu, et al. "Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks." arXiv preprint arXiv:2211.12588

W-3: Quality: Important details are missing from the paper, and there are some inconsistencies. I detail these further below in the “Questions” section.

Sorry for missing details. We give detailed information here, and we will revise as you suggested.

• After generating the seed Code CoT dataset, we fine-tuned Qwen2.5-Coder-7B to obtain the CodeGen model.
• The natural language instructions and answers mentioned in Section 3.3 were generated using Qwen3-8B (non-thinking mode).
• Thank you for the suggestion. We will replace vague mentions like "an LLM" with the specific model names throughout the paper, e.g. Qwen2.5-Code-7B, Qwen3-8B.
• Exactly — the 162K seed dataset includes the original source problems, which correspond to the “Original Math Problem” branch in Figure 2. Like the synthetic problems, we used Qwen3-8B to generate Language CoTs for them, and after filtering, 109K samples remained. The current description is unclear, and we will revise it in the next version.
• Thank you for the reminder. We will add confidence intervals to tables.

W-4 & Limitation: Quality: Given that the paper uses a larger model (Qwen-2.5-72B-Instruct) to generate the seed dataset of Code CoTs, I have a concern about whether the gains come from the knowledge from the more advanced model, or from the Caco procedure.

Thank you for raising this important point. To evaluate whether the performance gains are primarily due to knowledge transfer from the large teacher model (Qwen-2.5-72B-Instruct) rather than the Caco procedure itself, we conducted the following control experiment:

• We used Qwen-2.5-72B-Instruct to generate natural language CoT answers for the same seed questions, using multiple sampling rounds to create a dataset of 300K QA pairs, which we refer to as Qwen72B-Seed-distilled.
• We then fine-tuned the same student model on this dataset and compared it to a 300K subset of Caco, and the full Caco-1.3M dataset.

The results are shown below:

Note: Unlike the results reported in the original paper, we found after submission that the model requires 3 epochs of training to fully converge. The results shown here are based on this updated 3-epoch training.

We draw two key conclusions from this:

1. Caco outperforms the distilled data (66.2 vs. 65.5 AVG) even at the same dataset size, suggesting that prompt and reasoning diversity—enabled by Caco’s code-based augmentation—is beneficial beyond simply copying teacher traces.
2. Caco scales more effectively: When expanding to 1.3M samples, the model gains further improvement, which would be difficult to achieve using the seed-only natural language questions, demonstrating Caco’s scalability advantage in generating diverse and verifiable new problems.

These results confirm that the improvements come not just from the use of a strong teacher, but from the Caco generation procedure itself, which promotes data diversity, verifiability, and scalability.

W-5: Quality: The proposed procedure shares some similarities with STaR/ReST, which are much simpler methods. Ideally one iteration of STaR would be included as a baseline to confirm that the gains obtained from Caco are indeed due to leveraging the code representation. I discuss this concern in more detail below on the “Questions” section.

Thank you for the valuable suggestion. We would like to clarify that Caco’s primary focus is not on improving answers for a fixed set of questions (as in STaR or ReST), but rather on generating a much larger and more diverse set of new problems through code-based reasoning traces. Unlike STaR, Caco does not follow a traditional self-improvement paradigm. Rather than using model-generated answers to improve the same model, we train a dedicated code generator to produce executable CoTs from scratch—shifting the focus from answer refinement to scalable, verifiable data creation.

That said, we agree that the two directions are not mutually exclusive. In fact, we believe that applying STaR-style self-improvement on top of the more diverse prompts generated by Caco could yield further gains.

To directly address your suggestion of a single iteration of self-improvement as a control, we conducted the following experiment:

• Using DeepSeek-Math-7B, we generated 4 CoTs for each of the 109K seed examples.
• After filtering incorrect answers, we retained 363K verified solutions, from which we randomly sampled 300K (DS-7B-Seed-self-improve) for comparison.
• We compared this to 300K randomly sampled data from the full Caco dataset (DS-7B-Caco-300K).

The results are as follows:

We observe that the Caco-based dataset consistently outperforms the STaR-style self-improvement baseline across all evaluated benchmarks. We hope this additional baseline helps clarify the distinction and effectiveness of our approach.

Minor Points

These are some suggestions for improving the paper (but have not strongly influenced my score):

• The 65.4% refers to the average accuracy of Caco-1.3M-Qwen2.5-Math-7B across the six benchmarks in the main experiment. We will revise it to: “65.4% average accuracy on six benchmarks in Section 4.2.”
• Thank you for the reminder. We will discuss both forms of augmentation in Section 3.3 to better intuitively show the advantages of our method.
• Noted, we will move Section 3.4 to the appendix.
• Caco-100K is a subset of more difficult problems sampled from the full Caco dataset. We assess difficulty based on the average pass rate over three generations using Qwen2.5-Math-7B-Instruct.
• Yes, the experiment enables chain-of-thought reasoning when generating answers, and then trains on these longer traces. Our motivation is that test-time scaling methods like system-2 prompting can further benefit from our data, especially on extremely hard problems, e.g. AIME. We have added a comparison between the short-CoT and long-CoT fine-tuned models in the NsmM-W2 table.

Dear Reviewer,

We would like to kindly remind you to review our rebuttal for our submission. As the rebuttal phase is concluding, we hope to address any remaining concerns and clarify our points. Your feedback is highly appreciated.

Thank you very much for your time and consideration.

Best regards, Authors

We truly appreciate the time and effort you dedicated to reviewing our work. Below, we provide detailed responses to your concerns, and we hope this clarifies our design decisions and highlights the key contributions of our approach more clearly.

W-1: Lack of clarity in the Introduction: The main contributions of the paper are not clearly explained in the introduction. Q-4: Clearly articulate the main contributions of your work, including the dataset generation and any novel mechanisms introduced in Caco.

Thank you for pointing out the need for clearer articulation of our contributions. In the revised version, we will restructure the introduction to explicitly present the main contributions and novel mechanisms introduced by Caco. Below, we summarize these points clearly:

We propose Caco, a unified and scalable framework for generating large-scale, high-quality, and verifiable chain-of-thought (CoT) reasoning data by treating code as the central abstraction of logical reasoning. Unlike prior methods that begin with natural language questions and derive CoTs, Caco reverses this pipeline: it first generates executable code-based reasoning traces (Code CoTs) and then reverse-translates them into diverse, verifiable question-answer pairs. This inversion unlocks key benefits in scalability, verifiability, and cross-domain diversity.

To support this, we train a code-based CoT generator, which learns from a high-quality seed dataset and is capable of sampling millions of reasoning chains. This approach significantly improves the efficiency and diversity of data generation compared to prior methods that rely heavily on manual templates or language-only augmentation.

The key contributions of Caco are:

• Closed-Loop Generation with Built-in Verifiability. Caco introduces an end-to-end pipeline combining generation, validation, and translation. The process guarantees correctness by design through Code execution checks, AST-based structural filtering and Answer consistency verification. This makes every QA pair self-consistent, verifiable, and ready for downstream use, including reinforcement learning.

• Scalable Data Generation via a Trained Code Generator: We train a code-based CoT generator on a small set of verified seed examples. This model can efficiently sample millions of high-quality reasoning traces, enabling scalable and low-cost dataset creation without manual templates or annotations.

• Code-Centered Augmentation for Diversity and Generalization

  Caco uses code as a modular reasoning template, supporting two types of augmentation:

  • Pattern-level augmentation: sampling novel reasoning strategies beyond seen templates.
  • Problem-level augmentation: translating a single code trace into multiple linguistically distinct questions.

  This improves data diversity and model generalization across tasks and formats.

• Cross-Domain Reasoning Unification

  By representing both mathematical and algorithmic reasoning in Python code, Caco provides a shared reasoning format across domains. This promotes data source diversity, facilitates cross-domain learning, and broadens the applicability of the framework.

W-2: Inadequate problem formulation: The paper lacks a clear problem formulation. It is essential to explicitly define and describe the code-assisted reasoning problem, using appropriate notations to help readers understand the whole framework.

Q-5: Suggestion: Complete the problem formulation section to enhance clarity.

1. Problem Definition

• Let the problem space be $\mathcal{P}$, with each problem denoted as $p \in \mathcal{P}$ and its correct answer as $a^*$.
• Define the space of Code-based Chain-of-Thoughts (Code CoT) as $\mathcal{C}$, where each executable reasoning program $c \in \mathcal{C}$ produces the correct answer via Python execution: $F(c) = a^*$, where $F(\cdot)$ denotes running $c$ in a Python environment.
• Our goal is to generate diverse, verifiable new problems $p'$ and answers $s'$ from a small set of seed problems, with each $p'$ linked to a validating program $c'$.

2. Core Functions

• We use a general language model $G$ for conditional generation:

  $G_{p \to c}:\mathcal{P}\to\mathcal{C},\quad G_{c \to p}:\mathcal{C}\to\mathcal{P},\quad G_{p \to s}:\mathcal{P}\to\mathcal{S}$

  where $G$ can be any sequence-to-sequence model.

• An unconditional generator $U$ samples candidate programs:

  $U: \varnothing \to \mathcal{C}$

• $F$ is a code executor that filters valid programs through execution, AST checks, and output consistency:

  $F(c) = a^*$

3. Conventional Method and Its Limitations

The common data synthesis approach:

• Rewrite $p$ into $p'$: $P' = {G_{p \to p'}(p)}$
• Generate answer: ${p', s'} = {G_{p \to s}(p')}$

Limitations:

1. $p'$ depends on $p$, limiting diversity.
2. No guarantee of solvability of $p'$ or correctness of $s'$.

4. Caco: A Code-Driven Framework

Caco generates new problems by sampling new code instead of rewriting existing problems:

1. Seed Collection and CodeGen Fine-Tuning

   $\mathcal{C}_{\rm seed} = \{G_{p \to c}(p)\mid F(c) = a^*\}$

   Fine-tune $U$ on $\mathcal{C}_{\rm seed}$ to model valid code distributions.

2. Sampling New Programs and Reversing to Questions

   $\{(p', c')\} = \{G_{c \to p}(c')\mid c' \sim U()\}$

   Each $c'$ is translated into a question $p'$, with its correctness validated via $F(c')$.

3. Answer Generation and Filtering

   $\{(p', s', c')\}_{\rm valid} = \{G_{p \to s}(p') \mid G_{p \to s}(p') = F(c')\}$

   Answers are only accepted if consistent with code execution.

Benefits

• Diversity: Code serves as an abstract, language-independent reasoning template, enabling pattern-level and problem-level augmentation.
• Verifiability: Each problem is paired with executable logic, allowing strict automated validation.
• Scalability: The framework supports large-scale generation via unconditional sampling and efficient filtering.

Q-1: Why is your method not evaluated on the AIME dataset?

Thank you for the question. We initially did not evaluate on AIME because our models were trained only with short-CoT supervised fine-tuning, which is insufficient to yield strong improvements on AIME without more advanced System-2 reasoning strategies.

In early experiments using Qwen-Math-base, we observed only marginal gains on AIME(20% to 23.3%)—likely due to the fact that Qwen already exhibits strong performance on math reasoning tasks. However, when we tested LLaMA-based models (as shown in ZeGf-W2 Table), we observed a more substantial gain from 0% to 10.83% (Average@8 samples), suggesting Caco’s benefits are more pronounced on weaker base models.

Inspired by the System-2 reasoning paradigm, we later conducted experiments using Caco-Long, described in Appendix C.1. This involved generating 100K long-form CoTs using Qwen3 in “thinking mode”, and yielded some improvement.

After the submission, we further realized that long-CoT learning requires higher learning rates and longer training schedules due to its distinct reasoning pattern. We therefore re-trained the model using lr=5e-5 (10× the original), with 3 epochs and an expanded dataset of 300K long CoTs. This led to further performance gains, as shown in the updated results in NsmM-W2 Table.

Q-2: Code-assisted methods primarily address incorrect calculations in reasoning trajectories. Does the Caco framework specifically aim to reduce calculation errors? It would be beneficial to analyze whether improvements arise from other factors, such as Caco's influence on the reasoning patterns and behaviors of LLMs.

We appreciate the reviewer’s interest in the underlying factors behind Caco’s performance gains. Unlike many code-assisted methods that focus specifically on reducing calculation errors, Caco does not target a specific error type.

Instead, Caco emphasizes scaling diverse, verifiable reasoning traces. During code verification, all incorrect samples—whether due to calculation mistakes, logical flaws, or incomplete reasoning—are filtered out, ensuring that only correct and executable reasoning paths are retained.

Furthermore, to better diversify the reasoning behaviors and patterns represented in the training data, we deliberately included algorithmic problems in the seed dataset. These problems require reasoning patterns that are structurally different from typical math problems, such as recursion, dynamic programming, and search algorithms. By doing so, Caco enables the model to internalize a broader set of reasoning strategies, which we believe contributes meaningfully to its improved generalization and robustness across tasks.

Q-3: The introduction mentions the length inefficiency problem. Have you compared the efficiency metrics, such as response length and generation time, of Caco with those of other baseline methods?

When we mention “length inefficiency” in the introduction, we are referring to how long chain-of-thoughts (CoTs) often introduce redundancy and increase the risk of logical inconsistencies. In many long-CoT setups, the reasoning trace can span tens of thousands of tokens, even for relatively simple problems—resulting in unnecessarily verbose answers.

Regarding your question on response length, we analyzed the length of our generated code-based CoTs, which are much more compact. On average, each CoT contains 415 tokens, with a standard deviation of 284. Compared to long-CoT baselines, this represents a significant reduction in output length, suggesting improved efficiency in both reasoning compactness and model usage.

Thank you very much for your valuable feedback and thoughtful questions. We greatly appreciate your efforts in reviewing our paper. Below, we provide our detailed response to your concerns.

W-1: This paper is difficult to distinguish from some accepted works [1,2].

We appreciate the reviewer’s comment and acknowledge the growing popularity of code-assisted reasoning in recent literature. As noted, we reference relevant prior works [1,2] in Section 2.2 “Code Integration for Enhanced Reasoning”, which primarily use Python code for answer verification—not for data generation.

In contrast, our work reframes code as an abstract representation of reasoning patterns and logical templates for entire problem families. Specifically:

• Caco treats problem generation as code generation: Instead of verifying generated problems with code (as in [1,2]), we first generate executable Code CoTs, then reverse-translate them into diverse and verifiable QA pairs. This reflects a fundamentally different motivation and methodology.
• This shift enables several key benefits:
  • Scalability: By fine-tuning on a small seed set, we sample millions of structurally valid and semantically rich Code CoTs, automatically filtered without human annotation.
  • Diversity: Code abstracts reasoning beyond natural language, enabling generalizable problem-solving patterns
    1. Through pattern-level augmentation (e.g., as illustrated in Figure 3), the model generates novel reasoning strategies beyond template memorization.
    2. Each code trace can yield multiple natural language questions via reverse translation, increasing linguistic and topical diversity.
  • Verifiability: Executable code enables automatic verification through execution, AST analysis, and answer consistency checks, ensuring high-quality reasoning data.
  • Cross-Domain Unification via Code: Unlike prior math-focused work, we unify multiple domains (math, algorithms, etc.) into a shared code-based format, greatly expanding data diversity and applicability.

We will clarify these distinctions in the revised version and ensure both works are properly cited. For detailed definitions and contributions, please also see our W1 and W2 responses to Reviewer oWXi.

W-2 & Q-1: Evaluation limited to math.Despite the claimed generality, experiments are only conducted on math datasets. There is no evidence the framework generalizes to other reasoning domains (e.g., logic puzzles, science QA).

Thank you for the insightful question. While our primary experiments focused on mathematical reasoning, Caco is fundamentally a general-purpose framework for structured, code-based reasoning, and is applicable to domains that exhibit logical, symbolic, or programmatic structure—such as logic puzzles, scientific reasoning, and procedural tasks.

Logic Puzzles

Caco naturally extends to logic-based tasks like puzzles and rule-based games. In many of these problems (e.g., arithmetic expression puzzles, countdown problems), the underlying logic is shared across instances and can be encoded as parameterized templates. For example, in countdown puzzle [1], we can reuse the same program structure and vary only the inputs:

import random

def generate_expression_sample(min_len=3, max_len=6):
    ops = ['+', '-', '*', '/']
    count = random.randint(min_len, max_len)
    nums = [random.randint(1, 100) for _ in range(count)]
    ops_list = [random.choice(ops) for _ in range(count - 1)]
    expr = ' '.join(str(num) + ' ' + op for num, op in zip(nums[:-1], ops_list)) + f' {nums[-1]}'
    return {'numbers': nums, 'expression': expr, 'result': eval(expr)}

if __name__ == "__main__":
    for i in range(3):
        s = generate_expression_sample()
        print(f"Sample {i+1}:\n  Numbers: {s['numbers']}\n  Expression: {s['expression']}\n  Result: {s['result']}\n")

This aligns with Caco's central idea: code abstracts problem logic more compactly than natural language, enabling batch generation of diverse problem instances from shared reasoning templates. As such, logic puzzles are a natural fit for Caco’s generation and filtering pipeline.

Scientific Reasoning

To further evaluate Caco pipeline's generalization, we applied the framework to scientific QA tasks using the MegaScience dataset. After filtering out math-related instances, we retained 5.2K science reasoning samples as seeds. Using Caco’s pipeline:

• We sampled 50K Code CoTs by CodeGen tuned on 5.2K seed CoTs.
• After filtering, we obtained 37K valid science QA samples

When finetuned on these, the model showed strong performance improvements across several scientific benchmarks:

This demonstrates that Caco's pipeline generalizes effectively to scientific reasoning, provided that the reasoning logic can be encoded programmatically.

Broad Generalization Evaluation

We also benchmarked models trained with Caco-1.3M data across a wide range of tasks using the OpenCompass framework:

• Math (hard): AIME2024
• Code generation: HumanEval-Plus
• Science QA: ARC-c
• Logic puzzles: BBH, KorBench
• General knowledge & science: AGIEval

Except for AIME (8-sample majority voting), all evaluations used greedy decoding.

These results confirm that Caco-trained models generalize well beyond math, showing strong improvements in logic puzzles, scientific reasoning, and code tasks, across three different model backbones.

In summary, while Caco was designed with mathematical reasoning in mind, its core mechanisms are broadly applicable to any domain where logic can be codified, and our experiments show strong generalization across diverse reasoning domains. We will clarify this in the revised version and include additional non-math evaluations to highlight its broader applicability.

W-3: Scalability claim lacks resource metrics.

We appreciate the reviewer’s comment regarding the need to quantify the efficiency of our proposed method. To address this, we report the full computational cost for generating the Caco-1.3M dataset. All experiments were conducted on a machine with 8× NVIDIA A100 GPUs.

• Low Cost and Accessibility: First, the pipeline uses only open-source models, avoiding reliance on costly proprietary APIs.

• Efficiency and Scalability: Generating 1.3M verified samples in under 55 hours demonstrates strong scalability, with most steps parallelizable.

• Minimal Overhead over Standard Pipelines: the majority of the cost comes from answer generation—which is unavoidable in any instruction tuning or self-improvement setup. Prior works such as STaR【1】also incur similar or higher costs in the solution generation phase, especially when multiple samples per prompt are required.

  The additional steps specific to Caco (e.g., Code CoT generation and question reversal) are relatively lightweight (combined ~16h). This is due to natural language solutions are significantly longer than questions or Code CoTs.

• Clarification on Focus: While we acknowledge the value of data-efficient methods, Caco’s focus is on scalable, diverse, and verifiable reasoning data, not minimal data volume. Our aim is to enable generalization across domains by generating rich reasoning trajectories at scale.

[1] Zelikman, E., Wu, Y., Mu, J., & Goodman, N. (2022). STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems.

Q-2: The paper does not report the proportion of generated code-CoTs that fail execution or are filtered out due to semantic or structural issues.

As mentioned in Line 171 and Line 193:

“5.3M Code CoT samples. After filtering, we retained a high-quality subset of around 4.6M executable and structurally valid programs.”
“After the processes, it yields approximately 1.3M validated instruction-answer pairs...”

Therefore, the rejection rate during code generation is approximately 13%, and the rejection rate from code to question–answer pair generation is about 71%. This looks somehow with a high fail rate because we conduct a very hard and strict filtration that any failure during the code-to-question or question-to-answer generation steps is detected and filtered.

This also suggests that other data generation methods that do not include verification mechanisms may produce a large amount of incorrect or noisy data.

To present this more clearly, we will add a figure showing the retention rate at each stage in the final version of the paper.