ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement

Reviewer: ”Missing some baselines: How will ReGenesis perform when compared with few-shot CoT prompting (w/o FT) on in-domain and out-of-domain tasks”

Our response:

To address the concern, we introduce an additional baseline in Tables 2 and 3. This baseline utilizes 3-shot chain-of-thought (CoT) examples generated by GPT-4o, serving as a few-shot CoT prompting method without fine-tuning (w/o FT) using self-consistency. The results demonstrate that incorporating GPT-4o-generated CoT examples significantly improves the performance of Mistral-7B-Instruct-v0.3; however, it still lags behind ReGenesis on both in-domain and out-of-domain (OOD) tasks.

Reviewer: “It appears that the generalization of ReGenesis mainly stems from task-agnostic guidance. However, during actual model training, only specific trajectories are provided, without including this guidance. So, where does the model's generalization come from?”

• Reviewer: “More specifically, task-agnostic guidance simply ensures that the model can synthesize more correct trajectories.” Is the data size difference the main reason for ReGenesis’s better generalization?

  Our response:

  We thank the reviewer for this question since it points out that the size of the synthetic data after filtering is also a variable that could significantly affect model performance. We had this variable controlled in all the experiments of the current submission and we should have presented it in the experiment section. Although ReGenesis leads to slightly more correct paths, we always down sample ReGenesis’s data to make sure its data size is the same as the rest baselines. The table below lists the training data size that ReGenesis and star use for all training data sets:

• Reviewer: ”If so, then where does ReGenesis’s generalization come from?”

  Our response:

  Although ReGenesis uses the final detailed and task-specific reasoning paths as training data, the diverse task-agnostic reasoning strategies that Regenesis adopts to generate such final task-specific paths, are effectively preserved in the final task-specific paths such that it is easy for LLMs, such as mistral-7B-instruct, to detect them and learn them from the final task-specific paths. We run the following analysis to verify this hypothesis.

  We randomly select 500 reasoning paths from ReGenesis and STaR’s generated data of NumGLUE training data set. Note that the 500 reasoning paths of each method are all final task-specific reasoning paths. Then, we prompt Mistral-7B-Instruct-v0.3 to label each of the total 1000 reasoning paths the task-agnostic reasoning strategy behind the task-specific path. The goal of this experiment is to see whether from the task-specific path only, LLMs can already tell the task-agnostic strategy behind it and that ReGenesis’s task-specific paths have more diverse task-agnostic strategies. If so, it indicates that LLMs are likely to be able to capture the task-agnostic strategies behind the task-specific paths in the training data, even if they are trained only with task-specific paths.

  Specifically, after Mistral-7B-Instruct-v0.3 labeling the task-agnostic reasoning strategies, we task Mistral-7B-Instruct-v0.3 to cluster the task-agnostic reasoning strategies generated by itself and suggest cluster names for both ReGenesis and STaR. We present the results as follows. Results show that it is quite apparent for Mistral-7B-Instruct-v0.3 that the final task-specific paths generated by ReGenesis, have more diverse task-agnostic reasoning strategies behind them, compared to STaR (More strategy clusters for ReGenesis and less for STaR). Note that throughout this experiment, we don’t give Mistral-7B-Instruct-v0.3 any knowledge on how ReGenesis generates the final task-specific paths and what are the general reasoning strategies used by ReGenesis in the first two data generation steps. On this condition, Mistral-7B-Instruct-v0.3 can still back-engineer and mine the diverse task-agnostic reasoning paths behind them, suggesting that diverse task-agnostic reasoning strategies used by ReGenesis are effectively preserved in the final task-specific reasoning paths, and that Mistral-7B-Instruct-v0.3 can detect them easily.

  We have also run the same analysis at OOD inference on OpenBookQA, on models fine-tuned with ReGenesis and STaR separately on NumGLUE dataset to show that models trained with ReGenesis, actually learned to adopt more diverse task-agnostic reasoning strategies at inference in OOD settings. We present the results below. Results suggest that models trained with ReGenesis’s task specific reasoning paths, have learned the diverse task-agnostic reasoning strategies that ReGenesis has adopted for path generation and can apply them on OOD tasks.

  We show the clustering results on NumGLUE training data and OpenbookQA OOD inference data below separately.

We appreciate Reviewer TzKs's valuable feedback, which has guided us to conduct additional empirical analysis that strengthens our submission. In response, we present the following analysis:

1. Why ReGenesis works: We conduct empirical analysis to demonstrate why ReGenesis outperforms competing baselines, emphasizing that data size is not a contributing factor to our model's success.
2. Additional Baseline: Few-shot CoT Prompting: Following the reviewer's suggestion, we introduce an additional baseline: few-shot CoT prompting (w/o FT), using GPT-4-generated CoT examples for testing the Mistral-7B-Instruct-v0.3 model.
3. Fine-tuning on GSM8k’s Gold Trajectories: We fine-tune the Mistral-7B-Instruct-v0.3 model on GSM8k’s Gold Trajectories and show that it still underperforms compared to the ReGenesis-trained model in both in-domain and out-of-domain scenarios.
4. Reasoning Path Hints & Adverse Effects: We explain why providing hints to LLMs during final reasoning path generation has adverse effects, supported by relevant examples.
5. Training-Testing Dataset Complexity: We discuss the reviewer’s concern regarding our use of more complex training datasets and simpler test datasets, and we would like to clarify that this is not the case.
6. Typo Correction: We provide a correction for the typo ("A+S") noted by the reviewer.

These revisions are intended to provide further clarification of our methods and findings, and we hope they adequately address all the points raised by the reviewer.

This is a gentle reminder that the discussion period closes soon. We have thoroughly addressed the concerns raised, provided detailed rationales, and conducted additional experiments and case studies to strengthen our work. We would be greatly encouraged if the reviewers either raised their scores or engaged in further discussion with us. Thanks a lot.

We appreciate Reviewer tjkA's detailed and insightful feedback. In response, we will make the following response:

• Figures and Tables: We will revise the figures and tables as suggested, including adding borders and incorporating an Average Score column in Tables 2 and 3.

• Advanced Models for Reasoning Paths: we conducted experiments using more advanced models (GPT-4o) to generate reasoning paths for training data. Notably, employing GPT-4o for this generating reasoning paths with ReGenesis resulted in significant performance improvements in both in-domain and out-of-domain tasks.

• More Out-of-Domain Experiments: As per the reviewer's suggestion, we have also run additional OOD experiments on code generation benchmarks (MBPP and HumanEval). The results, reported as pass@10, show that after using ReGenesis, performance on code generation tasks is improved over both LLMs (Mistral-7B-Instruct-v0.3) without fine-tuning and those trained with STaR.

We believe these changes will address the reviewer's concerns and enhance the clarity of our work. Thanks.

We thank Reviewer tjkA for pushing for OOD results on more distant tasks such as code generation. We apologize for the delayed response due to our limited GPU resources in the ICLR rebuttal season.

We agree that including broader OOD tasks like code generation would enhance the evaluation. To address this, we conducted additional evaluations on two code generation benchmarks: MBPP and HumanEval. The results, reported as pass@10, show that after using ReGenesis, performance on code generation tasks is improved over both LLMs (Mistral-7B-Instruct-v0.3) without fine-tuning and those trained with STaR. We will include the results in our revised version.

We hope that these results effectively address the reviewer's concerns. Thanks so much.

Reviewer: “ReGenesis is designed to construct reasoning solutions that incorporate general task-agnostic reasoning guidance. However, it is unclear how the ReGenesis-generated reasoning chains retain this generalizability across tasks. According to Table 25, the generated reasoning chains appear more specific and detailed.” -> More analysis on why ReGenesis works for OOD tasks”

Our Response:

Although ReGenesis uses the final detailed and task-specific reasoning paths as training data, the diverse task-agnostic reasoning strategies that Regenesis adopts to generate such final task-specific paths, are effectively preserved in the final task-specific paths such that it is easy for LLMs, such as mistral-7B-instruct, to detect them and learn them from the final task-specific paths. We run the following analysis to verify this hypothesis.

We randomly select 500 reasoning paths from ReGenesis and STaR’s generated data of NumGLUE training data set. Note that the 500 reasoning paths of each method are all final task-specific reasoning paths. Then, we prompt Mistral-7B-Instruct-v0.3 to label each of the total 1000 reasoning paths the task-agnostic reasoning strategy behind the task-specific path. The goal of this experiment is to see whether from the task-specific path only, LLMs can already tell the task-agnostic strategy behind it and that ReGenesis’s task-specific paths have more diverse task-agnostic strategies. If so, it indicates that LLMs are likely to be able to capture the task-agnostic strategies behind the task-specific paths in the training data, even if they are trained only with task-specific paths.

Specifically, after Mistral-7B-Instruct-v0.3 labeling the task-agnostic reasoning strategies, we task Mistral-7B-Instruct-v0.3 to cluster the task-agnostic reasoning strategies generated by itself and suggest cluster names for both ReGenesis and STaR. We present the results as follows. Results show that it is quite apparent for Mistral-7B-Instruct-v0.3 that the final task-specific paths generated by ReGenesis, have more diverse task-agnostic reasoning strategies behind them, compared to STaR (More strategy clusters for ReGenesis and less for STaR). Note that throughout this experiment, we don’t give Mistral-7B-Instruct-v0.3 any knowledge on how ReGenesis generates the final task-specific paths and what are the general reasoning strategies used by ReGenesis in the first two data generation steps. On this condition, Mistral-7B-Instruct-v0.3 can still back-engineer and mine the diverse task-agnostic reasoning paths behind them, suggesting that diverse task-agnostic reasoning strategies used by ReGenesis are effectively preserved in the final task-specific reasoning paths, and that Mistral-7B-Instruct-v0.3 can detect them easily.

We have also run the same analysis at OOD inference on OpenBookQA, on models fine-tuned with ReGenesis and STaR separately on NumGLUE dataset to show that models trained with ReGenesis, actually learned to adopt more diverse task-agnostic reasoning strategies at inference in OOD settings. We present the results below. Results suggest that models trained with ReGenesis’s task specific reasoning paths, have learned the diverse task-agnostic reasoning strategies that ReGenesis has adopted for path generation and can apply them on OOD tasks.

We show the clustering results on NumGLUE training data and OpenbookQA OOD inference data below separately.

We thank Reviewer KRUg for their insightful questions and comments, which have greatly contributed to improving our submission. Below, we summarize our responses to address your concerns:

1. Task-Agnostic Reasoning Strategies - Why ReGenesis works: we run additional experiments demonstrating potential reasons of ReGenesis's superior OOD performance. We found that although ReGenesis trains LLMs with the final task-specific reasoning paths, the diverse task-agnostic reasoning strategies employed by ReGenesis in earlier generation steps are actually effectively preserved in the final task-specific reasoning paths, enabling LLMs to detect and learn these strategies easily. Even when trained exclusively on the final reasoning paths, the LLM successfully learns such task-agnostic reasoning strategies and adopts them to OOD tasks.

2. Sensitivity to Guideline Selection and robustness to low-quality reasoning paths: We address the sensitivity of ReGenesis to the selection of inappropriate guidelines, as detailed in our response. Additionally, we provide a table illustrating how different training datasets exhibit slight variations in preferences for specific seed reasoning guidelines.

3. Performance Comparison on GSM8K: A comprehensive case study on why ReGenesis outperforms STaR on the GSM8k dataset suggests that ReGensis leverages its diverse answering strategies to meet GSM8k’s requirements effectively.

4. Mixing Training Datasets: As shown in Appendix A.1, combining different training datasets slightly improves the LLM’s out-of-distribution performance.

We appreciate the opportunity to further clarify and refine our work based on your valuable feedback.

We show the clustering results of task-agnostic reasoning strategies mined from ReGenesis's OpenbookQA OOD inference data. (We only include examples for less straightforward cluster names due to the limit of the space.)

[ReGenesis] Clusters with examples:

• Cluster 1 Process of Elimination: This cluster involves strategies that use the process of elimination to systematically remove incorrect options until the correct one is identified.

• Cluster 2 Step-by-Step Reasoning: This cluster includes strategies that focus on breaking down problems into smaller, manageable steps or components, allowing for a systematic evaluation and logical progression towards the solution.

• Cluster 3 Logical or Deductive Reasoning: This cluster contains strategies that involve using logical or deductive reasoning to analyze problems, draw specific conclusions from general premises, and make well-informed decisions based on structured logical thinking.

• Cluster 4 Critical Thinking: Strategies in this cluster involve critically analyzing each option, evaluating their implications and effectiveness, and considering external and internal factors to arrive at a logical conclusion based on evidence and reasoning.

• Cluster 5 Compare and Contrast or Evaluative Reasoning: This cluster includes strategies that involve comparing and contrasting options based on specific criteria to evaluate their pros and cons and determine the best choice.

  • Question: What has more gravity force than Earth but less than the sun? (A) Jupiter (B) the moon (C) a space station (D) a comet
  • ReGenesis Answer: Identify the Objective: We need to find an object that has more gravity than Earth but less than the Sun. Define Known Values: - Earth's gravity: $9.8 \text{ m/s}^2$ - Sun's gravity: $274 \text{ m/s}^2$ Establish the Range: - We are looking for an object with a gravity force between Earth's and the Sun's gravity. Consider Possible Objects: - Jupiter, the moon, a space station, and a comet are all potential candidates. Evaluate Each Object: - Jupiter: Jupiter's gravity is approximately $24.8 \text{ m/s}^2$. ... The Moon: The moon's gravity is approximately $1.6 \text{ m/s}^2$. ... A Space Station: The gravity of a space station is negligible compared to Earth's, typically less than $1 \text{ m/s}^2$. Therefore, it is .... A Comet: Comets generally have a gravity close to zero, so they do not have significant gravity compared to Earth or the Sun. Conclude the Evaluation: - Jupiter is the only object that satisfies the condition of having more gravity than Earth but less than the Sun. Verify the Outcome: - Double-checking the values, Jupiter's gravity is indeed $24.8 \text{ m/s}^2$, ... Therefore, the object with more gravity than Earth but less than the Sun is (A) Jupiter.
  • LLM Summarized Strategy: How can I systematically evaluate and compare each option to find the correct answer?

• Cluster 6 Use of Domain Knowledge or Common Knowledge: Strategies in this cluster focus on leveraging existing knowledge, understanding of a subject, or common principles to answer questions accurately and confidently.

• Cluster 7 Systematic thinking and evaluation: This approach involves identifying the problem and the parameters and systematically evaluating each option to eliminate incorrect answers and arrive at the correct choice.

  • Question: What has more gravity force than Earth but less than the sun?(A) Jupiter (B) the moon (C) a space station (D) a comet
  • ReGenesis Answer: Jupiter: Jupiter is the largest planet in the solar system, with a mass much greater than Earth (about 318 times the mass of Earth). Its gravity on the surface is 24.79 m/s², which is more than Earth's gravity (9.81 m/s²), so it has more gravitational force than Earth. The Moon: The Moon's mass is only about 1/81 of Earth's, and its gravity is about 1/6th of Earth's gravity (1.62 m/s²). Thus, the Moon has far less gravity than Earth. A Space Station: A space station, like the International Space Station (ISS), is in orbit around Earth. The force of gravity at the ISS's altitude (about 400 km above Earth's surface) is still about 90% of the Earth's surface gravity. However, astronauts feel weightless due to the free-fall condition, not because there is no gravity. The gravitational force here is comparable to Earth's gravity, not more. A Comet: Comets are small, icy bodies in space with masses much smaller than Earth. The gravitational force they exert is much weaker than Earth's gravity. Therefore, a comet's gravity is much less than Earth's. Conclusion: Jupiter has more gravity than Earth and the Sun has much more. Therefore, Jupiter is the only option that has more gravity than Earth but less than the Sun. Correct Answer: (A) Jupiter
  • LLM Summarized Strategy: How can I systematically evaluate and compare each option to find the correct answer?

We show the clustering results of task-agnostic reasoning strategies mined from STaR's OpenbookQA OOD inference data. (We only include examples for less straightforward cluster names due to the limit of the space.)

[STaR] Clusters:

• Cluster 1 Process of Elimination: Focuses on systematically ruling out incorrect options to identify the correct answer.
  • Question: Where might a bunny live? (A) a thicket (B) atop palm trees (C) a sewer system (D) a deserted island
  • STaR Answer: A bunny would most likely live in a thicket, which is a dense growth of bushes or undergrowth, providing shelter and food. The other options are unlikely, as palm trees are tall and not typically found in habitats suitable for bunnies, sewer systems are not a suitable habitat for any animal, and a deserted island would be too isolated and lack the necessary food and shelter for a bunny. Therefore, the answer is (A).
  • LLM Summarized Strategy: Process of elimination: Assess each potential answer choice and eliminate those that are clearly incorrect or implausible. This strategy involves evaluating the likelihood or suitability of each option based on the information given or general knowledge, narrowing down the choices to the most reasonable one.
• Cluster 2 Deductive Reasoning: Focuses on using general principles or known facts to draw specific conclusions.
• Cluster 3 Identifying Core Concepts: Focuses on identifying the fundamental concepts or knowledge required to solve the problem.
• Cluster 4 Analytical Reasoning: Focuses on analyzing options or scenarios using critical thinking.

Thank you very much for your detailed responses and extensive experiments. Your thorough explanations have resolved my doubts and addressed my concerns. I have increased my score accordingly.

We show the clustering results of task-agnostic reasoning strategies mined from STaR's NumGLUE training data. (We only include examples for less straightforward cluster names due to the limit of the space.)

[STaR] Clusters with examples:

• Cluster 1: Direct Calculation and Simplification

• Cluster 2: Algebraic and Formula-based Approaches

  • Question: Claire decides to go to a nearby animal farm that has a total of 19 animals. He is standing in a queue near the entry gate, from the gate he could only see legs of the animals in the farm and he counted the total number of legs to be 56. He already knows that there are only ducks and horses in this farm. From this information find the number of ducks
  • STaR Answer: Let's say the number of ducks is D and the number of horses is H.\n\nSince each duck has 2 legs and each horse has 4 legs, the total number of legs can be represented by the equation:\n\n2D + 4H = 56\n\nSince there are 19 animals in total, we can set up another equation:\n\nD + H = 19\n\nWe can solve this system of equations by substituting D with 19 - H, which gives us:\n\n2(19 - H) + 4H = 56\n\nExpanding the equation and simplifying, we get:\n\n38 - 2H + 4H = 56\n2H = 18\nH = 9\n\nNow that we know the number of horses (H = 9), we can find the number of ducks (D) by substituting H into the first equation:\n\n2D + 4(9) = 56\n2D + 36 = 56\n2D = 20\nD = 10\n\nSo, there are 10 ducks in the farm.
  • LLM Summarized Strategy: How can I create equations that represent the problem and use them to find a solution?

• Cluster 3: Stoichiometry and Chemical Reactions

  • Question: How many moles of CH4 are required to react with 2 moles of Cl2 to form 2 moles of CH3Cl along with 2 moles of HCl
  • STaR Answer: The equation for the reaction is:\n\nCH4 (g) + Cl2 (g) \u2192 CH3Cl (g) + HCl (g)\n\nFrom the equation, we can see that:\n\n1 mole of CH4 reacts with 1 mole of Cl2 to form 1 mole of CH3Cl and 1 mole of HCl\n\nSince the question asks for 2 moles of CH3Cl and 2 moles of HCl, we can see that 2 moles of CH4 are required to react with 2 moles of Cl2 to form 2 moles of CH3Cl and 2 moles of HCl.\n\nTherefore, the answer is 2 moles of CH4.
  • LLM Summarized Strategy: Use direct application of stoichiometry: Identify the chemical reaction, balance it if necessary, and use the stoichiometric coefficients to directly calculate the required amount of reactants or products based on the given data.

• Cluster 4: Problem Decomposition and Step-by-Step Calculation

• Cluster 5: Logical Reasoning and Pattern Recognition

Reviewer: “Why is there such a large difference between STaR and the proposed method on GSM8k? (46.3 vs 63.6)”

Our Response:

We first conducted a case study focusing on questions where our method succeeded but STaR failed. Our analysis revealed that our method employs much more diverse strategies, which are absent in LLMs trained with STaR.

1. ReGenesis answers questions by systematically breaking down the problem into smaller parts: gathering data, calculating potential increases, and comparing profits. This approach demonstrates critical thinking and analytical reasoning, effectively leading to the correct conclusion.

• Example question: A merchant wants to make a choice of purchase between 2 purchase plans: jewelry worth 5,000 or electronic gadgets worth 8,000. His financial advisor speculates that the jewelry market will go up 2.5% while the electronic gadgets market will rise 1.2% within the same month. If the merchant is looking to maximize profit at the end of this month by making a choice, how much profit would this be?
• STaR answer: Let's calculate the future value of both options: 1. If the merchant chooses jewelry worth 5,000, the future value will be: 5,000 + (0.025 * 5,000) = <<5000 + (0.025 * 5000)=525>>5,250. 2. If the merchant chooses electronic gadgets worth 8,000, the future value will be: 8,000 + (0.012 * 8,000) = <<8000 + (0.012 * 8000)=808>>8,080. Comparing both values, the merchant should choose electronic gadgets worth $8,000 since it has a higher future value of 8,080 compared to jewelry worth 5,250. The profit the merchant would make is 8,080 - 8,000 = <<8080-8000=80>>80. #### 80
• ReGenesis Answer: 1. Gather data: - Current prices: 5,000 (jewelry) and 8,000 (electronic gadgets) - Expected market growth rates: 2.5% for jewelry and 1.2% for electronic gadgets 2. Calculate the potential increase in value: - For jewelry: 2.5% of 5,000 = 125 - For electronic gadgets: 1.2% of 8,000 = 96 3. Calculate the potential profit for each option: - Profit for jewelry: 125 (increased value) - Profit for electronic gadgets: 96 (increased value) 4. Compare the potential profits: - The merchant will make a profit of 125 if they choose jewelry, or $96 if they choose electronic gadgets. 5. Evaluate other factors: - The merchant should consider additional factors such as personal preferences, budget, and other financial obligations when making the final decision. 6. Make a decision: - Based on the calculated profits, the merchant will make a higher profit by choosing jewelry. However, they should weigh this with their personal financial situation and other factors before making the final decision. The merchant will make a profit of 125 if they choose jewelry.

2. ReGenesis also uses more logical reasoning processes, which involves identifying a pattern.

• Example question: Alice likes to count the puffs of clouds in the sky while she eats her lunch outside at school. On Monday she counts just 3 puffs of clouds. Each day after that through Friday, though, she sees double the number of clouds in the sky as the day before. At the end of the week, how many clouds will she have counted in the sky at lunch across all five days?
• STaR answer: On Monday, Alice counts 3 puffs of clouds. Tuesday: 3 x 2 = <<32=6>>6 puffs. Wednesday: 6 x 2 = <<62=12>>12 puffs. Thursday: 12 x 2 = <<122=24>>24 puffs. Friday: 24 x 2 = <<242=48>>48 puffs. Adding all of them up: 3 + 6 + 12 + 24 + 48 = <<3+6+12+24+48=83>>83 puffs. #### 83
• ReGenesis Answer: 1. Identify the initial number of clouds Alice sees on Monday: 3 clouds. 2. Recall that each day after Monday, she sees double the number of clouds from the previous day. 3. Create a pattern for the number of clouds she sees on each day: Monday: 3 clouds; Tuesday: 3 * 2 = 6 clouds; Wednesday: 6 * 2 = 12 clouds; Thursday: 12 * 2 = 24 clouds; Friday: 24 * 2 = 48 clouds. 4. Add up the number of clouds she sees each day to find the total number of clouds across the week: Total clouds = Monday + Tuesday + Wednesday + Thursday + Friday; Total clouds = 3 + 6 + 12 + 24 + 48; Total clouds = 93 clouds. So, Alice will have counted 93 clouds in the sky at lunch across all five days.

Reviewer: ”Can you compare the size of synthetic data you use for all methods? You mentioned that 32 reasoning paths are sampled for STaR, but how many of them are used for training after filtering?”

Our response:

We thank the reviewer for this question since it points out that the size of the synthetic data after filtering is also a variable that could significantly affect model performance. We had this variable controlled in all the experiments of the current submission and we should have presented it in the experiment section. The table below lists the training data size that ReGenesis and STaR use for all training data sets:

The same training data size of ReGenesis and STaR indicates that the performance difference between ReGenesis and STaR is not caused by training data size difference.

We show the clustering results of task-agnostic reasoning strategies mined from ReGenesis's OpenbookQA OOD inference data. (We only include examples for less straightforward cluster names due to the limit of the space.)

[ReGenesis] Clusters with examples:

• Cluster 1 Process of Elimination: This cluster involves strategies that use the process of elimination to systematically remove incorrect options until the correct one is identified.

• Cluster 2 Step-by-Step Reasoning: This cluster includes strategies that focus on breaking down problems into smaller, manageable steps or components, allowing for a systematic evaluation and logical progression towards the solution.

• Cluster 3 Logical or Deductive Reasoning: This cluster contains strategies that involve using logical or deductive reasoning to analyze problems, draw specific conclusions from general premises, and make well-informed decisions based on structured logical thinking.

• Cluster 4 Critical Thinking: Strategies in this cluster involve critically analyzing each option, evaluating their implications and effectiveness, and considering external and internal factors to arrive at a logical conclusion based on evidence and reasoning.

• Cluster 5 Compare and Contrast or Evaluative Reasoning: This cluster includes strategies that involve comparing and contrasting options based on specific criteria to evaluate their pros and cons and determine the best choice.

  • Question: What has more gravity force than Earth but less than the sun? (A) Jupiter (B) the moon (C) a space station (D) a comet
  • ReGenesis Answer: Identify the Objective: We need to find an object that has more gravity than Earth but less than the Sun. Define Known Values: - Earth's gravity: $9.8 \text{ m/s}^2$ - Sun's gravity: $274 \text{ m/s}^2$ Establish the Range: - We are looking for an object with a gravity force between Earth's and the Sun's gravity. Consider Possible Objects: - Jupiter, the moon, a space station, and a comet are all potential candidates. Evaluate Each Object: - Jupiter: Jupiter's gravity is approximately $24.8 \text{ m/s}^2$. ... The Moon: The moon's gravity is approximately $1.6 \text{ m/s}^2$. ... A Space Station: The gravity of a space station is negligible compared to Earth's, typically less than $1 \text{ m/s}^2$. Therefore, it is .... A Comet: Comets generally have a gravity close to zero, so they do not have significant gravity compared to Earth or the Sun. Conclude the Evaluation: - Jupiter is the only object that satisfies the condition of having more gravity than Earth but less than the Sun. Verify the Outcome: - Double-checking the values, Jupiter's gravity is indeed $24.8 \text{ m/s}^2$, ... Therefore, the object with more gravity than Earth but less than the Sun is (A) Jupiter.
  • LLM Summarized Strategy: How can I systematically evaluate and compare each option to find the correct answer?

• Cluster 6 Use of Domain Knowledge or Common Knowledge: Strategies in this cluster focus on leveraging existing knowledge, understanding of a subject, or common principles to answer questions accurately and confidently.

• Cluster 7 Systematic thinking and evaluation: This approach involves identifying the problem and the parameters and systematically evaluating each option to eliminate incorrect answers and arrive at the correct choice.

  • Question: What has more gravity force than Earth but less than the sun?(A) Jupiter (B) the moon (C) a space station (D) a comet
  • ReGenesis Answer: Jupiter: Jupiter is the largest planet in the solar system, with a mass much greater than Earth (about 318 times the mass of Earth). Its gravity on the surface is 24.79 m/s², which is more than Earth's gravity (9.81 m/s²), so it has more gravitational force than Earth. The Moon: The Moon's mass is only about 1/81 of Earth's, and its gravity is about 1/6th of Earth's gravity (1.62 m/s²). Thus, the Moon has far less gravity than Earth. A Space Station: A space station, like the International Space Station (ISS), is in orbit around Earth. The force of gravity at the ISS's altitude (about 400 km above Earth's surface) is still about 90% of the Earth's surface gravity. However, astronauts feel weightless due to the free-fall condition, not because there is no gravity. The gravitational force here is comparable to Earth's gravity, not more. A Comet: Comets are small, icy bodies in space with masses much smaller than Earth. The gravitational force they exert is much weaker than Earth's gravity. Therefore, a comet's gravity is much less than Earth's. Conclusion: Jupiter has more gravity than Earth and the Sun has much more. Therefore, Jupiter is the only option that has more gravity than Earth but less than the Sun. Correct Answer: (A) Jupiter
  • LLM Summarized Strategy: How can I systematically evaluate and compare each option to find the correct answer?

We show the clustering results of task-agnostic reasoning strategies mined from STaR's OpenbookQA OOD inference data. (We only include examples for less straightforward cluster names due to the limit of the space.)

[STaR] Clusters:

• Cluster 1 Process of Elimination: Focuses on systematically ruling out incorrect options to identify the correct answer.
  • Question: Where might a bunny live? (A) a thicket (B) atop palm trees (C) a sewer system (D) a deserted island
  • STaR Answer: A bunny would most likely live in a thicket, which is a dense growth of bushes or undergrowth, providing shelter and food. The other options are unlikely, as palm trees are tall and not typically found in habitats suitable for bunnies, sewer systems are not a suitable habitat for any animal, and a deserted island would be too isolated and lack the necessary food and shelter for a bunny. Therefore, the answer is (A).
  • LLM Summarized Strategy: Process of elimination: Assess each potential answer choice and eliminate those that are clearly incorrect or implausible. This strategy involves evaluating the likelihood or suitability of each option based on the information given or general knowledge, narrowing down the choices to the most reasonable one.
• Cluster 2 Deductive Reasoning: Focuses on using general principles or known facts to draw specific conclusions.
• Cluster 3 Identifying Core Concepts: Focuses on identifying the fundamental concepts or knowledge required to solve the problem.
• Cluster 4 Analytical Reasoning: Focuses on analyzing options or scenarios using critical thinking.

We show the clustering results of task-agnostic reasoning strategies mined from ReGenesis's NumGLUE training data . (We only include examples for less straightforward cluster names due to the limit of the space.)

[ReGenesis] Clusters with examples:

• Cluster 1: Step-by-Step Process

• Cluster 2: Applying Logic and Formulas

• Cluster 3: Reflective Thinking

  • Question: Joshua became headmaster of a school in the month of May. Joshua stayed headmaster for 2 months before retiring in the same year. In which month did Joshua retire?

  • ReGenesis Answer: Step 1: Reflect on initial biases and assumptions I recognize that the common Western calendar year structure consists of 12 months, but I should be open to alternative structures and month lengths. Step 2: Reflect on past experiences and learning. I've encountered problems that required creative thinking and consideration of different scenarios and structures. I'm prepared to think creatively and consider alternative month lengths or school calendar systems. Step 3: Reframe the problem and ask specific and open-ended questions. We know that Joshua became headmaster in May ... Step 4: Analyze and evaluate the problem. Joshua was headmaster for 2 months, and we know he retired ... Step 5: Make a decision or solution. Based on the analysis, we can conclude that Joshua retired in July, considering the Western calendar structure, or July or August considering alternative month lengths and school calendar systems. Final Answer: \boxed{July}

  • LLM Summarized Strategy: Reflective Thinking: This strategy emphasizes stepping back from the problem, engaging in introspection and self-reflection, examining personal biases and assumptions, and considering alternative interpretations or structures.

• Cluster 4: Sequential and Chronological Analysis

• Cluster 5: Information Extraction

• Cluster 6: Problem Decomposition

• Cluster 7: Systematic Thinking

  • Question: How many moles of Sodium bicarbonate are required to react with 3 moles of Hydrochrolic acid to form 3 moles of Water, 3 moles of Carbon dioxide and 3 moles of Sodium chloride

  • ReGenesis Answer: Step 1: Define the System. Reactants: 3 moles of Hydrochloric acid (HCl) and x moles of Sodium bicarbonate (NaHCO3). Products: 3 moles of Water (H2O), 3 moles of Carbon dioxide (CO2), and 3 moles of Sodium chloride (NaCl). Step 2: Identify the Reaction. Chemical reaction between HCl and NaHCO3 to form H2O, CO2, and NaCl. Reactants: 3 moles of HCl and x moles of NaHCO3. Products: 3 moles of H2O, 3 moles of CO2, and 3 moles of NaCl. Step 3: Analyze Feedback Loops and Interdependencies. The reaction is irreversible, meaning reactants are consumed and cannot be reversed. Formation of H2O and CO2 affects reaction rate and NaCl formation. Formation of NaCl affects reaction rate and H2O and CO2 formation. Step 4: Develop a Holistic Solution. Set up a balanced chemical equation: 3 HCl + NaHCO3 → 3 H2O + 3 CO2 + NaCl. Step 5: Determine the Required Amount of NaHCO3. 1 mole of NaHCO3 reacts with 1 mole of HCl to form 1 mole of each product. To react 3 moles of HCl, 3 moles of NaHCO3 are required. Conclusion: 3 moles of Sodium bicarbonate (NaHCO3) are required to react with 3 moles of Hydrochloric acid (HCl) to form 3 moles of Water, 3 moles of Carbon dioxide, and 3 moles of Sodium chloride. Therefore, the answer is: \boxed{3 moles} of Sodium bicarbonate are required.

  • LLM Summarized Strategy: Use systems thinking: Consider the problem as part of a larger system and understanding the interconnectedness of various elements. Focuses on identifying the underlying causes, feedback loops, and interdependencies that influence the problem, and developing holistic solutions that address the system as a whole.

• Cluster 8: Questioning Assumptions and Critical Thinking

  • Question: Piper counted wings of all commercial planes placed in an air exhibition. If total wings were 90, how many planes were there?
  • ReGenesis Answer: I. Establish the known facts: Total number of wings = 90. Each plane has two wings. II. Question initial assumption: Is the problem asking us to find the total number of wings or the number of planes? III. Challenge and revise initial assumption: If each plane has two wings, then we can set up an equation: 2x = 90, where x is the number of planes. Divide both sides by 2 to solve for x: x = 90 ÷ 2 = 45. IV. Conclusion: Therefore, the correct answer is that there are \boxed{45} planes in the air exhibition.
  • LLM Summarized Strategy: Start by establishing known facts, questioning assumptions, and setting up a simple equation based on logical deduction. Simplify the problem by considering the relationship between the number of wings and planes, and solve for the unknown variable.

• Cluster 9: Mathematical and Analytical Calculation

• Cluster 10: Stoichiometry and Chemical Problem Solving

Reviewer: “Potential mismatch between the motivation and the proposed method. The authors hypothesized that current methods’ underperforms on OOD tasks because "their self-synthesized reasoning paths are too task-specific, lacking general task-agnostic reasoning guidance". However, even though they proposed a multi-level prompting method starting from general guidelines, in the end, they still used task-specific reasoning paths to train the model. While the authors suggest their method can enhance the diversity of reasoning chains, which helps with OOD performance, there is no clear analysis.”

Our Response:

Although ReGenesis uses the final detailed and task-specific reasoning paths as training data, the diverse task-agnostic reasoning strategies that Regenesis adopts to generate such final task-specific paths, are effectively preserved in the final task-specific paths such that it is easy for LLMs, such as mistral-7B-instruct, to detect them and learn them from the final task-specific paths. We run the following analysis to verify this hypothesis.

We randomly select 500 reasoning paths from ReGenesis and STaR’s generated data of NumGLUE training data set. Note that the 500 reasoning paths of each method are all final task-specific reasoning paths. Then, we prompt Mistral-7B-Instruct-v0.3 to label each of the total 1000 reasoning paths the task-agnostic reasoning strategy behind the task-specific path. The goal of this experiment is to see whether from the task-specific path only, LLMs can already tell the task-agnostic strategy behind it and that ReGenesis’s task-specific paths have more diverse task-agnostic strategies. If so, it indicates that LLMs are likely to be able to capture the task-agnostic strategies behind the task-specific paths in the training data, even if they are trained only with task-specific paths.

Specifically, after Mistral-7B-Instruct-v0.3 labeling the task-agnostic reasoning strategies, we task Mistral-7B-Instruct-v0.3 to cluster the task-agnostic reasoning strategies generated by itself and suggest cluster names for both ReGenesis and STaR. We present the results as follows. Results show that it is quite apparent for Mistral-7B-Instruct-v0.3 that the final task-specific paths generated by ReGenesis, have more diverse task-agnostic reasoning strategies behind them, compared to STaR (More strategy clusters for ReGenesis and less for STaR). Note that throughout this experiment, we don’t give Mistral-7B-Instruct-v0.3 any knowledge on how ReGenesis generates the final task-specific paths and what are the general reasoning strategies used by ReGenesis in the first two data generation steps. On this condition, Mistral-7B-Instruct-v0.3 can still back-engineer and mine the diverse task-agnostic reasoning paths behind them, suggesting that diverse task-agnostic reasoning strategies used by ReGenesis are effectively preserved in the final task-specific reasoning paths, and that Mistral-7B-Instruct-v0.3 can detect them easily.

We have also run the same analysis at OOD inference on OpenBookQA, on models fine-tuned with ReGenesis and STaR separately on NumGLUE dataset to show that models trained with ReGenesis, actually learned to adopt more diverse task-agnostic reasoning strategies at inference in OOD settings. We present the results below. Results suggest that models trained with ReGenesis’s task specific reasoning paths, have learned the diverse task-agnostic reasoning strategies that ReGenesis has adopted for path generation and can apply them on OOD tasks.

We show the clustering results on NumGLUE training data and OpenbookQA OOD inference data below separately.

We deeply appreciate Reviewer w4BA’s insightful comments, which has guided us to add more analysis to strengthen our motivation and to show lights upon the potential reason why ReGenesis works. We respond to Reviewer w4BA’s comments as follows with more analytical results.

• Preservation of task-agnostic Reasoning Strategies: We conducted experiments to demonstrate that the task-agnostic reasoning strategies used by ReGenesis are effectively preserved in the final task-specific reasoning paths, allowing LLMs to easily detect and learn them. Despite being trained only with the final task-specific reasoning paths generated by ReGenesis, LLMs successfully learn these task-agnostic strategies and generalize them to out-of-distribution (OOD) tasks.
• Data Size Consistency Across Models: We present a table showing that the size of the synthetic data, after filtering, remains consistent across different baselines and our models.
• Performance Without Ground-Truth Data: As shown in Table 5, ReGenesis outperforms the baseline models even without relying on ground-truth data.
• Case Study on GSM8k: We also conducted a comprehensive case study, which reveals why ReGenesis significantly outperforms STaR on the GSM8k dataset.

This is a gentle reminder that the discussion period closes soon. We have thoroughly addressed the concerns raised, provided detailed rationales, and conducted additional experiments and case studies to strengthen our work. We would be greatly encouraged if the reviewers either raised their scores or engaged in further discussion with us. Thanks a lot.