TypedThinker: Diversify Large Language Model Reasoning with Typed Thinking

Thank you for your thoughtful feedback and for acknowledging the novelty of our framework and the thoroughness of our experiments. We will revise the manuscript to clarify the role of the memory during training and inference and better outline the framework flow from input to output, addressing any potential ambiguities. Your comments will help us refine the paper, and we greatly appreciate your insights.

W1: Although the idea of the paper is novel, the technical contribution seems to be limited. The authors seem to simply fine-tune a model to select the reasoning type and use it to search the examples and specify the prompt.

A1: Thank you for recognizing the novelty of our paper's idea of introducing reasoning types to diversify thinking processes. As outlined in the Introduction, effectively enhancing reasoning with reasoning types involves two significant challenges: (1) How to identify the appropriate reasoning type for a general reasoning problem, and (2) How to apply a specific reasoning type in problem-solving.

To address these challenges, we propose TypedThinker, a simple yet effective framework. TypedThinker uses a meta-thinker to select reasoning types and collect demonstrations for each type. However, the absence of ground-truth human annotations for the reasoning types required by each problem and their corresponding reasoning processes presents a significant hurdle. To overcome this, we curate a fine-tuning dataset by diversifying reasoning experiences with reasoning types and introducing an effectiveness score to estimate the suitability of each type. This provides pseudo-labels for reasoning types and demonstrations without requiring human annotations (Section 3.2).

Thus, our contributions extend beyond introducing typed thinking to include an effective method for optimizing type selection and typed reasoning without human supervision. The ablation study (Table 2) demonstrates the importance of each TypedThinker component, while the comparison between prompt-based type selection (CoT Selection) and TypedThinker (Table 1) highlights the critical role of the fine-tuned meta-thinker.

---

W2: Some technical details may be misleading. For example, the memory usually refers to data buffers frequently updated in inference, …, It would be better to clarify the framework from the input to the output for training and inference.

A2: Thank you for this suggestion. To avoid potential confusion, we will revise the term “Memory” to “Collection of Demonstrations” to emphasize that it represents a static collection of examples used to guide the reasoning process for specific reasoning types. We will also clarify the distinction between training and inference in the final paragraph of Section 3.1 in our revised manuscript.

Additionally, we have elaborated on the accuracy measurement in Lines 256–258:

On logical benchmarks, we compare the response with the ground truth using an exact match. For mathematics benchmarks, we follow Toshniwal et al. (2024) to compute accuracy.

We have detailed the memory size in Table 9 (Empirical Dataset) in the Appendix and included anonymized code as supplementary material in the original submission to ensure reproducibility.

Thank you for your positive evaluation and for highlighting the strengths of our framework, including its clarity, novelty, and ability to improve LLM reasoning performance. We appreciate your constructive feedback and hope these concerns are thoroughly addressed in our response below.

W1: Scalability and resource requirement due to the maintenance of the memory buffer could increase computational cost.

A1: Thank you for raising this concern. Our memory buffer is constructed during the training phase by collecting successful trajectories and remains fixed during inference. The memory size is approximately 10k instances for the logic domain and 5k for the math domain, which is manageable and does not impose a significant maintenance burden. Furthermore, TypedThinker can be applied to new logic or math tasks without requiring updates to the memory buffer, as demonstrated in Section 4.5. For retrieval, we employ the sentence transformer to calculate similarity, a lightweight process compared to the computational cost of LLM inference. This ensures the framework remains efficient and scalable.

---

W2: How generalizable is the learned experience to other different domains? In sect. 4.5 the authors discuss the applicability of typedThinker to ContextHub however this remains a collection of problems for logical reasoning. There is no issue with the proposed framework being suited only for mathematical and logical reasoning, but then this needs to be specified.

A2: We have discussed this concern in the limitations section. This paper focuses specifically on logical and mathematical reasoning. While we believe that extending TypedThinker to broader tasks, such as reasoning in code or general problem-solving domains, is an interesting and promising avenue, it is beyond the scope of this study. We will ensure this limitation is clearly articulated in the revised manuscript.

---

Q1: Can this method be applied to other types of reasoning (symbolic, commonsense, etc)?

A: Thank you for the question. Our TypedThinker framework can indeed be extended to other reasoning domains, such as symbolic or commonsense reasoning. For example, the propositional logic benchmark ContextHub, used in our generalization tests, includes symbolic reasoning tasks (e.g., the abstract category). Below is a simple example:

(wqeq or mnze) → zkx. (NOT ttjmx) → kottz. (kottz or zkx) → pofk. Given pofk is False, what is the value of ttjmx ?

In addition to the overall performance shown in Table 4, we present specific performance (+ SC @5) on the abstractcategory of symbolic reasoning below. TypedThinker outperforms baseline methods, even without further fine-tuning the meta-thinker and reasoner for this symbolic reasoning task.

For commonsense reasoning, while the LogiQA dataset contains problems requiring commonsense reasoning, the dataset lacks explicit annotations (such as a specific category) for such tasks. For example:

Traditionally, the most highly sought cars have been the sports cars and similar two-door models. Nevertheless, Zincstone Motors has chosen to eliminate the last two-door models and produce only four-door models. Which of the following would, if true, most help to explain Zincstone Motors' strategy?
Options:
(A) In almost every instance, Zincstone Motors models lead all comparable models of competitors in fuel efficiency and have lower average maintenance costs as well.
(B) After a spate of recent additional safety requirements, the cost of the frame and doors of Zincstone Motors' standard two-door models are now three times as expensive as standard four-door frames and doors.
(C) Many of Zincstone Motors models are exported and sold overseas, including in some countries like Japan, which imports a significant number of cars into the United States.
(D) As American consumers lose access to car manufacturers who produce two-door cars, and as two-door cars occupy smaller and smaller shares of the United States car market, American consumers' tastes tend to shift from two-door cars.

This task requires commonsense knowledge about manufacturing costs, market dynamics, and consumer preferences.

Thank you for your thoughtful feedback and for acknowledging the strengths of our paper. We appreciate your concerns and would like to address them in detail, as we believe our approach offers valuable contributions to the field.

W1: The paper lacks novelty: the idea of introducing different thinking types is not new at all. For example, Meta-Prompting introduced the concept of applying different experts to solve different tasks. Self-Discover proposed to select and compose different reasoning modules to customize the solving of each reasoning task.

A1: You mentioned that our idea of introducing different reasoning types is not novel, pointing out work like Meta-Prompting and Self-Discover. We acknowledge that these works explore similar ideas, such as applying different reasoning modules or experts[1,2,3]. However, we would like to clarify our contribution and how it differs from prior work.

Our key novelty lies in introducing and formalizing four distinct reasoning types (deductive, inductive, abductive, and analogical) for instance-level reasoning and how to effectively apply them to diverse reasoning problems. The challenge we address is not just selecting a reasoning module, but selecting the most appropriate reasoning type for each individual problem and then applying it to improve LLM performance. Unlike previous methods, we explicitly focus on instance-specific reasoning by utilizing a meta-thinker and fine-tuned reasoners. This allows us to optimize reasoning types for each task, which, as shown in our experiments, leads to a more effective reasoning process.

We will add a more detailed comparison to Meta-Prompting and Self-Discover in the related work section to make the distinctions clearer.

To further address the concern on Self-discover, we included Self-Discover as a baseline and present its performance below, following its official implementation [4].

Results show that Self-Discover performs poorly on Mistral 7B and LLaMA 3 8B compared to TypedThinker. This may be due to the difficulty these models face in identifying a reusable high-level reasoning structure for diverse instances, especially in datasets like LogiQA, where reasoning structures are highly varied. Additionally, the generated structures often introduce excessive complexity, confusing the models under zero-shot settings, especially for weaker models like Mistral 7B.

[1] Improving Factuality and Reasoning in Language Models through Multiagent Debate

[2] Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding

[3] SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures

[4] https://github.com/kailashsp/SELF-DISCOVER

---

W2: The fundamental assumption of matching each task into one of the 4 reasoning types is flawed. There are many ways of solving reasoning beyond the 4 reasoning types. In addition, each task could involve several different reasoning types at different steps.

A2: We understand your concern that the assumption of categorizing reasoning into four types may be limiting. However, our focus on logical reasoning has been informed by established frameworks in cognitive science and logic [5, 6], where these four types serve as fundamental approaches to problem-solving. While we agree that reasoning can be diverse and multi-faceted, our method is designed to enhance reasoning by structuring it around these four core logical types, which have proven effective in various domains. For other categorizations, our approach is extendable and could incorporate additional reasoning structures in future work.

Furthermore, as we noted in Section 5, multi-step reasoning scenarios are a natural extension of our framework. One approach is to use the same reasoning type throughout, aligning with our current experimental settings. Alternatively, TypedThinker could dynamically identify the most effective reasoning type at each step, using the meta-thinker to guide reasoning type selection and intermediate problem-solving. Our work lays the groundwork for such multi-step investigations, which we believe is a promising future direction.

[5] Logical Reasoning ISBN 10.0534176887

[6] Logical Reasoning and Learning, ISBN 9781441914279

Q5: Regarding the novelty of the method, the major claims from the authors are four reasoning types and instance-specific reasoning. Neither is novel in the sense that they are just natural extensions/expansions of general reasoning modules and task-level reasoning that are around in many papers already.

A5. Thank you for your thoughtful comments. We respectfully disagree that our work can be viewed as “natural extensions/expansions of general reasoning modules and task-level reasoning”.

First, our focus on logical reasoning types stems from well-established cognitive strategies inspired by human mental processes. We systematically analyze their role in diversifying LLM thinking (Figure 1, Section 4.3) and demonstrate their importance for improving reasoning performance.

Second, we highlight the instance-level reasoning identification and application is necessary but non-trivial because:

• Task-level reason strategy insufficient for diverse tasks, as seen in the performance of Self-Discover, where task-level reason structure fails for instance-level diversity. Similarly, reasoning type identification via prompting (e.g., CoT Selection) often struggles. To overcome the lack of annotations for reasoning type labels in finetuning, we propose effectiveness estimation(Section 3.2).
• Apply a specific reasoning type is challenging*,* as shown by the performance differences between zero-shot and few-shot MoR and the ablation studies. To address this, we introduce typed reasoning collections and fine-tuning (Section 3.2).

Furthermore, we provide a comprehensive investigation of the benefits of reasoning type diversity (Section 4.2), analyze each component’s contributions (Sections 4.3 and 4.4), and evaluate generalization capabilities (Section 4.5). All these observations highlight the contribution of our TypedThinker.

Unlike prior studies, which focus on enhancing specific reasoning types or benchmarking individual reasoning strategies, our work studies how instance-specific reasoning types can improve general logical and mathematical problem-solving. We believe this focus has not been explored in prior research, highlighting the contribution of TypedThinker. Thank you for raising this important point, and we hope this clarifies our contributions.

---

Please let us know if you have further concerns and we are happy to provide more detailed information in the discussion phase.

Thank you for your thoughtful review and for recognizing the novelty of our framework, the robustness of our experiments, and TypedThinker’s generalization capabilities. Regarding the concern about limited model validation (W1), we have conducted additional experiments with Qwen 2-7B-Instruct, a model comparable to Mistral 7B and LLaMA3, and found consistent performance improvements. For the request to compare with existing methods and other baselines (W2 and W3), we provide a more detailed analysis. Lastly, addressing other reasoning datasets (W4), we evaluate other methods on a new benchmark Livebench with 3 different reasoning tasks. We hope these clarifications and updates address your key concerns.

W1: Accuracy improvement of 3.4% on Mistral 7B seems rather minimal, though improvements on LLaMA3 were quite notable. It would have been more beneficial to have more models to validate the findings on this paper convincingly.

A1: Thank you for your valuable feedback. In response, we conducted further experiments using Qwen 2-7B-Instruct as our backbone LLM. The Qwen series of open-source large language models have demonstrated comparable or even superior performance to the Mistral and LLaMA families across multiple tasks [1]. The results are summarized below:

As shown, our method achieves approximately 7% improvement over the few-shot baseline in both single-generation and majority-vote settings (+SC @5). These results demonstrate that TypedThinker is a general and effective method for enhancing the reasoning capabilities of various LLMs.

[1] Qwen Technical Report

---

W2: It would have been very effective to compare these existing methods with this framework. This could be possibly used to measure effectiveness for/against existing methodologies and would have made this work a lot more relevant and connected with existing work.

A2: Thank you for this insightful suggestion. In our revised manuscript, we will expand the discussion on relevant work (in Related Work). Existing studies often focus on using a single reasoning type to address specific problem categories, which limits their generalizability to broader reasoning tasks [2, 3, 4].

For instance, Hypothesis Search [2] is designed for tasks solvable by a single program, such as ARC and SyGuS. While this approach generalizes well for these tasks, it is infeasible for general reasoning challenges, like logical reasoning and math problems, which cannot be solved with a single universal program.

In contrast, our work emphasizes improving LLMs’ ability to identify and apply diverse reasoning types to solve a wider range of logical and mathematical problems. This contribution distinguishes our approach from existing methods.

[2] Hypothesis Search: Inductive Reasoning with Language Models  https://arxiv.org/pdf/2309.05660

[3] Case2Code: Learning Inductive Reasoning with Synthetic Data

[4] UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations

---

W3: Add baselines for simply prompting the model to simultaneously state and explore all these reasoning type for final prediction.

A3: Thank you for the suggestion. As described in Section 4.1, we have already included two baselines to evaluate the influence of reasoning types:

1. CoT Selection: A two-step approach where the LLM first selects the most appropriate reasoning type and then applies it to solve the problem.
2. Mixture-of-Reasoning (MoR): The LLM applies all five reasoning types (including an empty type) and uses majority voting to determine the final answer.

These baselines can be considered enhanced versions of “simultaneously stating and exploring all reasoning types.”

The results, summarized in Table 1 (also shown below), reveal that prompt-based approaches struggle to identify suitable reasoning types, while a simple mixture of all reasoning types does not significantly improve reasoning performance.

These comparisons highlight the advantages of our TypedThinker framework.