Thinking LLMs: General Instruction Following with Thought Generation

We thank the reviewer for the constructive feedbacks. Below, we address your concerns and propose our revisions.

---

Failure case analysis is limited. The paper does not discuss scenarios where TPO might fail, such as backtracking-heavy tasks or reasoning tasks requiring multiple revisions. Automated evaluation reliance—the paper depends on GPT-4-based judges, which may introduce evaluation biases that are not addressed.

Our results on GSM8K, where both TPO and standard DPO degraded performance, illustrate a limitation. We attribute this primarily to the reward model's limitations in accurately judging complex, multi-step reasoning, highlighting that reward model quality is critical, especially for such tasks. We also acknowledge the potential for inherent biases in automated evaluations using LLM-based judges like GPT-4. To mitigate this, we implemented controls where feasible; for instance, output length was controlled across compared methods to prevent models from exploiting potential length bias in the automated evaluator.

Essential References Not Discussed

We agree these references are relevant and appreciate the reviewer pointing them out. We'll discuss these papers in our final draft.

The paper does not explore failure cases in depth. It would be helpful to provide more insights into when and why TPO fails, particularly in tasks that require iterative backtracking or fine-grained numerical precision. The evaluation relies entirely on GPT-4-based reward models, which introduces potential biases. A complementary human evaluation would help confirm the effectiveness of TPO in real-world scenarios.

We agree that a deeper analysis of failure cases would be valuable. Although the patterns TPO exhibits in general are generally helpful, such as drafting a reminder list for the answer (Figure 6 & 15), refining the response (Figure 5 & 16), and reflections, we also observed cases where TPO failed. Notably, in Figure 18, we found the model exhibits non-terminating behavior by iteratively criticizing its own answer without being able to produce a fix. This indicates the model might have a tendency to deviate from the thought-answer structure, despite being specifically trained to follow the structure.

The work primarily focuses on GPT-based models. It remains unclear how well TPO generalizes to open-source models such as LLaMA, Mistral, or Claude. While the method is technically sound, there is no theoretical analysis of whether TPO converges to an optimal thought process or if certain errors persist over time.

Regarding the request for theoretical analysis, we are not aware of any theoretical analysis on the optimality of TPO. But our intuition why TPO worked is it provides more freedom for the LLM to explore during the RL process by putting no constraint on the thought part (as the judge does not know what is inside the thought), though some errors might persist, e.g. not following format, non-stopping behaviors, etc.

Does training with TPO introduce significant additional overhead compared to standard instruction tuning?

TPO training doesn’t add additional complexity in implementation compared with DPO, but potentially generates slightly more tokens than standard DPO due to the thought part.

The paper could benefit from more real-world application demonstrations, such as deploying TPO-trained models in interactive agents or dialogue systems.

These are interesting ideas to explore in our future work.

How does TPO handle cases where the initial thought generation is incorrect? Does the model attempt self-correction, or does it remain committed to the initial flawed reasoning path?

Figure 18 provides an example where the model recognized an error but failed to recover or correct it.

What are the primary failure modes of TPO? Are there specific task types where the thought-generation process degrades performance instead of improving it?

One primary failure mode stems from the reward model's limitations. If the reward model struggles to accurately judge the final answers for certain types of tasks (like complex reasoning), it can provide poor guidance, causing TPO to optimize in the wrong direction and potentially degrade performance more. Another observed failure, illustrated in Figure 18, involves process instability where the model might enter non-stopping loops, such as repeated self-criticism without resolution, or deviate from the intended thought-answer structure.

Can TPO be integrated with retrieval-augmented generation (RAG) systems? Would the model's internal thought process benefit from external knowledge retrieval, and how would that affect optimization?

Thank you for the insightful comment; unfortunately, we are unable to provide a direct answer at this time, but it is an interesting topic worth exploring in the future.

---

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work.

We sincerely thank the reviewer for the valuable and insightful feedback. We address your concerns as follows:

---

How would simply doing this SFT compare with using TPO?

That's an insightful question regarding potential training alternatives. The Llama 70B model in our study functions strictly as a reward model, similar to the other RM evaluated (ArmoRM), providing reward signals rather than generating target outputs for supervision. Our TPO framework utilizes these reward signals and is agnostic to the specific nature of the reward model (e.g., a traditional RM or an LLM-as-a-Judge). This contrasts fundamentally with the proposed alternatives, which would use the 70B model as a generator to create a dataset for supervised fine-tuning (SFT). Because TPO operates via preference optimization driven by reward signals, while the suggested approaches rely on SFT using generated targets, a direct comparison might not be appropriate.

How would TPO compare with parallel sampling techniques? I.e. sample multiple answers and use the judge model to select the final answer.

We appreciate the reviewer raising the comparison with inference-time techniques. TPO as a training-time optimization method, is orthogonal to inference-time techniques such as sampling multiple responses and selecting the best using a reward model. These approaches are not mutually exclusive; applying inference-time sampling and selection strategies to a model already optimized with TPO is a plausible approach. Investigating the interplay and relative benefits of TPO training versus various inference-time selection methods remains an interesting direction for future work.

authors mention several prior works on thinking tokens that also do not require any labeled data (STaR, Quiet-STaR), is there a reason why authors didn’t compare empirically with those two baselines?

Thank you for asking about the comparison with STaR and Quiet-STaR. STaR requires ground truth answers, and Quiet-STaR relies on supervised fine-tuning data. In contrast, TPO requires only unlabeled prompts and preference signals from a reward model. These fundamental differences in data make a direct empirical comparison challenging. Furthermore, STaR and its variants were primarily evaluated on reasoning-intensive tasks, whereas our work focuses on general instruction-following capabilities.

Can we also see a variant of Figure 4 with win rate against the seed model? As authors have already mentioned, COT only helps with math/logic related tasks, so seed model should be a stronger baseline here for most of the categories shown.

We understand the reviewer's interest in seeing a direct comparison against the base seed model. The 'seed model' refers to the base pre-trained model prior to any preference optimization. The 'direct baseline' actually presented in Figure 4 is this seed model after it has undergone standard Direct Preference Optimization (DPO) training directly on final answers. This DPO-optimized direct baseline is generally stronger than the original un-optimized seed model, making it the more relevant and challenging point of comparison for evaluating TPO.

Authors show TPO do worse than baselines in GSM8K, any ideas why it does better than the baseline in the “math and calculation” subset of ultrafeedback?

This is a keen observation regarding the differing performance on math-related datasets. These two datasets exhibit significant distributional differences in the types of mathematical problems presented. We hypothesize it’s because the reward model we used is better at judging questions in the ultrafeedback distribution but bad on the GSM8k distribution.

Why is it problematic that “the seed model … uses CoT anyway due to its instruct training”? If it is because the accuracy is too high for further improvement, can authors also evaluate on MATH, which is a harder task than gsm8k?

We understand the reviewer's query regarding the implications of the seed model's inherent CoT capabilities. The goal of that experiment was to understand the effect of thinking and CoT on gsm8k performance. In particular, we tried to measure the accuracy of the seed model when directly outputting answers without any CoT. However this was tricky to measure because the model often did CoT before answering, sometimes even when explicitly instructed to not to do CoT

Authors mentioned that the poor performance on GSM8K might be “due to our setup not being oriented toward such tasks”. Have the authors tried to simply add the training examples in GSM8K as a part of the training data for TPO?

That's a relevant suggestion concerning the training data composition for specific benchmarks. We did not include examples from the GSM8K training set within the dataset used for TPO training in this study. This is an interesting direction which we’ll leave for future work.

---

We deeply appreciate the reviewer’s feedback and hope our responses fully address your concerns.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each concern raised and propose revisions:

---

Are the thoughts interpretable and aligned with the optimal or correct human-like reasoning steps?

While our methodology does not impose explicit constraints on the structure or content of the thoughts during reinforcement learning, our analysis revealed the emergence of patterns that resemble effective human problem-solving strategies. These include:

• Generating Checklists/Key Points: As seen in Figures 6 and 15, the model often formulates preliminary lists of elements to include or emphasize in the final answer.
• Iterative Refinement: Figures 5 and 16 show the model refining potential answers within the thought process, adding arguments or examples.
• Self-Correction and Evaluation: Figure 18 illustrates the model engaging in self-correction, sometimes even overriding pre-specified formats.

It's better to include a qualitative analysis of the generated thoughts to assess their interpretability, alignment with human reasoning...

We agree that a dedicated qualitative analysis assessing thought interpretability is a valuable suggestion. However, rigorously and explicitly evaluating the intrinsic quality of machine-generated thoughts presents significant challenges. First, benchmarks specifically designed for assessing thought processes are not well-established. Secondly, defining objective criteria for a "good" thought is inherently complex. For instance, is a thought process containing flawless step-by-step logic but ultimately unhelpful superior to one that suggests a promising overall direction despite containing intermediate errors? Developing methods for more direct evaluation of thought quality itself remains an important direction for future research.

It builds heavily on existing techniques like Direct Preference Optimization (DPO) and Reinforcement Learning from AI Feedback (RLAIF)...

We wish to clarify that the core innovation of TPO is not presented as a fundamentally new type of reinforcement learning or preference optimization algorithm itself. The primary objective of TPO is to demonstrate that substantial performance gains are achievable by affording the model this unjudged "thinking space," without requiring significant alterations to the underlying training algorithm architecture.

We provides empirical evidence supporting this claim through direct comparisons with standard DPO and a CoT approach (representing the iteration 0 before TPO). As presented in our results, optimizing the hidden thought process via TPO yields significant performance improvements over these methods.

...specific thought prompts perform slightly better than generic prompts, but the difference is marginal...

Specific thought prompt introduce negligible implementation complexity because the TPO training methodology remains identical irrespective of the prompt format. While average performance gains across all tasks may appear modest, specific prompts yield substantial improvements on challenging benchmarks, notably ArenaHard.

It is better to also evaluate the computational cost...

We’ll add discussion and plots of the generation cost vs the final performance in our final draft.

investigate why TPO underperforms on math tasks...

We hypothesize this may stem from the deficiency of the reward model we used, which focused on general instruction following rather than specialized mathematical reasoning. For domains requiring strict correctness like mathematics, directly evaluating answer accuracy against reference solutions, might be more effective than relying solely on the preference judge. Incorporating such correctness checks represents a promising direction for future work.

the core idea builds heavily on existing techniques like DPO and RLAIF...

TPO's contribution lies not in proposing a new optimization algorithm variant, but in demonstrating the methodology of optimizing an free-form thought process based only on judgments of the final answer. This indirect supervision approach facilitates the emergence of complex reasoning behaviors during training, which are not seen in standard answer only training.

Are there examples where the thoughts are nonsensical or misleading, despite leading to good responses

We did not observe cases where nonsensical thoughts produced high-quality final answers; we hypothesize the KL regularization used during training helps maintain coherent and interpretable thought structures by penalizing significant deviations from the base model's.

It's better to evaluate the computational cost of generating and optimizing thoughts relative to the improvements in response quality.

We’ll add analysis in the final draft.

---

We greatly appreciate the reviewer’s constructive feedback, which improves the clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions and clarifications.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:

---

Lack of model support. All the experiments for the model generation are done on 1 model: Llama 3.1 8B Instruct...

We acknowledge the reviewer's concern regarding model diversity. While we aim to conduct further evaluations involving larger models such as the latest Llama 3.3 70B and Deepseek R1, our current model selection is primarily constrained by limited training resources. Subject to the availability of time and resources, we intend to expand our experimental scope to include a wider range of models. However, this expansion may be relegated to future work.

Lack of understanding or insight into why thoughts help... For example, what kinds of patterns or additional context emerge from the thoughts that contribute to higher quality responses?

We agree that understanding why thoughts contribute to better responses is crucial. Our analysis identified several patterns within the thought processes that correlate with higher-quality final responses:

• Generating Checklists/Key Points: As illustrated in Figures 6 and 15, the model often drafts a preliminary list of essential elements or topics it determines should be included or emphasized in the final answer, effectively creating a structured plan.
• Iterative Refinement: Figures 5 and 16 demonstrate instances where the model refines its potential answer. This includes augmenting initial drafts with stronger arguments, incorporating more specific examples, or restructuring the content for clarity.
• Self-Correction and Evaluation: Figure 18 provides an example of the model engaging in self-correction and evaluation. Notably, this reflective behavior can emerge even when it leads to deviations from pre-specified output formats.

These observed behaviors are consistent with established strategies known to enhance the quality and reliability of responses generated by large language models.

Do different patterns emerge for different topics? How robust is this emergence? ...

We appreciate the reviewer inquiring about the robustness and specificity of these patterns. We didn’t statistically evaluate and classify these thought patterns because it requires eye-balling and does not have a way to do it at scale. However, we want to emphasize that some of the behaviors exhibited by our thought model are nearly never seen by a direct answer model, for example like the self reflection, refinement and self reminder behaviors, which we believe are the main reasons why our model can perform better.

I appreciated the study on length-control and DRO vs IRPO, but some parts of the method came across a bit ad-hoc...

Regarding length control, our approach incorporates established mechanisms documented in prior work. Extensive research has demonstrated that without effective length constraints, model performance can be significantly impacted, often negatively. We adopted these standard techniques primarily to ensure fair and meaningful comparisons between different methods evaluated in our study, as consistent length controls were applied across all conditions. This also enhances the practical relevance of our findings by preventing models from artificially inflating reward scores through excessive verbosity.

if this is a result strictly from preference optimization (TPO), or could we get Thinking LLMs via purely an outcome-based signal ...

Regarding the outcome-based signal, it aligns with methodologies explored in recent literature, such as DeepSeek-V1. Our proposed framework, which explicitly separates the generation process into distinct 'thought' and 'answer' components, is indeed adaptable to such outcome-based evaluation paradigms. By evaluating only the final 'answer' part, the framework grants the model greater flexibility in the 'thought' generation phase, encouraging more exploration during reinforcement learning, potentially leading to more diverse reasoning strategies and ultimately improving final task performance. Furthermore, alternative optimization algorithms like KTO or RLVR could readily replace DPO within our framework; our initial selection of DPO was based on its established effectiveness and relative simplicity of implementation.

The method comparison is also a bit lacking, where I think the authors should at least compare against STaR.

STaR requires ground truth answers, in contrast, TPO requires only a reward model. These fundamental differences in data make a direct empirical comparison challenging. Furthermore, STaR and its variants were primarily evaluated on reasoning-intensive tasks, whereas our work focuses on general instruction-following capabilities.

---

We greatly appreciate the reviewer’s constructive feedback, which enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions and clarifications.