Style over Substance: Distilled Language Models Reason Via Stylistic Replication

Thank you for your detailed feedback!

How do you justify that style replication is the key contributing factor, given that the synthetic traces (ST-HC) still underperform compared to the emergent traces (ST)?

The reviewer is correct in pointing out that the results are not exactly the same between ST and ST-HC.

If we were to think of the ST reasoning traces as an ideal distribution, then the replication of ST in ST-HC is inherently an approximation. This approximation is not perfect, as can be seen in the lower performance of ST-HC. Still, while the synthetic traces do indeed consistently underperform the emergent ones, the gap between them is quite narrow, indicating that it is a good approximation of emergent reasoning.

To lend additional credibility to our argument, we add another baseline to our paper, with the goal of showing the impact of the particular style replication we propose versus distilling from regular CoT. To do this, we instruct the same LLM used for our synthetic data generation to think step-by-step (SBS), following [1], without utilizing anything proposed by us (e.g., pivots). We then use the resulting CoT for distillation the same way we use the emergent (ST) and synthetic (ST-HC) reasoning traces.

Here is the updated results table with SBS for Llama 3.2, Ministral, and Qwen2.5:

SBS, while beneficial compared to the base model, noticeably underperforms ST/ST-HC by a large margin. This demonstrates that the ‘structure’ adopted for ST-HC is key in achieving improved performance. We hope this addresses the reviewer’s concerns and will update our paper accordingly.

Additionally, have you experimented with using other teacher models besides GPT-4o to generate synthetic reasoning traces?

For teacher models with instruction-following capabilities similar to GPT-4o, we did not find the choice of model to have an impact on performance.

We again thank the reviewer for taking the time to thoroughly review our paper.

References

[1] Kojima, T., et al. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 22199-22213.

With the end of the discussion period approaching, we wanted to reach out and ask the reviewer if they believe our response and new experiments sufficiently address their concerns? Please let us know if there is anything else we may clarify.

We thank the reviewer for their positive comments, as well as their acknowledgment that our paper deepens the community's understanding of LLM reasoning.

What I'm missing is comparing with distillation on GPT's CoT data, generated by "Let's think step-by-step" prompt. How close will it be to ST-HC results?

We agree that this additional baseline could be useful information for readers and would better substantiate the advantages of our approach. We have since conducted additional experiments on this. Here we instruct the LLM to think step-by-step (SBS), following [1]. We then use the resulting basic CoT for distillation — similarly to how we use the emergent (ST) and synthetic (ST-HC) reasoning traces.

Here is the updated results table with SBS for Llama 3.2, Ministral, and Qwen2.5:

From these results, it is clear that while generating synthetic data with a standard SBS prompt is beneficial compared to the base model, there is a significant performance gap between SBS and our ST/ST-HC methods. This demonstrates that the performance improvements are indeed substantially driven by the specific stylistic patterns we identified, rather than solely by distilling GPT's general reasoning abilities. We hope this addresses the reviewer’s concerns and will update our paper accordingly.

We again thank the reviewer for taking the time to thoroughly review our paper.

References

[1] Kojima, T., et al. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 22199-22213.

Thank you for your insightful and positive review.

Can you truly separate the style of verification from the act of verification?

This is a great question without an easy answer. When dealing with something as complex as language, it is very difficult to definitively say that specific words or phrases are entirely related to style or substance, for example. This mirrors long-standing debates in linguistics from syntax versus semantics [1] to form versus meaning [2]. We believe it is beneficial to draw such a distinction in order to clarify how LLMs reason – which is what our work aims to do. We’d be more than happy to continue this discussion at the conference.

If models improve significantly even with ST-HC-W, what exactly are they learning?

Our ST-HC-W experiment (Sec 4.2) suggests a deeper learning mechanism. Models trained on ST/ST-HC/ST-HC-W internalize a structure (or “meta-cognitive scaffold”) from the preserved stylistic and structural reasoning patterns – even with incorrect final training answers – as the trace body often contains valid intermediate reasoning. We provide a deeper exploration of this in Appendix B and in our new experiments (see below).

How much of the "style" benefit comes from the explicit prompting versus the inherent (and significant) reasoning capabilities of GPT-4o itself?

To gauge how much of an impact GPT-4o’s reasoning has, we conducted an additional experiment, where we instruct the LLM (GPT-4o) to think step-by-step (SBS), following [3]. We then use the resulting basic CoT for distillation – similarly to how we use the emergent (ST) and synthetic (ST-HC) reasoning traces.

Here is the updated results table with SBS for Llama 3.2, Ministral, and Qwen2.5:

As we can see, using GPT-4o as a teacher to generate SBS CoT traces provides a benefit over the base student model. There is a further, notable gap in performance between ST/ST-HC traces compared to SBS, confirming that the performance improvements are indeed substantially driven by the specific stylistic patterns we identified. We will update our paper accordingly.

Real traces (ST) come from R1, a model already optimized for reasoning.

We agree that existing reasoning models have a distinct style that emerges during reinforcement learning. As this has been found to be effective, we aim to emulate it (ST-HC).

How well would these "stylistically trained" models perform on problems requiring genuinely novel insights or reasoning patterns not captured by the identified styles? Is there a risk of overfitting to a particular way of expressing reasoning, which might be brittle?

Regarding generalization and potential brittleness, our work argues that the distilled 'style' is not arbitrary but rather embodies effective and generalizable reasoning processes. The cognitive stages we identify (problem framing, exploration, verification, synthesis) and their associated lexical pivots are fundamental to successful problem-solving, as evidenced by their emergence in high-performing RLMs and their grounding in cognitive science. Our ST-HC-W experiment, showing improvement even with incorrect final answers, strongly supports that the learned structure itself is beneficial and transferable.

We again thank the reviewer for taking the time to thoroughly review our paper.

References

[1] Chomsky, N. (1955). Logical syntax and semantics; their linguistic relevance. Language, 31, 36–45.

[2] Bender, E.M. and Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In ACL, pp. 5185-5198.

[3] Kojima, T., et al. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 22199-22213.

We thank the reviewer for their encouraging feedback. We are glad to hear that they believe the paper is inspiring.

More qualitative study (e.g., case study) could be helpful to further support the findings of this work. Besides, it would be great if the authors could provide more insights about the lexical pivots.

While we make an initial effort to give an overview of the pivots found in emergent traces (Appendix B), we agree that additional insights into pivots could be useful to readers. We will add additional qualitative examples to the appendix, breaking down pivots by type.

We again thank the reviewer for taking the time to thoroughly review our paper.