Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time

Question4: Is there a "sweet spot" for the capability gap?

Sure, we show the sweet spot is 1.5B+14B on MATH500.

• Thank you for the question. Due to the current availability of pretrained reasoning models, most open-source checkpoints are concentrated at 7B, 14B, and 32B scales. There are relatively few models with evenly spaced capability levels, especially below 7B. Therefore, in our experiments, we focus on evaluating speculative thinking using a 1.5B speculative model paired with 7B, 14B, and 32B target models. The results on the MATH500 dataset are presented below:

• We observe that the 1.5B + 14B combination achieves the best trade-off between speed and accuracy. This may be because the 14B and 32B models exhibit similar accuracy on MATH500, so the smaller 14B model provides nearly equal guidance at a lower computational cost. While 32B achieves slightly higher accuracy, its marginal gains are offset by a notable drop in inference efficiency.

Question5: Comparison with Speculative Decoding when n=5.

Here are some comparisons and we show the two advatages of speculative thinking. Although speculative decoding may offer marginally better accuracy in some cases, speculative thinking is more versatile and practical in scenarios where vocabulary mismatch or architecture heterogeneity prevents token-level alignment.

• We include the following results to compare speculative thinking with speculative decoding (SD) at different values of n. As shown, speculative decoding with n=5 achieves the highest accuracy on both MATH500 and AIME datasets:

• While speculative decoding can achieve slightly higher accuracy under certain configurations, speculative thinking offers broader applicability:
  • No Shared Vocabulary Requirement: Speculative decoding requires the speculative and target models to share a tokenizer and vocabulary. This constraint makes it infeasible for combining models such as DeepSeek-Qwen2.5-1.5B and DeepSeek-Qwen2.5-32B, whose vocab sizes differ.
  • No Token-Level Alignment: Speculative thinking operates at the reasoning level and does not require token-level output agreement between models, unlike speculative decoding. This flexibility allows for integration of models from different families, such as DeepSeek-LLaMA3-8B (speculative) and DeepSeek-Qwen2.5-32B (target).

Question1: Limited Target Model Diversity of Speculative Thinking.

Speculative thinking could use models in different families and we added some additional experiments to verify it.

• We apologize for the writing ambiguity in the original submission. In our current implementation, we primarily used Qwen-family models, which share the same tokenizer vocabulary. This means that the speculative model and the target model use consistent token IDs, though they maintain separate KV caches due to architectural differences such as hidden sizes.
• However, speculative thinking is fundamentally a reasoning-level collaboration rather than a token-level alignment. As such, there is no strict requirement for the speculative and target models to share the same tokenizer or vocabulary. To evaluate speculative thinking with models from different families, we conducted experiments with the following cross-family setups:
  • Deepseek-Qwen-1.5B + Phi-4-reasoning-plus (LLaMA3 speculative, Phi target)
  • Deepseek-LLaMA3-8B + Deepseek-Qwen-32B (LLaMA3 speculative, Qwen target)
• The results on MATH500 are shown below. These results clearly demonstrate that speculative thinking can be successfully applied even when the speculative and target models come from different architecture families.

Question2: Dependence on Specific Cues of "\n\n".

Sure, speculative thinking relies on such specific cues, but these cues—such as '\n\n'—are in fact commonly used across many state-of-the-art reasoning models, as shown by our additional experiments.

• To support this, we analyzed three representative open-source reasoning models—QwQ-32B (Qwen), Phi-4-reasoning-plus (Microsoft), and AceReason-Nemotron-14B (NVIDIA)—on the MATH500 benchmark. Specifically, we examined the most frequent tokens appearing directly before reasoning indicators like "wait". The results, shown below, demonstrate that all three models rely heavily on linebreak-based patterns (e.g., ".\n\n", " \n\n", "\n\n"), which serve as natural boundaries between reasoning steps.

• This suggests that the "\n\n" pattern is not a brittle or model-specific artifact, but rather a widely adopted structural convention for segmenting thoughts in multi-step reasoning. As such, speculative thinking’s reliance on this cue does not limit its generality—it reflects an emergent norm shared across diverse model families.

Question3: Complexity of Hyperparameter Tuning.

Yes, speculative thinking does need hyperparameter tuning.

• speculative thinking requires adjusting certain hyperparameters to align with the characteristics of different models or datasets in order to achieve optimal performance.
• here are our responses about questions of these parameters

Classification Keyword Selection (e.g., Affirmation, Reflection, Verification).

We demonstrate how these keywords are selected, and further show how the choice of keywords influences the effectiveness of speculative thinking.

• We selected these categories by first analyzing the next sentence after "\n\n" in model generations on reasoning datasets. We then tokenized and ranked the frequent leading words and manually grouped representative ones under the three categories.
• We acknowledge that the speculative performance is indeed sensitive to this keyword choice. A richer and more diverse keyword set improves category detection accuracy, whereas an overly narrow keyword list leads to degradation. While more advanced techniques (e.g., model-based classification) may help, we deliberately keep speculative thinking training-free and opt for simple lexical heuristics to preserve generality and efficiency.
• To quantify this, we conducted an ablation in which we restricted each category to just one keyword:
  • Affirmation: "confident"
  • Reflection: "wait"
  • Verification: "verify"

• This performance drop confirms that speculative thinking benefits from a broader set of semantic cues, and that careful—but simple—keyword selection remains important in the absence of training.

The Selection of Negativity Counter Threshold in Excessive Reflection Takeover.

We present how the negativity counter threshold is chosen and analyze its impact on the performance of speculative thinking.

• The negativity counter threshold c is set to 15 in our implementation. This value was manually estimated based on empirical observation and is kept fixed across all models and datasets in our experiments for simplicity and consistency.
• That said, we acknowledge that c is a tunable hyperparameter. Making it adaptive or model-specific may further improve performance. To evaluate its impact, we tested three different values of c—a low threshold (3), our default (15), and a high threshold (50)—and observed the corresponding effect on performance:

• These results suggest that the choice of c does affect the balance between early termination and generation coverage. Our fixed setting of 15 offers a good trade-off in practice, though adaptive variants remain a promising direction for future enhancement.

Application to Non-Reasoning Models without the Initial Generation.

The initial generation is critical when using a reasoning-capable model to assist a non-reasoning model in speculative thinking.

• Thank you for the insightful question. We agree that the interaction point between the non-reasoning target model and the reasoning model is crucial.
• We conducted ablation experiments to evaluate the effect of skipping the initial generation phase. The results show a substantial drop in both accuracy and reasoning quality:

• These findings suggest that the initial 100-token generation from the target model provides essential scaffolding for the reasoning model to effectively align and contribute.

speculative thinking relies on structural cues such as the "\n\n" pattern.

Yes, speculative thinking relies on such a pattern, but it is prevalent across most state-of-the-art reasoning models, as shown by our following additional experiments.

We analyzed three representative open-source reasoning models—QwQ-32B (Qwen), Phi-4-reasoning-plus (Microsoft), and AceReason-Nemotron-14B (NVIDIA)—on the MATH500 benchmark to investigate how they structure their reasoning. Specifically, we examined the most frequent tokens that precede reasoning indicators like "wait".

As shown in the table below, all three models commonly use "\n\n" and related formatting as structural cues, suggesting that speculative thinking can leverage this shared behavior even across model families.

• To further validate its generalizability, we applied speculative thinking to two heterogeneous model pairings:
  • Deepseek-Qwen-1.5B + Phi-4-reasoning-plus (using LLaMA3-family model to predict, Phi as the target)
  • Deepseek-LLaMA3-8B + Deepseek-Qwen-32B (LLaMA3 speculative, Qwen target)
• The table below reports the performance on MATH500. These results show that speculative thinking is both structurally and empirically effective across different model architectures. The consistent use of structural reasoning patterns like "\n\n" enables successful coordination even between models from separate families.

• We do acknowledge that speculative thinking depends on the presence of such structural signals. In rare cases where neither the speculative model nor the target model exhibits any recognizable reasoning format, our method may not be applicable. Nonetheless, given that most modern reasoning models—regardless of origin—naturally follow these patterns, speculative thinking remains broadly applicable in real-world deployment.

Question 1: envision a scenario of speculative thinking.

Sure, we appreciate the concern. Speculative thinking is designed for scenarios where efficiency, cost, and scalability are also critical, and some trade-off in accuracy is acceptable.

We highlight below several practical contexts where such a setup is not only reasonable but highly desirable: (1) Limited computational resources. (2) Low latency or high throughput requirements. (3) Cost-efficiency. (4) Accuracy demands are high, but not absolute.

Example use cases include:

• Educational QA platforms serving thousands of students simultaneously, where small models handle simple questions and speculative calls to the 32B model handle difficult cases.
• Enterprise productivity assistants or internal copilots that prioritize speed and cost-efficiency while preserving reasonable answer quality.
• Edge-cloud hybrid deployments, where the 1.5B model runs on-device, and the 32B model (hosted remotely) is only queried when necessary—minimizing cloud compute costs and network latency.

Question2 : why not directly use a 14B model?

Using a single 14B model does not fully address the efficiency and deployment constraints that speculative thinking resolves.

• inference speed with speculative thinking (1.5B + 32B) consistently surpasses that of the 14B model, as shown in the table below. This speedup arises because the majority of tokens are generated by the lightweight 1.5B model, with the 32B model queried only intermittently for guidance:

• in edge computing scenarios, deploying a 14B model locally is often infeasible due to memory and resource constraints. In contrast, a 1.5B model can run efficiently on-device, with occasional access to the 32B model hosted in the cloud. This design not only fits within hardware limits but also reduces cloud usage costs, as each call to the large model is minimized.

Question3 : Is Speculative Thinking Pareto-optimal?

We thank the reviewer for raising this important point. We have conducted a detailed Pareto frontier analysis on the trade-off between accuracy and speed, as shown in the figure (click to view): MATH500 Accuracy vs Speed . The result shows our method (e.g., 1.5B+14B and 1.5B+32B) lies on the Pareto frontier—our method achieves substantial speed gains with minimal accuracy degradation, making it a Pareto-optimal solution in the accuracy–efficiency trade-off space..

Specifically, while the 14B model achieves the highest accuracy (93.8%), our 1.5B+14B approach maintains a competitive accuracy (89.0%) with more than double the inference speed (134.64 vs. 60.1 tokens/sec). Similarly, compared to the 32B model (92.8%, 32.2 tokens/sec), our 1.5B+32B method offers a 3x speedup (96.61 tokens/sec) with only minor accuracy degradation (89.4%).

Question 4: Dose speculative thinking generalize to different families of models?

Yes. Speculative thinking can generalize to different families of models, as demonstrated by two additional experiments described below.

First we show that this pattern is not specific to a single model family, but is in fact widely adopted across modern reasoning models.

Then, We further demonstrate that speculative thinking generalizes across different model families by combining small and large models from heterogeneous architectures.

• We examined the behavior of three strong open-source reasoning models—QwQ-32B (Qwen), Phi-4-reasoning-plus (Microsoft), and AceReason-Nemotron-14B (NVIDIA)—on the MATH500 benchmark, and analyzed the most common tokens preceding reasoning cues such as "wait".
• The table below summarizes the top-10 prefixes for the "wait" token in each model. Across all three models, structural tokens like ".\n\n" and "\n\n" consistently appear among the top prefixes, suggesting that multi-line breaks are commonly used to separate reasoning steps.

• Specifically, we tested two cross-family settings:
  • Deepseek-Qwen-1.5B + Phi-4-reasoning-plus (Qwen speculative, Phi target)
  • Deepseek-LLaMA3-8B + Deepseek-Qwen-32B (LLaMA3 speculative, Qwen target)

The results on MATH500 are summarized below. It shows that speculative thinking remains effective even when the speculative and target models come from different families, consistently improving both accuracy and speed over the small model, while remaining significantly faster than the large model alone.

• Even if a reasoning model does not explicitly generate the "\n\n" + "wait" pattern, speculative thinking can still be applied as long as the model emits key reasoning tokens (e.g., "wait", "alternatively") after other identifiable structural clues. These clues serve as natural segmentation points and can be leveraged to synchronize speculative and target models.
• In cases where the speculative model lacks any reasoning structure, it is still possible to apply speculative thinking if the target model exhibits such reasoning patterns. As shown in Table 3, a reasoning-capable target model can enhance the performance of a non-reasoning speculative model through its structural guidance.
• However, if neither the speculative model nor the target model demonstrates any consistent reasoning pattern or structural segmentation behavior, then speculative thinking would no longer be applicable in its current form.

We sincerely thank you for your thoughtful and constructive feedback. We have carefully addressed all the points you raised, and we hope that our responses have clarified the concerns. We would greatly appreciate it if you could kindly take a moment to review our reply. If any questions remain, we would be happy to further elaborate.

Thank you for your follow-up and for taking the time to review our work. We appreciate your comments and feedback.