Hawkeye: Model Collaboration for Efficient Reasoning

We sincerely thank you for your review and constructive feedback! Below, we provide detailed responses to each of your comments.

1. Cost of two models' forward passes: We sincerely thank the reviewer for raising this concern. Coordinating two models indeed introduces additional overhead, which mainly depends on the deployment scenario—particularly the communication cost and the small model's prefilling overhead.

In Table 1, the reported latency under concurrent requests already includes the coordination cost of both models. Our experiments were conducted on the same machine within a single cluster, so the communication cost is negligible. The primary overhead comes from the small model’s prefill.

When handling a single request, the cost of prefill and decoding for the small model is significantly lower than that of decoding with the large model:

$T_{\text{SLM, prefill}} + T_{\text{SLM, decode}} \ll T_{\text{LLM, decode}}$

For example, in our setting with SLM = 0.5B and LLM = 7B, the ratio
(T_SLM_prefill + T_SLM_decode) / T_LLM_decode
is roughly 0.2 to 0.3. As the size of the LLM increases, the efficiency gains become even more pronounced. We will add this explanation to the final version of the paper.

2. More benchmarks: We indeed focused solely on mathematical reasoning in this work, as most state-of-the-art reasoning models currently concentrate on the math and code generation domains. These domains involve complex reasoning and provide standard answers, making them ideal for evaluating model reasoning capabilities.

Our choice aligns with other works on efficient CoT modeling, which has similarly focused on math tasks. Nonetheless, we plan to extend our evaluation to include more general domains in future versions to examine whether Hawkeye-trained models generalize beyond math.

We hope our responses have addressed your concerns! :)

We sincerely thank you for your review and constructive feedback. Below, we provide detailed responses to each of your comments.

1. Effect of length penalty factors and dataset size: We evaluated different length penalty factors during training by varying the target compression rate and the results indicated a compression rate around 0.3 is a reasonable choice. Please refer to the visualization at: https://imgur.com/a/BoMxyca. When the compression rate is set below 0.3, GRPO tends to overly prioritize the reward associated with shorter CoTs, which in turn causes the model to neglect whether the small model can still produce the correct final answer. Nonetheless, we observed a consistent trend across different datasets and training scales, including both MATH and GSM8K. This suggests that the compression behavior is relatively robust to dataset size, at least in terms of guiding the model to produce more concise CoTs. We also talked about this in Reward computation point in kmZC's response.

2. Limited baselines: (You can refer to Limited baslines" point we replied to rwTc's comments) We consider both CoT-Valve and O1-Pruner as concurrent works that have not undergone peer review. For this reason, we did not include them as baselines in the current paper. However, we will incorporate a detailed comparison with both methods in the next version. It is worth noting that both CoT-Valve and O1-Pruner are offline methods, whereas Hawkeye is an online approach that is lightweight and deployment-friendly.

CoT-Valve requires a complex pipeline, including pairing long-form “standard CoT” chains with shorter variants, combined with LoRA or distillation to learn a compressed representation. This adds considerable complexity to the training process.

O1-Pruner is based on PPO and is highly sensitive to hyperparameters such as the λ value, the number of samples 𝐾, and the PPO clipping range. It also relies on tens of thousands of training examples, making the process costly and less scalable.

In contrast, Hawkeye uses GRPO-based online learning, requiring only 2K samples, a simple reward function, and smaller models. Despite the simplicity, it achieves better token savings and comparable or better compression ratios.

For completeness, we provide a side-by-side comparison using the results reported in their drafts. While models and setups differ, we focus on compression rates and accuracy for fairness. We believe Hawkeye would outperform both methods under equal model scales.

CoT-Valve VS Hawkeye:

O1-Pruner VS Hawkeye

Q1: We indeed focused solely on mathematical reasoning in this work, as most state-of-the-art reasoning models currently concentrate on the math and code generation domains. We plan to include experiments in more general domains in future work to investigate whether a method trained on math also generalizes effectively to other tasks.

Q2: We will make necessary formatting adjustments to improve it. We list the missed sentence of the 3rd point from the contribution part here for your reference: “We curate a high-quality CoT dataset and fine-tune a reasoning model using reinforcement learning to optimize CoT generation. This approach reduces CoT length by over 75% compared to the original large model while maintaining performance, with only around a 4% drop in accuracy on the evaluated dataset”.

We hope our responses have addressed your concerns! :)

Q3:

In theory, the optimal loss would be achieved if the model could directly generate the correct answer as its CoT output (i.e., c=answer). However, in practice, this leads to unstable and brittle optimization, especially under GRPO training with a length penalty. In GRPO, the model samples multiple CoTs and selects the one with the highest advantage. Occasionally, a sampled CoT may skip the reasoning process entirely and still arrive at the correct answer. Due to its short length and correctness, this CoT receives a high reward and is used to update the model. However, this creates a dangerous optimization path: the model quickly learns to skip reasoning altogether to minimize token usage and maximize reward. Even though our GRPO is on a simple dataset. This could be a problem because for the majority of questions, reasoning is essential to arrive at the correct answer. Once the model starts omitting the reasoning step, it tends to reinforce that behavior across all subsequent generations. As a result, the model degenerates into producing short, ungrounded answers that frequently fail, which is the result of reward hacking.

Our solution is to decouple reasoning and response generation, and compute the reward based on whether the small model (SLM), given the CoT, produces the correct answer. In this setup, if a CoT skips reasoning, the SLM is unlikely to recover the answer, and the model receives no reward. This discourages the model from exploiting lucky shortcuts and instead incentivizes it to include sufficient reasoning steps to support reliable downstream responses.

Q4:

Yes, you are correct.

Q5:

As we mentioned in a previous response, we thank for the reviewers to find out the typo and we will fix it in our next version. We list the missed sentence of the 3rd point from the contribution part here for your reference: “We curate a high-quality CoT dataset and fine-tune a reasoning model using reinforcement learning to optimize CoT generation. This approach reduces CoT length by over 75% compared to the original large model while maintaining performance, with only around a 4% drop in accuracy on the evaluated dataset”.

We hope our responses have addressed your concerns! :)

We sincerely appreciate your thoughtful review and constructive feedback! Below, we provide detailed responses to each of your comments.

1. Further explaination about token budget: We agree with your observation. The example shown in Appendix A is indeed obtained by removing parts of the original CoT. The model would not necessarily generate this compressed CoT using the same token budget. Directly asking the model to produce a CoT with a token limit can reasily result in incorrect reasoning. However, our experiments results presented in Fig. 3 are not intended to imply that the model can naturally generate such a short CoT from scratch. We are intent to illustrate overthinking does exist and removing CoT redundancy in reasoning is empirically effective. In our reinforecement learning pipeline, shortening of CoTs is not performed in a single step, it is gradually—guided by the GRPO. The GRPO samples 64 candidate CoTs and selects the one with the highest advantage (i.e., a shorter CoT that still leads the small model to produce the correct answer) to update the model. Through this process, the model can gradually learn how to generate shorter CoTs from longer ones. It reduces the probability of producing incorrect CoTs when operating under limited token budgets, as it is not forced to immediately generate compressed CoTs without prior reasoning experience.

2. Initial Hawkeye thinking and 2nd LLM serves: It is unpredictable whether the Hawkeye thinking could produce the correct answers or not. While Hawkeye thinking can potentially produce the final answers, it does not always do so explicitly. In some cases, the answer is embedded within the reasoning process. Our observations show that some Hawkeye thinking traces do not yield the correct answer directly, and responses without the SLM tend to be of lower quality. In such scenarios, the SLM serves an important role by retrospectively interpreting the thinking process to extract the correct answer and convert it into a more readable and user-friendly response. Furthermore, if the reasoning process terminates prematurely, is incomplete, or fails to arrive at a correct conclusion, the SLM can perform additional inference to recover or generate an appropriate answer (provide an extra turn to reasoning more). This flexible collaboration mechanism between two LLMs offers several practical advantages. For example, you can easily control the model's output by just controlling the SLM, not the large model. Fine-tuning the smaller model is easiser and flexible. If the final response needs to undergo safety screening, incorporate RLHF for improved readability, or follow specific stylistic or formatting constraints, these requirements can be fulfilled by fine-tuning only the 0.5B small model—without altering the reasoning capabilities of the full system.

In summary, Hawkeye thinking plays the critical roles in heavy reasoning, whle Hawkeye SLM enables high flexibility and modularity, as the reasoning component remains consistent while the response generation can be easily customized. We are preparing an updated version of Fig. 6 that compares outputs with and without the SLM, highlighting its impact on improving response quality. It will be available soon and stay tuned!

Q1:

The workflow should be the opposite direction: the larger model is responsible for generating the CoTs, while the smaller model expands and rewrites the final response based on that.

Q2:

As we mentioned in our previous responses, the CoT should not be the final content delivered to users during inference, even though it contains answers sometimes, it still hard for users to quickly understand (not to mention sometimes it dosen't). Some existing LLMs even encode the CoT into formats that are not human-interpretable in order to make reasoning more efficient.

Te purpose of the SLM can be summarized:

• During training, it serves to prevent reward hacking by ensuring that the model optimizes for useful intermediate reasoning rather than simply maximizing a reward signal.

• During inference, it converts the CoT into a more natural, human-friendly response.

The expansion is thus not meant to generate a different answer, but to play a distinct role in shaping how the final output is presented. For additional discussion on the small model's utility, please refer to our detailed reply to Comment 2.

We sincerely thank you for your review and constructive feedback. We have carefully considered each of your comments and provide our detailed responses for each of them below:

1. Smaller model is better?: It's ture that smaller models cost less, but their performance concerns still exist especially when directly train them via SFT or RL for reasoning. First, even if after distillation, the base model’s capacity ultimately determines the performance ceiling. For example, DeepSeek-R1-Distill-Qwen-1.5B is largely behind (~20% on AIME) DeepSeek-R1-Distill-Qwen-7B (https://huggingface.co/deepseek-ai/DeepSeek-R1). Second, in our evaluation, although Hawkeye uses a smaller model to generate the final response, it can still achieve comparable performance to a larger model (Fig. 5 and Fig. 6). We belive it provides evidence that our pipeline outperforms direct SFT or RL on smaller models. Additionally, directly fine-tuning small models introduces further challenges, such as a limited context window, which becomes a significant bottleneck when producing extended chains of thought. For relatively simple and common tasks (e.g. GSM8K, MMLU), small models are indeed sufficient to meet most practical needs. The limitations raise significantly when tackling more complex tasks.

We also acknowledge that the reviewer’s observation is well-founded and aligns with this perspective.

2. clear declaration: You are correct, the larger model we used is DeepSeek-R1-Distilled-Qwen-7B-Hawkeye, the smaller model is Qwen2.5-0.5B-Hawkeye. We will clarify this explicitly in the next version, thank you for your suggestions!

3. Further explaination on Hawkeye thinking: Hawkeye thinking refers to the concise CoT generation trained with our pipeline (the [Think]...[/Think] content produced by the GRPO post-trained model). In contrast, original thinking corresponds to the long-form CoT generation from the original model without Hawkeye. Since the model has been explicitly trained to treat the [Think]...[/Think] segment as the reasoning component, combining Hawkeye thinking with the original model's response would likely incur additional computational cost while offering limited benefit. This is because the original model is optimized primarily for strong reasoning capabilities, but not necessarily for generating concise or high-quality final responses. A key feature of our method is response generation, where we explicitly disaggregate reasoning ([Think]...[/Think]) from the final response and delegate the response generation task to a smaller model. This design not only reduces inference cost but also preserves quality in the user-facing output.

In fact, there are specific reasons why we do not use the original model’s response in this setting. This choice reveals a potentially important insight: models that undergo extensive reasoning-focused post-training—especially on tasks like code generation and mathematical problem solving, which dominate current reasoning benchmarks—may exhibit degraded performance on other natural language tasks such as summarization or text rewriting. As a result, we argue that a smaller model, not overly optimized for reasoning tasks, may be better suited for generating the final user-facing response (since the CoT does not necessarily need to be presented to the user). Our results in Figure 6, which compare different models on response generation, could support this claim. However, we emphasize that this claim currently reflects our hypothesis. The extent to which reasoning-focused post-training causes catastrophic forgetting in other natural language tasks remains an open question and requires further empirical investigation.

4. Reward computation: The value of 0.3 is selected based on the analysis in Fig. 3, where we observed that most models exhibit approximately 70% redundancy in their CoT. This value is treated as a hyperparameter. The 0.3 ratio refers to the proportion of generated thinking tokens relative to the original CoT tokens, not relative to the final response length.

The reward should be computed as:

$R = \text{EM}(\hat{a}, a) - \lambda \cdot \max\left(0, \text{len}(c) - \alpha \cdot \text{len}(c_{\text{orig}})\right)^2$

Where:

• $\hat{a}$: the predicted answer
• $a$: the ground-truth answer
• $c$: the generated (compressed) CoT
• $c_{\text{orig}}$: the original, uncompressed CoT
• $\alpha$ (set to 0.3): the target compression ratio
• $\lambda$: the penalty weight

We will revise the text and notation in the paper to clearly convey this definition and avoid further ambiguity.

We appreciate your time in reading our rebuttal and look forward to further discussions.

1. Reward argument selection: please refer to 4. Reward computation in our last response

2. Fig.3 looks similar: This is a very good question, the curves in Fig. 3 are indeed similar. We observe these three LLMs share similar core reasoning logic. They differ only slightly in stylistic expression. This similarity may stem from the relatively simple and straightforward nature of GSM8K questions, which limits the observable variation between full and compressed CoTs. We anticipate that more complex reasoning tasks may reveal clearer distinctions but the trends could be similar in compression behavior. We hope these could address your concerns :)

We provide one of our CoT breakdowns for Fig. 3 (also avaliable in Appendix A.1):

Qwestion:

Problem: Rory orders 2 subs for $7.50 each, 2 bags of chips for $1.50 each, and 2
cookies for $1.00 each for delivery. There’s a 20% delivery fee added at checkout,
and she wants to add a $5.00 tip. What will her delivery order cost?

Claude

I'll work through this delivery order problem step by step, calculating each component carefully.

Step 1: Calculate the cost of the subs
Rory orders 2 subs at $7.50 each, so I need to multiply:  
Cost of subs = 2 × $7.50 = $15.00

Step 2: Calculate the cost of the chips
She orders 2 bags of chips at $1.50 each:  
Cost of chips = 2 × $1.50 = $3.00

Step 3: Calculate the cost of the cookies 
She orders 2 cookies at $1.00 each:  
Cost of cookies = 2 × $1.00 = $2.00

Step 4: Find the subtotal of food items  
Now I'll add up all the food costs to get the subtotal before fees:  
Subtotal = $15.00 + $3.00 + $2.00 = $20.00

Step 5: Calculate the delivery fee
There's a 20% delivery fee added to the subtotal. I need to find 20% of $20.00:  
Delivery fee = 20% × $20.00 = 0.20 × $20.00 = $4.00

Step 6: Add the tip 
Rory wants to add a $5.00 tip on top of everything else.

Step 7: Determine what gets added together for the final total 
The final cost will include:  
- The food subtotal: $20.00  
- The delivery fee: $4.00  
- The tip: $5.00  

So the calculation for the total would be: $20.00 + $4.00 + $5.00

GPT-4o:

To find the total cost of Rory’s delivery order, we will break it down step by step.  
First, we calculate the cost of the subs. Rory orders 2 subs at $7.50 each, so the cost of the subs is:  
Cost of subs = 2 × 7.50 = 15.00

Next, we calculate the cost of the chips. Rory orders 2 bags of chips at $1.50 each, so the cost of the chips is:  
Cost of chips = 2 × 1.50 = 3.00

Then, we calculate the cost of the cookies. Rory orders 2 cookies at $1.00 each, so the cost of the cookies is:  
Cost of cookies = 2 × 1.00 = 2.00

Now, we calculate the subtotal. The subtotal is the sum of the costs of the subs, chips, and cookies:  
Subtotal = 15.00 + 3.00 + 2.00 = 20.00

Next, we calculate the delivery fee. There is a 20% delivery fee added at checkout, so the delivery fee is:  
Delivery fee = 0.20 × 20.00 = 4.00

Finally, Rory wants to add a $5.00 tip. Therefore, the total cost including the tip is:  
Total cost with tip = 20.00 + 4.00 + 5.00

At this point, the total cost of Rory’s delivery order can be calculated.

Grok3:

Step 1:  Rory orders 2 subs, and each sub costs 7.50. To find the total cost for the subs, multiply the price per sub by the number of subs:  
2 × 7.50 = 15.00  
So, the subs cost $15.00.

Step 2:  Rory orders 2 bags of chips, with each bag costing 1.50. Multiply the price per bag by the number of bags:  
2 × 1.50 = 3.00  
The chips cost $3.00.

Step 3: Rory orders 2 cookies, and each cookie costs 1.00. Multiply the price per cookie by the number of cookies:  
2 × 1.00 = 2.00  
The cookies cost $2.00.

Step 4: The subtotal is the sum of the costs for the subs, chips, and cookies:  
15.00 + 3.00 + 2.00 = 20.00  
The subtotal for the items is $20.00.

Step 5: The problem states there’s a 20% delivery fee added at checkout. A 20% delivery fee means 20% of the subtotal. To compute this, convert 20% to a decimal (20% = 0.20) and multiply by the subtotal:  
0.20 × 20.00 = 4.00  
The delivery fee is $4.00.

Step 6: Add the delivery fee to the original subtotal to get the total before the tip:  
20.00 + 4.00 = 24.00  
The total before the tip is $24.00.

Step 7: Rory wants to add a 5.00 tip:  
24.00 + 5.00

We sincerely thank the reviewer for the constructive feedback. Below, we address each of the concerns you raised.

1. Missed part on the section of contribution & Fig.2 caption: Thank you for your feedback regarding the paper's formatting and typo. We will make necessary formatting adjustments to improve it. We list the missed sentence of the 3rd point from the contribution part here for your reference: “We curate a high-quality CoT dataset and fine-tune a reasoning model using reinforcement learning to optimize CoT generation. This approach reduces CoT length by over 75% compared to the original large model while maintaining performance, with only around a 4% drop in accuracy on the evaluated dataset”. The captions of Fig.2 is indeed align. The 182 tokens is calculated by summing thinking tokens (72 tokens) and response tokens (110 tokens), they are both involved in Hawkeye (full).

2. Showcase the actual accuracy numbers: Fig. 5 can actually demonstrate there is no or minor accuracy loss in practice. We also verify the similar situation in our real-world deployment. We further provide more evaluation results on CNMO and HMMT, two recent math datasets (https://huggingface.co/deepseek-ai/DeepSeek-R1-0528). The accuracy numbers are shown in the table below:

Due to the small size of the HMMT (only 30 questions), a single incorrect answer can impact the overall accuracy significantly, thus the accuracy on it exhibits a relatively large drop (7%). This can be eliminated by multi-turn reflexion.

3. Limited baselines: We thank the reviewer for pointing this out, as we also noticed other methods have been proposed these days. We consider both CoT-Valve and O1-Pruner as concurrent works that have not undergone peer review. Furthermore, both CoT-Valve and O1-Purner are offline methods while Hawkeye is an online methods that could be deployed easily. For example, CoT-Valve requires a complex setup for pairing long-form "standard CoT" chains with shorter CoT variants. This process, combined with LoRA or distillation, enables the model to first learn a shortened CoT representation, making the overall pipeline even more complex. O1-Pruner relies on PPO, where hyperparameters such as the λ value, the number of samples 𝐾, and the PPO clipping range significantly fluctuate the results. Its offline training demands tens of thousands of CoT examples. In contrast, Hawkeye performs GRPO-based online learning and can be finished with only 2k samples. Hawkeye can achieve better token saving and compression rate than O1-Pruner by using a simpler reward function, fewer samples, and smaller models. For the sake of completeness, we provide a comparison based on the results they reported on the drafts. Since the models and configurations are different, we care about compression rates and accuracies on the benchmarks. Hawkeye can achive better compresstion ratio compared with O1-Pruner and similar value on CoT-Valve with a 7B model. Commonly, larger models have is more potential to have better space to get compressed. We have confidence that Hawkeye can perform bettwe than the baselines with the same model.

CoT-Valve VS Hawkeye:

O1-Pruner VS Hawkeye

4. The distinction between HAWKEYE (Full) and HAWKEYE (CoT-only): The tokens generated by Hawkeye can be divided into the thinking tokens enclosed between [Think] and [/Think], and the the response tokens. Hawkeye (full) refers to the complete reasoning process. It includes both the Hawkeye CoT reasoning (i.e., the [Think]...[/Think] content generated by the large model) and the responses generated by the small model.

We will incorporate this clarification into the paper to help readers better understand the structure of the generated outputs.

Once again, we sincerely thank you for your valuable review. We hope that the above responses sufficiently address your concerns :)

I sincerely thank the authors for their detailed responses. Most of my concerns have been solved. I will keep my score.