The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

Thank you for your review and for acknowledging the novelty and practicality of our method. We address your concerns below, particularly a key misunderstanding about our "unsupervised" setup and new experiments that refute concerns about model generalization.

W3: While the method is labeled as 'unsupervised,' it still implicitly relies on ground-truth answers for tuning. It may not be proper to mention an unsupervised tuning method.

We must respectfully clarify a critical misunderstanding here. Our method is fully unsupervised and does NOT use ground-truth answers for training.

In our unsupervised setting (the main focus of our paper), we generate a single reasoning path for each question and use its prefix for fine-tuning. This process is "unfiltered"—the sample is used regardless of whether the final answer is correct or not. The training data is therefore noisy and contains many incorrect solutions. This is the core challenge our method is designed to address and is fundamentally different from supervised methods like RFT, which filter for correct answers. We will make this distinction even clearer in the paper.

W4&Q3: While the proposed method is interesting, the performance gain is limited, especially on challenging datasets (e.g., Math500, AIME 2024, and GPQA) using weak LLM backbones (e.g., Llama-3.1-8B-Instruct ). That means the proposed method may rely on the original capacity of backbones, e.g., if a similar rule exists on a 0.5B-size model. The potential limited generalization of the method reduces the paper’s impact.

This is a fair question. To address it, we ran new experiments on much weaker LLM backbones: Llama-3.2-1B and 3B.

The results show that UPFT provides consistent and significant improvements even on these smaller models, demonstrating that it does not simply rely on the high capacity of large models.

As before, SFT is unstable and often hurts performance, while UPFT consistently yields gains across all benchmarks. This confirms our method is effective even for weaker models.

W2: A more holistic cost analysis (e.g., including the training time and cost) would strengthen the efficiency claims.

Excellent suggestion. We now include training time analysis, which further highlights UPFT's efficiency. By using much shorter sequences, UPFT drastically cuts training time.

On average, UPFT reduces training time by over 70% compared to standard SFT on the same data, in addition to the 95%+ sampling cost reduction compared to RFT. We will add this table to the paper.

Q1: Why does the model's performance drop so sharply when the structure tuning ratio is set to 0? In my perspective, such senarios is standard fine-tuning.

There seems to be a misunderstanding of this ablation. A structure tuning ratio of 0 means we fine-tune exclusively on prefixes (100% prefix-only), with no full-trace examples. This is the opposite of standard SFT.

Performance drops sharply in this setting because the model is never exposed to a complete thought process:

• It never sees a final, boxed answer, so it doesn't learn to produce one.
• The training sequences are artificially short, teaching the model to terminate prematurely.
• It loses the high-level instruction-following format of a full solution. This ablation confirms that a small amount of full-trace data is necessary to preserve the model's structural and formatting capabilities, which is why UPFT includes it.

Q2: Have the authors considered applying sampling multiple prefix span for fine-tuning?

This is an insightful idea. We ran a new experiment on Qwen2.5-Math-7B-Instruct where we used three prefixes per question (3xN) instead of one (1xN).

Using more prefixes provides a modest but consistent boost, confirming your intuition that prefix diversity can be beneficial. This also supports our theoretical claim about reinforcing consistent reasoning steps. Thank you for the great suggestion!

W5: More baseline comparisons (e.g., Quiet-star [1] and so on) could be conducted.

Thank you for the pointer. Quiet-STaR is an interesting recent work on self-teaching. However, its focus is different from ours; it aims to generate rationales before answering, rather than improving efficiency via unsupervised prefix tuning. Its goals are not directly comparable to our efficiency- and unsupervised-focused evaluation against RFT and SFT. We will, however, add a discussion of Quiet-STaR and other related self-improvement methods to our related work section to better situate our contribution.

W1: While concepts like prefix self-consistency are important to the method, the mathematical formulations in Sections 3.1 and 3.2 are dense and may overwhelm readers. The paper would benefit from a well-defined preliminary.

We agree and plan to introduce clearer definitions, brief preliminaries, and inline intuition for the Bayes formulation in a revision.

Thanks for the reply, and I have no further comments. This paper proposes an effective fine-tuning method and only compares UPFT under standard SFT. Under this experimental setting, the performance and efficiency are validated. However, I keep my concerns about whether other types of SFT methods can achieve much better performance (e.g., the performance gain for Llama in Math 500 is 52.0->53.4), which may make the efficiency advantage less valuable. I raise these concerns in the rebuttal stage, which is not accepted by the authors. Therefore, I will keep my negative score, and good luck.

We want to express our sincere gratitude for your positive and encouraging review. We are thrilled that you found our paper "excellent" and our contributions clear and significant. Your understanding of our core insight and appreciation for our comprehensive experiments are very motivating. We kindly ask for your continued support during the internal discussion phase. Your endorsement will be invaluable in contextualizing our contributions for the other reviewers and the Area Chair.

We thank you for your feedback and the opportunity to clarify our contributions. While you found the idea interesting, your main concerns were about the practical utility and statistical significance of our results. We respectfully disagree with the assessment that the gains are "marginal" and provide substantial new evidence below to address your concerns.

W1: SFT seems to decrease performance, and UPFT only marginally improves it. In this setting, fine-tuning itself appears largely ineffective.

We respectfully disagree. The fact that standard SFT often fails in this unsupervised setting is precisely what makes UPFT's consistent and significant gains so valuable.

Our setting is fully unsupervised, meaning we train on a single, unfiltered sample per question, which often contains incorrect reasoning. Standard SFT struggles with this noisy data, frequently degrading performance. In contrast, UPFT is designed to be robust to this noise by focusing on the more reliable initial tokens.

Our results in Table 2 (reproduced below with average gains) show that UPFT consistently and significantly outperforms the base model, while SFT often hurts it.

These are not marginal improvements; gains of +3.1 and +4.4 on strong base models are substantial. Furthermore, this is achieved with massive efficiency gains over supervised methods like RFT (95% less sampling cost, 75% less training cost), highlighting its practical utility.

W2&Q1: Concern about statistical significance and variance from random seeds, especially on AIME.

This is a crucial point. We ran new experiments with 5 random seeds to confirm the statistical significance of our results. We agree that single-run results can be noisy.

The table below shows the mean and standard deviation over 5 seeds for DeepSeek-R1-Distill-Qwen-7B on the LIMO dataset. We performed additional significance testing experiments. In each test, 200 samples were randomly drawn from both SFT and UPFT inference results for comparison. This procedure was repeated 1000 times. UPFT consistently and significantly outperforms SFT across all benchmarks, with p<0.01.

Furthermore, to mitigate sampling variance during evaluation, we also report avg@32 results, which show even more stable and pronounced gains for UPFT. These results confirm our findings are robust and not due to random chance.

W3: The only meaningful baseline appears to be the supervised methods like RFT and V-Starm, but even there, the improvement is minimal. Also, why not test with additional training datasets in Table 2 like U-Hard?

• On Baselines: Our primary claim is effectiveness in an unsupervised setting. Therefore, the most meaningful baseline is unsupervised SFT, which UPFT consistently beats. We also added an SC-CoT (Self-Consistency) baseline, a widely-used unsupervised inference method. As shown below, UPFT is complementary and further boosts performance.

• On U-Hard: The reason U-Hard is not in the supervised comparison (Table 3) is that U-Hard is an unsupervised dataset with no ground-truth answers. It cannot be used for supervised methods like RFT, which require answer verification.

Q2: Does SFT refer to sampling one response from the model and using that as training data?

Yes, that is correct. In our unsupervised setting, SFT means fine-tuning on a single, complete response generated by the model for each question, without any filtering based on correctness.

Q3: I’d like to see in-domain performance. How about using the MATH training dataset directly for training?

This is an excellent question that highlights the core principle of our method. UPFT is designed for a self-sampling regime, where it reinforces the model's own consistent reasoning patterns. When using external, gold-standard data like the MATH training set, this self-alignment property is lost.

We ran this experiment. On the MATH dataset, standard SFT outperforms UPFT.

This result, combined with our strong performance in the self-sampling setting below (e.g., Llama3.1 on PRM: SFT 48.4 -> UPFT 52.0 on MATH500), confirms that UPFT's strength lies specifically in unsupervised self-improvement from noisy, model-generated data.

Q4: Any qualitative results? Any analysis of the samples that become solvable after fine-tuning?

Yes. We find that UPFT increases the model's reliability. For many problems, the base model might generate a correct answer in only 1 out of many attempts. After UPFT, the model's probability of generating the correct answer increases significantly.

Take the case below as an example, for a GPQA question about optical activity, the base Qwen2.5 model answered correctly in only 1 of 16 attempts. After UPFT, it answered correctly in 3 of 16 attempts. This demonstrates that UPFT solidifies nascent correct reasoning abilities. We will add more case studies to the appendix.

***Question***
How many of the following compounds will exhibit optical activity?

(Z)-1-chloro-2-methylbut-1-ene
(3aR,7aS,E)-8-(chloromethylene)hexahydro-4,7-methanoisobenzofuran-1,3-dione
(2R,3S)-2,3-dimethylsuccinic acid
(2R,3R)-2,3-dimethylsuccinic acid
(R)-cyclohex-3-en-1-ol
(1s,3s,5s)-cyclohexane-1,3,5-triol
1-cyclopentyl-3-methylbutan-1-one

***Ground_Truth***
3

Dear reviewer fBpq,

We apologize for disturbing you and thank you again for your thoughtful review and feedback, which we have carefully addressed above.

We believe these clarifications fully address your main concerns and would appreciate it if you could indicate whether they sufficiently resolve your points and might positively impact your score.

As the rebuttal period is drawing to a close, please feel free to let us know if you have any further concerns or questions—we would be happy to discuss them and provide additional clarification if needed.

Thank you again for your time and consideration.

Best regards, The Authors

We sincerely thank the reviewer for their continued engagement and thoughtful follow-up questions. We are glad to hear our initial rebuttal helped clarify that UPFT excels in the unsupervised, noisy data setting, and that unsupervised SFT is the key baseline. We will address your remaining points below.

On the Fairness of Baselines (Unsupervised SFT and RL)

"SFT is not really designed for an unsupervised setting... I believe it would be more appropriate to compare against rl with random rewards... or consistency-based rewards"

This is an excellent point about the taxonomy of methods. Our response has two parts:

1. Why Unsupervised SFT is the most direct baseline: You are correct that SFT stands for "Supervised" Fine-Tuning. However, applying SFT to unlabeled, model-generated data (which we term "unsupervised SFT") is a common and intuitive baseline to establish the difficulty of a problem. It answers the question: "What happens if we naively fine-tune on the model's own output?" Our results show this often fails, which is precisely what motivates our work. UPFT is fundamentally an SFT-based method, using the same cross-entropy loss. Therefore, comparing it to unsupervised SFT provides the most direct, apples-to-apples comparison of the training strategy itself, isolating the benefit of our prefix-tuning objective.
2. Why RL is an orthogonal comparison: RL-based methods introduce entirely different components (reward models, policy gradient optimizers like PPO) and significantly higher computational overhead. While interesting, comparing our SFT-based method to an RL-based one would be more of a comparison between training paradigms (SFT vs. RL) than an evaluation of our specific contribution. Furthermore, as noted in recent work [1, 2], the effectiveness of some RL techniques can be confounded by other factors, and simple reward schemes like random rewards have been shown to be ineffective [3]. We believe our chosen baselines provide a clearer and more direct assessment of UPFT's value.

On Test-Time Sampling vs. Training-Time Improvements

"how many samples are used for consistency? Couldn’t simply increasing the sampling budget have a similar effect to UPFT?"

This is a crucial question that gets to the heart of what UPFT accomplishes.

UPFT is a training-time algorithm, while self-consistency (SC) is a test-time decoding strategy. They are complementary, not competing. All our experiments, for both baselines and UPFT, used a standard SC-CoT setup with N=4 samples to ensure a fair comparison.

The key difference is this:

• Increasing the test-time sampling budget (e.g., from N=4 to N=32) gives a model more chances to find a correct answer that it could already generate with some non-zero probability. It doesn't make the model fundamentally better.
• UPFT, as a training method, improves the model itself. It increases the base probability of generating correct reasoning steps. This means that each sample drawn at test time is more likely to be correct. As our results show, UPFT boosts performance at N=4 and N=32, demonstrating it makes the model more reliable, which is a more fundamental improvement than simply sampling more.

Dear Reviewer fBpq,

As the rebuttal period is drawing to a close, please let us know if you have any remaining concerns or questions that you feel we have not addressed, and which might affect your judgment of our paper's value.

We would be more than happy to discuss them further and provide additional clarifications as needed. We believe this would be very helpful for improving the quality of our paper.

If all your concerns have been adequately addressed, please kindly consider updating the score to reflect your current assessment. We appreciate your time and consideration!

Best regards, The Authors

On the Significance of Our Gains (vs. Prompt Engineering)

"I think the claim that “+3.1 and +4.4 on strong base models are substantial” may be overstated, considering that such improvements can sometimes be achieved through better prompting alone"

We understand the skepticism, as prompting can indeed have a large effect. We argue our gains are substantial for three reasons, including a new experiment to address this directly.

1. We ran a new prompt engineering experiment. To test your hypothesis, we used Deepseek-v3 to generate four diverse, high-quality system prompts for the Qwen2.5-Math-7B-Instruct model. We evaluated them against the model's default prompt on the challenging AIME24 and GPQA benchmarks. The results below show that the default prompt is already optimal, and that prompt engineering does not easily yield further gains on these hard tasks.

   System Prompt	AIME24 Acc@8	GPQA Acc@8
   0 (Default - Our Setting)	15.8	10.1
   1 (Generated by DS-v3)	12.3	7.9
   2 (Generated by DS-v3)	13.8	9.1
   3 (Generated by DS-v3)	14.6	9.3
   4 (Generated by DS-v3)	13.7	8.8

   This experiment confirms that our reported improvements come from our method, not from a suboptimal prompt.

2. Gains on challenging benchmarks are hard-won. As literature confirms [5, 6], the effectiveness of prompt engineering diminishes on complex, distribution-shifted data like AIME24 and GPQA. Achieving a +3 to +4 point absolute improvement on these benchmarks through a training modification is a significant result.

3. Our gains are consistent across multiple model families. We demonstrated UPFT's effectiveness on Llama3.1, Qwen2.5, and DeepSeek-R1 models. This consistency, especially on models like Llama that are less prone to training/test data contamination [1, 2], shows that UPFT provides a generalizable improvement to reasoning, not a model-specific artifact.

We thank you again for the rigorous and constructive feedback. We are confident that our method provides a practical and significant contribution to unsupervised model improvement, and we will integrate these clarifications and new results into the final paper.

References

[1] Chandak et al. Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims. Notion, 2025.

[2] Wu et al. REASONING OR MEMORIZATION? UNRELIABLE RE-SULTS OF REINFORCEMENT LEARNING DUE TO DATACONTAMINATION. arxiv, 2025.

[3] Li et al. PRESERVING DIVERSITY IN SUPERVISED FINE-TUNING OF LARGE LANGUAGE MODELS. ICLR, 2025.

[4] Shao et al. Spurious Rewards: Rethinking Training Signals in RLVR. arxiv, 2025.

[5] Li et al. Robust Prompt Optimization for Large Language Models Against Distribution Shifts. EMNLP, 2023.

[6] Ghosh et al. A Closer Look at the Limitations of Instruction Tuning. ICML, 2024.

We sincerely thank you for your thoughtful review and recognition of our work's novelty and efficiency. Your constructive feedback is very helpful. We address your questions below.

W1&Q1: The experimental evaluation is limited to mathematical reasoning tasks, which raises questions about the generalizability of the proposed method to other domains.

Yes, our method shows strong generalization to non-mathematical domains.** We'd like to highlight a key result that may have been missed:

• Our evaluation already includes a non-math, cross-domain benchmark. The GPQA dataset, as described in the paper, consists of PhD-level questions across biology, physics, and chemistry, not math.
• UPFT demonstrates strong Out-of-Distribution (OOD) generalization. Our models were trained exclusively on mathematical reasoning data. The consistent performance improvements on GPQA (e.g., +4.6 for DeepSeek on LIMO, see Table in response to R_fBpq) show that UPFT enhances a general reasoning capability that transfers OOD.

While we did not test on code generation, the strong OOD results on GPQA suggest UPFT is not limited to a single domain and could be promising for other reasoning tasks like code generation, which we leave for future work.

W2: While the title “The First Few Tokens Are All You Need” may overstate the claim. In practice, the SFT procedure still involves structure tuning with the full reasoning trace, suggesting that early tokens are important but not sufficient on their own.

Thank you for this valuable suggestion. Our title was inspired by "Attention is All You Need" to be memorable, but we agree it could be interpreted more literally than intended. We acknowledge that the small percentage of full-trace tuning is important for preserving the model's output structure. We will revise the title to be more precise in the final version, e.g.. “The First Few Tokens Matter”.

Q2: Is there a performance difference when applying UPFT to models trained with long vs. short CoT?

Yes, and our results suggest UPFT is broadly effective regardless of the CoT length of the training data. As shown in Table 2, the training datasets have vastly different average CoT lengths (e.g., LIMO at 491.8 vs. U-Hard at 393.3 for Qwen2.5). Despite these differences, UPFT consistently delivers gains:

• Qwen2.5 on LIMO (long CoT): Base 51.4 -> UPFT 52.5 (+1.1 avg.)
• Qwen2.5 on U-Hard (shorter CoT): Base 51.4 -> UPFT 54.5 (+3.1 avg.)

This indicates that UPFT's benefit, which comes from reinforcing correct initial reasoning steps, is not dependent on the length of the full reasoning traces in the source data.

Q3: If responses are distilled from a stronger model to fine-tune a weaker one, is training on only the first few tokens still important?

Not necessarily. Our method's core assumption is that we are fine-tuning on the model's own generated samples. This self-sampling is key to reinforcing the model's inherent, correct reasoning pathways. Distilling from a stronger, external model is a different setting where this assumption does not hold.

To verify this, we ran a new experiment where we fine-tuned Llama3.1-8B-Instruct but generated the prefixes from stronger models.

As shown, performance degrades when using prefixes from an external model, even a much stronger one. This confirms that UPFT is most effective in the self-improvement setting, where it aligns the model's distribution with its own most consistent reasoning starts.

W3: Typo about line 108.

Thank you for catching this. We will correct it in the final version.

Dear reviewer JFVJ，

Thank you again for your thoughtful review and feedback, which we have carefully addressed above.

We believe these clarifications fully address your main concerns and would appreciate it if you could indicate whether they sufficiently resolve your points and might positively impact your score. If not, we welcome further discussion to better understand any remaining issues.

Thank you again for your time and consideration.

Best regards,

The Authors