Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

We thank the reviewer for the positive evaluation and constructive suggestions.

On Novelty and Conceptual Contribution

Our approach is a paradigm shift rather than a simple change of training data. By utilizing critique-based data, CFT unlocks the model’s reasoning potential and effectively mitigates the overfitting risk associated with SFT. Instead of biasing the model toward a narrow set of reference answers, CFT exposes it to teacher critiques across diverse candidate responses, allowing the model to learn from various perspectives and error patterns. This fosters deeper understanding and more robust analytical capabilities.

Connection Between CFT Objective and Improved Reasoning

Regarding the connection between the CFT objective and improved reasoning performance, we offer a theoretical framework to explain this relationship:

The fundamental difference between SFT and CFT lies in their information structure during learning. Traditional SFT requires models to learn a direct mapping from questions (x) to answers (y), without intermediate reasoning steps. This resembles asking students to produce final answers without showing their work.

CFT, however, operates on richer information: it trains on triples consisting of a question (x), an initial answer attempt (y'), and a detailed critique (c) of that attempt. This approach parallels educational practices where students learn by analyzing worked examples with instructor feedback.

From an information-theoretic standpoint, CFT provides more structured guidance for the learning process. The model must process not only the question and correct answer, but also understand common error patterns, solution strategies, and verification techniques through the critique process. This conditioning on additional context (the initial answer) provides a more constrained learning objective that focuses specifically on reasoning failures and their corrections. This view is also supported by findings in "Critique-Guided Distillation: Improving Supervised Fine-tuning via Better Distillation," which demonstrates that critique-driven supervision provides richer learning signals and leads to better generalization.

Our experimental results validate this theoretical advantage - models trained with CFT consistently outperform their SFT counterparts across diverse benchmarks, demonstrating more robust reasoning capabilities with significantly less training data. This efficiency suggests that learning from critiques enables more effective knowledge transfer than simply imitating correct answers.

We will expand this analysis in our revised manuscript to provide a clearer theoretical justification for CFT's effectiveness.

Thank you for your careful reading and thoughtful comments. We address your main concerns as follows:

Inference Procedure Clarity

At inference time, as described in Appendix A.4.1, we use the same prompt and procedure as standard SFT baselines: the model is directly prompted to generate a final answer, not a critique. Interestingly, even though the CFT training objective differs from the evaluation task, models fine-tuned with CFT demonstrate superior performance across benchmarks. This suggests that learning to critique diverse solutions helps the model internalize better reasoning patterns, which transfer to direct answer generation. This phenomenon is similar to how students deepen their understanding by grading others' work and subsequently perform better in exams.

To clarify this important point, we will add more details about the inference procedure in the main text of the revised manuscript.

Positioning Against Prior Work

Lan et al. (2024) propose MultiCritique, a pipeline that aggregates critiques from multiple agents to enhance the critique ability of language models. Their approach involves both supervised fine-tuning and reinforcement learning stages, utilizing multi-agent feedback to improve critique quality. In contrast, our Critique Fine-Tuning (CFT) method trains a single model to critique noisy responses using GPT-4o-generated critiques, eliminating the need for multi-agent aggregation and reinforcement learning, thereby simplifying the training process.

Hu et al. (2024) introduce Self-Refinement Tuning (SRT), where a base model generates initial responses that are critiqued and refined by a more advanced model, enabling the base model to learn from this feedback. This process creates a feedback loop that promotes model improvement. While SRT relies on a teacher-student framework and iterative self-critique, our CFT approach trains a single model end-to-end to critique and improve upon noisy responses, reducing dependency on more advanced models for feedback and enabling the model to directly generate improved answers without iterative refinement.

Feng et al. (2023) present CITING, where a teacher model creates rubrics and revisions to guide the instruction tuning of a student model. This method emphasizes curriculum learning through rubric-based feedback. In contrast, our CFT approach focuses on training models to critique noisy responses directly, without the need for rubric generation or a separate teacher model, streamlining the training pipeline.

We will clarify how CFT is distinct in its paradigm and practical implementation, and discuss its relative strengths and novelty compared to these frameworks.

Comparison With High-Quality SFT

We have included experiments comparing CFT with high-quality SFT data, as shown in Table 5:

Results show that CFT remains more effective, even when compared to these strong SFT baselines. Additionally, Table 3 has shown comparisons with numerous high-quality SFT baselines, including NuminaMath-7B-CoT, Eurus-2-SFT, rStar-Math@Greedy, and AceMath-Qwen2.5-Math:

These results demonstrate CFT's advantages over high-quality SFT approaches, while using significantly fewer training examples (50K vs. millions).

We will incorporate these improvements and clarifications into our revised submission.

We thank the reviewer for the positive and encouraging feedback. We acknowledge the limitations regarding multilingual evaluation. In future work, we will conduct CFT experiments on other language versions of WebInstruct and then evaluate the performance on other language tasks in benchmarks like MMLU-Pro, Math, to explore multilingual capabilities.

We sincerely thank Reviewer XEXb for the thoughtful and constructive feedback. Below, we address each concern point-by-point.

Rigorous Theoretical Explanation for CFT's Effectiveness

We can explain why CFT outperforms SFT from a Bayesian perspective that SFT and CFT differ fundamentally in the amount of information available for learning.

• SFT learns the mapping P(y|x): given only the input x, the model is expected to produce the correct answer y. This setting is analogous to inferring an output solely from a prior, without any additional evidence.
• CFT, in contrast, learns P(c|x, y'): given the input x and the model's initial answer y', the model is trained to generate a critique c. Here, y' provides an explicit instantiation of the model's prior reasoning, and the critique c serves as feedback or "evidence" that highlights errors or gaps in reasoning.

In Bayesian terms, conditioning on more evidence (here, y') reduces uncertainty in the model's output distribution. The critique c can be seen as an update based on the "observation" of y', sharpening the model's internal representations and focusing learning on specific reasoning failures.

Formally, the conditional entropy H(C|X, Y') is always less than or equal to H(C|X), meaning that the output distribution in CFT is more constrained and easier to learn than in SFT. This additional conditioning enables CFT to develop more precise and diagnostic internal representations, which empirically leads to improved reasoning performance.

This view is also supported by findings in "Critique-Guided Distillation: Improving Supervised Fine-tuning via Better Distillation," which demonstrates that critique-driven supervision provides richer learning signals and leads to better generalization.

Reliance on GPT-4o Critiques and Data Quality Concerns

We explore the impact of teacher model quality on CFT effectiveness in Table 7, where we compare different critique models:

While stronger teachers (GPT-4o-1120) yield better results, even weaker teachers (GPT-4o-mini) significantly outperform SFT baselines. Additionally, we conducted experiments using WebInstruct-verified and DeepScaler datasets, which contain verifiable data to validate critique accuracy. After filtering out noisy data, performance remained largely unchanged, suggesting that the model learns the pattern of critical evaluation rather than depending on the precise accuracy of individual critiques. We will expand this discussion in our revised manuscript.

Evaluation Beyond Math/Reasoning Benchmarks

We evaluate performance on general instruction-following tasks in Table 8:

Results show that CFT, despite being trained on WebInstruct, effectively improves general instruction-following capabilities, significantly outperforming SFT baselines on both IF_Eval and MT-Bench metrics.

Analysis of Self-Critique Mechanisms

Regarding self-critique mechanisms, our Table 11 experiments demonstrate that models trained with CFT do not further benefit from self-critique methods during inference:

We hypothesize that during CFT training, models already internalize critical reasoning abilities. Therefore, additional self-critique at inference time does not yield improvements beyond the model's existing capabilities. Direct inference with lower temperature settings consistently yields the best performance.

We appreciate the additional paper references and will incorporate them into our related work discussion in the revised manuscript.