LIMO: Less is More for Reasoning

We sincerely thank you for your detailed review and for acknowledging the clarity and implementability of our methodology. We appreciate your thoughtful analysis of our work's contributions and are pleased to address your specific concerns.

Regarding the significance and novelty of our findings: You raise an important point about similar results in the field. While it's true that the concept of small, high-quality datasets improving model performance has gained recognition recently, we would argue that this understanding was not always intuitive, particularly for complex reasoning tasks earlier. Historically, the predominant belief in the community was that challenging reasoning capabilities—especially mathematical reasoning—required massive datasets to achieve meaningful improvements through scaling. Our work makes a significant contribution by systematically demonstrating and formalizing this paradigm shift specifically for reasoning tasks at a crucial moment in the field's development. We provide not just empirical evidence but also a principled framework for understanding when and why small, curated datasets can be more effective than large-scale approaches. This systematic formalization and timing represent important contributions to the community's evolving understanding of efficient training methodologies.

Regarding the reported improvement numbers: Thank you for this excellent question, which highlights a crucial insight about the current state of foundation models. The apparent discrepancy you've identified actually illuminates a key phenomenon in our field: as foundation models become increasingly powerful, the quality gap between existing open-source datasets and what these models can generate themselves has widened significantly. For datasets like NuminaMath, the original solutions are often of substantially lower quality compared to what state-of-the-art models can now produce. This quality gap means that training on such datasets can actually degrade performance relative to the base model's capabilities—which explains why you observe that models trained on NuminaMath (6.5% and 59.2%) perform worse than the base Qwen2.5-32B-Instruct model (16.5% and 79.4%). This phenomenon actually reinforces our core LIMO hypothesis: that strong foundational capabilities are essential for effective reasoning elicitation, and that low-quality training data can be counterproductive when working with capable base models. Our carefully curated LIMO dataset avoids this pitfall by focusing on truly high-quality reasoning chains that complement rather than detract from the model's existing capabilities. This observation underscores the critical importance of our data curation methodology and validates our approach of prioritizing quality over quantity in the current era of increasingly capable foundation models.

Thank you again for your insightful questions, which have allowed us to clarify these important aspects of our work and its broader implications for the field.

We sincerely thank you for your thoughtful review and constructive suggestions. We are pleased that you appreciate our emphasis on data quality over quantity, which indeed represents a core contribution of our work.

Regarding data construction method comparison: Thank you for highlighting this important aspect. We want to clarify that LIMO's core data filtering methodology is both elegant and effective, as detailed in Section 3 and illustrated in Figure 2. Our approach consists of two key components: question filtering and reasoning chain filtering for supervised fine-tuning.

We have conducted extensive ablation studies and comparative analyses to validate the effectiveness of our design choices. Specifically, in Section 6.3.1 (RQ1: Impact of Reasoning Chain Quality) and Section 6.3.2 (RQ2: Impact of Question Quality), we provide comprehensive experimental evidence demonstrating the efficacy of our filtering methods. For reasoning chain quality, we validated that our rule-based scoring system consistently identifies higher-quality reasoning chains that lead to superior model performance. For question quality, we confirmed the effectiveness of our difficulty-based filtering approach through systematic evaluation. These ablation studies serve as implicit comparisons with alternative data construction strategies, showing that our carefully designed filtering criteria significantly outperform random sampling or less sophisticated selection methods applied to the same source data.

Regarding the scope of reasoning evaluation: You raise an excellent point about the breadth of reasoning capabilities. While mathematical reasoning indeed represents one of the most challenging and prominent branches in the field—particularly crucial for advancing toward AGI—we acknowledge the importance of demonstrating generalization across different reasoning paradigms. To address your concern about logical reasoning tasks, we have conducted additional evaluations on the benchmarks you specifically mentioned:

These results demonstrate that LIMO's effectiveness extends well beyond mathematical reasoning to logical reasoning tasks. The consistent improvements across both English and Chinese logical reasoning benchmarks, as well as reading comprehension requiring logical reasoning, strongly support the generalizability of our approach. This suggests that our high-quality reasoning chain curation methodology successfully captures fundamental reasoning patterns that transfer across different domains and reasoning types.

Thank you again for your valuable feedback, which has helped us provide a more comprehensive evaluation of our methodology's broader applicability and effectiveness.

We sincerely thank you for your thorough review and positive evaluation of our work. Your recognition of LIMO's relevance and the convincing nature of our experimental results is greatly appreciated.

Regarding evaluation on entirely out-of-domain tasks: Thank you for this excellent suggestion! You raise a crucial point about the broader applicability of our training paradigm. While our training dataset contains exclusively mathematical samples, we did evaluate on some non-mathematical benchmarks in our paper, including the multidisciplinary GPQA and STEM-focused Minerva benchmarks. However, to fully address your concern and further demonstrate the remarkable generalization capabilities of our approach, we conducted additional comprehensive evaluations on a diverse range of entirely out-of-domain tasks.

Specifically, we evaluated our model on the following benchmarks that span far beyond mathematical reasoning:

• MMLU-Pro: Spans 14 diverse domains including physics, chemistry, law, engineering, psychology, and health
• SuperGPQA: A comprehensive benchmark evaluating graduate-level knowledge and reasoning capabilities across 285 disciplines
• SciBench: Contains collegiate-level scientific problems from mathematics, chemistry, and physics domains
• LogiQAv2 & LogiQAv2_zh: Challenge datasets for machine reading comprehension with logical reasoning
• ReClor: A reading comprehension dataset requiring logical reasoning

The results are presented below:

These results demonstrate that LIMO not only maintains strong performance across diverse domains unrelated to mathematical reasoning but actually achieves superior performance compared to models trained on much larger datasets. This suggests that our high-quality, reasoning-focused training paradigm effectively enhances the model's general reasoning capabilities rather than narrowly specializing it for mathematical tasks. The consistent improvements across logical reasoning, scientific knowledge, and multidisciplinary comprehension tasks indicate that LIMO successfully elicits and strengthens the fundamental reasoning patterns that transfer broadly across domains.

These findings strongly support the wider applicability of our training paradigm for general-purpose language models, addressing your important concern about domain transfer and reinforcing the practical value of our approach for the broader LLM community.

Thank you once again for your insightful feedback, which has helped us provide a more comprehensive evaluation of our method's capabilities.

Thank you very much for your thoughtful and comprehensive review. We greatly appreciate your recognition of our core contribution and the potential implications of our work for the broader community.

Regarding writing quality and presentation: We sincerely acknowledge your feedback on the writing and diagram quality. We are committed to significantly improving the presentation, clarity, and visual elements in the revised version to better communicate our contributions and methodology.

Regarding SOTA performance and scaling: We are pleased to clarify that we have indeed conducted extensive scaling experiments that directly address your questions. As detailed in Section 6.3.3 and particularly Section 6.3.4, we performed comprehensive experiments across different model sizes to investigate the scaling behavior of our approach. Our scaling experiments reveal several key findings: First, as model size increases, the performance consistently improves, which strongly supports one of LIMO's core hypotheses—that strong foundational capabilities are essential for effective few-shot reasoning elicitation. Second, we observe that the 32B model represents an optimal sweet spot in our experiments, demonstrating a significant performance leap from 14B to 32B, while the improvement from 32B to 72B shows more modest gains. This suggests that there may be diminishing returns beyond a certain model size threshold, which has important implications for computational efficiency in practical deployments. These results not only validate our approach but also provide valuable insights into the relationship between model scale and reasoning capability elicitation, further strengthening the theoretical foundation of our work.

Thank you again for your valuable feedback and constructive evaluation!