Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models

Q1: Why LLaMA3.1-8B collapses?

Existing studies use Qwen for incentivizing reasoning, and there exists few studies that report successful LLaMA performance gains. Why? It does not work well on LLaMA and training easily collapses halfway [97]. As we reported, the training logs (Fig. 55) show a collapsing at step 1K that pushed $π_θ$ into a degenerate distribution.

The collapsed LLaMA exhibits the symptoms:

1. it keeps repeating reasoning contents without jumping to final answer;
2. it fails to output valid response but only the assistant-like call-closing statements in Chinese.

We believe such phenomenon is related to:

• limited multi-language capability of LLaMA (especially in Chinese comprehension and generation)
• insufficient basic knowledge of step-by-step reasoning (lack of pretraining on massive mathematic corpus).

In this case, we should interpret the results of LLaMA3.1-8B as the evidence of collapsing and cannot make any conclusion on the generalizability of our method. To avoid mis-interpretation, we will add more analysis into the manuscript.

Q2: 1.5B is already good w/ SFT, why we bother w/ RL?

SFT might be a good starting point, but it does not promise generalizability.

• Our method outperforms SFT on all 1.5 B models (+3.68 pp on Qwen1.5B, +6.85 pp on DeepSeek1.5B, +6.85 on DeepscaleR 1.5B).
• Our method outperforms SFT on all 7~8B models where SFT can barely boosts LLMs consistently on all benchmarks.

To improve the generalizability of SFT, one must prepare millions of instructions for injecting a broad range of reasoning patterns into existing instructed models. With RLVR, online rollout is performed for preference alignment as long as the complex instructions and their verifications are prepared. It encourages LLMs to self-develop reasoning that enjoys a higher level of generalization.

Q3: Comparison with SOTAs on training and testing-time compute?

We report the training and testing-time compute in the table below (Qwen2.5-7B-Instruct). We followed the official implementation (if publicly released) and used the same instruction datasets for fair comparison on ComplexBench.

We observe that:

• Most training-free methods use manually designed workflows (e.g., alternating reasoning and response drafting, iterative refinement, or consistency-based sampling) to increase test-time compute. However, as discussed in the manuscript (Fig. 1, lines 36–48), naive CoT performs poorly on instructed models with complex instructions. Shallow reasoning leads LLMs to focus only partially on the original instruction and overlook sub-instruction structure. Self-refinement [99] also fails to improve results, as LLMs tend to be overconfident and stop refining after just 2 turns on average (max 6).

• The self-consistency method (10 samples) uses ten times more tokens but shows only a slight performance drop, indicating that sampling multiple responses does not improve performance on ComplexBench. Since complex instructions involve multiple constraints and nested structures, deep reasoning is needed for comprehension, not just more sampling.

• The SDC method, by separating reasoning from answer generation, improves performance. Prompting the LLM to generate only reasoning first allows it to better analyze sub-instructions and constraints. All reasoning is then combined with the original instruction for answer generation.

• Oracle decomposition tests if LLMs perform better with simplified instructions (removing compositional structures). Results show slightly longer responses and better scores, suggesting that redundant constraints in complex instructions can confuse LLMs. However, such decomposition requires human annotation and is not practical in real use.

• Most training-required methods use SFT+DPO to improve LLMs on complex instructions, requiring ground-truth SFT responses and carefully selected DPO pairs [13][14]. In contrast, our GRPO-based RLVR does not need contrastive pair design and benefits from online rollout for easier pipeline setup of optimization.

• Compared to CNFR [14] and FC [13], our method increases test-time compute by an average of 299 tokens due to reasoning, which we believe is crucial for LLM generalizability and worth the modest cost. For training compute, CNFR [14] uses the most SFT examples, while SFT often mixes general and complex samples to prevent forgetting [14]. Thus, training compute does not guarantee better performance if the SFT dataset lacks diversity in instruction constraints and structures.

We will add the above comparison on training and test-time compute into the manuscript. More explanations will be added into the discussion section.

Q4: What is oracle decomposition?

We follow ComplexBench [11] to implement oracle decomposition. It is designed to test whether the decomposition of instructions and executing them step-bt-step can improve the performance of LLMs.

• And: remains as it is;
• Chain: is divided into sequential tasks where each task corresponds to one sub-instruction, and then executed one by one each time;
• Selection: is divided into selection and execution branches where only the true valid sub-instructions are kept;
• Nested: is processed as a combination of the above.

We compare the performance of LLMs between the vanilla I/O (excution of the entire complex instruction in one step) and the oracle decomposition (execution of the decomposed instructions step-by-step).

In practice, there exists no ground-truth decomposition and therefore the oracle decomposition can be interpreted as how the cumulative errors would be if multi-round interactions are performed for LLMs to sequentially respond to each sub-instruction.

Q5: Which LLMs are involved in evolving of instructions?

Three LLMs (see Table 7):

• Seed Instruction Selection: TagLM [87]
• Evolving with Rules and Constraints: Qwen2.5-72B-Instruct [95]
• Response Gneration and Quality Check: DeepSeek R1 [18]

The choice of LLMs in different stages can be determined by the available LLMs at hand. It is free to choose any competitive open-source or proprietary LLMs.

We also agree with the reviewer oTH1 that the expression of self-evolving is not precise because we use larger LLMs for instruction evolving rather than the LLMs used in training. Therefore, we would like to replace the term self-evolving with LLM-based instruction evolving.

Q6: How to interpret behavior cloning?

We would like to resolve any mis-understanding about line 289.

First, the experimental results of the ablation on behavior cloning (BC) are provided in Table 3. Both line 7 (w/o BC) vs. line 9 (w/ BC) and line 8 (w/o BC) vs line 5 (w/ BC) confirm the necessity of applying behavior cloning.

Second, with respect to Without proper guidance, cold-start LLMs can only repeat their inherent shallow thinking and consistently receive negative feedback, we are saying that the instructed models (i.e., non-reasoning model) are not pretrained on reasoning corpus in general domains and therefore they exhibit shallow reasoning during RLVR. In this case, we perform behavior cloning on instructed models so that 1) the adherence to the expected format allows successful parsing and grading of answers at an early stage; 2) the deep reasoning on complex instructions distilled from DeepSeek R1 can be imitated; 3) the potential reward hacking can be mitigated to prevent over-optimization.

Third, it is unfair to compare the cold-start and warm-start models with different model sizes. Under the same model size (1.5B), both our cold-start instruct model (Qwen2.5-1.5B-Instruct) and warm-start reasoning models (DeepscaleR-1.5B and DeepSeek-Qwen1.5B) exhibit significant superiority over baselines.

Fourth, with respect to the 7B-scale models, althought their performance gains are smaller than 1.5B models, the consistent improvement across cold-start and warm start models are non-trivial. One could easily verify that previous studies are validated on very few benchmarks (mostly IFEval, FollowBench or InfoBench) and their generalizability is not fully examined. Our improvement on the extensive SEVEN benchmarks is hard to achieve, thanks to the powerful tool of RLVR which allows LLMs to generalize reasoning-inspired instruction following rather than overfitting.

Last but not least, we would like to explain that for larger models, the current dataset size (13K) might not be enough. The TULU 3 authors use 15K RLVR instructions for the IFEval-style instructions and their optimized LLMs exhibit limited generalizability on OOD constraints. In contrast, we only construct 13K instructions and achieve generalizability on brand-new constraints from IFBench (see our new OOD evaluation in our response to reviewer oTH1). Therefore, we believe with an increase of dataset size, 7B-scale models are expected to achieve superior performance according to the scaling law (Kimi-arxiv 2501.12599 & OpenThoughts-arxiv 2506.04178). In the future, we plan to construct mixed RLVR dataset with size over 100K for scaling up.

More explanations will be added to make it clearer.

Q1: Does the proposed method generalize on OOD Constraints?

We find it quite interesting to investigate the generalizability of the proposed method in handling constraints that are brand-new and human-annotated. We follow the reviewer's suggestion and carefully study the paper of Tulu 3 (Lambert et al.-arxiv 2411.15124) for experimental settings. We use their most recently released benchmark IFBench (Pyatkin et al.-arxiv 2507.02833) as OOD evaluation dataset because:

1. it is proposed by AI2 that releases Tulu 3;
2. it is an extension to the IFEval-OOD dataset which contains 58 new, diverse, and challenging constraints that are different from existing benchmarks like IFEval;
3. it is released in July and there exists no possibility of data contamination with respect to our work.

Based on their official implementation, our evaluation results are shown below:

Our method achieves consistent improvement on the instruction following capabilities across model sizes and families, demonstrating the generalizability on new constraints. It suggests that the proposed method does NOT overfit the constraints of IFEval.

The Tulu 3 paper reports that their optimized LLMs overfit IFEval and exhibit performance degradation compared with models before RLVR (Table 30 in their paper). This contradicts with our findings, which also highlights our advantages over existing RLVR methods.

We believe the generalizability of RLVR for solving complex instructions lies in three key aspects:

• The diversity of constraints. Tulu 3 fully exploits the constraints in IFEval to augment existing instruction datasets (e.g., Wildchat). We do not know their exact construction details, but we are aware that the distribution of constraints in IFEval benchmark is severely unbalanced. In this case, as disclosed in our detailed construction process (Sec. A.3), we refer to the atomic constraint definitions of IFEval (Fig. 7) and generate the evolved instructions with multiple, diversified constraints. We do NOT overfit the distribution of constraints in the testing set to sacrifice the generalizability of the proposed method.
• The diversity of compositional structures. Tulu 3 simply considers the IFEval constraints that are merely code verifiable. However, we consider semantic constraints by introducing a LLM for judgement. In addition, Tulu 3 only considers the "AND" composition for multiple constraints. On the contrary, we consider all structures including "AND", "SELECTION", "CHAIN", and "NESTED". Such semantic constraints verified by LLM-as-a-Judge also benefit our method in reducing the over-optimization problem (see Fig. 26, 27 in Tulu 3 paper).
• The power of deep reasoning. Tulu 3 uses RLVR to directly optimizes the responses without deliberate reasoning. In contrast, we focus on improving "reasoning" of LLMs for complex instructions, which is a "pre-requisite" to the final answers. It is noted that the reasoning is a very powerful tool for generalizability, which has been confirmed in our study (see Sec. A.7.2).

We will add the above Table and its explanations into the manuscript as generalization study.

Q2: What are the differences between the proposed method and Tulu3?

To improve clarity, we will add Tulu 3 in related work and highlight our differences.

• Problem Definition. We aim at solving a broad range of complex instructions with various constraints and their compositions via reasoning. We are dealing with the shallow reasoning problem of existing LLMs under this scenario. Tulu 3 focuses only on aligning model responses with IFEval-style instructions and therefore exhibits limited generalizability.
• Constraint Definition and Structure. We follow CFBench [8] and ComplexBench [11] respectively for the constraint categories (e.g., word, lexical, and semantics) and composition types (e.g., And, Selection, Chain, Nested). Tulu3 is limited in code-verifiable constraints with "and" structures (all constraints have to be satisfied simultaneously).
• Reward modeling. Both code-execution and LLM-as-a-Judge are involved for verification. This greatly expands the tasks that can be handled in our method. It is noted that no previous studies ever evaluate the capability of solving complex instructions on such a scale where seven comprehensive benchmarks are involved. As pointed out by Tulu 3, it is very challenging to generally improve the instruction following of LLMs as over-optimization could occur during RL. Our semantic constraints with LLM-as-a-Judge verifications are complementary to the lexical and word constraints in Tulu 3, which prevents reward hacking.
• RLVR algorithm. The Tulu 3 is designed to directly optimize the response via the vanilla PPO algorithm without deliberate reasoning. However, we are aimed at incentivizing reasoning for solving complex instructions. We have to point out that the reasoning is NOT natural under our scenarios as it is under mathsmatic problems. Since the LLMs are prone to shallow reasoning and tend to muddle with a few sentences that ignore the constraints and their structures. In this case, we design the filtering mechniasm that only selects responses with deep, true reasoning. During rollout stage, we generate responses with and without reasoning at the same time. If the responses with reasoning are scored lower than those without reasoning, we believe the reasoning content is negatively associated with the answer and should be discarded to encourage the beneficial ones.

Q3: What is self-evolving exactly?

First, we would like to explain that we DO NOT simply augment existing instructions as Tulu 3. We DO NOT follow their simple technique of adding verifiable constraints into the original instructions. Tulu 3 only involves IFEval-style constraints and ignores semantic ones. They do not consider the structures of constraint organization at all. In addition to the naive "AND" composition, we build "SELECTION", "CHAIN", and "NESTED" compositions, which are missing in existing instruction datasets such as WildChat. Therefore, evolution of instructions is performed to incorporate multiple sub-instructions.

Second, with respect to "self-evolving", we would like to emphasize that these complex instructions are generated by LLMs and are evolved from the ones in datasets like WildChat. All the prompts for evolving instructions have been described in Sec. A.3.

Third, we agree with the reviewer that that "self-" is not accurate since we use large LLMs (see Table 7) for generation of the instructions. In this case, we will modify the "self-instruct" expression with "LLM-based evolving" for clarity.

Q4: Is the problem definition precise enough?

The current problem definition follows ComplexBench [11] and the terms might not be straightforward for people not familiar with the concept.

We agree with the reviewer that the introduction will be polished with clearer problem definition.

Q5: Can we support the claim on generalization with a citation?

We will add TULU 3 for citation as the reviewer mentions the generalizability of their work is limited.

prior approaches are prone to over-fitting constraints in the training set and fail to generalize to the unseen ones [Lambert et al., 2024, Tulu 3]"

1) For three out of four instructed (i.e., non-reasoning) models, the vanilla CoT does not bring gains but performance drop on the new constraints of IFBench, which is in line with our findings in Table 1. The shallow reasoning nature of instructed models often cause misinterpretation of the original instructions and neglect of certain constraints. They only briefly summarize the original instruction without truly analyze the rules to be obeyed for generation.
2) The SDC decouples the thinking and execution via two separate prompting. Such isolation allows the models to review the original instruction again, where the previous shallow reasoning only acts as a part of the context. Therefore, the SDC performs better than CoT but it is still prone to the unperfect reasoning.
3) For reasoning models, SFT brings a slight improvement as they learn the distilled reasoning patterns from a stronger reasoning model (e.g., DeepSeek R1). However, such distillation cannot generalize to instructed (non-reasoning) models. We believe the reason behind is that the distribution of our SFT dataset (e.g., reasoning patterns) is quite different from that of shallow reasoning patterns inherent in the instructed models. 
4) **As demonstrated by DeepSeek R1 [arxiv 2501.12948], if one is expected to achieve gains on instructed models solely from SFT, a dataset of around 800K curated samples is required.** It relies on huge amount of diverse SFT data to completely transform an instructed model into a reasoning model. On the contrary, if we perform RLVR on instructed models, we can achieve gains with only 13K samples. Such benefit comes from the stimulation of self-developed reasoning trajectories for instructed LLMs to acquired rewards during RL.

Dear Reviewer oTH1,

We sincerely thank you for the review and your constructive suggestions.

In our response, we have addressed your concerns point by point, especially around the new findings of generalization on OOD constraints (which are conducted on the IFBench released by TULU 3 team). More explanations and discussions are incorporated into the manuscript for consistent quality improvement.

As the discussion period is nearing its end, we would like to kindly check whether there are further questions. We would be happy to continue the discussion.

Best regards, Authors

Q1: How to justify the applicability of the proposed method with different improvement across model size?

First, we would like to point out that the evaluation of LLMs in solving complex instructions on seven benchmarks is NOT as simple as that in maths problems. Each benchmark is unique and challenges LLMs with quite different tasks. It is non-trivial to achieve consistent improvement on nearly all benchmarks. Our preliminary study reveals that supervised-finetuning (SFT) often causes a seesaw-like phenomenon where LLMs overfit cetain (easily-learned) tasks and lose generalizability on the other ones. In this case, we resort to reinforcement learning (RL) for incentivizing reasoning which we believe is a powerful and general tool.

Second, the aggregated statistics of Table 1 tell a different story, which futher substantiates the generalizability of our RL over SFT on both small and large LLMs.

• Three out of four 7–8 B models still benefit from RL-incentivized reasoning.
• The single drop (LLaMA3.1-8B) is an outlier whose score collapses even with SFT, suggesting a training-stability issue of LLaMA rather than conceptual limitation (see line 255).

Third, the diminishing returns are expected under the same training dataset size and step. Larger models already saturate many tasks (I/O ≈ 72–78 on 7B), so absolute gains naturally shrink. More importantly, our method consistently outperforms SFT on 7B models (see Table 1, bolded entries), showing complementary rather than redundant value. To further improve performance, one could generate more diverse complex instructions and train longer steps.

Fourth, task-specific gains remain explicit even for big models. These are non-trivial for complex instruction benchmarks as they contain tasks from various domains and high-stakes applications. As far as we are concerened, there exist no previous studies that extensively, comprehensively conduct experiments on multiple benchmarks for performance analysis.

To improve clarity, we will add more explanations into the manuscript.

Q2: Why LLaMA3.1-8B degenerates on multiple domains as an outlier?

Existing studies use Qwen models for incentivizing reasoning, and there exists no studies that report successful LLaMA performance gains. Why? It does not work well on LLaMA and training easily collapses halfway [97]. As we reported, the training logs (Fig. 55) show a collapsing at step 1K that pushed $π_θ$ into a degenerate distribution.

The collapsed LLaMA exhibit the following symptoms:

1. it keeps repeating reasoning contents without jumping to final answer;
2. it fails to output valid response but only the assistant-like call-closing statements in Chinese.

• IFEval

{"key": 1108, "prompt": "Are hamburgers sandwiches? Please respond using only the Kannada language, no other language is allowed.", "instruction_id_list": ["language:response_language"], "kwargs": [{"language": "kn"}], "response": "<think>\n \nOkay, the user is asking if hamburgers are sandwiches......So I need to translate the answer into Kannada. Let me structure it. Start with the question, then the answer, then a note on the debate. But the user might not want the note. Just the answer. \n\nSo the answer is yes, hamburgers are sandwiches. But some people argue against it. But the main point is yes. So in Kannada: ಹ್ಯಾಂಬರ್ಗರ್ ಸ್ಯಾಂಡ್ವಿಚ್ ಆಗಿದೆ. ಆದರೆ ಕೆಲವರು ಇದನ್ನು ಸ್ಯಾಂಡ್ವಿಚ್ ಎಂದು ಪರಿಗಣಿಸುವುದಿಲ್ಲ. ಆದರೆ ಪ್ರಾಥಮಿಕವಾಗಿ ಹ್ಯಾಂಬರ್ಗರ್ ಸ್ಯಾಂಡ್ವಿಚ್ ಆಗಿದೆ. \n\nBut the user might want a simpler answer. Maybe just \"ಹ್ಯಾಂಬರ್ಗರ್ ಸ್ಯಾಂಡ್ವಿಚ್ ಆಗಿದೆ.\" But the user wants a response using only Kannada, no other language. So the answer should be in Kannada, but the user might want a detailed explanation. But the user's instruction is to respond using only Kannada, no other language. So the answer should be in Kannada, but the user might want a clear yes or no. But the answer is yes, but with a note. \n\nSo the final answer in Kannada is: ಹ್ಯಾಂಬರ್ಗರ್ ಸ್ಯಾಂಡ್ವಿಚ್ ಆಗಿದೆ. ಆದರೆ ಕೆಲವರು ಇದನ್ನು ಸ್ಯಾಂಡ್ವಿಚ್ ಎಂದು ಪರಿಗಣಿಸುವುದಿಲ್ಲ. ಆದರೆ ಪ್ರಾಥಮಿಕವಾಗಿ ಹ್ಯಾಂಬರ್ಗರ್ ಸ್ಯಾಂಡ್ವಿಚ್ ಆಗಿದೆ......"}

• ComplexBench

{"main_id": 83, "group": "complex_instruction_eval_41", "idx_in_group": 1, "instruction": "Please translate the following into English and use Markdown format，分点转化为Markdown的无序列表。请记住你的所有输出都是英文，不能含有中文字符:\n\n要点总结如下:\n\n1. 泡影性病变是一种常见的医学影像学表现，可能由多种原因引起，包括感染、自身免疫性疾病和恶性肿瘤等。\n\n2. 根据提供的 CT 扫描结果，巨大泡影性病变位于右侧上肺尖，应考虑恶性肿瘤的生长。\n\n3. 建议进一步进行影像学检查，如 MRI 或 PET-CT 等，以确定病变的性质和范围，并为后续治疗提供指导。\n\n4. 泡影性病变并不一定意味着恶性肿瘤，也可能是其他疾病或情况的表现。因此，在进行任何诊断之前，需要进行进一步的检查和评估。", "task_types": "Professional Writing", "constraint_dimensions": ["Markdown Format", "Helpfulness", "Target Language", "Factuality"], "composition_types": ["And"], "category": "And", "response": "<think>\n\n"用户要求将提供的要点翻译成英文，并用Markdown的无序列表格式。需要确保输出完全为英文，不能包含中文。首先，理解每个要点的内容，然后准确翻译。注意术语如“泡影性病变”翻译为“ground-glass opacities”，确保专业术语正确。检查格式是否符合无序列表，使用正确的符号和缩进。最后，确认没有遗漏任何要点，并且翻译准确无误。</think>\n\n您的问题已经深入思考并详细回答，我们希望这些信息对您有所帮助。\n\n祝您一天愉快！"}

• ARC-C

{"prompt": [
    {
        "role": "HUMAN",
        "prompt": "Question: Juan and LaKeisha roll a few objects down a ramp. They want to see which object rolls the farthest. What should they do so they can repeat their investigation?\nA. Put the objects in groups.\nB. Change the height of the ramp.\nC. Choose different objects to roll.\nD. Record the details of the investigation.\nAnswer:"
    }
],
"origin_prediction": "<think>\n \nOkay, so Juan and LaKeisha are trying to figure out which object rolls the farthest down a ramp......But the question is about repeating the same experiment. So maybe the answer is to use the same objects each time, so the variables are controlled. But the question is about which object rolls farthest. So they need to test different objects. But the answer choices are about what they should do to repeat the investigation. So maybe the answer is to......"}

We believe such phenomenon is related to:

• limited multi-language capability of LLaMA (especially in Chinese comprehension and generation)
• insufficient basic knowledge of step-by-step reasoning (lack of pretraining on massive mathematic corpus).

In this case, we should interpret the results of LLaMA3.1-8B as the evidence of collapsing and cannot make any conclusion on the generalizability of our method. To avoid mis-interpretation, we will add more analysis into the manuscript.

Q3: Can we confirm the generalizability of the proposed method?

We additionally conduct experiments on a brand-new benchmark IFBench to test the generalizability of the proposed method. Results confirm that the reasoning indeed improves the proposed method consistently on such diverse constraints (see our response to the reviewer oTH1 for details).

Q4: What is the role of replay buffer filtering?

The replay-buffer filtering is the bridge between reasoning and instruction-following quality in our setup.

First, we would like to highlight the challenges of incentivizing reasoning under complex instruction tasks. Unlike math, a CoT is not automatically required to produce a response. A model can spit out an answer that superficially satisfies the prompt without any deep reasoning. Therefore, naively keeping every CoT rollout can pollute the buffer with long but useless rationales. These rationales might not be positively associated with the improved responses (e.g., some shallow, non-sense descriptions about the complex instructions). We do NOT allow such reasoning samples to be incorrectly encouraged during RL.

Second, the mechanism of filtering is to compare the reward of $\hat{R}^{i}_{acc}$ of the vanilla (empty-CoT) answer and the reward of $R^{i}_{acc}$ of the CoT answer.

• If any of the $R^{i}_{acc}, i=1,...,G$ is larger than $\hat{R}^{i}_{acc}, i=1,...,G$, the CoT actually improved instruction adherence (e.g., format, constraints, etc.). The rollout sample is kept in the buffer for RL training.
• If all the CoT rollouts are evaluated worse, the model’s current reasoning capacity is insufficient for this particular complex instruction. We discard this sample where shallow, incorrect reasoning (e.g., wrong decomposition and analysis of the complex instruction) corrupts the original instruction alignment capabilities.

Third, our ablation study (Table 3) confirms that the superior CoT enforcement is effective with gains of +1.39~3.28 on average of all seven benchmarks.

To improve clarity, we will add the following explanations into the manuscript:

The replay buffer filtering guarantees that every retained CoT is causally linked to better instruction-following performance rather than just longer text. It prevents the policy model from generating verbose but non-compliant chains for reward hacking. The dynamically curated replay buffer ensures that the effective training signals come from the reasoning-aligned responses with genuine instruction-following gains.