Beyond Two-Stage Training: Integrating SFT and RL for Improved Reasoning in LLMs

Thank you for your thoughtful and constructive feedback on our work. We are encouraged that you found our motivation sound, idea interesting, and paper well-written. We have conducted extensive additional experiments and analyses to address your primary concerns regarding the experiments.

Summary of Additional Experiments

In response to your suggestions, we have:

1. Extended experiments to additional LLMs: We tested on two additional models: We evaluated BRIDGE on Qwen3-8B-Base and Llama-3.2-3B-Instruct, and BRIDGE achieved improvements of 13% and 31% over RL-zero and Cold-start respectively on LLaMA-3.2-3B, and 16% and 10% improvements on Qwen3-8B-Base.

2. Added in-distribution experiments: We trained and evaluated on three datasets with in-distribution testing. BRIDGE demonstrated consistent improvements of 6% over RL-zero and 3% over Cold-start.

3. Integrated DPO-style training: We verified compatibility with DPO algorithms, where BRIDGE achieved a 39% improvement over RL-zero, confirming our bi-level framework is algorithm-agnostic.

Below, we address each point in detail.

Response to W1: Experiments on Additional LLMs

We appreciate your suggestion. To assess generalization across model families and sizes, we conducted additional experiments on Qwen3-8B-Base and LLaMA-3.2-3B-Instruct, beyond the original Qwen2.5-3B.

Tables 1 and 2 present these results. On Qwen3-8B-Base, BRIDGE achieves an average accuracy score of 49.9, representing a 16% improvement over RL-zero and a 10% improvement over Cold-start. On LLaMA-3.2-3B-Instruct, BRIDGE outperforms RL-zero by 13% and Cold-start by 31%, demonstrating strong generalization across model families.

Table 1. Experiments on Qwen3-8B-Base

Table 2. Experiments on Llama-3.2-3B-Instruct

Response to W2: In-Distribution Evaluation

Thank you for the valuable suggestion. To further support our generalization claims, we conducted additional training on three datasets: GSM8K, MATH (Level 3–5), and DeepMath, each with 3k training examples. We then evaluated performance on their respective test sets.

Table 3. In-Distribution Performance Across Three Datasets

As shown in Table 3, BRIDGE consistently outperforms the baselines on all datasets, demonstrating robust in-distribution generalization in addition to out-of-distribution performance.

Response to W3: Compatibility with DPO-Style Methods

Thank you for raising this important point. Our bi-level framework is algorithm-agnostic and can be integrated with a variety of RL methods.

To demonstrate this flexibility, we conducted experiments using DPO as the underlying RL algorithm. As shown in Table 4, BRIDGE achieves a 39% relative improvement over DPO-zero, confirming compatibility beyond PPO/GRPO paradigms.

Table 4. Experiments on DPO

Minor Correction

Thank you for pointing out the typo. We will correct "RL-zere" to "RL-zero" in Line 247.

We sincerely appreciate your thoughtful and constructive feedback, which has substantially strengthened our evaluation. We hope these additional results adequately address your concerns and welcome any further questions or suggestions.

Dear Reviewer qL6k,

Thank you again for your valuable and constructive feedback. We have conducted the additional experiments you suggested, including:

1. Multiple model families and sizes: We extended our experiments to Qwen3-8B-Base and LLaMA-3.2-3B-Instruct, showing consistent improvements (13–31%) over baselines.

2. In-distribution evaluation: We trained and evaluated on GSM8K, MATH, and DeepMath datasets, achieving 6% improvement over RL-zero.

3. DPO compatibility: We successfully integrated our framework with DPO, yielding a 39% improvement over DPO-zero.

We sincerely hope these results address your concerns regarding the evaluation scope. With the discussion period ending on August 6, we would greatly appreciate it if you could take a moment to review our updates and share your thoughts.

Thank you again for your time and consideration.

Best regards,
The authors

I appreciate the author's efforts on providing more experiments in supporting the effectiveness of the algorithm, also recommend the authors to incorporate these results in their future revisions. I will improve my score accordingly.

Thank you for your positive and insightful feedback on our work. We are encouraged that you found the method interesting, the motivation strong, and the performance improvements notable across benchmarks. We appreciate your recognition of the clear method explanation and effective combination of SFT and RL advantages.

Summary of Response

In response to your questions, we have:

1. Clarified LoRA usage: We confirmed that baseline methods do not use LoRA and demonstrated through ablation that LoRA's impact on baseline performance is minimal (< 0.2% difference).

2. Provided comprehensive cost-benefit analysis: BRIDGE demonstrates superior training efficiency, achieving 44% faster training than Cold-start on Qwen2.5-3B with 13% better performance, and 14% faster training than Cold-start on Qwen3-8B-Base with 10% better performance. While BRIDGE introduces moderate memory overhead, these significant improvements in both efficiency and accuracy justify the trade-off.

Response to Q1: Use of LoRA in Baselines

Thank you for your question. All baseline methods in our paper are trained using the base model only, without LoRA. To assess the influence of LoRA on performance, we also trained the baselines with LoRA, which account for only 0.5% of the total parameters. As shown in Table 1, the performance impact is minimal.

Table 1. Effect of LoRA on RL-zero

Reponse to Q2: Cost-Benefit Analysis

Thank you for this insightful suggestion. We agree that averaging performance across epochs is not a standard metric for evaluating the performance-efficiency trade-off. In response, we conducted a comprehensive cost-performance analysis reporting wall-clock training time, average GPU memory usage per device, and final convergence performance across two model scales: Qwen2.5-3B (trained on 4×80GB A100 GPUs) and Qwen3-8B-Base (trained on 8×192GB MI300 GPUs).

Table 2. Cost-performance analysis of Qwen 2.5-3B

Table 3. Cost-performance analysis of Qwen 3-8B-Base

Key Findings:

1. Cold-start suffers from significant inefficiency due to data distribution mismatch between SFT and RL stages. During SFT, the model learns from teacher-distilled data (R1 model), producing long reasoning traces. This leads to excessive response lengths during early RL training, increasing computational costs. We observe a "dip-then-rise" pattern in response length, suggesting that the model first somewhat forgets the expert behavior acquired during SFT and then slowly explores new behaviors. This contributes to the inefficiency and suboptimal performance of cold-start training.

2. BRIDGE achieves superior training efficiency by dynamically integrating both SFT and RL data distributions within a single-stage process. This design mitigates the data distribution and response-length mismatch problem in the two-stage pipeline, resulting in:

   • 44% faster training than Cold-start on Qwen2.5-3B with 13% better performance
   • 14% faster training than Cold-start on Qwen3-8B with 10% better performance
   • Even outperforming RL-zero by 13% in speed on larger models

3. The computational trade-off is favorable: While BRIDGE introduces additional memory overhead, the substantial gains in both training efficiency and final performance justify this cost.

Reponse to Q3: Computational Cost Comparison

Please refer to our response to W1 for detailed computational cost analysis.

We sincerely appreciate your thoughtful comments and suggestions, which have significantly enhanced our work. We hope our responses and new results effectively address your concerns, and we welcome any further feedback you may have. Thank you again for your review.

Dear Reviewer XEdK,

Thank you for your positive and constructive review. We're glad you found the method interesting and well-motivated, and we appreciate your thoughtful questions regarding training efficiency and implementation complexity.

Following your suggestions, we conducted a detailed cost-benefit analysis across two model scales. Despite the use of bilevel optimization, BRIDGE achieves 44% faster training than Cold-start on Qwen2.5-3B and 14% faster on Qwen3-8B, while also delivering 13% and 10% better final performance, respectively. We believe this demonstrates a favorable trade-off between efficiency and accuracy in practice.

We also clarified that baseline methods do not use LoRA. Ablation experiments show that adding LoRA leads to negligible performance change (<0.2%), confirming that LoRA is not a confounding factor in our comparisons.

With the discussion phase ending on August 6, we just wanted to ensure you had a chance to see these new results. If there's anything else we can clarify, we'd be very happy to discuss further.

Thank you again for your careful review--it has helped us strengthen the paper.

Best regards,

The authors

Thank you for your thoughtful feedback on our work. We appreciate your recognition of our method as original and intuitive, with clear theoretical formulation and promising empirical results. We are pleased that you found our paper well-written and easy to follow.

Summary of Responses

We have addressed your concerns by:

1. Providing comprehensive implementation details for all baselines and BRIDGE, including previously unspecified hyperparameters
2. Clarifying the fundamental challenge that motivates our bilevel formulation: the data distribution mismatch in two-stage training
3. Explaining the role assignment in BRIDGE based on the natural flow from SFT initialization to RL optimization
4. Analyzing learning dynamics through comparative reward and KL penalty trajectories

Below, we address each point in detail.

Response to W1: Implementation Details

We appreciate the opportunity to clarify the implementation details. Here are the complete configurations:

Cold-Start (Two-Stage Pipeline)

• SFT dataset: Distilled from R1 model
• SFT stage: Batch size = 32, learning rate = 1e-5, epochs = 2 (best validation checkpoint selected after 1 epoch)
• RL stage: Batch size = 64, mini-batch size = 64, learning rate = 5e-7, epochs = 6, rollouts = 5, maximum sequence length = 5,000

Naive Alternating (Algorithm 1)

• SFT step: Batch size = 64, learning rate = 1e-5
• RL step: Batch size = 64, mini-batch size = 64, learning rate = 5e-7
• Switching interval: 1 iteration (alternating every iteration)

BRIDGE (Algorithm 2)

• SFT: Batch size = 64, learning rate = 1e-5
• RL: Batch size = 64, mini-batch size = 64, learning rate = 5e-7
• Penalty coefficient: λ = 0.5
• LoRA configuration: Rank = 16, α = 16
• All other hyperparameters are identical to those used in the Naive alter

Additional training details are provided in Section 4.1. We will incorporate these specifications into our revised manuscript and release our code for reproducibility.

Response to Q1: Challenge of the Two-Stage Pipeline

Thank you for the insightful question. The primary limitation of the two-stage paradigm is the data distribution mismatch between SFT and RL stages.

In the first stage, the SFT data $D_{sft}$ is distilled from a stronger teacher model (R1), producing long and well-structured reasoning traces. In contrast, the RL data $D_{rl}$in the second stage is sampled from the student model itself, which initially generates shorter and less complete responses.

This mismatch creates two critical issues:

1. Training inefficiency: Early in RL training, the model tends to generate overly long responses, increasing computational cost. We observe that the response length initially drops, then gradually recovers as training progresses.

2. Distribution shift: This “dip-then-rise” pattern suggests that the model first somewhat forgets the expert behavior acquired during SFT and then slowly explores new behaviors. This mismatch contributes to the inefficiency and suboptimal performance of cold-start training.

Table 1. Training Time and Performance Comparison

As shown in Table 1, these effects result in nearly 2× longer training time and 13% lower performance compared to BRIDGE. We believe this stems from the lack of interaction between the SFT and RL stages, which prevents smooth adaptation. This observation motivates our design of a more integrated training framework.

Response to Q2: Role Assignment in BRIDGE

Our role assignment follows the natural training flow where SFT provides initialization for RL optimization. In the standard pipeline, SFT creates a strong starting point for subsequent RL training.

BRIDGE extends this principle through meta-learning: at each iteration, upper-level SFT provides an improved initialization for lower-level RL exploration. This design preserves the established SFT→RL flow while enabling continuous interaction.

Reversing the roles (RL as upper-level, SFT as lower-level) would contradict this fundamental principle, as it would imply RL provides initialization for SFT—contrary to the general principle.

Response to Q3: Learning Dynamics Analysis

Following your suggestion, we analyzed the reward trajectories and KL penalties for BRIDGE versus Naive Alternating. Tables 2 and 3 present these dynamics.

Table 2. Mean Reward Dynamics

Table 3. KL Penalty Dynamics

Our analysis reveals that BRIDGE demonstrates more efficient exploration in the early training stages, with faster reward growth accompanied by larger KL penalties compared to the Naive Alternating approach. As training progresses toward completion, BRIDGE achieves higher average rewards while the KL penalties of both methods converge to similar levels.

These findings suggest that BRIDGE's training dynamics enable more effective policy optimization through better balancing of exploration and constraint satisfaction. We will incorporate these comparative learning curves and discussion in our revised manuscript to provide the community with deeper insights into the underlying mechanisms.

We sincerely appreciate your constructive feedback, which has substantially strengthened our work. We hope these additional details and analyses adequately address your concerns and welcome any further questions.

Dear Reviewer 1amM,

Thank you again for your insightful review and thoughtful questions. We have carefully responded to your comments in our rebuttal, including:

1. Implementation details: We provided comprehensive configurations for all baselines (Cold-Start, Naive Alternating, and BRIDGE), including previously unspecified hyperparameters.

2. Learning dynamics: We presented an analysis of reward trajectories and KL penalties between BRIDGE and Naive Alternating, highlighting how BRIDGE enables more efficient early-stage exploration and achieves higher final rewards.

3. Bilevel motivation: We clarified the fundamental challenge of data distribution mismatch between SFT and RL stages, which leads to longer training time and suboptimal performance in traditional pipelines.

We hope these responses can address your concerns. With the discussion phase ending on August 6, we would greatly appreciate any thoughts or feedback you might have on our responses. Please feel free to let us know if there are any remaining questions we can help clarify.

Thank you again for your time and engagement.

Best regards,
The authors

Dear Reviewer 1amM,

Thank you for your efforts in reviewing our work and for your insightful questions. Following your suggestions, we have elaborated on our implementation details and discussed the challenges of the two-stage baseline, the motivation behind our design, and the learning dynamics. We truly hope that our response helps clarify these issues.

As the discussion deadline is approaching, please let us know if there is anything else you need further information on or any additional concerns you might have.

Best regards,

Authors

Thank you for your thoughtful feedback and for engaging deeply with our work. We appreciate your openness to further discussion.

We'd like to clarify our approach from two complementary perspectives and extended empirical evidence:

1. From RL Perspective: Addressing Exploration Challenges

Q: Why introduce SFT?

RL optimization faces fundamental challenges: (1) due to the unknown environment and non-convex optimization, practical RL cannot reach the optimal π_θ2* and often becomes trapped in local optima; (2) it relies on inefficient trial-and-error to discover solutions that yield positive rewards.

In hard reasoning problems, agents exploring independently often get stuck, unable to find reward-yielding answers. Even with SFT initialization, RL frequently reaches performance plateaus—particularly on difficult questions where agents fail to produce any positive solutions.

BRIDGE addresses this by providing adaptive SFT guidance, helping agents discover solution paths they cannot find alone. This design avoids inefficient exploration and provides external guidance to escape local optima, with empirical results showing significant performance enhancement.

2. From Meta-Learning Perspective

Q: What is the meta-learning objective?

BRIDGE does not simply optimize toward π_θ1 and π_θ2 simultaneously. Instead, BRIDGE follows a meta-learning structure: we optimize SFT specifically to help RL converge better to π_θ2.

Since SFT is not always beneficial for RL—naively alternating SFT and RL updates does not guarantee that every SFT update will improve the RL objective. To address this, BRIDGE alternates between: (1) updating LoRA adapter weights to explicitly maximize the reward gap between joint SFT + RL training and RL alone, and (2) updating base model parameters following a blend of SFT and RL gradients.

Q: Why is single-stage meta-learning superior to two-stage Cold-Start?

The two-stage cold-Start follows a naive continual learning paradigm with inherent catastrophic forgetting—the model loses valuable SFT knowledge during RL training. In contrast, BRIDGE's meta-learning paradigm:

(1) Avoids forgetting: Single-stage training prevents the wasteful forgetting-relearning cycle

(2) Provides better initialization: Each SFT update is specifically designed to provide initialization points that help RL converge toward π_θ2*

The key insight: SFT's role in BRIDGE is instrumental—it exists to facilitate RL's optimization toward π_θ2*.

3. Empirical Evidence

To validate BRIDGE's advantages, we conduct additional experiments on Qwen3-8B and LLaMA-3.2-3B-Instruct, in addition to the previously evaluated Qwen2.5-3B.

These consistent improvements across diverse LLMs demonstrate that BRIDGE's meta-learning approach enables better exploration and guaranteed helpfulness of SFT updates to RL objectives, leading to superior performance compared to Cold-Start's direct but less efficient optimization.

We hope these clarifications address your concerns and welcome any further questions.

Thank you for your positive and constructive feedback on our work. We are encouraged that you found the method novel, the motivation sound, and the performance consistently strong across baselines. We appreciate your recognition of BRIDGE's sophisticated integration of imitation learning and reward-driven exploration. Your main concern regarding training efficiency is well-taken, and we have conducted comprehensive analysis to address it, as detailed below.

Summary of Response

• Provided comprehensive cost-benefit analysis: BRIDGE demonstrates superior training efficiency, achieving 44% faster training than Cold-start on Qwen2.5-3B with 13% better performance, and 14% faster training than Cold-start on Qwen3-8B with 10% better performance. While BRIDGE introduces moderate memory overhead, these significant improvements in both efficiency and accuracy justify the trade-off.

Reponse to W1: Cost-Benefit Analysis

Thank you for this insightful suggestion. We agree that averaging performance across epochs is not a standard metric for evaluating the performance-efficiency trade-off. In response, we conducted a comprehensive cost-performance analysis reporting wall-clock training time, average GPU memory usage per device, and average final convergence performance across two model scales: Qwen2.5-3B (trained on 4×80GB A100 GPUs) and Qwen3-8B-Base (trained on 8×192GB MI300 GPUs).

Table 1. Cost-performance analysis of Qwen 2.5-3B

Table 2. Cost-performance analysis of Qwen 3-8B-Base

Key Findings:

1. Cold-start suffers from significant inefficiency due to data distribution mismatch between SFT and RL stages. During SFT, the model learns from teacher-distilled data (R1 model), producing long reasoning traces. This leads to excessive response lengths during early RL training, increasing computational costs. We observe a "dip-then-rise" pattern in response length, suggesting that the model first somewhat forgets the expert behavior acquired during SFT and then slowly explores new behaviors. This contributes to the inefficiency and suboptimal performance of cold-start training.

2. BRIDGE achieves superior training efficiency by dynamically integrating both SFT and RL data distributions within a single-stage process. This design mitigates the data distribution and response-length mismatch problem in the two-stage pipeline, resulting in:

   • 44% faster training than Cold-start on Qwen2.5-3B with 13% better performance
   • 14% faster training than Cold-start on Qwen3-8B with 10% better performance
   • Even outperforming RL-zero by 13% in speed on larger models

3. The computational trade-off is favorable: While BRIDGE introduces additional memory overhead, the substantial gains in both training efficiency and final performance justify this cost.

Reponse to W2: Computational Cost Comparison

Please refer to our response to W1 for detailed computational cost analysis.

Response to Minor Issues

Thank you for pointing this out. We will correct the typo on line 247 ("zere" → "zero") and ensure consistent usage of λ and γ in Section 3.2 in the revision.

Response to Q1: Missing SFT Curve in Figure 1

Thank you for this careful observation. You are absolutely correct—the curve for SFT alone was inadvertently omitted from Figure 1. While the Cold-start curve includes an initial SFT phase, we agree that adding a dedicated SFT curve would provide clearer evidence for SFT's rapid initial learning dynamics (line 105) and strengthen our comparative analysis.

We will revise Figure 1 in the final version to include the standalone SFT curve, making the learning dynamics comparison more transparent and complete.

We sincerely appreciate your thoughtful comments and suggestions, which have significantly enhanced our work. We hope our comprehensive responses and additional analyses can address your concerns. We welcome any further feedback you may have.

Dear Reviewer 73wX,

Thank you for your positive review and recognition of BRIDGE's novel integration of SFT and RL. We truly appreciate your constructive feedback.

Following your excellent suggestion, we have provided a comprehensive cost-benefit analysis. BRIDGE achieves 44% faster training than Cold-start on Qwen2.5-3B with 13% better performance, and 14% faster training on Qwen3-8B with 10% better performance, clearly demonstrating its practical advantages. We've also acknowledged the missing SFT curve in Figure 1 and will include it in the final version for clearer comparison.

With the discussion period ending on August 6, we wanted to ensure you had a chance to see our responses. If you have any additional thoughts or suggestions, we'd be happy to address them.

Thank you again for your supportive and constructive review.

Best regards,

The authors

We sincerely thank you for your insightful and constructive feedback. We appreciate your positive assessment of our formulation of SFT-RL cooperation within a bilevel optimization framework and recognition of our efficient first-order training algorithm. Your acknowledgment of the method's empirical effectiveness across multiple baselines is especially encouraging.

Summary of Response

In response to your suggestions, we have:

1. Extended experiments to additional LLMs and non-math benchmarks: We evaluated BRIDGE on Qwen3-8B-Base and Llama-3.2-3B-Instruct, achieving average improvements of 13% and 31% over RL-zero and Cold-start respectively on Llama-3.2-3B, and 16% and 10% improvements on Qwen3-8B-Base. We also expanded to larger training datasets (8.5K samples) and scaled training setups (8 rollouts, 8K token length). Additionally, we tested on two non-math benchmarks (LCB and GPQA), where BRIDGE achieved 9% and 56% improvements over RL-zero and Cold-start respectively.

2. Provided comprehensive cost-benefit analysis: BRIDGE demonstrates superior training efficiency, achieving 44% faster training than Cold-start on Qwen2.5-3B with 13% better performance, and 14% faster training than Cold-start on Qwen3-8B with 10% better performance. While BRIDGE introduces moderate memory overhead, these significant improvements in both efficiency and accuracy justify the trade-off.

3. Clarified our technical contributions: We explained how our cooperative meta-learning framework differs from existing bilevel approaches through (a) the specific design for addressing the "data distribution mismatch challenge" in reasoning model training pipeline, and (b) the augmented architecture that enables true SFT-RL co-adaptation rather than simple MAML-style updates.

4. Conducted sensitivity analysis: We performed ablation studies on LoRA hyperparameters, demonstrating that BRIDGE maintains stable performance across configurations, confirming the robustness of our approach.

5. Discussing the challenge of Cold-start pipeline: We detailed the fundamental challenges of two-stage pipelines—particularly the data distribution mismatch that leads to training inefficiency and distribution shift—with concrete evidence of prolonged training time and degraded final performance.

Below, we address each point in detail.

Response to W1: Expanded Experimental Setup

Thank you for highlighting these limitations. In response, we have expanded our experiments to include additional LLMs (Qwen3-8B and Llama3.2-3B-Instruct), a larger training dataset (Math Level 3-5, 8.5K samples), and a scaled training setup (8 rollouts per prompt, 8K maximum token length). As shown in Tables 1 and 2, BRIDGE consistently outperforms baselines across diverse LLMs and configurations, achieving 16.3% improvement over RL-zero and 9.7% over Cold-start on Qwen3-8B, and 13.5% improvement over RL-zero and 30.9% over Cold-start on Llama-3.2-3B.

Furthermore, we evaluated our method on two additional benchmarks—LCB and GPQA—as presented in Table 3. BRIDGE achieves 9.2% improvement over the best baseline on these non-math tasks, further highlighting our approach's generalization capability across domains.

Table 1. Results on Qwen3-8B-Base

Table 2. Results on Llama-3.2-3B-Instruct

Table 3. Results on Non-Math Benchmarks

Response to W2: Contributions in Formulation and Implementation

Thank you for the insightful comment. While our formulation shares the general bilevel structure with prior work, our implementation introduces a key customization to enable effective cooperation between SFT and RL in training reasoning models.

Conceptually, we propose a cooperative meta-learning framework in which, at each iteration, the upper-level SFT provides an improved initialization for RL exploration, while the lower-level RL further updates based on this initialization. This dynamic cooperation addresses the significant data distribution mismatch problem in two-stage pipelines (as further discussed in our response to Q1).

From an implementation perspective, to enable this cooperative learning, we introduce an augmented model architecture consisting of two components: a base model componet and a LoRA component. This separation allows the upper- and lower-level objectives to co-adapt during training, as illustrated in Figure 2. Without this design, this formulation collapses into a standard MAML-style setup, where the lower-level solution reduces to a single gradient step used to update the upper-level SFT parameters. In this case, RL learning is disabled, and the cooperation between SFT and RL is lost.

We acknowledge that we did not provide a theoretical analysis of Danskin's conditions and instead applied the approximation in practice. We will clarify this assumption in the revised version.

Response to W3: Cost-Benefit Analysis

Thank you for your insightful suggestion. We agree that averaging performance across epochs is not a standard or robust metric for evaluating the trade-off between performance and training efficiency. In response, we conducted a more precise cost-performance analysis by reporting both the wall-clock training time, average GPU memory usage per device, and final convergence performance across two model scales: Qwen2.5-3B (trained on 4×80GB A100 GPUs) and Qwen3-8B-Base (trained on 8×192GB MI300 GPUs). The results are summarized in Tables 4 and 5.

Table 4. Cost-performance analysis of Qwen 2.5-3B

Table 5. Cost-performance analysis of Qwen 3-8B-Base

As shown in Table 4, Cold-start requires nearly 2× the training time of RL-zero, despite the short SFT stage. This overhead stems from long sequence lengths induced by offline SFT (see Q1 response). In contrast, BRIDGE achieves 44% time savings compared to Cold-start while delivering 13% performance improvement.

Table 5 demonstrates that BRIDGE requires 14% less training time than Cold-start on larger models, with only 11% additional memory usage. Given BRIDGE's consistent and noteble final performance gain, this trade-off is favorable in practice.

Response to Q1: Challenges of the Two-Stage Pipeline

Thank you for the insightful question. The main limitation of the two-stage training paradigm lies in the data distribution mismatch between the SFT and RL stages.

In the first stage, the SFT data $D_{SFT}$ is distilled from a stronger teacher model (R1), producing long and well-structured reasoning traces. In contrast, the RL data $D_{RL}$ in the second stage is sampled from the student model itself, which initially generates shorter and less complete responses due to its limited capacity.

This mismatch leads to two main issues:

1. Training inefficiency: Early in RL training, the model tends to generate overly long responses (inherited from SFT), increasing computational cost. We observe that the response length initially drops, then gradually increases as training progresses.

2. Distribution shift: This “dip-then-rise” pattern suggests that the model first somewhat forgets the expert behavior acquired during SFT and then slowly explores new behaviors. This mismatch contributes to the inefficiency and suboptimal performance of cold-start training.

These effects lead to approximately 2× longer training time and up to 13% weaker performance compared to BRIDGE on Qwen2.5-3B model. Our integrated framework addresses this through continuous SFT-RL interaction, preventing these inefficiencies.

Response to Q2: Wall-clock Time

Please refer to our response to W3 for detailed timing analysis.

Response to Q3: Sensitivity Analysis of LoRA Hyperparameters

Following your suggestion, we conducted ablation studies on LoRA hyperparameters using Qwen3-8B-Base. We observe that overall performance remains relatively stable across settings.

Table 6. Performance with Different LoRA Configurations (Rank R, Alpha A)

We sincerely appreciate your thoughtful comments, which have significantly enhanced our work. We hope these comprehensive responses and additional results can address your concerns and welcome any further feedback.

Dear Reviewer CmPU,

Thank you for your thorough and technically insightful review. Your detailed feedback has undoubtedly helped us strengthen our work. We have conducted extensive additional experiments and analyses to address each of your concerns:

1. Expanded experimental scope: We extended evaluation to Qwen3-8B-Base and Llama-3.2-3B-Instruct, scaled to 8.5K training samples with 8K token length, and included non-math benchmarks (LCB and GPQA), demonstrating consistent improvements across larger scales and diverse settings.

2. Cost-benefit analysis: We provided detailed wall-clock time and memory usage comparisons, showing BRIDGE achieves 44% faster training than Cold-start on Qwen2.5-3B while delivering 13% better performance--a favorable trade-off in practice.

3. Technical contributions: We clarified how our cooperative meta-learning framework with augmented architecture design enables true SFT-RL co-adaptation, addressing the specific challenge of "data distribution mismatch" in traditional two-stage pipelines.

4. LoRA sensitivity analysis: We conducted ablation studies confirming BRIDGE's stability across different LoRA configurations.

We hope these comprehensive responses--particularly the expanded experiments and quantitative cost analysis--effectively address your concerns about scope and practical viability.

With the discussion phase ending on August 6, we would greatly appreciate any thoughts or feedback you might have. Please feel free to let us know if there are any remaining questions we can help clarify.

Thank you for pushing us to strengthen our work through your rigorous review.

Best regards,

The authors

Dear Reviewer CmPU,

Thanks for acknowledging our rebuttal. We've addressed the experimental scope, LoRA ablation, cost analysis, and technical clarifications as requested.

Please let us know if there are any outstanding concerns, and we would be happy to discuss. We would appreciate if our efforts could be taken into consideration in your final evaluation.

Best,

Authors