Thank you for your thoughtful feedback! We appreciate your recognition of our paper’s strengths, especially the novel combination of supervised fine-tuning and reinforcement learning for training critics and the robust experimental design supporting our claims. Below, we address your specific concerns in detail:

Scalability to broader tasks beyond code generation:

We agree that demonstrating generalization beyond code generation would strengthen our paper. We have conducted additional experiments on IFEval [1], a benchmark commonly used for evaluating alignment capabilities [2]. Our results show that, while not explicitly trained on instruction-following tasks, CTRL improves performance on alignment tasks through iterative refinement, demonstrating our approach can generalize to tasks beyond code generation. We will incorporate these findings in our final version to better illustrate CTRL's broader applicability.

[1] Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models

[2] Yang A, Yang B, Zhang B, et al. Qwen2. 5 technical report

Sandbox reliance of our method:

While we leverage sandbox execution during training, CTRL can be adapted to tasks without such verification tools through 1) learned reward models for tasks like safety; and 2) reference-based evaluation for tasks with reference responses (e.g., translation) using metrics like ROUGE or BLEU.

In fact, our framework is agnostic to the specific reward mechanism as long as it can distinguish between successful and unsuccessful revisions.

Ablation of components (e.g., SFT vs. RL):

Tables 1 and 2 provide an initial component analysis comparing CTRL-SFT and full CTRL performance. Together with our additional experimental results on Pass@5 performance in the anonymous link, Table 2, we observe that:

• SFT improves discrimination (F1: 61.19% → 68.55%) and establishes the critique format
• RL significantly improves single-turn critique-revision (Pass@1: 8.36% → 11.76%) while reducing regression ($\Delta_\downarrow$: 3.03% → 0.85%)
• For multi-sample scenarios, SFT shows larger gains in Pass@5, while RL improves single-sample effectiveness

This demonstrates that while SFT provides initial improvement in critique format and discrimination, RL training is crucial for generating feedback that leads to successful revisions.

Essential references not discussed:

We thank the reviewer for suggesting highly relevant papers we overlooked. We will include and discuss these works in our final version:

• While CodeDPO improves code generation through DPO with self-generated validation, our work differs by specifically training a critic model to provide human-readable feedback. Besides, CodeDPO relies on larger models for data generation, whereas our approach requires only self-generated critiques for critic training.
• MultiCritique shares our goal of training critic models but utilizes a multi-agent framework with GPT-4 for meta-critique. In contrast, CTRL achieves strong performance without requiring access to more powerful models during training. Besides, our method enjoys the simplicity of having one critic model instead of four different models.
• While RLEF uses execution feedback to improve generator models directly, we focus on enhancing the feedback capability of LLMs. In this regard, our approaches are complementary - RLEF trains better generators, while CTRL trains better critics.

Contributions relative to recent advancements in generative reward models and self-correction:

We discuss our position relative to generative reward models in Appendix B, Table 6. Specifically, our approach uniquely unifies discrimination and refinement by generating actionable critiques without direct human supervision. Our work differs from recent self-correction approaches:

• SCoRe [3] employs multi-turn online RL to improve self-correction, but focuses on the generator directly correcting its outputs. CTRL decouples critique and generation, allowing specialized training of the critic and demonstrating signs of scalable oversight.

• GLoRe [4] decomposes refinement into when, where, and how to refine. However, they focus on training both global and local refinement models separately, while CTRL trains a specialized critic model that can identify errors and provide actionable feedback to any generator.

[3] Kumar A, Zhuang V, Agarwal R, et al. Training language models to self-correct via reinforcement learning

[4] Havrilla A, Raparthy S, Nalmpantis C, et al. Glore: When, where, and how to improve llm reasoning via global and local refinements

We again appreciate your thoughtful review and will incorporate your suggestions to strengthen the final version of our paper.

Thank you for your comprehensive and thoughtful review! We appreciate your recognition of our work's solid mathematical and theoretical foundation and adequate experimental validation. We would like to address your concerns in detail:

Theoretical analysis:

Our work is primarily empirical. While we use mathematical frameworks to motivate our approach, our main contributions are demonstrating a practical framework for critic training and empirically validating its effectiveness.

To strengthen our theoretical analyses, we discuss more in detail:

• Markov chain analysis: We provide theoretical analyses of Markov chain convergence properties in the anonymous link complementing Figure 3's empirical findings.
• GRPO properties: We show that GRPO achieves variance reduction by a factor of $G$, explaining its stability with large critique spaces. Our analysis also demonstrates that GRPO ensures monotonic improvement and converges to a local optimum under standard assumptions.
• Weak-to-strong generalization: Our findings that weaker critics can guide stronger generators align with recent work (Burns et al., 2023; Kenton et al., 2024), though theoretical conditions for when this works remain an open research question.

CTRL does reach the highest level of innovation:

Thank you for recognizing our novel, mathematically-based, empirically validated method. Our primary contributions include:

• Problem formulation: We identify and formalize the critique generation problem through a Markov chain lens, distinguishing between discrimination and critiquing abilities.
• System design: We propose a decoupled critic-generator architecture and a two-stage training pipeline specifically designed for critique learning.
• Robust training: We demonstrate GRPO's suitability for critique training due to reduced variance in policy gradients, and demonstrate its effectiveness empirically.
• Empirical findings: We provide extensive empirical evidence demonstrating that our framework enables test-time scaling, mitigates compounding errors, and facilitates weak-to-strong generalization.

Applying CTRL to domains where feedback is less binary:

For domains where feedback is less binary, CTRL can leverage alternative reward mechanisms:

• Learned reward models: Training a reward model from pairwise preference data (as in RLHF approaches) for tasks like safety or preference alignment.
• Reference-based evaluation: For tasks with reference responses (e.g., translation, summarization), we can use metrics like log-likelihood on reference outputs or automated evaluation metrics (e.g., ROUGE and BLEU) as reward signals.

CTRL is agnostic to the specific reward mechanism as long as it can distinguish between successful and unsuccessful revisions. This flexibility extends CTRL beyond domains with clear verification tools.

Notably, we observe positive generalization results on IFEval, a benchmark outside our training distribution, as shown in our anonymous link.

Sensitivity to GRPO parameters:

We relied on prior work in RL training that identified the most sensitive parameters for training stability. Due to computation constraints, we performed a limited grid search (<10 runs), with final parameter selection based on performance on a TACO test subset. We found our method relatively robust to GRPO hyperparameters within reasonable ranges:

• Group size: Larger sizes improve performance (we used 8) but increase computation time
• KL coefficient: While unnecessary, KL improves stability and KL=0.001 balances exploration and stability in our experiments.
• Learning rate: Most sensitive parameter; we found 1e-5 to be optimal via the grid search.

Qualitative assessment of critiques:

To better assess the quality of the generated critiques, we manually evaluated critiques on 50 randomly sampled critiques from CTRL against Qwen2.5-Coder's solutions on CodeContests. Our analysis revealed CTRL employs diverse debugging strategies: algorithm improvement (43 instances), static analysis (38), strategic debugging (11), and dynamic testing (10).

This distribution reveals that CTRL primarily focuses on structural code improvements and algorithmic enhancements rather than superficial issues. The prevalence of static analysis and refactoring strategies also suggests CTRL learns debugging approaches that align with human reasoning patterns.

Robustness to noisy feedback:

While we didn't explicitly test noisy feedback scenarios, our low regression rates ($\Delta_\downarrow$) demonstrate CTRL can distinguish correct implementations from problematic code, avoiding misleading critiques that worsen solutions. This inherently suggests robustness to noise, though we agree more systematic evaluation of noise tolerance is an important direction for future work.

Thank you again for your detailed review. We hope our response addresses your concerns and questions.

Thank you for your detailed review and valuable feedback! We appreciate your recognition that our main claim is well supported by the experiments with a number of interesting analyses and that the paper very clearly motivates its approach by citing relevant related work. We address your primary concerns below:

Proof: Variance of the gradient:

We provide a short proof in our anonymous link that justifies our claim about gradient variance scaling with $|\mathcal{Y}| \cdot |\mathcal{C}|$. The key insights from this proof are:

• When analyzing the variance of our policy gradient estimator, we find that under mild assumptions it scales proportionally with the product of answer and critique space sizes ($|\mathcal{Y}| \cdot |\mathcal{C}|$).
• In cases where covariance terms are significant, the scaling becomes even worse than linear, potentially leading to a super-linear increase in the overall variance.

This theoretical analysis provides a mathematical foundation for why standard policy gradient approaches struggle with critique-revision optimization, directly motivating our CTRL design choices that address this variance challenge. We will incorporate the proof in our final version.

Dataset overlap between TACO and CodeContests:

Thank you for raising this question! We agree that while we discuss this issue in the paper and propose a decontamination method to mitigate this, it deserves more attention. We would like to clarify a few points:

• Actual overlap: While the TACO paper mentions a significant overlap with CodeContests overall, the critical distinction is that we evaluate on CodeContests test set (165 problems), while the reported overlap primarily exists between TACO and CodeContests full set. We excluded the 47 problems we identified as direct overlaps between the TACO train set and the CodeContests test set.

• Performance on out-of-distribution benchmarks: We would like to highlight that our method demonstrates significant improvement over the zero-shot baseline on LiveCodeBench with Qwen2.5-Coder as the generator model (30.54% -> 33.21% in Pass@1), outperforming GPT-4o and Qwen2.5-Coder as critic models. This demonstrates that our method does learn useful critiquing abilities that transfer beyond its training distribution.

• Additional experiment: To further validate CTRL's generalizability, we have conducted additional experiments on IFEval [1], a benchmark for instruction-following capabilities. The results are presented in the anonymous link, Table 1. Despite not being trained on instruction-following data, CTRL boosts accuracy by over 2% for single-turn critique-revision with Qwen2.5-Coder as the generator, outperforming all the baselines including GPT-4o.

We agree that scaling up the data size and diversity is a promising direction to further improve the results. and consider this an important direction for future work, which we will highlight in the final version.

[1] Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models[J]. arXiv preprint arXiv:2311.07911, 2023.

Typos and suggestions: Thank you for your careful review! We appreciate your suggestion about the naming convention, and will rename $Q$ and fix the spotted typos in the final version.

Thank you again for your thorough review. Your feedback has helped us strengthen our work and identify important areas for improvement.

Thank you for your comprehensive and well-thought-out review! We appreciate your recognition of our work's novelty, effectiveness, experimental design, clear motivation, and solid supplementary material. Your positive feedback is truly encouraging.

Below, we address your concerns about computational costs and system complexity:

Cost for majority voting in JudgeBench evaluation:

The base critic used for majority voting is our CTRL-trained Qwen2.5-Coder-32B model, the one used throughout our evaluation. The majority voting approach is primarily used for our JudgeBench evaluation (Section 4.3) to calculate pairwise accuracy.

While generating multiple critiques may seem costly, modern inference engines significantly reduce this overhead through prefix caching (i.e., the same prompt is prefilled for generating multiple critiques). This majority voting approach is also used in recent work on generative verifiers [Zhang et al., 2024], which observe similar results to ours that majority voting boosts accuracy significantly.

Our experiments show that performance plateaus after approximately $2^6$ critiques, with additional samples providing diminishing returns (64.3 at Maj@$2^7$ vs 64.0 at Maj@$2^6$). For cost comparison with Claude 3.5 Sonnet (which performs similarly on JudgeBench), we calculate per-problem inference costs using Together AI pricing for Qwen2.5-Coder-32B and Anthropic pricing for Claude 3.5 Sonnet:

CTRL with Maj@$2^6$:

• Average prompt length: 2,067.17 tokens
• Average response length per critique: 231.72 tokens
• Input/output cost: $0.80/1M tokens
• Cost: (2067.17 + 231.72 * 64) * 2 * 0.8 / 1M = $0.027 per problem

Claude 3.5 Sonnet:

• Average prompt length: 1,627.74 tokens
• Average response length: 652.97 tokens
• Input cost: $3/1M tokens
• Output cost: $15/1M tokens
• Cost: 1627.74 * 2 * 3 / 1M + 652.97 * 2 * 15 / 1M = $0.029 per problem

Note that the factor of 2 accounts for calculation on both responses in CTRL and removing positional bias for Claude 3.5 Sonnet.

This analysis demonstrates that our method achieves comparable performance to state-of-the-art proprietary models at a similar or slightly lower cost.

Cost for fine-tuning:

We acknowledge that RL training is computationally intensive, especially for large models. To manage this, we implemented several optimizations:

• SFT stage: Our SFT on self-critique with execution feedback provides a strong initialization, significantly reducing the need for extensive RL exploration.
• Optimization techniques: We leveraged fully sharded data parallelism (FSDP), gradient checkpointing, and sequence packing to reduce memory requirements and speed up training. At each step the critic model and the generator model are offloaded to CPU to save memory, allowing us to train 32B models with a minimum of 8 x 80G GPUs.

As open-source RL training frameworks [1,2] continue to evolve, we expect these costs to decrease further in the future.

[1] Sheng G, Zhang C, Ye Z, et al. Hybridflow: A flexible and efficient rlhf framework[J]. arXiv preprint arXiv:2409.19256, 2024.

[2] von Werra L, Belkada Y, Tunstall L, et al. TRL: Transformer Reinforcement Learning[CP/OL]. GitHub repository. GitHub, 2020. https://github.com/huggingface/trl.

Clarification on "multiple critics":

We want to clarify that our approach uses only a single critic model during training, not multiple critics as may have been implied. We sample multiple critiques from this single model to estimate advantages in GRPO.

Hyperparameter tuning:

For hyperparameter selection, we relied on prior work in RL training that identified the most sensitive parameters for training stability: learning rate, KL coefficient, and group size for GRPO. Due to computation constraints, we performed a very limited grid search over these key parameters (< 10 runs), with final parameter selection based on performance on a TACO test subset. We report the final hyperparameters in Table 9.

We agree that more extensive hyperparameter search could potentially improve results. We will improve the final version of our paper with a more detailed description of our grid search process.

Sandbox outputs as correctness signal:

Regarding using sandbox outputs as correctness signals, we would like to clarify two contexts of “reward calculation”:

• During CTRL training: We do use sandbox outputs (pass/fail) as the reward signal to train our critic model, as described in Section 3.1.
• For JudgeBench evaluation: High-quality unit tests are not available for many tasks, especially general-domain questions. This practical limitation is precisely why CTRL is valuable - it internalizes sandbox signals during training, allowing critic models to generalize to new domains where explicit test cases are not available.

Thank you again for your insightful review and positive assessment of our work.