Training Language Models to Generate Quality Code with Program Analysis Feedback

We sincerely thank Reviewer qzWn for their positive and thoughtful assessment. We appreciate the recognition of our paper’s clarity, the novelty of our joint optimization framework, and the significance of addressing both functionality and security in code generation.

We address the reviewer’s concerns as follows:

Response to Weakness

“Can the authors provide insight into the following observation: Based on the results, the SFT model seems to be a very strong competitor model, especially for the functionality metric on SecCodePLT+.”

This is a critical point! Indeed, the SFT model performs strongly on functionality in SecCodePLT+, even surpassing REAL. We attribute this to the relatively straightforward nature of the benchmark: SFT benefits from direct supervision using ground-truth solutions, allowing it to learn precise patterns that match test cases.

In contrast, REAL, trained via reinforcement learning, does not rely on ground-truth answers. Instead, it explores candidate solutions and learns from reward feedback. While this can lead to slightly lower functionality in simple benchmarks, it offers greater flexibility in satisfying complex and diverse constraints—such as the 17 CWEs and broader code quality goals. As shown in our results, REAL excels in security and maintainability, demonstrating its ability to go beyond pattern-matching and adapt to real-world quality requirements.

In short, while SFT is strong at mimicking known solutions, REAL is designed to generalize and optimize for broader, verifiable objectives without relying on heavy supervision.

Response to Question

“Will the authors open-source the tool they created for the detection of 17+ vulnerabilities?”

Yes, we recognize the importance of open-sourcing to support the research community. We plan to release the vulnerability detection tool after completing internal validation and adding documentation to ensure its usability and reliability. This will help promote reproducibility and enable future work on quality code generation.

Response to Limitation

“Although there seems to be an in-depth discussion on the limitations of related work, I did not find any explicit paragraph on the limitations of REAL, and also the possible negative societal impacts of their training strategy.”

Thank you for raising this important point. A clear discussion of REAL’s limitations and potential societal impacts will indeed help strengthen the paper.

• Limitations: REAL currently focuses on single-turn code generation, where the model outputs a complete solution in one step. However, many real-world scenarios involve multi-turn or interactive coding, where bugs and vulnerabilities may emerge through complex interactions across multiple code fragments. Detecting and resolving such issues is more challenging and remains an open area for extension of our framework.
• Potential Societal Impact: While REAL is designed to enhance code quality and security, it could be misused by malicious actors to generate more sophisticated or harmful code. Moreover, improved automation may reduce the need for human oversight in certain development tasks, potentially impacting entry-level programming roles. These risks underscore the importance of responsible deployment—through access control, robust evaluation safeguards, and continued human oversight in the code generation process.

I have read the author's response, and I appreciate the author's clarifications. I will keep my score unchanged.

We sincerely thank Reviewer 5a54 for the insightful review and detailed suggestions. We are grateful that the reviewer carefully checked our paper and found our method novel, scalable, and empirically grounded.

The concerns are mainly about detail clarifications.

We first answer the questions and then address the remaining concerns.

Response to Question 1

“How is the hybrid reward function (e.g., ) calibrated? I would recommend that the authors experiment with multiple values of α to assess its sensitivity.”

In our experiments we fixed α = 0.5, which delivered a strong balance of correctness and quality. To probe its sensitivity, we swept α over {0, 0.3, 0.5, 0.7, 1.0} on two benchmarks (SecCodePLT+ and SafeSQL).

As α increases, the functionality score predictably declines, but by tuning α just above or below 0.5 we can actually surpass the default’s joint “Func-Qual” performance.

(1) SecCodePLT+

(2) SafeSQL

Response to Question 2

“How did the authors validate the correctness and robustness of the custom detectors (e.g., for sanitization or code style)? Do they generalize outside the training distribution?”

We validate our detectors both in-domain and out-of-distribution:

• In-Domain Validation (Security Detector)

  Generally, we evaluate the detector using positive and negative pairs. For example, we leverage the SecCodePLT+ test set, which comprises 164 coding problems—each annotated with one “safe” and one “unsafe” solution for a specific CWE. For each detector, we compute true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), then derive precision, recall, and F1-score.

  Below is a representative subset of results, comparing our detector against two popular open-source tools (Bandit and Bearer):

  CWE ID	Metric	Bandit	Bearer	Ours
  77	Precision	0.6721	0.1961	1
  77	Recall	0.8039	0.3279	1
  77	F1-score	0.7321	0.3279	1
  770	Precision	0.5	0	0.9808
  770	Recall	0.0196	0	1
  770	F1-score	0.0377	0	0.9903

• Out‐of‐Distribution Generalization (Maintainability Detector)

  To assess generalization, we evaluate on the HumanEval benchmark, whose problem styles differ markedly from our training data (APPS+). Crucially, none of the HumanEval samples were seen during training. We measure both functional correctness and code quality to compute a hybrid “Functionality-Quality” score.

  Model	Function	Quality	Function-Quality
  Qwen-Coder-Instruct-7B	0.7927	0.2561	0.2317
  REAL-7B	0.8476	0.9573	0.8171

Response to Question 3

“Can the authors confirm if they measured or observed the RL-trained model exploiting brittle or sparse reward patterns, i.e., engaging in reward hacking?”

We explicitly monitored for reward‐hacking behaviors and confirmed that our performance gains were not due to exploiting brittle or sparse reward signals.

We did observe reward‐hacking when using only the static‐analysis signal, and here is an example code that the model generates null code to bypass the static-analysis tool:

def generate_post_html(author: str, title: str, body: str, post_tags: list[str]) -> str:
    # Implement the function to generate HTML for the post
    pass

We fixed this by adding correctness unit test and the model cannot simply generate null code to achieve a high reward. We manually checked and found the code was functional and high-quality.

Response to Question 4

“Why were strong recent baselines (e.g., DeepSeekCoder, Claude-Code, StarCoder2) excluded from comparison? Would adding them affect the paper's conclusions?”

• Our method focuses on improving LLMs' capability to generate both correct and high-quality code (security and maintainability), not chasing SOTA performance. Performance comparisons against base model and SFT show significant gains, demonstrating our method's effectiveness.
• These stronger models will not affect our conclusion — we have shown that REAL works across different model scales (0.5B, 3B, 7B), so stronger backbone models will also benefit from our training paradigm.

Response to Question 5

“Can the author report statistical confidence in the evaluation metrics (e.g., via multiple seeds, bootstrapping, or CI bars) to rule out that observed gains are not noise?”

We can confirm that the observed gains are not from randomness or data noise. Extensive experiments on three different datasets (SecCodePLT+, SafeSQL, and APPS+) and three different model scales (0.5B, 3B, and 7B) show consistently significant gains.

We find that most of the Weaknesses are already addressed in the answers above. We respond to the remaining uncovered ones as follows:

Response to Weaknesses 3 & 4

“…Failure case breakdowns are essential…” & “… best judged by humans…”

• For the failure cases, we will add more in the revision. The failure cases mostly covers (1) null code with empty implementation, (2) correct but unsafe code, and (3) incorrect and unsafe code.
• For human evaluation, we recruited two experts to do a winning-rate–based evaluation via pairwise comparisons on 50 coding problems from APPS+. For each problem they picked between the Qwen2.5-7B base output and the Qwen2.5-7B+REAL output, judging maintenance, readability, and fault tolerance (e.g., comments, type annotations, brevity, and error resilience). The REAL-trained model won 80% of the time. We will include more details in the revision.

Response to Weakness 7

“The paper treats code quality and functionality as a single metric ... Pg. 5 line 184 simply provides the interval [0,1]”

The reviewer’s comment conflates the hybrid training reward with our evaluation metrics. In fact, we evaluate functionality, quality, and their combination separately; each is reported as a pass rate and defined in the paper (Line 213, Page 6).

Response to Weakness 8

Algorithmic Contributions: Overall, it seems that REAL suffers a bit of novelty concerns …

We do not claim novelty in the general application of RL to code generation. Our contribution lies in what we optimize and how we do it.

• We are the first to leverage RL with Program Analysis feedback to train LLMs that generate code that is not only functional but also secure and maintainable—a critical yet under-addressed problem, as highlighted by Reviewers 5a54 and qzWn. While previous RL-for-code efforts focus almost exclusively on functional correctness using unit tests, we tackle code quality, including security and maintainability, which has been largely overlooked.
• Additionally, we are the first to introduce Program Analysis as a reward signal. In contrast to unit tests—which are narrow, task-specific, and hard to scale—program analysis provides a general, scalable, and task-agnostic way to detect security flaws and enforce maintainability (e.g., checking for type hints). This makes it an ideal signal for guiding RL in this domain.

Response to Weakness 9

“It is not clear to me how the paper addresses potential human biases in the annotation process …”

Thank you for pointing this out. The SafeSQL dataset was constructed through a multi-step process to reduce bias:

• Seed Tasks: Human experts wrote 10 seed tasks covering SQL injection–prone scenarios across diverse query types.
• LLM Expansion: We used GPT-4.1 to mutate these tasks (e.g., changing constraints or merging cases), followed by deduplication and expert verification.
• Cross-Verification: Multiple annotators reviewed each example, and disagreements were resolved via discussion. We also applied program analysis tools to enforce consistency.
• Functionality Tests: For each task, we generated I/O pairs from reference solutions to create unit tests.

While we did not compute IRR scores, we reduced annotation variance through multi-pass review and plan to include formal IRR in future work.

We sincerely thank the reviewer for the detailed feedback and constructive suggestions. We believe our comprehensive responses and additional experimental results address all the raised concerns. We welcome any further questions and are committed to addressing any remaining issues.

We appreciate the reviewer's consideration of our responses and hope they find our clarifications satisfactory.

We sincerely thank the reviewer for their time and thoughtful feedback. We are pleased to hear that the reviewer finds our method meaningful and appreciates the clarity of our writing. Below, we address the reviewer’s concerns in detail:

Response to Question 1

“Why does the model perform so well on the SafeSQL dataset? Does this suggest overfitting or that the task itself lacks sufficient challenge?”

• This is not overfitting, as we use separate data for training and evaluation. Moreover, during RL training, the model receives no ground-truth answers—only feedback from program analysis and unit tests.
• The strong performance likely comes from the dataset focusing on a single CWE (SQL injection), enabling the model to specialize in avoiding this vulnerability.
• Despite targeting one CWE, SafeSQL remains challenging. As shown in Table 2, untrained base models perform poorly, with joint pass rates of only 20%, 27%, and 58% for 0.5B, 3B, and 7B models, respectively.

Response to Question 2

“Why are the scores in the Quality column significantly higher than those in the other two columns?”

Functionality and quality capture different aspects of code generation and are not directly comparable. Their difference arises from how each metric is defined:

• Functionality: Measures the percentage of prompts where the generated code passes all unit tests—an all-or-nothing criterion.
• Quality: Measures the percentage of prompts passing all program analysis checks. However, this metric can be gamed—for example, by generating empty code—so it should not be used in isolation.
• Joint (Functionality & Quality): Requires passing both unit tests and analysis checks, making it the strictest metric. As a result, its score is naturally lower than either individual metric.

Response to Weakness 1

“The idea of using RL to train code models is not particularly novel.“

• We do not claim novelty in applying RL to code generation broadly. Instead, our contribution lies in what we optimize and how.
• We are the first to use RL with Program Analysis feedback to train LLMs that generate not just functional, but also secure and maintainable code—an urgent and under-explored problem, as also recognized by Reviewers 5a54 and qzWn. Prior RL-for-code work focuses primarily on functionality via unit tests. In contrast, we address quality—i.e., security and maintainability—which prior work overlooks.
• We are also the first to propose using Program Analysis as the reward signal. Unlike unit tests, which are impractical and problem-specific, program analysis offers a general, scalable, and task-agnostic mechanism to detect vulnerabilities and enforce maintainability standards (e.g., type annotations), making it uniquely suited for RL optimization in this setting.

“In particular, it remains unclear whether reinforcement learning — especially preference optimization algorithms like PPO — is truly beneficial for code-related tasks.”

• RL has demonstrated clear benefits for coding tasks: Recent models like DeepSeek-R1 [1], Kimi-K2 [2], and Qwen3-Coder [3] are all post-trained with RL (typically using unit test feedback) and show strong performance on code benchmarks. While details are not public, Anthropic’s blog suggests Claude also uses RL [4]. While the broader impact of RL on reasoning remains open, its value for math and coding tasks is well supported.
• Clarifying PPO: PPO is not inherently a "preference optimization" algorithm—it is a general-purpose RL method. In RLHF, it’s often used to optimize learned preference models, but in our case, PPO directly optimizes hybrid rewards based on program analysis and unit test feedback. This encourages the model to generate code that is not just correct, but also secure and maintainable.

[1] DeepSeek: Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[2] Kimi: Team, Kimi, et al. "Kimi K2: Open Agentic Intelligence." arXiv preprint arXiv:2507.20534 (2025).

[3] Qwen Team: "Qwen3-Coder: Agentic Coding in the World." Blog post. Qwen Blog, 2025.

[4] Anthropic Team. “Claude’s Constitution.” Anthropic News blog post. May 9, 2023. Anthropic.

“REAL does not appear to deliver significant performance improvements.”

We respectfully disagree. REAL consistently yields substantial gains over both the vanilla base model and the strongest baseline (SFT), especially when evaluated on the Joint Pass-Rate, a strict metric requiring the code to pass both unit tests and static analysis checks.

Across all datasets and model sizes, REAL improves over the vanilla model by up to +57.7%, and over SFT by up to +16.5%. Notably, for the 7B model, REAL outperforms SFT by +15.3% on SecCodePLT+, +12.9% on SafeSQL, and +15.4% on APPS+. These consistent gains across benchmarks and scales clearly demonstrate the effectiveness of our program-analysis-guided RL framework.

Response to Weakness 2

The implementation details can be found in Appendix A.2 and we add more details below to help improve the quality of our paper and we will definitely include more in the final revision.

“how was reinforcement learning applied?”

We employ Proximal Policy Optimization (PPO) on top of a pretrained code model (Qwen2.5‑Coder) using our hybrid reward. We use the VeRL framework to implement this and we mainly focus on the reward design. All the code will be released and we will also add more details in the paper.

“What data was used as seed data? ”

To curate the SafeSQL dataset, we asked human experts to write 10 database programming tasks that are potentially vulnerable to SQL injection. These tasks span diverse, realistic scenarios—including filtering, sorting, and aggregation—to ensure broad coverage of query types.

Each prompt was carefully reviewed by both human experts and a strong LLM (GPT‑4.1) to ensure clarity and solvability. For example, one task asks the model to query a books table for entries within a user-specified price range and genre, simulating a typical real-world use case prone to SQL injection risks in the code solutions.

“How were the experiments conducted?”

Our experiment setup is introduced in Section 4.1 and more implementation details are provided in Appendix A.2. Generally, we implement ReaL based on the widely used VeRL framework and conduct all experiments on an 8 x H100 GPUs server. The backbone model used in our experiments is Qwen2.5-Coder-Instruct. We will elaborate more on the experiment details in the final revision.

“How does the hyperparameter α affect the trade-off between code quality and functional correctness?”

Our experiments reveal a clear trade-off: optimizing solely for correctness or quality often hurts the other. To study this, we vary the weighting factor α in the hybrid reward and evaluate on SecCodePLT+ and SafeSQL:

SecCodePLT+

SafeSQL

As shown, low α (favoring correctness) leads to poor quality scores, while high α (favoring quality) degrades functional correctness. Both extremes result in lower joint pass rates. In contrast, moderate values (e.g., α = 0.3–0.5) strike a better balance, yielding the strongest overall performance across scales and datasets.

These findings validate our hybrid reward design, demonstrating that RL can flexibly optimize different objectives through careful tuning of α. In our main results, we use α = 0.5 as a balanced default, but the framework remains adaptable to varied priorities.

We hope this analysis addresses the reviewer’s question. We’re happy to clarify further if needed.

We thank Reviewer sExN for their thorough review and insightful comments. We are pleased that the reviewer recognized ReaL’s well-structured experiment settings and superior performance over baselines. We address the reviewer’s concerns as follows:

Response to Weakness 1

“More implementations details are needed for better clarity.”

We provide additional technical details to help the reviewer better understand our work. In the REAL framework, we fine‑tune Qwen2.5‑Coder‑Instruct using Proximal Policy Optimization (PPO) with a hybrid reward that balances functional correctness and code quality (security and maintainability). At each training step, the model generates candidate solutions for coding prompts, evaluates them using unit‑test pass rates and static analyzers, and combines these scores into a normalized hybrid reward. The static analyzer leverages control‑flow graphs and taint analysis to detect unsafe data flows from user input to execution. Based on the reward signals, we compute advantages using Generalized Advantage Estimation (GAE) and update the policy with PPO’s clipped objective, augmented by KL‑penalty and entropy regularization to ensure stable learning and maintain exploration. This framework allows REAL to improve steadily using scalable and verifiable feedback.

Response to Weakness 2

“Please make it more clear that as benchmarked on 3 curated datasets as SecCodePLT+, SafeSQL and APPS+, proposed method outperforming because of these specifically curated dataset or for employing a RL-based framework or combining both. Generalization over real-world raw dataset should be explored.”

We clarify that the datasets used in this paper are not specifically curated for our method, but are used to establish a new experimental setting and to train and evaluate all baseline methods, ensuring a fair comparison. The original SecCodePLT and APPS datasets remain unchanged; we only add new verifiers (security and maintainability) as extra evaluation dimensions. The SafeSQL dataset was curated solely to study SQL development due to the lack of existing datasets. And it is also used to train the SFT baseline. Thus, REAL’s superior performance stems from our RL‑based framework with hybrid reward mechanisms, not from data differences.

To further validate generalization, we trained REAL on APPS+ and evaluated it on the unseen HumanEval [1] benchmark. The result is as follows. REAL‑7B achieves substantial improvements across all metrics despite not being trained on HumanEval, demonstrating strong generalization.

[1] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

Response to Weakness 3

“Authors should also points towards current limitations and potential breakpoints of the REAL framework (if any), particularly in the context of real-world quality code generation applications”

We appreciate the reviewer’s request. We would like to highlight our novelty in our focus on the quality code and the use of program analysis in reward mechanism, and discuss current limitations of the REAL framework.

• Focus on quality code: REAL is the first to apply reinforcement learning to optimize for code quality—specifically security and maintainability—rather than just functional correctness. This is increasingly critical as coding assistants are integrated into real-world workflows. Prior RL efforts mainly focus on solving tasks via unit tests, overlooking real risks like unsafe or unmaintainable code. Our work addresses this important and under-explored problem.
• Use of program analysis in reward: Unlike prior works that rely on unit tests as rewards—which are task-specific and insufficient for quality aspects—we leverage program analysis (e.g., taint analysis and control-flow graphs) as a general-purpose reward signal. This enables scalable and automated evaluation of vulnerabilities and maintainability across diverse prompts, without the need for handcrafted test cases.
• Limitations and future directions: Our current focus is on single-turn quality code generation. We acknowledge that many real-world applications involve multi-turn interaction and iterative refinement. Extending REAL to such settings—e.g., through agent-based frameworks that incorporate feedback loops—is a key direction for future work.

Response to Question 1

“Instead of curated datasets, what if we evaluate on previous SecCodePLT [Yang e al., 2024] and APPS [Hendrycks et al., 2021] without any preprocessing. Can you please demonstrate similar benchmarking as Table 3-5?”

We’d like to clarify that we did not curate or preprocess new data for the SecCodePLT and APPS benchmarks. Our updated versions (SecCodePLT+ and APPS+) use exactly the same data—we simply add static analysis tools as an additional evaluation metric to assess code quality (security and maintainability), which was not covered in the original benchmarks that only used unit tests.

Dataset statistics are shown below:

“Can REAL, as a novel stand alone RL framework, applicable for any unknown real world raw datasets (generalizability) towards satisfying maintainability and vulnerability check (can we expect self-automated)?”

Yes, this is a core strength of the REAL framework. By integrating program analysis tools as reward functions, REAL can generalize to unseen, real-world datasets without manual curation. To demonstrate this, we trained on APPS+ and evaluated on HumanEval—a benchmark with different styles and no overlap in training data. Despite no exposure to HumanEval, REAL significantly improves code quality:

[1] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

Response to Question 2

“Authors mentioned as "with minimal manual intervention", however used 3 curated datasets for evaluation. How minimal was this "curation" as "by extending existing datasets or evolving data with LLM"? Is there any bias introduced due to these curated datasets in training? Can it anyhow misguide the RL agent in specific scenario, for complex logic codes especially in dynamic contexts?”

We’d like to clarify that “minimal manual intervention” refers to the use of program analysis tools as automated reward functions in REAL. These tools significantly reduce human effort in detecting vulnerabilities in generated code, especially when compared to manually crafting case-specific unit tests.

For example, given 1,000 SQL programs, a single static analyzer can check for SQL injection vulnerabilities—eliminating the need to manually write and annotate 1,000 individual test cases.

As for the datasets mentioned in the question, we’d like to clarify that the three mentioned datasets were not curated specifically for training REAL, but rather to establish a sound experimental setting for evaluating security and maintainability—both for our method and future work. We will include these clarification in our future version to make this point clearer.

Response to Question 3

“Program analysis-based feedback might not always provide explainable rationales for the improvements or rejections. Any concerns there?”

In REAL, we use program analysis–based feedback as part of the reward function, represented as a binary signal (1 for secure, 0 for vulnerable). While this feedback is not explicitly explainable, our experiments show that it is sufficient to guide learning and significantly improve code quality.

This practice aligns with standard approaches in the RL with verifiable rewards (RLVR) literature [1,2,3,4], where binary or scalar reward signals are commonly used. The advantage estimation mechanism in PPO effectively captures reward differences between successful and unsuccessful outputs, enabling the policy to learn the desired behaviors—even in the absence of detailed rationale.

[1] AI2: Lambert, Nathan, et al. "Tulu 3: Pushing frontiers in open language model post-training." arXiv preprint arXiv:2411.15124 (2024).

[2] DeepSeek: Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[3] Microsoft Research: Wen, Xumeng, et al. "Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs." arXiv preprint arXiv:2506.14245 (2025).

[4] Meta AI: Gehring, Jonas, et al. "Rlef: Grounding code llms in execution feedback with reinforcement learning." arXiv preprint arXiv:2410.02089 (2024).