Rethinking Verification for LLM Code Generation: From Generation to Testing

Dear Reviewer 2jtr,

We thank you for your feedback, and especially for recognizing the importance and relevance of the question we address, as well as the potential path and value our work offers for building better verifiers. Your comments have helped us identify several areas where the presentation clarity and impact of our paper can be substantially improved. We assure you that we have taken all your points seriously and have prepared the following clarifications, which we hope will address your concerns. We sincerely thank you for the time and effort you have invested in reviewing our manuscript.

We will address the weaknesses you pointed out one by one:

Weakness 1: "no discussion of related work"

Thank you for pointing out this issue. We would like to clarify right away: we do have a very comprehensive related work section, located in Appendix A, and we commit to moving a condensed summary into the main body in the final version, adopting a "main text summary + appendix details" structure.

The reason for this initial arrangement was that the main text space was insufficient to accommodate the full version. In order not to sacrifice the depth and completeness of the review—which we feel is critical for readers to fully understand the landscape—we made the difficult decision to place the full discussion in Appendix A. This section (A.1 and A.2) covers advancements in LLM-based code generation, various evaluation benchmarks (like HumanEval, EvalPlus), and both traditional (SBST, Fuzzing) and LLM-based TCG methodologies. We sincerely apologize that this information was not easily discoverable. In the final version of the paper, we will move a condensed yet thorough summary of this related work section into the main body to ensure all readers can understand how our work is situated within the broader academic context. Furthermore, we will cite the latest research in the updated version and sincerely welcome any relevant references you feel we may have overlooked. We will incorporate them into our final revision.

Weakness 2: Clarity of Section 2 and its metrics

We appreciate your feedback on the clarity of Section 2. Our initial intention was to build a comprehensive measurement framework, and we now recognize that our initial explanation was too dense. The four metrics (DR, VAcc, DEPC, AUC-AccN) are designed to assess test suite quality from multiple perspectives.

• DR and VAcc measure effectiveness (Can it find bugs?).
• DEPC and AUC-AccN measure intrinsic quality and efficiency (How diverse and potent are the tests?). The "error pattern vector" is a practical method for defining diversity: it is a vector that represents which incorrect programs a single test case can detect a bug in. We have provided more detailed descriptions of all these metrics, their formulas, and their interpretations in Appendix B. Based on your feedback, we will revise Section 2 in the main text to provide more intuitive explanations and examples, referencing the clearer descriptions in the appendix.
• The practical value of these metrics is demonstrated throughout the paper:
  • In Figure 6, we use them to precisely show why SAGA is superior to the baselines (as it achieves higher DEPC and AUC-AccN).
  • Appendix B provides their full mathematical derivations and practical interpretations.

Weakness 3: The logical link between Section 2.2 and Section 3

The assertion at the beginning of Section 3 that "Section 2.2 revealed limitations" is directly supported by the experimental results presented in that very section, specifically Figure 3 and Figure 4.

• Figure 3 shows the low quality of directly generated tests (low DR and VAcc).
• Figure 4 shows the homogeneity of tests from the Input-Interpreter paradigm, evidenced by detection rate saturation, which validates the theoretical upper bound for the most commonly used paradigm today. These results demonstrate what we refer to as "low test quality and homogeneity." We will rewrite the opening of Section 3 to explicitly reference these figures and findings, thereby establishing a clearer motivation for SAGA for the reader.

Weakness 4: Lack of detail on the SAGA method

Thank you for the emphasis. The power of SAGA lies in its structured, human-insight-driven process. The core mechanism of SAGA operates through detailed, structured prompts, which we have open-sourced in the supplementary materials (with full prompts for both Multidimensional and Differential Analysis). These prompts are effectively the "source code" of our method. To make it completely transparent and reproducible, we will add a new, detailed running example to the appendix of the final paper. This example will demonstrate:

1. How a problem description, a correct human solution, and a buggy human solution are fed into the SAGA prompts.
2. The LLM's structured output, including the constraint_analysis and the generated Case Scripts.
3. How these scripts are executed to produce the final test cases. This will provide the concrete, reproducible detail that you rightly pointed out was missing.

Limitations:

Thank you for your attention to this. A dedicated and detailed discussion of limitations and future work is located in Appendix J. We discuss the scope of human priors, generalization, computational cost, and other important factors. We recognize the importance of highlighting these limitations and will add a summary of this section to the conclusion in the main body to ensure this crucial context is not missed by readers.

We are confident that these clarifications and the planned revisions, directly guided by your valuable feedback, will address the issues you have raised and significantly strengthen our paper. Thank you again for your rigorous review and for helping us improve our work.

Best regards,

Authors of Submission 17344

Dear Reviewer 2jtr,

Thank you again for your invaluable guidance. Following your suggestion, we have posted a detailed response with a concrete example for our metrics.

We are very keen to ensure all your concerns are fully addressed. Please let us know if any part of our explanation remains unclear, as we would be grateful for the opportunity to provide further clarification.

Best regards

Thank you for your efforts and reviews. With the discussion now drawing to a close, we would be grateful to know whether our response has fully addressed your concerns.

If there are any further questions or areas you feel need additional clarification, please do not hesitate to let us know. We are more than happy to continue the discussion to address them promptly.

Conversely, if you feel that all your concerns have now been clarified, we would sincerely appreciate it if you would consider updating your evaluation to reflect the progress made.

Thank you again for your time and engagement throughout this process.

Best regards,

Authors of Submission 17344

Dear Reviewer Y2Dv,

We sincerely thank you for your thoughtful and constructive review. We are very encouraged that you recognize the importance of the problem we address, the theoretical novelty of the SAGA framework, the contribution of our new dataset, and the comprehensiveness of our experiments. Your insights on practical issues such as cost, generalization, and human effort are highly perceptive and align perfectly with our own considerations during this research. We greatly appreciate this opportunity to elaborate on how we have addressed these key points.

Below, we respond in detail to the weaknesses you identified:

1. On the Efficiency and Cost of SAGA: This is a critical practical concern, and thank you for raising it. We would like to clarify the operational flow of SAGA, which is designed to be more efficient than it might initially appear. The main cost of SAGA lies in its analysis phase. The LLM performs a single, powerful, and focused analysis of human code (correct and/or incorrect solutions) to generate a Python Case Script. This script is a standalone program that contains not only the logic for generating test inputs but also the Math Explanations and Self-Validation code.

Crucially, it is this generated Python script—not the LLM—that then programmatically generates a large volume of test inputs. This design avoids the high cost of repeatedly invoking the LLM for every single test case.

Furthermore, we completely agree with your concern about democratizing advanced TCG methods. This is precisely why we developed TCGCoder-7B. Our goal was to demonstrate that SAGA's complex reasoning capabilities can be distilled into a smaller, more efficient model. As our results in Table 2 show, TCGCoder-7B's performance significantly surpasses baseline methods that use much larger models. This proves that with specialized training, a small-sized model can provide top-tier TCG capabilities, allowing it to be easily deployed in most research environments and thus greatly lowering the computational barrier.

2. On Dataset Selection, Human Effort, and Generalization: You noted that our evaluations are based solely on competition-level datasets. This is correct, and we want to emphasize that this was a deliberate and strategic choice, not a limitation.

• High-Quality Ground Truth: In the task of Test Case Generation (TCG), the most critical challenge is evaluating the quality of the generated test cases themselves. Competitive programming platforms provide a massive source of near-perfect "ground truth" solutions, validated by millions of submissions. This is a level of reliability that no other source (like GitHub) can currently match. Without this reliable anchor, our research could not have conducted rigorous, reproducible, and quantitative evaluations.
• Rich Failure Modes: Our SAGA framework, especially its differential analysis, heavily relies on learning from diverse human errors. These platforms offer hundreds or thousands of real, labeled "wrong submissions" for each problem. This invaluable "database of failures" is an indispensable resource for training and evaluating our methodology, as it reveals the subtle mistakes human programmers make when facing complex logic.

On Generalization to Real-World Software: We completely agree that the ultimate goal is to generalize to real-world software like GitHub projects. However, we must recognize that this is a "walk before you run" process. The entire field is still in its very early stages when it comes to handling complex, multi-file, dependency-heavy real-world codebases (as evidenced by the struggles on benchmarks like BigCodeBench).

Therefore, our strategy is: First, master the art of generating high-quality tests in a "sandbox" that, while domain-specific, is extremely logically complex and has a perfect evaluation environment. We believe that by validating SAGA's effectiveness on competitive code, we are building the most solid methodological foundation for tackling the larger, messier real-world problems in the future. Our work demonstrates a clear and significant advantage in this critical domain, which is a key step toward that ultimate goal.

3. On the Train/Test Split for TCGCoder-7B: Thank you for raising this critical point for clarification. We assure you that we took meticulous measures to prevent data leakage and ensure a fair evaluation.

Our process is as follows:

• A Strict Chronological Split: The training data for TCGCoder-7B is sourced entirely from earlier problems on Codeforces and Nowcoder, the vast majority of which are from before 2023. Our evaluation set, TCGBench-Lite, on the other hand, consists exclusively of contest problems from June 2024 onwards. This strict temporal separation ensures that our model is evaluated on unseen problems, guaranteeing the impartiality of our results.
• Proactive Semantic Deduplication: We went a step further by using a SentenceTransformer model to deduplicate all problem descriptions, filtering out any pairs with a cosine similarity greater than 0.9 to ensure problem independence. Our recent manual checks revealed that even for these highly similar problems, their core solution logic and patterns are entirely different.

We hope these detailed clarifications fully address your concerns and demonstrate the rigor of our experimental design. Your practical and insightful questions have been invaluable in helping us to better articulate the contributions and context of our work. Thank you again for your diligent effort and feedback.

Best regards,

Authors of Submission 17344

Dear Reviewer p61z,

We sincerely thank you for your review, and we are especially grateful for your recognition of our in-depth analysis of the deficiencies in existing benchmarks and the innovative metrics we have proposed for evaluating test effectiveness.

Regarding Weaknesses and Reasoning Models (Your Weakness #2, Question #8):

This is an excellent point. We want to emphasize that the SAGA framework itself is fundamentally designed to guide LLMs in structured reasoning. In our main experiments, we chose DeepSeek-V3 as the backbone model based on a comprehensive consideration of performance, efficiency, and cost, which makes our method more easily reproducible. By compelling the LLM to deconstruct human defensive code patterns and analyze specific failure modes, SAGA employs a guided, programmatic Chain-of-Thought process, prompting the model to "think through various corner cases." The detailed prompts we have open-sourced serve as direct evidence of this reasoning-driven process.

To directly address your concerns about reasoning models, we conducted new experiments with a state-of-the-art reasoning model. The results on Qwen3-235B-A22B are shown in the table below:

We observed that the performance with the reasoning model actually decreased. After manual investigation, we found this was not because of a lack of reasoning ability, but due to practical engineering challenges—the model's inference sequence length exceeded the limits of our deployed model (our Qwen3 deployment has a sequence limit of 64k), causing some difficult problems to not be parsed successfully. This also indicates that using such a large reasoning model significantly increases inference and time costs, with limited actual performance improvement. This result highlights our rationale for choosing DeepSeek-V3: it is more accessible and reproducible for the community. We will add this experiment as a new section in the appendix.

On Biased Test Generation (Your Question #1):

Thank you for requesting specific examples. An illustration can be found in Figure 1(b) of our paper. The PCA analysis shows that errors caused by LLMs are highly clustered, indicating a systematic bias—they tend to make mistakes on similar types of problems. In stark contrast, human errors are diverse and widely distributed. This means that if test suites are generated without reference to human error patterns, the scores of LLM-generated code could be artificially inflated. This directly explains a key finding in our work: when we re-evaluated code that passed on LiveCodeBench on a real online judge platform, we found its verifier accuracy to be alarmingly low. This is precisely because the tests in LiveCodeBench, lacking insights from diverse human failure modes, missed bugs that a more robust verifier should have caught.

On the Clarity of Definitions (Your Question #2):

Thank you very much for pointing out where a clearer explanation was needed.

Let us re-clarify these two core metrics:

• Detection Rate, es(T): This is a solution-level metric that measures, "Can our test suite T find at least one bug for this specific buggy program?" Mathematically, it is the probability of the union of events. You correctly identified Ei as the event that "solution S fails on test case i." Therefore, es(T) is the probability that at least one of these failure events E1, E2, ..., En occurs, which is P(E1 ∪ E2 ∪ ... ∪ En). Our use of 1 - P(∩Ei_bar) in the paper was a way to calculate this union probability using its complement, where Ei_bar is the event that "solution S passes on test case i." We acknowledge this notation may be unintuitive and will revise the final version to explain it directly with simpler language and the P(∪Ei) form.
• Verifier Accuracy, VAcc(T): This is a stricter, problem-level metric that asks, "For a given problem, can our test suite T find bugs in all known incorrect solutions?" It evaluates the systematic completeness of the verifier. VAcc(T) equals 1 only if the Detection Rate is greater than 0 for every single incorrect solution for that problem, making it the gold standard for measuring verifier robustness.

We will revise this section with clearer language and examples in the final version to ensure the logic and distinction between these two definitions are perfectly clear.

On the Relationship Between Detection Rate and Test Scale for Paradigm 1 (Your Question #3):

We focused the "Detection Rate vs. Test Scale" experiment on Paradigm 2 due to the intrinsic differences between the two paradigms. The core advantage of Paradigm 2 (Input-Interpreter) is its ability to easily generate a massive and diverse set of test inputs. In contrast, Paradigm 1 (Direct Generation) struggles to produce a large number of unique and complete test cases (including both input and output), making it less suitable for an analysis that requires varying the test scale from 1 to 100. This is also why in our main experiments comparing different methods, we report performance at a fixed scale (e.g., @20 and @50).

On the Meaning of DEPC's Error Pattern Vector (Your Question #4):

The error pattern vector $v(t_k)$ for a test case $t_k$ is a binary vector where each element corresponds to one of the $N_p$ incorrect programs in our benchmark. If $t_k$ causes the j-th incorrect program to fail, the j-th element of the vector is 1; otherwise, it is 0. DEPC then counts only the number of unique non-zero vectors in the entire test suite. This allows us to see precisely which set of incorrect programs each test case exposes. If two test cases have identical error pattern vectors, it means they are functionally redundant.

On How Human Priors are Incorporated (Your Question #5):

This is detailed in our SAGA framework. Human priors ($S_{human}$) are insights derived from correct human-written code. We feed these correct solutions, along with the problem description, into our Multidimensional Analysis prompt (available in our anonymous code repository). The LLM is then tasked with reverse-engineering the defensive logic (e.g., boundary checks, special value handling) from this human code. These machine-extracted strategies are the "human priors" that guide the generation of new, more challenging test cases.

On the Workflow of SAGA (Your Question #6):

For Multidimensional Analysis, we provide correct solutions to give the LLM a broad perspective on robust coding patterns. For Differential Analysis, we provide a correct solution and its most recent preceding incorrect submission to help the LLM pinpoint specific failure modes. Decomposing strategies into formal constraints is a core task specified in our prompts; solutions are not input all at once but sequentially. We understand this part is complex, so we will add a detailed case study in the appendix of the final version to walk readers through the entire process with a concrete example.

On Extending SAGA to Cases Without a Ground Truth Solution (Your Question #7):

This is a key question about the scope of SAGA's application. SAGA currently requires a ground truth solution to act as an infallible interpreter. This is a limitation we acknowledge in Appendix J. Extending SAGA to settings without an oracle (e.g., through majority voting or by learning a verifier model) is an important and exciting direction for future work.

Once again, we express our deepest gratitude for your valuable feedback. Your guidance has been crucial, and we are confident that with your insights, the revised paper will be much stronger and clearer.

Best regards,

Authors of Submission 17344

Dear Reviewer p61z,

Thank you so much for taking the time to review our rebuttal and for providing your final thoughts. We are very grateful for your positive evaluation and for the insightful discussion throughout this process. Your feedback has been invaluable to us.

We would like to offer a final clarification on the two points you raised:

1. On the performance of the reasoning model: We completely understand why the result showing a reasoning model performing worse is counter-intuitive. We want to clarify that this is not due to a deficiency in the model's reasoning capabilities, but rather a practical engineering and resource constraint on our end.

   The "thinking" mode of models like Qwen3 generates extremely long chains of thought. Due to computational resource limitations, the maximum context length of our deployed model was configured to 64k. For many of the harder problems, the model's full reasoning process was still too long and resulted in truncated outputs. This led to incomplete or failed script generation for those specific hard cases, which in turn lowered its overall score.

   The reason we chose Qwen3 over DeepSeek-R1 was also based on resource constraints; our available computational resources were best suited for deploying Qwen3-235B. We believe that with a longer context length and more computational resources, a powerful reasoning model would indeed show its true potential. We will add a note to the appendix to clarify this important context behind the experimental results.

2. On the limitation of requiring a golden solution: We completely agree with you. For our current study, we believe relying on a golden solution is the most rigorous approach to establish our foundational framework and provide a theoretical basis and reference for the SAGA-based method. Fortunately, in our competitive programming domain, an official solution is often available, which can serve as a reliable oracle.

   However, overcoming this dependency for broader applications is a key goal for future work, and your comment on majority voting has inspired us to think more deeply about this. In such a future system, a rule-based mechanism would be key. First, we could generate a large number of test cases (e.g., via the method in Paradigm 2). These test cases would then act as "voters" to vet a large number of rollout solutions. A "majority vote" based on passing a supermajority of diverse tests would allow us to automatically label a small set of high-performing solutions as reliable "pseudo-gold" solutions. This imperfect "pseudo-gold" set could then serve as the input to our SAGA framework for refinement. Future work could consider implementing test case generation via this pseudo-labeling method, which would rely on the mathematical principles and metrics (like DEPC) that we establish in this foundational paper. Thank you for asking the precise question that allowed us to engage in this exciting discussion about the future. Your contributions to this dialogue are of great value to the entire community.

Thank you once again for your constructive feedback and for maintaining your positive evaluation. We sincerely appreciate the guidance you have provided in helping us strengthen our work.

Best regards, Authors of Submission 17344

Dear Reviewer UMHU,

We sincerely thank you for your positive review of our paper. We are especially grateful that you recognized the solid contributions, deep insights, and rigorous methodology of our work, as well as the value that TCGBench and CodeComPass bring to the community. Your constructive questions are crucial for us to further refine and clarify our work.

Below, we provide detailed answers to your specific questions:

1. Running Example and Workflow of SAGA:

Thank you for this question. Figure 5 provides a high-level overview, and we understand that a more detailed explanation of the concrete workflow is needed. Your understanding is correct: apart from the human-written solutions, other components such as the test case scripts, math explanations, and self-validation are indeed generated by the LLM.

Regarding the role of the Math Explanation that you specifically mentioned, the LLM transforms the implicit programming logic from human code into explicit, structured test instructions. This process ensures that we are not just randomly generating inputs but are purposefully targeting the core mathematical and logical properties of the problem.

For example, in the classic "Convex Hull" problem, SAGA would identify from human code that the algorithm's core relies on using the cross-product to determine the geometric orientation of points. Based on this mathematical principle, SAGA would infer two key testing directions: 1) Singularity Testing: When the cross-product is zero, the points are collinear. SAGA would intentionally generate a test case with a "trap" point lying exactly on an edge of the true convex hull to test the algorithm's precision. 2) Extremum/Degenerate Case Testing: When all points are collinear, SAGA would generate a test case where all points lie on a single line to test if the algorithm can correctly handle this degenerate input. This process allows SAGA to systematically discover and test logical singularities and structural extrema determined by the problem's intrinsic mathematical structure, a depth that is unattainable by simple random testing.

We will add a detailed case study to the appendix of the final version to walk readers through this entire process, showing how these prompts are used to transform inputs into final, rigorous test cases. We will also elaborate more on the role of the math explanation in the main text.

2. How to Combine Multidimensional and Differential Analysis:

This is an excellent question. These two analyses are performed in parallel. Multidimensional Analysis utilizes correct solutions ($S_{human}$) to explore the solution space and defensive logic, while Differential Analysis uses pairs of correct and incorrect solutions ($S_{wrong}$) to target known failure modes. Each generates its own Case Scripts. These scripts are then pooled and deduplicated to form a comprehensive and diverse set of test inputs, thereby maximizing both coverage (from Multidimensional Analysis) and discriminative power (from Differential Analysis).

3. Test Case Generation and Filtering:

Yes, we run the generated Python Case Scripts to produce multiple test inputs. The filtering is done through the Self-Validation script, which is also generated by the LLM as part of the SAGA process (as shown in Figure 5). This script programmatically checks if the generated inputs conform to the problem's constraints (e.g., input format, value ranges). Any input that fails this validation is discarded, ensuring the quality and relevance of the final test suite. After obtaining the test case outputs, we also use multiple correct solutions to verify the test cases; only those that are passed by all correct solutions are retained.

4. Details on Simple Human Priors in Table 1:

The human priors here refer to adding prompts such as "Refer to the correct human solution and analyze the boundary conditions" into the prompt.

5. Human Analysis on LLM-generated Math Explanations and Scripts:

This is an important aspect. While we have not conducted a formal, large-scale human analysis on the correctness of every explanation and script, their quality is indirectly validated by the effectiveness of the test cases they produce. The high performance of SAGA (e.g., high DR and VAcc) indicates that these LLM-generated artifacts are, on the whole, correct and effective at guiding the generation of useful tests. Our self-validation retention rate is typically above 60%.

6. Detection Rate with Mixed LLM Test Cases:

We were also very interested in this point and conducted this experiment, with the results presented in Appendix F.1 and Figure 12. The findings demonstrate that mixing test cases from different LLMs (which have different cognitive biases) can reduce the inter-test correlation ρ and significantly improve the overall quality of the test suite (higher AUC@50). This confirms the theoretical foundation of our work. We will add a sentence to the main text to highlight this finding and point to the appendix.

7. Evaluating the Latest Thinking Models on CodeComPass:

We evaluated Qwen3-235B-A22B-thinking-2507, and the experimental results are shown in the table below. This again validates the effectiveness of CodeComPass as a high-difficulty benchmark and reveals that even top-tier "thinking" models still have room for improvement in generating truly robust code, highlighting the value of our work.

Thank you once again for your thoughtful and constructive review. Your feedback has been crucial in helping us improve the clarity and completeness of our paper.

Best regards,

Authors of Submission 17344

Hi reviewer UMHU,

Do you find the response to the rebuttal convincing?

Dear Area Chair and Reviewers,

Thank you for initiating this important discussion. We appreciate the opportunity to clarify our unwavering commitment to reproducibility and address any lingering concerns.

We believe our work is easy to reproduce because we designed it to be open and clear. We made sure of this in three main ways:

1. Clear Algorithm Description in the Paper: The paper gives a full description of our SAGA framework. We believe the text in Section 3.1 and the picture in Figure 5 give a clear overview of the algorithm's logic. To make it easy to reproduce, we also give the exact data setup, like using 10 correct solutions for one analysis and pairing a correct solution with the most recent incorrect one for another (details at the end of Page 7). Also, the math for our metrics is in Section 2 and Appendix B.

2. Transparent Experimental Setup: We have provided all the details needed to reproduce our results, with clear pointers to where they are in the paper:

   • Models: The LLMs we used (DeepSeek-V3-0324, Qwen series, etc.) are named in Section 2.2 (Page 4) and Section 3.2.1 (Page 7).
   • Datasets: How we built our datasets, TCGBench (1840 problems) and TCGBench-Lite (270 problems), is explained in Appendix C.4 and C.5 (Page 18).
   • Generation Settings: We used greedy decoding (temperature=0), as stated in Section 2.2 (Page 4), and Pass@1 for all tests.
   • Training Details for TCGCoder-7B: All training details, like dataset size (15k problems), epochs (3), and learning rate (5e-6), are in Appendix G (Page 23).
   • Full Implementation: To further help with reproducibility, our anonymous repository contains the complete prompts (the source code of our method) and a working demo. We also provide a guide in the README for setup and running.

3. A Concrete Plan for the Community: Our goal is bigger than just this paper. We promise to open-source everything to give the community a new set of powerful tools and data. After the paper is published, we will create a special Hugging Face organization to share:

   • The complete, open-source SAGA framework code.
   • The full datasets we used.
   • The model weights for TCGCoder-7B.
   • A large number of SAGA-generated problems and test cases, including the full LLM response history, to provide high-quality data for future research.
   • We will also create and keep up a public leaderboard for CodeComPass to track new models.

We hope this explanation shows our full commitment to reproducibility. We are sure that our paper, along with our planned open-source contributions, gives the research community everything it needs to build on our work.