Defects4C: Benchmarking C/C++ Faults to Assess LLM-Based Program Repair

We believe the reviewer has significantly misunderstood aspects of our paper, including the key content reading, the scope of ICLR, and foundational concepts in the program repair domain. While we appreciate the time taken to review our work, some comments suggest a lack of careful reading and understanding of the paper’s contributions. To be frank, we are disappointed by the reviews provided thus far but have made every effort to clarify these misunderstandings and address all concerns comprehensively.

We respectfully request that the reviewer reevaluate the paper with a fair and unbiased perspective. Thank you!

Q: Overall, I think this paper is a dataset paper that could fit MSR dataset track well. However, it would not be qualified for a ICLR technical paper cause it contains little technical contribution and What is the technical contribution of this paper?

Please refer to the ICLR 2025 Call for Papers: https://iclr.cc/Conferences/2025/CallForPapers, which explicitly defines the conference's scope to include topics such as datasets and benchmarks.

We believe the reviewer may have some misunderstanding or bias regarding the research topics addressed in our paper. As summarized in our general response, our contributions include a general bug collection framework, a high-quality dataset, and a large-scale empirical study. These contributions are highly relevant to the dataset and benchmark scope outlined by ICLR.

We respectfully request the reviewer to reevaluate the paper from the perspective of the dataset and its alignment with the conference’s stated scope.

Q: The description of the collection process of the vulnerability dataset is rather vague. I did not clearly understand how it was constructed. From the Introduction, it seems that this sub-dataset is directly reused from the official CVE database. Later, in the human annotation part, it seems that it was collected by humans, which makes me very confusing. Another point is that 248 bugs were chosen from 3,785 candidates, but 102 vulnerabilities were chosen from 102 condidates. What leads to such a big difference in this ratio?

We would like to clarify the dataset collection process and address your concerns.

1. Data Sources:

   • We collect bugs (248) and vulnerabilities (102) from two primary sources:
     • GitHub real-world repositories (see Introduction, lines 88–97).
     • The official CVE database (see Introduction, lines 99–103).
   • These sources have been described in separate paragraphs in the Introduction to clearly distinguish their origins.
   • We name any samples from these two sources as defects4C_bug and defects4C_vul, respectively. (see Introduction, lines 081-087).

2. Collection and Annotation Process: Both bugs and vulnerabilities follow a similar pipeline for verification and quality control, as illustrated in Figure 1 and detailed in Section 3:

   • Pipeline Steps:
     • Availability Validation
     • Single-function filtering
     • Test case filtering
     • Unit test matching
     • Human Annotation: It is important to note that this step refers to human annotation rather than human collection. The pipeline ensures multi-level filtering, and human experts then annotate the filtered data to confirm their correctness.

3. Filtering Process and Ratios: The filtering process explains the difference in the ratio of selected bugs to vulnerabilities:

   • Bug Filtering Pipeline: 40M → 9M → 76K → 3,785 → 248 .
   • Vulnerability Filtering Pipeline: 14,488 → 249 → 102 → 102 .

4. The larger difference in the final ratios between bugs and vulnerabilities arises from the initial dataset sizes: The raw bug dataset starts with 40M samples and the vulnerability dataset starts with only 14,488 samples. It is because vulnerabilities are inherently rarer in real-world data.

We hope this explanation clarifies the collection and annotation process as well as the difference in ratios.

Q:The manual annotation leads to Cohen’s Kappa values like 0.48 and 0.60 which are very low. This means that the annotation quality is rather low.

We believe there has been a misunderstanding of the content in paragraphs 246–250 of our paper. In our paper, we clearly show that we conducted three rounds of annotation, during which the Cohen’s Kappa values progressively improved as follows:

• First round: 0.48
• Second round: 0.60
• Third round: 0.88 (final value, reported at line 249).

The final Cohen’s Kappa value of 0.88 demonstrates a high level of agreement and indicates the reliability of our annotations after iterative refinement. This progression highlights our commitment to quality control and the effectiveness of our multi-round annotation process.

We respectfully encourage the reviewer to revisit the relevant section of the paper, as it clearly explains the steps. We hope this clarification addresses your concerns and facilitates a more accurate evaluation of our work.

Q:In the annotation process, CWE types were used to help participants decide whether each commit is bug-related. that bugs and vulnerabilities are different concepts. Why use this type of information for bug identification?

Thank you for raising this important point. We agree that bugs and vulnerabilities are distinct concepts, which is why we have treated them as separate ones, with different sources for their collection and analysis (as discussed in our previous response).

We are sorry for any potential confusion in the wording of the paper. To clarify:

• CWE types were not used directly to determine whether a commit is bug-related.
• Instead, CWE types served as reference during the annotation process to help construct a taxonomy of our collected bugs and vulnerabilities. We refer to CWE and build our taxonomy to classify the root causes of both bugs and vulnerabilities systematically. These CWE types were used as an initial foundation to guide our classification efforts rather than as definitive criteria for identifying bugs.

We will rephrase the relevant sentences in the revised paper.

Q: The prompts used for evaluation are simply wrong. Before fixing a bug, we would never know how many code entities we need to modify. That means what we can do is only providing a unified prompt for fixing. cannot ask to perform modifications at a specific level because we do not have this prior knowledge. The experiments conducted in this study were thus totally wrong. and Why can we use prompts of different levels for fixing bugs?

We would like to clarify that in the program repair domain, fault localization (identifying the fault lines) and repair (generating patches) are typically treated as separate tasks, as established in many state-of-the-art works [1–8]. The standard problem statement for program repair assumes fault localization is provided as input, enabling researchers to focus on the repair process. Our paper follows this widely-used setting, evaluating repair tasks under the assumption that the fault localization is given.

Even under your setting where fault localization is not given, our conclusions remain similar:

• Despite providing perfect fault localization, the repair performance is still far below expectations, with only a small number of bugs successfully fixed.

• Hence, if fault localization were not given, we anticipate the results would be similarly poor or even worse, highlighting the inherent challenges of automated program repair.

To further address your suggestion, we conducted an additional experiment to evaluate the fault localization (FL) capabilities of large language models (LLMs). The results, presented in Rebuttal Table 5, indicate that LLMs exhibit low accuracy in identifying fault lines, which negatively impacts the overall repair process. These findings emphasize that separating FL and patch generation allows for targeted improvements in both tasks independently, ultimately enhancing the overall. Specifically, the lower performance of fault localization significantly affects downstream repair quality. By isolating these tasks, researchers can address their distinct challenges more effectively.

We will include these discussions in the revised-version. We hope this clarification helps the reviewer better understand the distinction between the two research problems, the commonly-used settings, and the importance of clearly articulating this separation in our work.

RebuttalTable 5. Fault localization performance report on Top@1 of Greedy and TopK=[1,3,5] with temperature 0.8.

References
[1] Rahul, Soham, et al. “DeepFix: Fixing common C language errors by deep learning”. AAAI 2017.
[2] Sumith, Panupong, et al. “SPoC: Search-based Pseudocode to Code” NeurIPS 2019.
[3] Harer, Jacob et al. “Learning to repair software vulnerabilities with generative adversarial networks” NeurIPS 2018
[4] Yasunaga, Michihiro et al. “Graph-based, self-supervised program repair from diagnostic feedback” ICML 2020
[5] Berabi, Berkay et al. “TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer” ICML 2021
[6] Yasunaga, Michihiro et al. “Break-it-fix-it: Unsupervised learning for program repair”, ICML 2021
[7] Zimin, Steve, et al. “Neural Transfer Learning for Repairing Security Vulnerabilities in C Code” IEEE Transactions on Software Engineering 2022
[8] Fu, Michael, et al. “VulRepair: A T5-based Automated Software Vulnerability Repair” ESEC/FSE 2022

Dear Reviewer,

As the deadline for the discussion period approaches, we kindly request your feedback on our response. Your insights would be greatly appreciated.

Thank you for your time and consideration!

Q: The novelty and contributions are limited. In recent years, a lot of APR benchmarks have been proposed, covering a variety of programming languages, types (real-world, programming assignments, competitions), difficulty levels, and so on. Table 1 of the paper shows some existing APR benchmarks. There are some more, such as DebugBench (Tian et al., 2024) , EvalGPTFix (Zhang et al., 2023), FixEval (Haque et al., 2023), QuixBugs (Lin et al., 2017b), etc. Therefore, the contributions of this paper are rather incremental.

Thank you for your detailed feedback and for pointing out these additional benchmarks.

1. Please refer to our general response for a detailed explanation of the key novelties and contributions of Defects4C. While we acknowledge that as a dataset paper, our work may have limited technical novelty, our primary focus has been on creating a high-quality, realistic, verifiable, highly usable, and sustainable dataset for C/C++. These features, in our view, are crucial criteria for a dataset to have a lasting impact on the research community.

2. Among the benchmarks you mentioned, we found that DebugBench [8] (Tian et al., 2024) is the only one that includes C/C++ defects. However, these defects are collected from LeetCode and predominantly focus on programming contest problems, which are simpler and may lead to overfitting with modern LLMs. As shown in our experiments (see general response), LLMs like GPT-4 and GPT-3.5 achieve a 90–94% pass rate on DebugBench, highlighting that these datasets might not be able to effectively evaluate the current state of APR methods.

• The other benchmarks, EvalGPTFix, FixEval, and QuixBugs, focus on defects in Python or Java. Our work specifically addresses C/C++, a domain with unique characteristics:

  • C/C++ programs are more prone to bugs due to their complexity, low-level memory management, and lack of safety features.

  • The collection, confirmation, taxonomy, and even repair methods for C/C++ differ significantly from those for Python and Java.

We sincerely appreciate your contribution to DebugBench and will ensure it is added to the related work section in our revised paper. We also believe that Defects4C addresses critical gaps by providing a robust, realistic, and challenging dataset for C/C++, which sets it apart from existing benchmarks.

Q: Lack of evaluations on sub-tasks: The paper only evaluate models on the entire program repair task, without describing the effectiveness of each step, such as bug localization and patch generation. Particularly, given the challenges in identifying the exact location of faults in C/C++ code, the paper could evaluate LLMs' capabilities in bug localization, which is important for real-world program repair.

Thank you for your insightful suggestion and for highlighting the importance of evaluating sub-tasks like bug localization. Initially, we did not include fault localization evaluations due to resource limitations. However, we have followed the reviewer's suggestion and conducted additional experiments to assess LLMs’ capabilities in fault localization.

Experimental Setup: Fault localization aims to identify suspicious locations in the code (e.g., single or multiple statements). For evaluation, we use the Defects4C patch commit line numbers as the ground truth. We provide the complete function content to the LLMs and prompt them to suggest the fault locations. To avoid data leakage, we do not include error messages or compiler feedback in the fault localization prompts.

We measure the Top@1/3/5 accuracy. For multi-line faults, we merge the Top-K most suspicious lines ranked by the LLM’s probability and then compute the metric scores, following prior works [9,10,11].

The results are presented in RebuttalTable 5. and indicate that FL performance still lower in larger general LLM, like Grok-beta. These findings highlight the potential and limitations of LLMs in fault localization, a critical step for real-world program repair tasks.

We will include these results and discussions in the revised version of the paper.

RebuttalTable 5. Fault localization performance report on Top@1 of Greedy and TopK=[1,3,5] when temperature 0.8.

Q: The categories of bugs as listed in Table 2 are too rough. I suggest to categorize bugs using finer-grained taxonomy of bugs, which would help reflect the capability of models more comprehensively.

Thank you for your valuable suggestion.

We appreciate your point about using a finer-grained taxonomy. In fact, we have already expanded our categorization into a three-level hierarchy, encompassing 35 categories. Due to space constraints, we only presented the high-level categories in the current version of the paper.

For your reference, the following table shows the detailed fine-grained categories, which have been tagged in the evaluation dataset. These finer categories provide a more comprehensive breakdown of bug types, enabling a deeper understanding of model capabilities. please ref to GitHub repository for fine-grained [12].

We will update the paper to include an explanation of this fine-grained taxonomy and how it is used in the dataset.

Q:The dataset includes a large collection of bug-relevant commits (e.g., 9M in total), 248 high-quality buggy functions and 102 vulnerable functions paired with test cases for reproduction. How do you ensure adequate quality control through the human annotation process?

Thank you for this insightful question. Ensuring the quality of the final evaluation dataset was a key focus of our efforts, and it is precisely why we retained only a small set of high-quality buggy functions (350) after rigorous filtering and annotation. Below, we describe our quality control mechanisms:

1. We recruited 4 C/C++ experts with over five years of C/C++ programming experience. Two of these experts are also researchers specializing in software testing, which enhanced their ability to analyze and annotate cases effectively.

2. Our annotation process involved multiple rounds to ensure accuracy and consistency. Disputes were resolved through in-depth discussions, focusing on the root causes based on code semantics and commit messages.

3. Note that each annotation was validated by running the associated test cases on both the buggy and fixed versions of the code. This process allowed us to debug and verify whether a given commit truly represents a bug, ensuring high confidence in the annotations.

4. As described in the paper, we removed ambiguous cases during the annotation process, including those that were either not clear bugs or too difficult to understand or explain. This step reduced the original set of 3,785 commits to the final high-quality dataset.

5. To measure consistency and reliability among annotators, we calculated Cohen’s Kappa, which reached a score of 0.88. This is considered a high level of agreement and demonstrates the robustness of our annotation process.

We sincerely thank you for your suggestion and will add these details to the revised version of the paper to provide a more comprehensive explanation of our quality control process.

References:
[1] Guo, Yuejun, et al. "An investigation of quality issues in vulnerability detection datasets." In 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)

[2] Croft, Roland, et al "Data quality for software vulnerability datasets." In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

[3] Kuehn, Philipp, et al "OVANA: An approach to analyze and improve the information quality of vulnerability databases." In Proceedings of the 16th International Conference on Availability, Reliability and Security

[4] Croft, Roland, et al. "Data preparation for software vulnerability prediction: A systematic literature review." IEEE Transactions on Software Engineering 49, no. 3 (2022): 1044-1063.

[5] Lin, Derrick, et al. “QuixBugs: A multi-lingual program repair benchmark set based on the Quixey Challenge” ACM SPLASH 2017

[6] Zhang, Zhang, et al. “A critical review of large language model on software engineering: An example from chatgpt and automated program repair” arXiv:2310.08879

[7] Haque, Md Mahim Anjum et al. “Fixeval: Execution-based evaluation of program fixes for programming problems” IEEE/ACM International Workshop on Automated Program Repair 2023

[8] Tian, Ye, et al. “Debugbench: Evaluating debugging capability of large language models” arXiv:2401.04621

[9] Xia, Deng, et al. “Agentless: Demystifying llm-based software engineering agents” arXiv:2407.01489

[10] Wu, Li, et al. “Large language models in fault localisation” arXiv:2308.15276

[11] Yang, Le, et al. “Large language models for test-free fault localization” the 46th IEEE/ACM International Conference on Software Engineering 2024

[12] https://github.com/defects4c/defects4c/blob/master/minor_category.md

Dear Reviewer,

As the discussion period nears its last moment, we would be grateful for your feedback on our response. Your valuable insights will help us further strengthen our work.

Thank you for your time and consideration.

Q1. Given that most people are interested in migrating to Rust (because of memory safety features), and there has been research on code translation between C to Rust (https://arxiv.org/pdf/2404.18852), why do you think a dataset of bugs for C/C++ would be worthwhile?

Thank you for raising this interesting and relevant question.

We would like to address this concern from several perspectives:

1. The value of our dataset depends on whether C/C++ remains widely used. As long as these languages are in active use, tools for their support, including automated repair, will be necessary. According to a recent study in 2024 [1], C and C++ are still among the top 5 most popular programming languages, and their combined usage would rank them first overall. This continued popularity is why research in areas such as testing, vulnerability detection, code summarization, debugging, and automated repair for C/C++ projects remains vibrant.
2. C has the highest number of reported vulnerabilities among programming languages, accounting for over 50% of all disclosed open-source vulnerabilities since 2019 [2]. This statistic highlights the critical need for tools and datasets that support automated repair in these languages, as they directly impact software security.
3. While there is exciting research on translating C/C++ to Rust, this technology is still in its early stages and not yet mature. Fully translating complex C/C++ programs into Rust remains highly challenging. Moreover, due to some performance considerations, and practical deployment issues such as hardware constraints and execution speed, Rust is unsuitable in certain contexts, meaning that many companies will likely continue using C/C++ in the foreseeable future. Even if the migration to Rust accelerates in the future, the transition will not be immediate or universal. C/C++ is deeply embedded in many legacy systems, critical applications, and performance-intensive domains. Until these languages are entirely replaced—a scenario that is far from imminent—our dataset will remain valuable for ongoing research and development.

We believe that C/C++ will continue to be popular and relevant in the coming years, making our dataset a significant contribution to the field.

Q2. Have you performed some more analysis on why a model like Magicoder produces 0.0 as result? Do you think the right prompt is not used when inferencing with Magicoder?

We conducted the investigation on the magicoder result, the main reason is that magicoder’s output did not follow the LLM regular response which encloses the code by three backticks, i.e. ```, the preprocessing method to extract code content from LLM’s response is a failure, so the result is zero. While this uniform code extractor is employed in most LLMs, i.e. the preprocessor is fair to every LLMs. So if failure to detect the three backticks``` and it will be marked fail in pass rate.

We conduct a re-evaluation on magicode for robustness analysis purposes. Assume the magicoder response on the C/C++ bug did not always include any three backticks```, we report two different scores: utilize the whole response as code even failure to extract code from backticks, or keep the regular preprocessor as other LLMs to only extract code from three backticks```.

We will revise the submission with different preprocessor results, and explain the magicoder’s zero scores.

Q3. Why didn't the authors perform APR on existing C/C++ datasets. What if overall LLMs are bad on all existing C/C++ datasets including Defects4C? If that is the case, what is the one thing that makes Defects4C stand out?

Thank you for the suggestion. Please refer to the general response.

Reference:
[1] https://distantjob.com/blog/programming-languages-rank/

[2]https://www.mend.io/most-secure-programming-languages

@@ Instruction
fix code in line XX.

@@ Response
```

Q5: Including evaluations of the latest general-purpose LLMs, like GPT-4o(-mini), Claude3, Gemini 1.5, or even o1-preview, would be helpful. Testing one or some of these models could demonstrate how recent models perform on C/C++ APR tasks.

Thank you for your valuable suggestion.

In response, we have conducted additional evaluations on some of the latest general-purpose LLMs, specifically GPT-4o-mini, o1-preview, and Grok. As shown in RebuttalTable 4, while these recent models exhibit promising performance compared to existing LLMs like DeepSeek and CodeLlama, they still face significant challenges in addressing C/C++ APR tasks.

RebuttalTable 4 More latest general-purpose LLMs. (original Tab4.)

We appreciate your recommendation and will include the results and detailed discussions in the revised version of the paper to provide a more comprehensive evaluation of these models.

Q6:Given that Defects4J is an older dataset, it’s possible that knowledge contamination is affecting model performance. Conducting a case study on this potential contamination could provide valuable insights. Furthermore, are there methods to prevent your benchmark from contaminating future LLM knowledge?

We appreciate the reviewer’s insightful comments regarding the potential contamination impact on our dataset.

Indeed, we acknowledge that there is knowledge contamination in the case of Defects4J used in our results. For example, we observed that some models could still repair buggy functions even when we removed necessary variables or definitions. We will incorporate these findings and their implications in the paper.

Regarding our own dataset, we have implemented several measures to mitigate contamination risks:

1. We ensured that the patches include corresponding timestamps, allowing us to enforce a knowledge cut-off date. This enables users to evaluate the temporal difference between the dataset and the LLM training data.
2. Our dataset construction process is fully automated for collecting new bugs (after the training of LLMs), with only the human annotation phase requiring manual effort.
3. We implemented a decontamination script based on Magicoder’s algorithm, which identifies and removes samples containing high-similarity substrings from the training corpus, thereby reducing overlap.
4. To further mitigate contamination risks, we are developing a mutation-based approach to semantically preserve the dataset while introducing changes such as variable renaming, stylistic adjustments, and code sequence modifications. This will ensure that the bugs remain unaffected while making the dataset less susceptible to contamination.

We appreciate the reviewer’s suggestion and will include these measures and discussions in the revised paper to address contamination concerns comprehensively.

References:
[1] Yasunaga, M., Liang, P.. Graph-based, self-supervised program repair from diagnostic feedback. In ICML’20.

[2] Xia, C. S., Wei, Y., & Zhang, L. Automated program repair in the era of large pre-trained language models. ICSE’23

Q1 In lines 248-249, you mention that half of the bugs were reviewed in the third round. What about the other half? Does this imply that some parts of the dataset received two rounds of annotation, while others received only one? Additionally, is the reported Cohen’s Kappa calculated based on only the reviewed half or all annotated commits?

Thank you for pointing out this confusion in our description.

To clarify, during the third phase of annotation, we aimed to improve annotation quality by splitting the entire dataset into two halves. Initially, we annotated the first half and checked the agreement on these results, conducting discussions as needed. This approach allowed us to ensure that, if the agreement on the first half had been unacceptably low, additional refinement (i.e., the fourth round) would be performed. However, we found that the agreement on the first half was already acceptable, so we proceeded to annotate the second half using the same process.

Importantly, the reported Cohen’s Kappa value was calculated based on the entire dataset, encompassing both the first and second halves.

We will revise the unclear phrase.

Q2 line 255, the 248 commits is a significant reduction from the original 3,785 commits. Does remaining commits in the 3,785 were deemed non-bug-related, that could affect training set quality, or were only a portion of the 3,785 commits annotated by human experts?

Thank you for your insightful question.

As explained in the paper, we employed a rigorous and conservative human annotation process to ensure the highest quality for the evaluation dataset. This process resulted in the selection of 248 confirmed commits.

As discussed in Lines 251–256, the reduction from 3,785 commits is due to the following:

1. A substantial portion of the commits were non-bug-related, as verified through human inspection.

2. Some commits contained bug fixes that were too vague or ambiguous to be definitively classified as real bugs.

To avoid introducing noise or uncertainty into the evaluation dataset, we prioritized quality over quantity and included only the 248 commits that were unequivocally confirmed as bug-related.

We will clarify it more clearly in the paper.

Q3 Could you include some traditional APR methods as baselines in Table 3? The paper frequently claims that LLMs perform well, and a traditional baseline would strengthen the claim.

Thank you for this thoughtful suggestion.

Before submitting the paper, we evaluated 10 traditional APR methods as potential baselines. Unfortunately, the majority of these methods (e.g., Prophet, Darjeeling, F1X, Verifix) could not be successfully executed for the following reasons:

1. Many traditional APR tools are designed for small programs with a single function and lack support for building and verifying large-scale projects.

2. Some tools rely on resource-intensive techniques, such as symbolic execution, which are not scalable for large projects. Ultimately, we were able to successfully execute two traditional tools, VRepair and VulRepair. As shown in RebuttalTable 2, these tools performed significantly worse compared to LLM-based methods, successfully repairing only one bug.

In fact, recent advancements in APR research have consistently demonstrated the superior performance of LLM-based methods, which are now regarded as the mainstream approach [2]. This rationale underpins our decision to focus on multiple LLMs for the evaluation.

RebuttalTable 2. New results of non-LLM baselines( original Tab.4)

We appreciate your feedback and will incorporate the results of non-LLM methods, along with detailed discussions, in the revised version to further strengthen our claims.

Q4: For close values in Table 6, standard errors are needed to support the claim that fine-tuning improves model performance.

Thank you for your constructive suggestion.

In response, we have included the standard errors in RebuttalTable 3, calculated based on results from 5 random seeds. The results indicate that while fine-tuning shows some improvements in certain cases, it does not consistently enhance performance across all metrics. We appreciate your feedback and will ensure that the updated table and discussion are included in the revised version of the paper.

RebuttalTable 3. Comparative Results of LLMs With Fine-Tuning (original Tab 6.).

Dear readers

Defects4C is live on arXiv https://arxiv.org/abs/2510.11059. Try the C/C++ benchmarking suite for your LLM evaluation and share feedback. Thanks to the reviewers and program chairs for their support.

Best regards

authors