EditLord: Learning Code Transformation Rules for Code Editing

We really appreciate your time and effort in leaving constructive comments.

Q1: The three tasks it focuses on may limit real-world applicability because they are not highly frequent in real-world editing scenarios.

The three editing tasks we considered are extensively studied in the literature [1-9], even evaluated on the same datasets as we do. They are broadly applicable already. For example, Scalene, the popular Python profiler, has already integrated GPT to suggest code optimizations for short code snippet [11]; LLMs have been integrated into existing decompilers as plugins [8,9]; Large companies are actively hosting competitions on LLMs to produce secure code [10]

[1] Tan, Hanzhuo, et al. "LLM4Decompile: Decompiling Binary Code with Large Language Models." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

[2] Hu, Peiwei, Ruigang Liang, and Kai Chen. "Degpt: Optimizing decompiler output with llm." Proceedings 2024 Network and Distributed System Security Symposium. Vol. 267622140. 2024.

[3] He, Jingxuan, and Martin Vechev. "Large language models for code: Security hardening and adversarial testing." Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. 2023.

[4] Peng, Jinjun, et al. "CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation." arXiv preprint arXiv:2501.08200 (2025).

[5] Shypula, Alexander, et al. "Learning Performance-Improving Code Edits." ICLR. 2024.

[6] Huang, Dong, et al. "Effilearner: Enhancing efficiency of generated code via self-optimization." Advances in Neural Information Processing Systems 37 (2024): 84482-84522.

[7] Peng, Yun, et al. "PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback." arXiv preprint arXiv:2412.03578 (2024).

[8] aiDAPal: IDA Pro plugin that uses a locally running LLM that has been fine-tuned for Hex-Rays pseudocode to assist with code analysis, https://github.com/atredispartners/aidapal

[9] GhidraAssist: An LLM extension for Ghidra to enable AI assistance in RE, https://github.com/jtang613/GhidrAssist

[10] Amazon Nova AI Challenge accelerating the field of generative AI (LLM for secure code generation), https://www.amazon.science/amazon-nova-ai-challenge-accelerating-the-field-of-generative-ai

[11] Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals, https://github.com/plasma-umass/scalene

Q2: Despite the manual inspection, the quality of these rules has not been formally verified.

This is a valid concern. While there is no formal guarantee on the generated editing rules’ correctness, our evaluation shows our rule set consistently improves editing performance (Sec. 3.2). We included additional results to show that our meta-rule set learning algorithm can bring improved robustness against the perturbations of the meta-rule set (see below). We do agree that further interacting with formal verifiers to guarantee the correctness of editing rules can be an exciting future work.

Q3: In practical software engineering, people apply code edits in a repository. In addition, the code edits are triggered by an issue (see GitHub issue). How can this research push forward in this direction?

EditLord focuses on file-level local edits rather than repo-level editing. File-level edits are already valuable in many scenarios during software development. For example, the developers frequently refactor their code at the local context, e.g., by editing the function they have just written to make it more efficient. While we do believe repository-level editing is indeed an exciting work, we do not attempt to overclaim that EditLord is readily useful for repository-level edits. Instead, we would like to emphasize that the tasks we considered in this paper are already nontrivial and useful, as we have also argued in our response to Q1.

Q4: Essential references not discussed.

Thanks for pointing out these related works. We found the main difference between EditLord and many existing editing works (e.g., CoEdPilot, Grace, and CODIT) is that these works mainly focus on repairing functionality, while our tasks focus on preserving original functionality while introducing changes in other dimensions, e.g., performance and readability. That said, these are all very related and should be discussed. We have added discussions in the related works of these papers in our draft.

Thank you so much for your taking the time and effort to read our paper and leave constructive comments.

Q1. Why extracted editing rules are effective in improving code efficiency or what the rules actually look like.

These extracted editing rules serve as editing guidelines that help the model reason about the useful steps and actions to take to generate the edited code. Please refer to Table 8 in Appendix B for some examples of the rules. We have also added the detailed samples to the anonymous repo.

Q2. The figures are confusing and do not clearly illustrate the methodology, e.g., the input/output of each subtasks, rule usage, and the editing process.

We apologize for the potentially unclear figure presentation. To clarify, we take the performance optimization task as an example in Fig 1. The input is a slow (pre-edit) code, while the expected output is semantically equivalent but faster (post-edit) code. Now consider inferencing by an EditLord finetuned model. Instead of directly generating the faster code (post-edit code), the model will first generate functional specifications and editing rules, and then these generated is in the context, and the model keeps on generating the post-edit (faster) code.

For the decompilation task, the input is an unreadable, non-idiomatic ghidra-decompiled (pre-edit) code, while the expected output is a more readable, idiomatic (post-edit) code (see Fig 3).

For the security hardening task, the input is vulnerable code, while the expected output is secure code.

In the upper part of Fig 2, we illustrate how we prepare the functional specifications and editing rules. We will take the pre-edit and post-edit code pair from the training data as input and use LLMs to annotate the functional specification and the editing rules for each pair.

Since editing rules are often shared among different samples, our Alg. 1 describes how an LM serves as a rule learner by iteratively refining the raw meta-rule set and producing a more concise meta-rule set by using operations ADD, MERGE, and PRUNE. This step is done independently for each task. We also state the input and output for each editing mode more clearly in the updated figure for our improved workflow figure.

Q3. The dataset used in their experiment is unclear and causes confusion (e.g., HQ, HumanEval).

We evaluate different editing tasks with different datasets, as different objectives require different evaluations. Specifically, for performance optimization tasks, we train on the HQ dataset and evaluate on the test split from PIE. For the decompilation task, we train on AnghaBench and test on HumanEval-Decompile. For the security hardening task, we train on SVEN, and test on CWEval. We will update our Section 3.1 to describe the setting more clearly.

Q4. More benchmarks should be included for better evaluation.

Thanks for pointing this out. Per our response to reviewer raGi, we evaluated EditLord’s finetuned DeepSeek-Coder 1.3B in CodeEditorBench, the benchmark we did not consider in the submission. We follow their metrics by focusing on 1) accuracy: the percentage of problems with correct edits; 2) OptScoreTime: the execution time improvement; and 3) OptScore, the improvement computed by the averaged time and memory together.

Surprisingly, without further finetuning EditLord on this benchmark, but simply running inference using a finetuned EditLord model on a completely different training set, it substantially outperforms the finetuned baseline by 22.5%, 1.8%, and 1.1%, respectively.

Q5. Repo seems empty.

Thanks for your interest in our artifact. We would hope to clarify that the repository is not empty. The code and prompts were uploaded along with the paper submission. It is likely that the README is blank, which makes the repo appear to be incomplete. We have already uploaded other important files and updated the README. We hope this addresses your concern.

We are grateful for your effort in leaving us such encouraging comments.

Q1. It’s unclear how robust the meta rule creation process is. And there aren’t really any guarantees on the quality without human interaction.

Great point. We observed that if we simply bootstrap per-sample rules, the rules often end up being noisy, repetitive, and not generic. This leads the resulting meta-rule set to be too large to fit in the model context, and the resulting per-sample manifested rules become extremely susceptible to the order of training samples. This motivated us to mitigate this issue by developing the meta-rule learning algorithm to keep the meta-rule set concise and generalizable. As you have also noted, our evaluation in Sec 3.6 shows that exposing the meta-rule set during inference allows us to invite human intervention to further improve the quality of the edited code.

While there is no formal guarantee on the generated editing rules’ correctness, our evaluation shows our rule set consistently improves editing performance (Sec. 3.2). In our response to your Q4, our additional results show that our meta-rule set learning algorithm is relatively robust against the perturbations of the training samples. We do agree that bringing provable guarantees to the editing rules, e.g., by interacting with formal verifiers brought by the compiler techniques, is an exciting future work.

Q2. It’s unclear how annotation-driven performance improvements scale with varying amounts of finetuning data.

This is a great suggestion. We have added an experiment on the decompilation task when using only 50% of the finetuning data. The results show that EditLord, with only 50% of the training set, still outperforms the finetuned baseline with 100% training samples by 5.9%. Its readability results remain 13.5% and 16.4% higher than the finetuned baseline on char- and token-level readability. This result demonstrates that the sample efficiency brought by EditLord, requiring less than 50% of training samples while achieving comparable performance.

Q3. What is the performance of larger, non-finetuned LLMs w/ or w/o rules/specifications appended?

We added an experiment exploring the GPT-4o-mini’s (2024-07-18 version) performance when incorporating our meta-rule set and functional specifications. As shown below, simply including them for non-finetuned GPT-4o-mini will improve its correctness and readability by 11.9% and 8.6%, respectively.

Q4. How stable is the convergence of the meta-rule set? Will shuffling introduce high variance?

We have added an experiment that randomly shuffles the initial dataset before passing it into Alg. 1 and obtaining a pair of meta-rule sets. We then measure the similarity between these two sets. Specifically, for each rule in one set, we compute its average semantic similarity to the top 5 most similar rules in another set and average them again to obtain the overall similarity. The semantic similarity between two rules is computed by the cosine distances of the rule embeddings computed by CodeSage, following the similar setting as in the readability metrics in Sec. 3.1. The resulting semantic similarity between the shuffled rule sets is 0.87, close to 1.

We also show the end-to-end results on the robustness of EditLord introduced by the resulting meta-rule set. The perturbed rule sets lead to less than 2.4% performance changes.

Q5. What is DIS in Alg. 1?

DIS is the internal embedding distance calculated by the LLM. We intended to use DIS to describe that we ask the LLMs to discover the rules that share similar semantics to decide whether to directly ADD the current rule to the rule set or MERGE it with an existing rule in the rule set. We acknowledge that this can lead to confusion, and we have updated the algorithm at link.

Q6: Essential References Not Discussed.

Thanks for pointing out these related benchmarks. We have added the discussion to our draft. We view code generation tasks as complementary to ours as their input is not the code, but usually natural language specification, similar to our functional specification.

Thanks for pointing out skill discovery works, e.g., TroVE and GENOME! We were inspired by TroVE in the very early stage of our project but ended up narrowing down our focus to primarily code-editing works. We should definitely discuss both, and we have added them to the draft.

We really appreciate your time and effort in reviewing our paper and giving us constructive comments!

Q1: More well-known benchmarks should be included.

Thanks for pointing this out. We originally focused on individual tasks where the corresponding papers also proposed tailored solutions (e.g., PIE) to ensure we compare to the state-of-the-art baselines.

That said, we evaluated our finetuned DeepSeek-Coder 1.3B on the Code Polish task in CodeEditorBench. We follow their metrics by focusing on 1) accuracy: the %problems with correct edits; 2) OptScoreTime: the execution time improvement; and 3) OptScore, the improvement computed by the averaged time and memory. EditLord, even without extra finetuning on this dataset, outperforms the finetuned model by 22.5%, 1.8%, and 1.1%, respectively.

Q2: How does EditLord work on out-of-domain generalization (e.g., unseen vulnerabilities)?

We ensured our training and evaluation came from two data sources. As described in Sec 3.1, our training comes from SVEN, but our testing is from CWEval with unseen CWEs. We add below a breakdown of the baseline and EditLord’s performance on seen/unseen CWEs. EditLord consistently generalizes better than the baseline, outperforming it by 7.5% and 38.1%, respectively. We also add generalization tests on unseen languages (Python/Java) in performance optimization in CodeEditorBench. EditLord achieves improvement in both seen and unseen languages, outperforming it by 2.05% and 0.61%, respectively.

Along with the length generalization results in Sec 3.5, our added experiments here show that EditLord maintains strong generalization in various settings. We will include them in the draft.

Q3: The granularity of the input code (function/class/file level) for each task is unclear.

The input code is at the file level for all tasks. This ensures the code can be compiled to measure the functional correctness. Please see detailed examples here.

Q4: How robust is EditLord to noise in rule manifestations during inference?

Great point. We observed that learning rules for each sample often leads to noisy and repetitive rules, which can degrade performance. This motivated us to propose Alg.1 to disentangle the meta-rule learning and the per-sample rule manifestation. While there is no formal guarantee on the generated editing rules’ correctness, our evaluation shows our rule set consistently improves editing performance (Sec. 3.2). That said, generating provably correct rules in formal languages with formal verifiers is indeed an exciting future work.

We include some preliminary results on the robustness of EditLord against randomly shuffled rules for training. The following shows that perturbed rule sets lead to less than 2.4 performance changes in the decompilation task.

Q5: How large is the initial rule set G^0, and the final discovered rule set G? Can you share the entire rule set for each task?

The initial rule set $G^0$ has 2.9K, 1.9K, and 1.2K rules for performance, decompilation, and security hardening, respectively, while the final rule set G has 221, 228, 237, respectively. Table 8 shows the example rules. Thanks for your interest. We will definitely release the full set once the paper is ready to be published.

Q6: Will extending CoT reasoning with their rule set give better results?

Yes, we included this study in the paper. As described in Fig.2, the prompting editing mode refers to CoT prompting with our meta-rule set. In Fig.4 (R=0 means CoT prompting without iteratively refining the generated code), we compare this CoT setting with zero-shot prompting w/o CoT (i.e., w/ vs w/o EditLord). Including our rules improves the CoT performance by 46% across all the tasks.

Q7: Related work not discussed.

Thanks for pointing out these benchmarks. We have included preliminary results on CodeEditorBench (Q1), showing EditLord’s potential to generalize to unseen benchmarks. We will add new results and discussions of these benchmarks in our paper.