CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale

Dear Reviewer Jxkj,

We would like to express our sincere gratitude for your thoughtful and constructive feedback. We have addressed all of the comments and thoroughly presented our most recent experimental findings.

---

Q1: The paper provides limited details on how 220 APIs are selected from the 6,036 tracked updates. What specific criteria are used for filtering? More details on building the dataset would be beneficial to assess the benchmark's representativeness and reliability.

A1: As detailed in Appendix B.1, we filtered out APIs with insufficient real-world usage. Specifically, we selected 220 APIs that each had at least 15 retrieved real-world invocation examples.

---

Q2: Since libraries continue to evolve, how will CodeSyncBench remain up to date? Is there a plan for periodic updates, or will the dataset become outdated as newer API changes emerge?

A2: Yes, we plan to maintain and periodically update CodeSyncBench to reflect ongoing API changes, ensuring it remains a relevant and reliable benchmark for evaluating LLMs' ability to stay synchronized with evolving API knowledge over time.

---

Q3: What is the "Parameter Mapping" that is referred to in Appendix B.1.2. Would this method include some APIs that are not updated?

A3: As described in Appendix B.1.2, parameter mapping is a standard technique in software engineering for detecting API signature changes. For each pair of APIs across versions, we analyze:

• Parameter types: positional-only, keyword-only, or both
• Parameter attributes: required vs. optional

We then construct a mapping from old-version parameters to their corresponding new-version parameters. If no mapping can be established—for example, due to newly added or removed parameters—this indicates an update. Additionally, if mapped parameters differ in type or attribute, we classify the API as changed. We will clarify this in the next version of our manuscript.

---

Q4: Could the authors provide a clear and detailed explanation of the process for identifying valid method API invocations, given that method APIs often involve dynamic binding?

A4: As stated in Appendix B.1, we initially collected 6,036 updated APIs and retrieved invocation samples for each. The next step is to identify target API invocations, as data retrieval relies on string match. We then analyze the code context to determine the actual API calls. For Method-Level APIs, precisely identifying their invocations through static analysis alone is extremely challenging. Therefore, we limit our focus on cases where parameter types can be inferred from the code context. Specifically, we track the dataflow and control flow of each instance. Then, we bind class type and identify method calls for each. This method makes sure to capture actual API calls.

---

W5: This paper assumes that LLMs should always be updated with the latest API versions. However, it may not align with real-world development needs, since many projects continue using older versions for stability and compatibility. If CodeSync only fine-tuned on the latest API changes, it might lead to over-specialization.

A5: Thanks for your comments. We think there may be a misunderstanding here. Actually, CodeSyncBench selected the current time as a snapshot to highlight the importance of API evolution. However, our framework is highly flexible—users can specify any preferred version ranges of a library to generate customized benchmarks. As we have shown below, users can set different start times and deadlines of API evolution for each library. The training set generated by CodeSync helps LLMs to synchronize with target API knowledge. Therefore, in the real-world development scenarios, users can design their own training sets and benchmarks and fine-tune models using LoRA methods for further application and exploration.

Dear Reviewer oxwH,

We sincerely appreciate your thoughtful review and your recognition of our work’s contributions to the community.

---

Q1: Why is the HumanEval score for Qwen2.5-Coder-7B-Instruct so low? The results from the Qwen team show that it achieves 88.4% on HumanEval.

A1: In fact, the HumanEval score we reported for Qwen2.5-Coder was obtained using the bigcode-evaluation-hardness framework, which applies stricter evaluation criteria than those used by the Qwen team. Notably, it does not automatically add import statements and enforces stop tokens, leading to a more rigorous evaluation.

To ensure fairness, we have updated our evaluation setup to align more closely with the official criteria and re-evaluated all base and updated models. The revised scores are as follows:

The updated results align with trends from the original settings, indicating that our conclusions remain valid.

---

Q2: How is the training data related to the benchmark? Does every function change in the benchmark problem have a corresponding data point in the training set?

A2: As stated in lines 255–262, for each API update, we collected 15 legacy-updated invocation pairs—10 for training and 5 for evaluation—ensuring that every API change in the benchmark has corresponding data points in the training set. However, this does not imply data leakage, as the training and benchmark data points use distinct code contexts, as discussed in our response to Reviewer HLiK (W4).

---

Q3: Is it possible to include RAG + prompting baseline?

A3: Thank you for the suggestion. We have included RAG baseline results in our response to Reviewer HLiK (Q3).

---

Q4: How much human effort is needed for creating the synthetic legacy update code from the latest API invocation data?

A4: As stated in Sec. 2.3, to ensure benchmark quality, two authors manually verified the divergence between legacy and updated API invocations synthesized by DeepSeek-V3. On average, verifying and guiding re-synthesis required ~1 minute per invocation pair.

---

W5: The paper uses BLEU, ROUGE, and relative edit-distance scores for the code completion and error correction tasks. Not including exact match score or semantic-aware matching score (exact match with respect to parameter ordering) is not ideal.

A5: Thank you for the suggestion. We have addressed this concern by incorporating AST-based semantic matching in our evaluation in our response to Reviewer HLiK (Q1).

---

W6: The paper only focuses on function signature changes, whereas API changes may also include changes to function semantics.

A6: While API updates that involve purely semantic changes without signature modifications do occur, they are relatively uncommon and undocumented. Thus, our benchmark mainly focuses on API signature changes, as these typically reflect substantial semantic modifications and can be systematically tracked.

---

W7: Another limitation of the benchmark is the fixed legacy version (Jan 2023), which may favor LLMs with later knowledge cutoff dates.

A7: While LLMs with later knowledge cutoffs may have greater exposure to recent API updates, this does not contradict the benchmark’s objective—evaluating LLMs' ability to access and adapt to real-time API updates. In fact, this setup reflects a realistic scenario for assessing up-to-date knowledge. Nevertheless, even models like DeepSeek-R1 with recent cutoffs (July 2024), achieve low BLEU scores (e.g., 19.32), indicating limited effectiveness in handling evolving APIs. This further supports the need for explicit knowledge updating methods.

---

W8: The paper only includes 6 popular Python libraries, which may be a limitation of the benchmark.

A8: Although CodeSyncBench currently focuses on six widely-used Python libraries, our proposed CodeSync engine is generalizable. It can track API updates across any Python library over arbitrary time periods. Thus, the benchmark can readily be extended by broadening the version range and including more libraries as needed. We leave the extension to our future work.

Dear Reviewer HLiK,

We sincerely appreciate your suggestions and assessment of our work! Motivated by your feedback, we are committed to improving our manuscript and providing a more comprehensive benchmark and evaluations!

---

Q1: How do the results of the paper change if you use static analysis methods, such as AST matching, etc. instead of string-matching approaches for evaluation?

A1: Thanks for the suggestion. Actually, we have considered CodeBLEU to measure the effectiveness of API invocation updating from the perspective of AST matching. However, we found that, in practice, many LLM completions contained invalid syntax or natural language, causing them to fail the AST parsing (e.g., ~70% of Qwen2.5-Coder completions on our benchmark failed AST parsing).

As a result, we compute CodeBLEU using a fallback strategy: if an API invocation passes AST parsing, we compute CodeBLEU against the ground truth; otherwise, we assign a corresponding part score of zero. The results for Qwen2.5-Coder are shown below:

The experimental results align with trends from the original metrics, indicating that our conclusions remain valid under structure-aware evaluation. (Full results are available at: https://anonymous.4open.science/r/CodeSyncRebuttal-D65F ).

---

Q2: Why do you think current methods are limiting? What’s the expected upper-bound on performance? What kind of improvements are you suggesting will improve performance further?

A2: Thanks for your comments. Current knowledge-updating methods are limited in helping LLMs fully internalize and recall updated APIs, as evidenced by suboptimal CodeBLEU scores (<40) in the CCT task. The upper bound of performance can be quantified by evaluating LLMs’ ability to invoke legacy APIs. The empirical upper bounds for Qwen2.5 and Qwen2.5-Coder are as follows:

The results indicate a significant performance gap with upper bounds. To close it, we recommend exploring hybrid approaches, as discussed in response to Q3.

---

Q3: How does the simple RAG baseline compare with the methods discussed in the paper? When or when not should someone use your method as opposed to RAG to update API invocation knowledge?

A3: As stated in lines 104–109, unlike RAG which increases inference overhead by retrieval for each query, our CodeSyncBench focuses on evaluating and improving LLMs’ ability to internalize API updates. FT methods allows models to update without extra costs while inference, making them preferable in latency-sensitive or offline scenarios. Moreover, RAG's reliance on retrieval quality is problematic, especially with large real-world API updates. In contrast, CodeSync tracks API changes and integrates them into LLMs, enabling more reliable recall of updated API knowledge.

To address your concern, we test a RAG baseline with a vector database of 5,056 updated API signatures from six libraries mentioned in the paper, indexed with OpenAI’s text-embedding-3-large model.

Results show that RAG performs lower than FT methods (35.17), primarily due to a retrieval success rate of only ~60% (665/1100). However, RAG achieves better results (40.70), while combined with FT methods, highlighting the potential of hybrid approaches. In summary, knowledge updating methods outperform RAG in scenarios requiring efficient inference and precise API knowledge tracking.

We will incorporate these results and analysis into the revised manuscript.

---

W4: One concern that stands out for me is the lack of commentary on the leakage of samples from CodeSyncBench into the training data of models being evaluated.

A4: To demonstrate that there is no data leakage between training data and benchmark, we compute the n-gram similarity of all samples associated with each API. Lower n-gram (<15%) scores indicate minimal overlap and thus confirm the absence of leakage.

Specifically, after retrieving data for APIs, we duplicate all data and then split them into two parts. This approach has a clean separation and prevents any potential data leakage.

W5: The paper does not include contemporary works also focusing on API invocations and benchmarks developed.

A5: Thanks for the suggestion. We will include all the mentioned works in the next version of the paper.