[Weakness 1 / Question 1] High performance of Base model

See general response.

[Question 2] Question about evaluation procedure.

Your second interpretation is correct — “it only needs to solve one at a time, and these are treated as separate entries in the dataset (where each entry has 1 test synthesis problem and 2 training synthesis problems)”

[Weakness 2] Missing a minor baseline ablation of c=1 and r=2.

We ran the experiment and we found that the default setting of (c, r) = (2, 1) in our paper remains the optimal choice. See Table 5 and 11 of the updated draft.

[Weakness 3] The number of annotated failure cases instances in Table 8 (in previous draft).

We thank the reviewer for the very careful read. We examined 66 failed program synthesis examples, but each could have been assigned more than one failure reason. We added the clarification to Line 1764.

[Weakness 4] Scale of erroneous examples in the dataset

Yes, it’s only about 3% of the whole dataset at time of inspection.

Dear Reviewer,

As we approach the response deadline, we wanted to kindly check if our revisions have adequately addressed your concerns. If you have any remaining questions or concerns, we would be happy to address them.

Thank you for your time and consideration.

Best regards,

Authors

[Question 3] Example of random update in training examples in FT(PS) setting.

We give two examples as below: when training for update numpy.argsort, the target update has the following description:

Added a new parameter 'reverse' to the numpy.argsort function to support reverse sorting."

Compare to a random example taken from elsewhere in the dataset:

[random example to demonstrate task format]
[INST] Scenario: Peter is a physics teacher and is developing an application for his students to calculate real-world problems. During a test, a student entered a negative number causing the application to break since square roots of negative real numbers do not exist. Peter now plans to update his application such that it doesn't break but instead returns a predefined result when a student enters a negative number.
Problem: Peter wants you to write a function. This function should be designed in a way to handle both special and normal scenarios. In the usual situation, the function takes a real number and returns the square root of that number. However, in the special scenario, when students enter a negative number, instead of the program breaking due to an exception, it should return a predefined fallback number. The function must take three parameters: the first parameter is the number for which the square root has to be found, the second parameter is the fallback number and the third parameter will decide whether to use the fallback or not, if the first number is negative.
Solution signature: def calculate_sqrt(number: float, fallback: float, use_fallback: bool)
[TEST]
…
[/TEST]
[/INST]
### Response:
[PYTHON]
import math

def calculate_sqrt(number: float, fallback: float, use_fallback: bool):
    # if use_fallback is set to True and number is less than 0, return the fallback value
    if use_fallback and number < 0:
        return fallback
    # if use_fallback is not set and number is less than 0, let the math.sqrt function raise
    # a ValueError as it would traditionally do
    else:
        # The new math.sqrt function signature takes in two parameters
        # The first one is 'x' which is the number we need to find the square root of
        # The second one is 'fallback' which is an optional parameter that defaults to None
        # We use fallback parameter here to avoid raising an error
        return math.sqrt(number, fallback)

[/PYTHON]

when training for update torch.squeeze, the target update has the following description:

Change the name of the argument 'dim' to 'dimension'.

Compare to a random example taken from elsewhere in the dataset:

[random example to demonstrate task format]
[INST] Scenario: You're an engineer working in an AI based healthcare startup. Your team has been assigned with a task to analyze body scan images and extract diagonal arrays of pixel intensity values. These diagonal arrays will further be used to detect anomalies in human body scans.
Problem: Given a 2D array representing the pixel intensity of a medical scan image and a shift 'k' from the leading diagonal, you need to write a Python function that extracts the given diagonal. These diagonals serve as features for your AI models, where different diagonals might signify different medical conditions.
Solution signature: def extract_scan_diagonal(scan: numpy.array, shift: int)
[TEST]
…    
[/TEST]
[/INST]
### Response:
[PYTHON]
import numpy

def extract_scan_diagonal(scan: numpy.array, shift: int):
    # Use the updated numpy API function
    res = numpy.diag_extract_or_construct(scan, shift)
    
    return res

[/PYTHON]

[Question 4] High performance of Base model

See general response.

[Weakness 1 / Question 1] The model may be unfairly penalized by UPass@k

Thanks for the suggestion! In our updated draft, we also report Pass@k in Table 4; this new metric measures the execution success with updated API in the environment. Under this metric, the model can pass tests without using the updated API. Unsurprisingly, Pass@k reports higher values for all models.

In the following table, we address the reviewer’s concern by showing the model is NOT generating a correct program (without using updated function) more than it does in the Base model. We think the delta (Pass@k - UPass@k) reflects a certain number of problems where other solutions naturally exist. As we could see from the table, compared with Base, such gaps either remain on par, or shrink after (in-context or in-weight) updates. This implies that the updated models do not solve the problem without using updated function and thus we are not unfairly penalizing the model.

[Weakness 2] Quality control for a LLM-generated dataset.

See general response.

[Question 2] How many gradient updates are performed when training on update docstring? And is it optimal?

In Table 4, we perform 10 gradient update steps on docstring (FT(U)) with a causal language modeling objective. We ran the FT(U) experiment in Table 4 with various gradient steps (see Table 10 in updated draft) and concluded the choice of 10 gradient updates is optimal. We include the table below for convenience.

Dear Reviewer,

As we approach the response deadline, we wanted to kindly check if our revisions have adequately addressed your concerns. If you have any remaining questions or concerns, we would be happy to address them.

Thank you for your time and consideration.

Best regards,

Authors

[Weakness 1] Limited size and diversity of the dataset

While we agree that other languages and libraries could be included, we believe that the number of functions is sufficient to support our claims. We sample these from different packages and with different types of update to cover the space of update types. For demonstrating superiority of one editing method over another, our benchmark is large enough to show this for any practically significant performance delta, assuming that the dataset reflects representative use cases.

[Weakness 2] Core contribution

We would like to emphasize that our core contribution is a dataset that provides a testbed for future researchers to investigate in teaching code LLM about package update and ensure it could use the update for problem solving beyond regurgitation.

[Question 1] How would experiment results change if the training dataset is significantly increased?

Past work like [1] shows what happens in a different regime where far more data is available. Although our work supports multi-edit scenarios, we do not provide examples for conducting these types of experiments. We hope our work encourages future researchers to develop more sample-efficient algorithms to update code LLM methods. We believe this is important for the method to be scalable and deployed online (i.e., dealing with API updates from common libraries every day/week).

[1] Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Dear Reviewer,

As we approach the response deadline, we wanted to kindly check if our revisions have adequately addressed your concerns. If you have any remaining questions or concerns, we would be happy to address them.

Thank you for your time and consideration.

Best regards,

Authors

[Weakness 1] Effort to make synthetic API reflect realism

See general response.

[Weakness 2] Related work positioning.

We have updated the draft and moved the related work section; thanks for the suggestion.

[Weakness 3] Naming the benchmark “Arena”

While Chatbot Arena is the most well-known benchmark at the moment, multiple prior work (LongRangeArena (https://arxiv.org/abs/2011.04006), WebArena (https://arxiv.org/abs/2307.13854), Arena (https://ojs.aaai.org/index.php/AAAI/article/view/6216) used “Arena” to refer to a place to test the model (which reflects our use case) without invoking side-by-side comparison and human input.

[Weakness 5] Extend synthetic API generation to APIs like web applications and visualization packages

We mention this in our limitations paragraph. We foresee that how to evaluate the correctness of such scenarios automatically is a chief barrier to doing this, with no good solution easily available.

[Question 1] LoRA Finetuning (our work) vs Fully-Fine Tuning (FFT) with data augmentation

In our preliminary experiment on a different dataset, we found LoRA outperforms full fine-tuning (FFT). Our use of LoRA follows that of other recent editing work like [1]. We chose to use LoRA for our experiments as it is substantially more computationally efficient than full fine-tuning (10x). Nevertheless, we believe that sufficient data augmentation compared with full fine-tuning, like RAFT, could lead to improved performance.

[1] Gangadhar and Stratos. “Model Editing by Standard Fine-Tuning”. arXiv 2024.

[Question 2] Whether updated model becomes stronger in other capabilities

Since our core contribution is the dataset that opens a new domain for code generation, we think the answer to this intriguing question is left for future work. To our best guess, finetuning would not lead to improved capabilities, as the learning itself doesn’t involve teaching the model more things beyond the target update. Also, the vanilla finetuning algorithm doesn’t “propagate” new knowledge well — finetuning the model on docstrings doesn’t teach the model to solve related program synthesis problems.

[Question 3] The base model we chose to conduct the experiment is not strong enough.

We respectfully disagree with this point. We found DS-Coder-Instruct-7B is the strongest open model (at comparable size) so far; according to their paper (https://arxiv.org/pdf/2401.14196), it surpasses GPT-3.5 Turbo and are only less than GPT-4 by 6% on HumanEval (Python). Therefore, what we observed in DS-Coder-Instruct-7B may be reflective of what might happen to a strong code model like GPT-4.

Dear Reviewer,

As we approach the response deadline, we wanted to kindly check if our revisions have adequately addressed your concerns. If you have any remaining questions or concerns, we would be happy to address them.

Thank you for your time and consideration.

Best regards,

Authors

[Weakness 1 / Question 1] Justification of synthetic data instead of using historical API updates.

See general response.

[Weakness 2] Multiple versions of API can be useful for users

Thank you for bringing this point. However, we don’t see this as a weakness impacting our work. The current dataset tests the ability to overwrite an old API with a newer version. One could use this capability to inject knowledge associated with a particular library version. A knowledge injection method that works on our dataset can generalize to that use case as well.

[Weakness 3] Scalability of updating LLMs for each update

Since our benchmark contains multiple updates for multiple libraries, future researchers can use our dataset to investigate techniques that update multiple APIs at the same time or in sequential order (at least, on a subset of examples in our dataset that do not update the same function).

[Weakness 4 / Question 3] Robustness of evaluation metric

The robustness of our observation from result tables comes in expectation over the many synthesis examples of many updates — we test on 82 HumanEval examples (a quantity larger than prior work [1,2]), after we finish updating the model on each API update. We note that this is a common practice in other benchmarks; for example, RAG benchmarks don’t involve exhaustively answering questions about the entire content of a document. Additionally, we confirm such a difference with a bootstrap significance test with p < 0.05.

[1] Locating and Editing Factual Associations in GPT, Meng et. al, NeurIPS, 2022

[2] Propagating Knowledge Updates to LMs Through Distillation, Padmanabhan et. al, NeurIPS, 2023

[Weakness 5] Headroom of the benchmark

The goal of our benchmark is to test updating methods, not specific models. In this work, our LLMs like GPT-4 and Claude are prompted through RAG, which doesn’t do in-weight parameter updates. These models aren’t addressing the task at hand, so the high performance numbers have no bearing on knowledge updating.

[Question 2] Quality control for an LLM-generated dataset

See general response.

Dear Reviewer,

As we approach the response deadline, we wanted to kindly check if our revisions have adequately addressed your concerns. If you have any remaining questions or concerns, we would be happy to address them.

Thank you for your time and consideration.

Best regards,

Authors