CursorCore: Assist Programming through Aligning Anything

Thanks for your review. Please see our detailed response below:

This point about history (H) not being incorporated in prediction has been made previously in many papers in software engineering conferences

Our work focuses on proposing a framework that can integrate various types of information, rather than being limited to historical edit records alone. The software engineering community has made significant contributions to utilizing historical edit records, but most of these approaches either rely on prompt engineering techniques or involve modifications to encoder-based or encoder-decoder architectures such as CodeBert or CodeT5, which are not easily applicable to the current LLM training that primarily uses decoder-based architectures. We will update the related work section to reflect these points.

but there is no mention of releasing the benchmark set in the paper

We release all data collection pipelines, models, and benchmarks. Due to the anonymous review policy, we cannot disclose public code link. We will include the corresponding links in the abstract of the final published version.

inclusion of H almost always leads to worse performance

This is a normal result because the evaluation samples for different data types are distinct. During the benchmark creation process, in order to better assess the model's ability to utilize historical edits and integrate them with user instructions, we specifically collected data that requires relevant information for inferring user intent. For example, in the case of the H+C and H+C+U categories, these samples were designed to require related information to answer the user query. If a sample only includes data from a single category (such as only C or only U), it might be insufficient to answer the query due to a lack of information. The difficulty of utilizing historical edit information is even greater. Although it can be automatically captured without requiring user input, it reflects user intent in a more ambiguous way, or it may contain irrelevant or noisy records. Therefore, it is harder for the model to make use of this information, which is why the setting with H performs worse than without H.

The code (C) and user interaction (U) part is already part of most LLM-based programming assistants.

In the current implementation, the components C and U are mixed together in both the model's input and output. Many existing tools handle them through prompt engineering and post-processing techniques to achieve features like instructional editing. However, models like ChatGPT produce outputs in a free-form conversational style. While this format is user-friendly and easy to read, it may hinder automated parsing of the output. For example, to quickly highlight edited sections, LLMs might omit unchanged parts or split the content into multiple snippets, which can pose challenges for applications. To address this, we separate the C and U components in our framework design, making it easier to process the input and output of C.

I did not find any conversational interactions here / What is the conversational bit in the Assistant-Conversation framework?

We refer to the 'conversation interaction' here just as the interaction between human programmers and AI programming assistants (models). Our Assistant-Conversation framework is proposed as an extension of frameworks similar to ChatGPT's chatbot model. Unlike traditional chatbots, which typically involve only user instructions and model responses, our framework incorporates edit history, current content and user instructions.

Thanks very much for your response!

We will break down and detail the novel contributions of our work as follows, highlighting the innovative aspects:

1. Consolidate historical edits, current code, and user instructions into a unified format for edit prediction

   We agree with your point that leveraging historical edits has been explored in prior software engineering research. However, some programming assistance scenarios require the simultaneous consideration of historical edits, user instructions, and other contextual information. As we know, previous works have typically focused on a single type of information, such as historical edits or user instructions, without exploring frameworks or models that integrate multiple types of information. Our work is the first to explore it.

2. Novel Data Collection Pipeline

   We developed an innovative data collection pipeline to automatically gather datasets containing diverse information about the programming process. As reviewer fkg9 mentioned, fine-grained coding history data can be difficult to obtain, this synthetic approach is great for this task. And as we know,, such a method has not been explored in prior work.

3. Benchmark Contribution

   The benchmark we proposed is another significant contribution. It was made publicly available over a month ago, but due to the double-blind review policy, we are unable to include the corresponding link here. We will add the link in the abstract of the final public version of our paper.

4. Models

   We release the collection of trained models, which are useful for deployment in applications such as automated editing. As mentioned in the introduction section, existing state-of-the-art open-source Code LLMs are not well-suited for these tasks, and no closed-source APIs currently address these needs. As we know, this is the first fine-tuned Code LLMs specifically trained and released for such tasks.

5. Design the conversational framework based on the characteristics of modern inference engines for decoder-only LLMs

   Successfully addressing this task requires considering AI/LLM mechanisms (e.g., fine-tuning and inference), systems (e.g., prefix caching and speculative decoding), and software engineering (e.g., specific forms of programming assistance tasks) collectively. For example, tasks like pressing the Tab key for auto-editing demand extremely low latency, requiring us to leverage modern inference systems to optimize speed in the design of conversational framework. As mentioned in Section 2.3, prefix caching is an optimization technique applicable only to decoder-only LLM architectures. We specifically adjusted the conversational framework to accommodate this technique, similar ideas and principles has not been explored in prior work.

   Additionally, some previous works utilized encoder-based methods for localization, these approaches cannot be directly applied to modern decoder-only LLM architectures. Hope this clarifies the point you did not fully get.

This work spans multiple fields and incorporates a significant amount of content. Successfully integrating all these elements is essential to achieving the final application. We use terminology more commonly employed in the AI/LLM domain to align better with ICLR's focus. We fully understand the questions and concerns you raised while reviewing our work, and are deeply grateful for the time and effort you have dedicated to reading and understanding our work.

We sincerely appreciate your review again and look forward to your response!

Thanks for your review. Please see our detailed response below:

As a benchmark paper, this paper doesn't provide a very convincing evaluation dataset

The reviewer seems to have misunderstood our experimental setup. Our paper is not a benchmark paper, the proposed new benchmark is only part of the work; it encompasses a framework, data collection, training, and benchmarking, among other components. The training dataset we collected and the benchmark used for evaluation are entirely different. The training data was synthesized using LLMs, and during this process, techniques similar to AI Judge were employed to filter the data. The benchmark used for evaluation was manually annotated by human programmers. This benchmark naturally reflects real-world editing needs and is tested using execution-based metrics to ensure precise evaluation of functional correctness. These details are discussed in section 3 and 4 of our paper. The separation between training data construction and benchmark creation is a common practice in the alignment of LLMs.

The paper doesn't make clear comparison with dataset curated from github/coding submits and synthetic data

We present concrete examples in Figure 2 and provide statistical information from different data sources for further analysis in Figures 5 through 8. During the training of code LLMs, it is not feasible to perform cross-validation on the training data. Currently, evaluation of code models primarily relies on execution-based metrics such as Pass@k to ensure correctness, which requires the sample to be executable and accompanied by accurate test cases. However, the training data does not include corresponding test cases, and some code snippets have highly complex dependencies, making execution nearly impossible. Therefore, for training code LLMs, the approach typically involves training with different data combinations through ablation studies, and comparing results on benchmarks, which is the method used in our paper. And Due to the high cost of training large models, many current works omit comprehensive ablation experiments on the training data in order to accelerate the experimental process. [1] [2]

[1] OctoPack: Instruction Tuning Code Large Language Models, ICLR 2024

[2] Magicoder: Empowering Code Generation with OSS-Instruct, ICML 2024

We appreciate your review and look forward to your response!

Dear Reviewer fkg9,

Considering the author-reviewer discussion is ending very soon and the updated score you mentioned is not yet reflected in the system (currently still showing Rating: 5 and Confidence: 4), we want to follow up to ask if there is anything else you would like to discuss (or just a minor technical issue with updating the score in the system). Thank you again and look forward to your confirmation!

Thanks for your review. Please see our detailed response below:

Questions about our benchmark

We do not rely solely on code generation tasks; instead, we cover various code editing scenarios. We only use the testcases and function names from the HumanEval (+) benchmark, without utilizing any other content. The historical information, current code, and user instructions are provided by annotators based on the specified function functionality. For instance, our benchmark accommodates scenarios that require edits and historical context. We illustrate this with two simple examples to facilitate quick understanding.

Example 1:

# Current
def has_close_elements(n, t):
    for i in range(len(n - 1)):
        for j in range(i + 1, len(n)):
            if n[i] - n[j] < t or n[j] - n[i] < t:

This code check if in given list of numbers, are any two numbers closer to each other than given threshold. The current conditional logic in this code has an issue. We should check whether the absolute difference between the two values is less than t. Therefore, the model needs to recognize this and, in addition to generating the remaining code, also edit the erroneous code accordingly.

Example 2:

# History 1
def incr_list(l: list):
    return [x++ for x in l]

# Current
def incr_list(l: list):

This example corresponds to a situation where a programmer writes an incorrect piece of code and then deletes it to add the correct version. However, if we only retain the current code, the model cannot infer the specific purpose of this function. For instance, when passing the current code to ChatGPT, it would only respond with something like this:

def incr_list(l: list):
    # ... existing code ...
    {{ edit_1 }}
    # ... existing code ...

While by incorporating historical editing information, the model can infer the user's intent and make accurate code edits. In our benchmark, there are more complex scenarios where the model needs to combine various information to align with the programmer's intent.

During the process of creating the benchmark, in order to better evaluate the model's ability to utilize historical edits and integrate this information with user instructions, we collected samples for the H+C and H+C+U types that required the use of relevant historical information to accurately infer user intent. If a sample contained only a single category of data (such as only C or only U), it might be impossible to provide an adequate answer due to a lack of sufficient information. Leveraging historical edit information is particularly challenging, as it can be automatically captured without user input, but often reflects user intent in a more ambiguous manner or includes irrelevant or distracting records. Consequently, models find it more difficult to utilize this information effectively, and settings that include H perform worse than those without H.

The experimental results analysis is insufficient.

The reviewer may have misunderstood our experimental setup, and we would like to clarify. Our training is based on base models because fine-tuning LLMs is generally more effective when done on base models rather than on instruction models. The case mentioned by the reviewer pertains to specific categories in our 6B+ model comparison, where our model was compared against the official instruction-tuned versions, not with the base models. Our model significantly outperforms the base models across all categories (average 22.6%) and shows notable improvements in overall performance compared to the corresponding instruction models (average 9.7%), even though our training data is much smaller than the official instruction versions. Additionally, prior models required long prompts to handle this task, whereas CursorCore does not, which is highly beneficial for practical applications.

def if_continuous_modify(code1, code2, code3):
    dist1 = Levenshtein.distance(code1, code2)
    dist2 = Levenshtein.distance(code2, code3)
    dist3 = Levenshtein.distance(code1, code3)

    if dist3 == dist1 + dist2:
        return True
    else:
        return False

def blockwise_if_continuous_modify(code1, code2, code3):
    dist1 = Levenshtein.distance(code1, code2)
    dist2 = Levenshtein.distance(code2, code3)
    dist3 = Levenshtein.distance(code1, code3)

    past_diff_blocks = generate_diff_blocks(code1, code2)
    new_diff_blocks = generate_diff_blocks(code1, code3)

    if dist3 == dist1 + dist2 and len(past_diff_blocks) == len(new_diff_blocks):
        return True
    else:
        return False

Thanks for your review. Please see our detailed response below:

The improvements with using CursorCore is not clear

CursorCore demonstrates significantly improved performance compared to the base model (average 22.6%). It delivers the best results while requiring far less training data than official instruction-tuned versions (average 9.7%), as shown in Table 4. Additionally, previous models needed lengthy prompts to handle this task, whereas CursorCore does not—an important advantage in practical applications.

The conversational framework needs be motivated better as its hard to understand why this system was chosen

Thanks for the reviewer’s suggestions. The Introduction section has explained the challenges of the current model in practical applications and the necessary directions for improvement. We will further elaborate on the connection between these issues and real-world applications, as well as the motivation behind the design of the Assistant-Conversation Framework, in the transition between the Introduction and section 2. This will help readers gain a clearer understanding.

Formatting & Writing

We will update the relevant captions and the related work section to improve the overall presentation of the paper. We will make the following changes:

• Provide a detailed explanation of two examples in the caption of Figure 2, each using different types of information.
• Provides additional information on the statistics of H, C, and U in Table 1.
• In the caption of Figure 7-8, add analysis of the differences in distribution of different data sources.
• Add discussion of other work that utilizes historical information to the related work section.

We appreciate your review and look forward to your response!

We sincerely thank the reviewers for their insightful and positive feedback, and their time during the review process!

In response to the reviews and to provide further clarification, we have made the following revisions to our paper. Revisions are highlighted in blue for clarity.:

• Further elaborate on the connection between real-world applications and the motivation behind the design of the Assistant-Conversation Framework in the transition between the Introduction and section 2. (jgRp, zutF)
• Provide a detailed explanation of two examples in the caption of Figure 2, each using different types of information. (jgRp, zutF)
• Provides additional information on the statistics of H, C, and U in Table 1. (jgRp, zutF)
• Add analysis of the differences in distribution of different data sources. (jgRp)
• Add discussion of other work that utilizes historical information to the related work section. (jgRp, zutF)
• Add an explanation on how the benchmark data was collected for the targeted evaluation. (13qq, zutF)
• Add figures of benchmark examples. (13qq, zutF)

We hope these clarifications and updates address the reviewers' concerns. Thanks for your kind consideration!