CursorCore: Assist Programming through Aligning Anything

Thanks for your review. We sincerely appreciate your recognition of our work.

Thanks for your review. Please see our detailed response below:

Claims and Evidence & Q1

Thanks for the suggestion. We can provide the results of removing H from the samples in APEval that contain H, and compare them with the results obtained without removal. This ensures a fair comparison under the same conditions. The evaluation results are as follows:

The settings are consistent with those used in Appendix I. Additionally, the results presented in Appendix I further support it, as they provide evaluation results under different window lengths of H.

“let the LLMs judge whether each segment of M aligns with the user's purpose through principle-driven approaches” is ambiguous

Thanks for the detailed review. We did prompt the LLMs to first analyze the user's intent before making a judgment, as shown in Table 18. We have corrected this in the revised version.

It would be better if the benchmark included more complex tasks

Including more benchmarks is certainly beneficial! While, the prompts in the DS1000 and ClassEval lack historical context. Recently, a new dataset called Zeta has become available on HuggingFace (released just last month, so it was not possible to include it before the ICML deadline). Zeta is constructed from real engineering use cases, and better reflects the distribution of tasks in real-world scenarios. It include H (though not U). We have now conducted evaluations on this benchmark, and some results are as follows:

The reported metric is the average accuracy over all evaluation samples. We use GPT-4o to assess the correctness of the generated results based on the assertion texts.

Of course, we can also evaluate DS1000 and ClassEval, to assess the model's ability to leverage U and C. We choose to evaluate the performance of CursorCore on these benchmarks using the Inline and Tab modes, as they most closely resemble the original formats of them. Some results are as follows:

The reported metrics are averaged over all samples, with class-level generation evaluated using ClassEval. All generations are performed under the greedy decoding setting.

Appendix J also includes another open-domain code editing benchmark (CanItEdit). These results collectively demonstrate the strong effectiveness of CursorCore.

Performance of Qwen2.5 and Concerns about bias in APEval (Q2)

We have included a discussion of this issue in lines 1130 to 1135. Moreover, while instruction-tuned models are effective at aligning with prompts for program synthesis and code repair, they struggle to align with various forms of contextual information, such as historical context, which is commonly encountered in programming workflows. Therefore, the strong performance of Qwen2.5-Coder-7B-Inst on the synthesis and repair tasks, coupled with its weaker performance on APEval, is expected.

Regarding your concern about bias in APEval, we have already taken this into consideration during the annotation process. As described in Appendix C, we ensure that the current code is sufficient to indicate the user's next intent.

Explanation of user instruction (Q3)

For "user instruction," we refer to general prompts or feedback related to the code. Alternatively, from another perspective, if we consider the historical context as representing changes in the internal state, then the user instruction can be viewed as an external signal. The term is defined broadly—any type of prompt or feedback that may assist programming can be regarded as a user instruction, such as "write a quicksort algorithm," "translate this code into Java," or feedback messages like "Traceback...". In normal conversation templates of instruct models, this is commonly labeled as "user" or "instruction." To maintain compatibility and in consideration of a previous reviewer's suggestion, we choose to use it.

For HumanEval, we employed different modes, as described in lines 1093 to 1128. In the Tab mode, we leave the docstring as part of the code context; in contrast, we transform the docstring into a user instruction. This setting helps align the evaluation inputs more closely with those encountered in real-world applications.

Typo

We have fixed them. Thanks for your careful review.

We appreciate your review and look forward to your response!

Thanks for your review. Please see our response below:

Baseline of prompt-engineering to incorporate H is missing

The reviewer may have misunderstood our experimental setup. Our prompt engineering baseline does include H, as shown in Tables 19 and 20.

No empirical justification for incorporating H. Experiments show that (H, C) which includes historical context is poorer in performance compared to (C) which does not & Q1

A direct comparison between the C and H+C subsets is impossible, as they involve different problem sets. Detailed APEval annotations are in Appendix C. The H+C subset is more challenging due to ambiguous or irrelevant history, making it harder for models to leverage and resulting in lower performance.

While, as reviewer mFUu suggested, we can remove H from the H-included subset for direct comparison; please see our response to mFUu.

Historical information is essential in many cases. For example, during variable renaming, a model without access to edit history may mistakenly revert the name. Similarly, if a user writes and later deletes a draft, only the edit history reveals its intention.

No sufficient work to identify these cases

For the training data, we do not need to explicitly label which cases require historical information. In practice, the collected historical data naturally includes both helpful and unhelpful contexts. We must train LLMs to make predictions based on such inputs containing noise.

For the benchmark construction, we identify cases where historical edits are necessary to infer the intended changes, shown in Appendix C.

APEval does not thoroughly capture the relevance of historical user edits & It would not benefits from historical context

As noted above, we already considered this during annotation. We suspect the reviewer may have misunderstood our approach: we did not simply add historical edits to the original code. Since HumanEval docstrings include rich context like functionality and test cases, doing so would offer little benefit. Instead, we removed all docstrings and only showed annotators the signature and its purpose, making it significantly harder. In this setting, historical context becomes more helpful.

Collect user edits from GitHub and OJs is not reflective of data authors are seeking

The reviewer is concerned that the collected historical edits may already be “complete,” with no need for further edits. Our data collection pipeline has taken it into consideration. We adopt the following perspectives and methods:

1. For data from OJs, we retain only the user’s first correct submission and submissions preceding it.

2. As shown in Section 4.2, we use LLMs to judge whether code changes align with user intent based on historical and current context. If the current version is complete and changes are version updates (e.g., adding auther information), the LLMs are designed to filter out such updates, while meaningful edits are retained.

HumanEval-style of evaluations suffers from contamination & does not reflect real-world use cases & Q2

The base models’ technical reports confirm HumanEval contamination was addressed. We also cleaned our training data (Section 5.2). Besides, our benchmark inputs were significantly modified which further reduce contamination risk.

While HumanEval-style benchmarks differ from real-world scenarios in data length (function/file vs. full repositories) and use of third-party libraries/tools, we still chose to extend this benchmark because:

1. At the time of our experiments, no public benchmarks included all contextual elements like historical edits and user instructions. Thus, we chose to extend existing benchmarks as the most efficient way to assess models on basic tasks in this paradigm; directly creating a more complex benchmark might obscure performance differences.

2. Repository-level benchmarks (e.g., SWE-Bench) target different goals than ours (see historical context for agent).

Furthermore, a newly released dataset Zeta is constructed based on real-world engineering cases with historical edits; please see our response to mFUu.

Examples in Figure 3 are from a real-world source; we chose the shortest one to aid clarity and fit space constraints.

Should also include human annotation rubric/results in Section 3.1

Thanks, we agree it is important. Due to space limits, it's in the appendix for now, but we will add the detailed discussion to Section 3.1 in the camera-ready version.

Historical context for agent & Q3

Including such context could help the agent better understand user intent and adapt to coding style! While, our work focuses on single-call LLM usage with strict latency requirements, unlike agentic workflows, which involve multi-turn interactions and prioritize end-to-end performance. Integrating context into such workflows is a great direction for future work.

Typo

We have fixed it. Thanks for the careful review.

We appreciate your review and look forward to your response!