Nemotron-CORTEXA: Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity

We truly appreciate the time and effort you have put into reviewing our work and are grateful for your insightful comments that help improve our paper. For comparisons to SOTA works, please refer to our response to Reviewer WZSG. We would like to highlight that Cortexa outperforms Agentless in several key areas:

1. Patch Generation Efficiency: Our reported performance is with only 9 patches, whereas Agentless requires 40 patches. When generating a comparable number of patches as mentioned in response to Reviewer WZSG (32 patches), Cortexa significantly outperforms Agentless, achieving 58.8% on the Verified set (p-value = 0.006).
2. Localization Performance: While Agentless uses a combination of prompt-based and embedding-based approaches, our specialized embedding model achieves higher retrieval accuracies. From Table 2 in the Agentless paper, its localization cost is $0.15/instance, while Cortexa’s localization cost is $0.11/instance (including file and entity localization) as shown in Appendix A.2.
3. Recall in Oracle Entity Identification in Table 3: Our recall rate in identifying the oracle entity is statistically superior to Agentless, with a p-value of 0.0007.

We will release our code including all the prompts we used, full logs, and our embedding model upon the paper’s publication.

Additional baselines on retrieval: Here are the new baseline results on the Verified set:

1. Temperature sampling and ensembling across multiple stages instead of ensembling multiple models: We use DeepSeek-v3 here as it is individually the strongest in entity localization. Neither method can match Cortexa’s recall. The temperature sampling approach obtains a higher precision since it retrieves fewer entities. However, recall matters more for patch generation to provide contextual information. The results show that model diversity is the most effective approach here.

2. Agentic retrieval baselines: We select 4 agentic approaches here: OpenHands, AutoCodeRover-v2, MarsCode Agent, and SWE-agent-v1. For each agent we extract files from their official trajectory logs in the SWE-bench leaderboard. Since these methods can retrieve an arbitrary number of files, we denote the accuracy metric with recall@inf. In this comparison, we use recall@10 for Cortexa as other baseline methods retrieved at least 15 files on average.

For entity retrieval accuracy, these agentic approaches often do not explicitly retrieve entities. Moreover, there is no exact way to measure entity retrieval results from their logs as there is no fixed pattern. Thanks to its localization agent step, Cortexa provides a concrete way to measure the more granular entity retrieval accuracy, which proved helpful to iterate and improve the agent.

Impact of diversifying generations: To address this comment, we run the following experiments on the Verified set:

1. Taking the context (localization agent, LA) and the edit format (search/replace) that give us the highest pass rate with temperature 0 and generating 9 patches for it (1 greedy sampling and 8 temperature sampling).
2. Changing the edit format to use both edit formats.
3. Changing the context to other choices of DP (direct prompt), LA+DP, and File that uses the results of the embedding model.

Table below summarizes the pass@9 and pass@1 (after applying the filtering steps outlined in the paper) results for each combination. These results support our claim that diversifying both the edit format and context increases the number of correct patches, as each combination has its own strengths and weaknesses, and diversification allows us to leverage all of them.

Context filtering: In our prompts, we ask the agent to filter irrelevant context by issuing a special <keep> command after consuming a list of code context. We will provide these prompts upon code release.

Chunking strategy: We experimented with the SFR-Embedding-Mistral model starting with a chunk size of 4k and gradually reducing it. We observed a recall@k of 43.67% at a chunk size of 4k on the Lite set and it peaked at a chunk size of 450 with recall@k of 57.33%. Based on these results, we selected a chunk size of 450 for both training our model and final inference. Similar to the SFR-Embedding-Mistral model, our model's retrieval accuracy decreases with larger chunk sizes. For instance, with a chunk size of 4k, the retrieval accuracy on the Lite set is 35.67%, compared to 70.33% with a chunk size of 450. We will include the full experimental results in the paper.

Thank you for taking the time to review our work and for offering constructive feedback to help enhance our paper. Regarding the comparison to the SOTA works, please refer to our response to Reviewer WZSG. Additionally, our sample efficiency is demonstrated by achieving higher resolution rates than Agentless with only 9 patches, compared to Agentless's 40 patches.

Importance of different components: One of the main messages of our work is that improved localization of the issue is key to increasing resolution. Our initial experiments demonstrated that when GPT-4-turbo was given the ground truth file plus additional relevant files to fill the context window, it solved only 4 Lite set instances with 1 greedy patch. Providing only the oracle file increased this to 53 instances, and narrowing to ground truth functions/classes reached 72 instances. The dramatic increase in instance resolution from narrowing the context suggests that models are severely impacted by superfluous context and can benefit from a more localized edit location. This observation is the reason why we focused our work on better localization.

To verify if this phenomenon holds in our final end-to-end results, we ran new experiments by generating 9 patches with the results of the file localization vs the entity localization, as shown in response to Reviewer sboj. Using the entities retrieved by the localization agent (LA) as context, we get to pass@1 of 245, while the pass@1 with File retrieval results is 223. This is consistent with our early observations.

Our second key contribution concerns generating diverse candidate patches. We induce diversity by using the variety of different contexts obtained throughout the localization process, in addition to requesting different edit formats as outputs from the language models. Using the standard temperature sampling with the LA results and search/replace edit format, we obtain a pass@9 of 270 and pass@1 of 237 after the filtering steps mentioned in Section 3.5. Using our approach to maximize patch diversity we increase the pass@9 to 307 and the pass@1 to 263. A further advantage of this patch diversity method is that it is straightforward to adopt and makes more effective use of the existing sampling budget by leveraging the strengths of different contexts. We believe this technique has been overlooked by previous works and could easily be adopted independent of Cortexa.

As demonstrated in Table 2, our embedding model significantly influences file localization. To further evaluate its effect on entity localization, we performed new experiments on the Verified set by running the entity localization step with file retrieval results from the base model. The localization agent takes a list of files from the file localization step as the starting point and navigates the repository as defined in Section 3.3. Table below indicates that starting with more accurate relevant files enhances the agent’s success in identifying oracle entities.

Localization model: We chose NV-Embed-QA as our pretrained model as it is the strongest commercially-available text embedding model in our tests. Its commercial availability aligns with our goal of releasing Cortexa and its associated retrieval model. Our dataset includes 290k query-positive pairs from 19k issues across 35 non-test repositories with positive passages from oracle files in the golden patch. Additionally, we generated 125k coding question-answer pairs using DeepSeek-v2.5. We also included data from APPS, CoSQA, Text2SQL, CodeTransOcean, and StackoverflowQA. We will add a detailed table of all data to the paper. While we acknowledge the potential of the embedding model for broader coding tasks, its purpose here was to help with retrieving relevant files to an issue description. A more comprehensive analysis of the embedding model on diverse datasets falls out of the scope of this paper and will be addressed in a future work.

Claude for Table 2: We agree that several strong LLMs exist that may surpass GPT-4o here such as Claude, however we use Agentless as our baseline and thus wanted to facilitate comparisons by keeping the same LLM. Another important motivation in our choice of embedding models over prompt-based approaches is because many LLMs including GPT-4o and Claude have been trained on these exact repositories, allowing them to identify relevant files by name alone. By contrast, our embedding model does not suffer from this same test leakage and thus ensures that the gains cannot be attributed to memorization from a specific LLM. Furthermore, reading the full contents of the files with these LLMs is prohibitively costly, as shown in CodeMonkeys (Ehlrich et al. 2025), whereas our embedding model achieves equal or better accuracy using only a fraction of the cost.

Thank you very much for your valuable feedback. We appreciate your thoughtful comments.

Comparison to SOTA: In response to your comment on Cortexa's performance compared to other works, we would like to emphasize that our primary goal was to identify ways to increase the efficiency of coding agents. We observed that while free-form agentic flows are effective here, their vast pool of actions often leads to numerous costly iterations. Similar to Agentless, we aim to develop a more structured approach by identifying and improving key stages crucial for the final task, ultimately increasing efficiency and performance simultaneously. This approach also simplifies debugging and human intervention.

For instance, while OpenHands achieves a 26% resolve rate on SWE-bench Lite at $1.1/instance (with the cost for other submissions unreported), Cortexa achieves a 42% resolve rate at $0.51/instance. We recognize that Cortexa's performance can be further improved by refining the selection process and/or scaling the number of generated patches. These optimizations are orthogonal to our approach and are left as a future work. Currently, the majority voting in the patch selection process is suboptimal, especially since we diversify the solutions generated, and a correct solution may not appear multiple times. Indeed, we observe that Cortexa’s performance can be increased by adopting a better patch selection approach - we’ve switched from a majority vote to using an LLM for selection, and pass@1 on the Lite set increases to 42.67% and on the Verified set to 54.8%. Additionally, another approach that brings our results in line and beyond most open approaches is scaling the number of generated patches. For instance, increasing it to 32 (16 by Claude-3.5-Sonnet and 16 by DeepSeek-V3) increases the performance on the Verified set to 58.8%

Confidence interval: We used a temperature setting of 0 for all LLM calls, except for 4 out of 9 generated patches during the repair stage. Each of these 4 patches utilized different combinations of context and edit formats. To address the concern about stochasticity of LLM calls, we generated two additional patches for each combination (temperature sampling), resulting in 81 variations of the final 9 patches. The table below shows the peak performance statistics of these 9 patches on the Verified set. As mentioned in Appendix A.1, the patches used in the paper achieved a peak performance of 307 instances (61.4%) which falls within the confidence interval. We will include these results in the paper.

Ensemble increases precision: Indeed, this is a surprising effect we observe. In Table 4 we reported precision/recall averaged across instances. To illustrate why ensembling is helpful consider the following example retrieval task over 5 documents (A, B, C, D, E) and two queries and relevant documents {Q1, (A,B)} and {Q2, (C,D)}. Say method 1 returns (A, C) for Q1, and (A, B) for Q2; method 2 returns (C, D) for Q1, and (B, C) for Q2. Method 1’s mean precision is (1/2 + 0/2) / 2 = 1/4. Method 2’s mean precision is (0/2 + 1/2) / 2 = 1/4. The union for both methods results in (A, C, D) for Q1 and (A, B, C) for Q2. The ensemble mean precision is (1/3 + 1/3) / 2 = 1/3 greater than either methods’ precision (1/4). For each question instance, the union ensemble has a lower precision (1/3) than the highest individual precision (1/2). This phenomenon explains why our ensemble method benefits from models’ strengths in solving different problem subsets.