OrcaLoca: An LLM Agent Framework for Software Issue Localization

We thank the reviewers for their thoughtful feedback and for highlighting several key strengths of our work. We listed our responses for weaknesses and questions below:

Weakness 1: While the overall approach is innovative, parts of the paper are difficult to parse... Ensuring the paper is self-contained and introducing the agent framework more gradually would significantly improve readability.

Response: Thank you for your useful advice. We agree that Section 3.1 contains a lot of technical information in a small space, which may hinder readability. In the revision, we will summarize the APIs utilized in previous systems (Zhang et al., 2024b; Ma et al., 2024a) to make the paper more self-contained and accessible to a wider audience.

Weakness 2: The approach is tested entirely on Python issues. It’s unclear how well the code-graph approach generalizes to Java or cross-language settings.

Response: This is an important point. Currently, we rely on SWE-Bench, which is the state-of-the-art benchmark so far for evaluating software agents and is Python-based. We recently found more new benchmarks like SWE-Gym and LocBench, but these are also Python-based. We recognize the need for broader generalization and are actively working on developing a cross-language benchmark to evaluate agent performance in more diverse environments. Our graph-based framework could be improved further to support other languages with appropriate parsing and indexing backends.

Q1: How do you envision OrcaLoca’s performance scaling for extremely large repositories (e.g., tens of thousands of files, multi-million-line codebases)?

Response: This is an excellent question. Scaling to extremely large repositories such as Linux or Android codebases is an exciting but challenging direction. Key bottlenecks include efficient index management and the context management of CoT (Chain-of-Thought) reasoning during retrieval. In future work, we aim to incorporate an efficient Process Reward Model (PRM) to compress reasoning chains while reducing overall token consumption. We are also exploring hierarchical indexing and chunking strategies to handle large codebases more efficiently.

Q2: Do you anticipate any major challenges when applying OrcaLoca to other programming languages or multi-language repositories?

Response: Applying OrcaLoca to other single-language repositories generally presents no fundamental issues, as our system operates on the code structure and relationships. However, languages with strong type systems (e.g., C++, TypeScript) offer opportunities for type inference integration, which could enhance LLM reasoning by leveraging compiler-level information. This could reduce reasoning effort and improve localization accuracy.

For multi-language repositories—especially common in ML systems (e.g., Python + C++ via PyBind)—the main challenge lies in cross-language linking. We are currently extending our index to infer connections across language boundaries and enhance reasoning with cross-language semantic context. This will require both richer static analysis and more advanced reasoning strategies.

Q3: Your approach relies on the shortest paths in a code graph. Are these distances cached or computed dynamically?

Response: Currently, shortest-path distances are computed dynamically, as our current system serves primarily as a proof of concept. That said, we recognize the performance bottleneck this can introduce. In future iterations, we plan to add a caching layer and incremental computation mechanisms to improve efficiency, especially for large or frequently accessed repositories.

We thank the reviewers for their thoughtful and constructive feedback, as well as for highlighting several key strengths of our work. For the responses for weakness, we listed our analysis part by part below:

Weakness 1: The complexity of the approach

Response:

Thank you for your suggestions. We are actively exploring strategies to simplify the workflow while maintaining localization performance. One promising direction involves using a reward model to efficiently pre-score relevant code indices in a single pass, followed by a more powerful LLM to process only the high-value candidates. However, these methods are still under experiment. The current area is relatively new so there exists a big design space to explore and have adventure. We are also very interested to see if there exists a more elegant and efficient localization method.

Weakness 2: Computational overhead

Response:

Thank you for highlighting the importance of a detailed computational cost analysis. We agree that comparing resource usage across approaches is valuable for readers.

We chose token cost as our metric because LLM inference dominates our system's overall time and money expenses. For time analysis, since we are majorly leveraging the API service from model providers, it could seen as a metric proportional to the token count. In our revised manuscript, we will include a table summarizing the token cost for each agent:

Notably, over half of OrcaLoca's cost is attributed to the editing phase (0.90 out of 1.77), which is influenced by the specific edit tool used. Although we majorly target performance and accuracy issues in our paper, localization has a large potential for efficiency optimization shown by OrcaLoca-batch.

In OrcaLoca-batch, we implemented a batched actions optimization for the localization process, where we extracted the top batch steps from the priority queue. The table below summarizes the old and new token costs per instance, with the ratio (New Cost / Old Cost) indicating the improvement:

Due to the cost budget of experiments, we sampled 10 issues from the original SWEBench-Lite with different levels of token cost. Using weighted averages based on sampled instances in each cost bin, we estimate that the per-instance cost for localization was reduced by an average of 34% (from 0.87 to 0.58) with no adverse effect on localization correctness.

We are also committed to further refining OrcaLoca and will explore additional optimizations (e.g., implementing kv-cache techniques) in future work.

Weakness 3: Model dependency

Response:

We agree that model generalization is an important concern. In our experiments, we have actually tested with models like GPT-4o and Gemini 2.0, and they worked well in our test samples, proving our agent framework is model-agnostic. However due to budget constraints, we could not exhaustively evaluate across a wider range of LLMs. We majorly tested Claude because it was the best model with code reasonability at that time. In future work, we plan to include open-source models such as Qwen, LLaMA, and others—particularly once we fine-tune and serve our models locally, which will largely save cost for the repo-level benchmark evaluation.

We thank the reviewers for their thoughtful and constructive feedback, as well as for highlighting several key strengths of our work. We also thank you for raising this important point regarding the discrepancy between Function/File Match performance and the Resolved rate in baselines.

Generally speaking, higher localization performance usually offers a better background for the stage of downstream editing (Fig.1). However, since the editors are different across different implementations, the final resolved rate is not guaranteed to be high by providing a higher localization rate. For instance, although HyperAgent and RepoGraph agents scored exactly the same in the Function Match in Table.1, HyperAgent's inadequate editing capability results in a final resolved rate less than that of HyperAgent.

Another reason is caused by agent workflow and algorithm stage plan. In modular systems where localization is seen as a separate step, localization provides an exact starting point that can greatly improve the quality of the final changes, which is shown in Table 2. However, not all agents follow a strictly modular localization → edit pipeline. Some approaches, such as swe-search[1], interleave editing, and localization stages, where editing feedback can influence and refine the localization in an iterative loop. In such designs, the final Resolved rate may align more directly with localization performance because the two components are tightly coupled. Kodu-v1 may also have particular algorithms to set up a tighter link between localization and edit stages.

[1] SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement https://arxiv.org/abs/2410.20285

Note: We discovered that we omitted to mention this study since it was under the Moatless tool and not reported on SWE-Bench. Later on, we will include this in our revised version.

We thank the reviewers for their thoughtful feedback and for highlighting several key strengths of our work. For the weak points, we list our responses part by part, as follows:

Weakness 1: Heavy reliance on LLM outputs can lead to unpredictability and occasional hallucinations, affecting consistency in localization results. For example, different LLMs or even different LLM versions might yield different results.

Response: Thank you for this valuable point. We acknowledge that LLM-based agents may suffer from unpredictability or hallucinations. However, in our system, we have incorporated several design choices to mitigate these issues and improve reliability.

As illustrated in Figure 3, we introduce redundant action elimination to prevent the LLM from repeatedly generating previously executed (and potentially hallucinatory) actions. Furthermore, our agent adopts a context management framework, which enables pruning out noise and irrelevant data. Our agent has a self-consistency mechanism, where the LLM evaluates and refines its own intermediate search result to guide convergence, improving both stability and correctness. To evaluate consistency, we conducted a controlled experiment by selecting five representative instances from SWE-bench ([django__django-13551], [django__django-16255], [scikit-learn__scikit-learn-13439], [sympy__sympy-14774], [sympy__sympy-15011]). For each, we ran the agent five times and consistently observed the same function-level localization results under the same LLM configuration, demonstrating strong intra-model stability.

For inter-model consistency, we agree that different models varying a lot due to their reasonability and model weight discrepancy. To solve this problem, we would focus on improving LLM consistency by exploring consistency-aware prompting, robust fine-tuning, and more powerful code index and embedding in the future.

Weakness 2: The evaluation primarily focuses on SWE-bench datasets.

Response: Currently, we rely on SWE-Bench, which is the state-of-the-art benchmark so far for evaluating software agents. We recently discovered additional benchmarks such as SWE-Gym and LocBench, however they were released so lately that we do not have time to review the findings before submission. In the future, we plan to expand our method to include more benchmarks and experiment with other approaches.

Another reason is that the benchmark uses repo-based software, which increases the cost significantly. Experiments for 300 instances on SWE-bench lite will cost between $500 and $600, not considering debugging and reproduction for different baselines. To address this issue, we will continue to work on efficient models (such as small model distillation and finetuning) to better preserve tokens.

Weakness 3: The paper provides limited discussion on computational overhead and scalability, leaving questions about resource requirements for practical deployment.

Response:

Thank you for bringing this up. Our initial experiments primarily focused on accuracy and localization performance, so we did not perform extensive optimization for computational overhead. Currently, our system runs using API-based access to LLMs, which means that the local infrastructure requirement is minimal—a standard CPU server is sufficient to orchestrate the agent pipeline.

However, we acknowledge that scalability and cost-efficiency are crucial for practical deployment. In future work, we plan to serve our own models (e.g., fine-tuned variants) using lightweight LLM-serving frameworks such as Sglang, which would likely require GPU resources (e.g., A100 or H100) depending on the model size.

For additional details regarding cost analysis, including token usage and runtime cost on SWE-bench Lite, please refer to our response to Reviewer 3 (due to character limitations here). Thank you for your understanding!