Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks

W1: what is the baseline

There are two scenarios in our experiments - repository-level issue fixing (Section 5.1) and repository-level code completion (Section 5.2).

1. For Repository-Level Issue Fixing (Section 5.1, Tables 1 & 2):

Our primary baselines are the SOTA methods from the official SWE-bench Lite leaderboards (as of May 2025). This ensures we compare our work against the most competitive and relevant systems in the field. To provide a fair and nuanced assessment of our open-source, agentless approach, we categorize our comparisons as follows:

• Against Open-Source Systems: We compare CGM directly with other fully open-source systems, regardless of the underlying LLM they use. As shown in Table 1(a), our method (CGM-SWE-PY) ranks 2nd overall among all open systems on SWE-bench Lite.
• Against Methods Using Open-Weight Models: This is our most direct comparison group. Here, CGM-SWE-PY ranks 1st, outperforming the next-best method (Moatless+DeepSeek-V3) by a significant margin of 12.33%.
• Against All Systems (Including Proprietary Agents): We also situate our performance in the broader context of all published results, including those using powerful, closed-source agentic frameworks. Our method achieves a top-8 overall rank, demonstrating its competitiveness even against proprietary systems.

2. For Repository-Level Code Completion (Section 5.2, Tables 3 & 4):

Here, we compare against two categories of strong baselines to isolate the benefits of our CGM architecture:

• SOTA LLMs: In Table 3, we compare CGM against powerful standalone language models (e.g., DeepSeek-236B, Mistral-123B) using a standard retrieval method. This demonstrates that the architectural innovations of CGM provide a substantial performance boost beyond what can be achieved by simply using a powerful LLM backbone.
• SOTA RAG Methods: In Table 4, we conduct a head-to-head comparison against established RAG and context-retrieval methods for code, such as BM25, R2C2, and RepoFuse. This is a critical comparison, as it shows that CGM's method of directly integrating graph structure into the model is more effective than methods that also use graphs for retrieval but then flatten the context into linear text.

These baseline selections were deliberately chosen to rigorously test our central hypotheses and clearly demonstrate the unique advantages of the CGM framework. Full details of all baseline implementations are provided in Appendix C.6.4 and C.6.5.

W2 & W4 & L1: most significantly contributed factors & the contribution of the graph-aware attention mask and the graph-based retriever & appendix heavy

To clarify, we have conducted comprehensive ablation studies in Appendix C.7, listed as follows:

• Ablation of the high-level Graph RAG Framework: Verifying the contribution of each pipeline component.
  • Rewriter: Removing it dropped performance by 8.33%, confirming its value in clarifying issue descriptions.
  • Retriever: Its removal caused an 11.33% performance drop, highlighting its role in finding relevant subgraphs.
  • Reranker: Proved the most critical component; its removal led to a 24.67% performance drop.
  • Full RAG Pipeline (w/o R3): Removing all three modules caused a 33.33% drop, showing the RAG pipeline is essential to filter noise.
• Ablation of the Core CGM Reader: Analyzing the model's architecture, training, and generalization.
  • Model Architecture:
    • Semantic Integration: Progressively fine-tuning the model's Adapter, Decoder, and Encoder showed that training all components is essential for maximum performance.
    • Structural Integration: Replacing the graph-aware attention mask with the original causal mask hurt performance by 8.61% (Java) and 5.56% (Python).
    • Graph Modality: Replacing the graph modality input with pure text input (by flattening the text-rich graph as in RepoGraph and GraphCoder) resulted in a 37.67% performance decrease, validating CGM's superior design.
  • Training Strategy:
    • Pre-training Task: Removing the subgraph reconstruction task reduced performance by 7.65%, confirming its importance.
  • Generalization:
    • Backbone Models: CGM’s performance scales with the underlying model's coding ability across Llama3.1-70B-Instruct, Qwen2.5-Coder-32B/7B-Instruct, and Qwen2.5-72B, consistently enhancing each backbone.

From the above results, we can see that among all high-level graph RAG components, the Reranker has the most significant impact—removing it results in a 24.67% performance drop. This highlights the importance of precise file selection for effective issue resolution.

Within the core CGM Reader, as shown in Table 11, joint fine-tuning of the adapter, encoder, and decoder (i.e., semantic integration) brings the largest performance improvement. Notably, the adapter’s ability to map encoded node representations to the LLM input space is crucial for maximizing effectiveness.

Regarding the graph-aware attention mask (i.e., the ablation study of structural integration), using vanilla causal attention mask instead of the graph-aware attention mask leads to performance degradation on both Java and Python dataset (EM performance drops by 8.61% and 5.56%, respectively), as shown in Table 11.

Omitting the graph-based Retriever—by relying solely on the Rewriter and Reranker for file selection—resulted in an 11.33% reduction in issue fixing performance on SWE-bench Lite (see Table 10). This underlines the Retriever’s role in supplying contextually relevant, structurally coherent subgraphs for downstream modules.

Due to space constraints in the initial version, we were only able to include the most essential ablation findings in Section 5.3 of the main paper. For the camera-ready version (which allows one additional page), we will move more comprehensive ablation results and analyses into the main content, as you suggested.

W3: fundamental research questions

Our research is driven by observing the current landscape of automated software engineering, which is dominated by complex, agent-based systems that rely on proprietary LLMs (as discussed in Lines 26-36). This paradigm, while powerful, introduces significant challenges related to accessibility, cost, data privacy, and the inherent unpredictability of multi-step agentic reasoning.

This context gives rise to our primary research question, stated explicitly in the Introduction (Lines 37-38):
"Can open-source LLMs be employed in an agentless manner to effectively complete repository-level coding tasks?"

W5&Q1: reorganize the paper

We agree that a clear presentation following the Hypothesis → Experiments → Results → Conclusion framework is essential for conveying our contributions effectively. We believe our paper already adheres to this logical structure, and we appreciate the opportunity to clarify this flow.

Here is how the current sections of our paper map to the scientific structure suggested by the reviewer:

1. Hypotheses (Section 1: Introduction):
   Our primary hypothesis is explicitly stated as a research question in the introduction: "Can open-source LLMs be employed in an agentless manner to complete repository-level coding tasks?" (Lines 37-38). We further posit a core technical hypothesis that this is achievable if the LLM is empowered to comprehend the repository’s structural dependencies via a code graph (Lines 42-45, "the key lies in empowering the open-source LLMs to fully comprehend code repositories... but also the dependencies across functions and files").
2. Methodology to Test the Hypotheses (Sections 3 & 4):
   To test our hypotheses, we designed and built the Code Graph Model (CGM) and the accompanying Graph RAG framework.
   • Section 3 details the construction of the code graph, the fundamental data structure required by our hypothesis.
   • Section 4 describes the novel CGM architecture and the Graph RAG framework, which are the instruments we use to conduct our experiments. This section explains how we integrate semantic and structural information to test if this approach is effective.
3. Experiments & Results (Section 5):
   This section is dedicated to the empirical validation of our hypotheses.
   • We test our CGM on challenging repository-level tasks (Issue Fixing in Sec 5.1, Code Completion in Sec 5.2) and present the outcomes in Tables 1, 2, 3, and 4.
   • Crucially, the Ablation Studies (Section 5.3 and Appendix C.7) directly test the core components of our technical hypothesis by measuring the performance impact of our structural integration (the graph-aware attention mask) and other key design choices. The results presented there provide strong evidence supporting our claims.
4. Conclusion (Section 6):
   This section summarizes our findings, directly answering our initial research question. We conclude that our graph-integrated, agentless approach using open-source models is not only viable but also highly competitive, thus validating our central hypotheses.

While we have not used explicit "Hypothesis" or "Results" subheadings, we believe this logical progression is inherent in the paper's current organization. We hope this clarification helps illustrate the scientific rigor of our paper's structure, and we thank the reviewer again for prompting this helpful discussion.

Dear Reviewer HU7w,

Thank you for your positive feedback; we were very encouraged to hear that you are satisfied with our rebuttal!

While we appreciate your satisfaction with our response, we noticed the score seemed unchanged. We would like to know if this suggests there are other aspects of our work that we could still strengthen.

We are looking forward to hear from you.

Best regards,

Authors

W1 & Q1: Graph RAG for retrieval-based generation.

The core of our contribution is the CGM, which is a powerful graph-aware generator (or "Reader"). The Graph RAG framework (Rewriter, Retriever, Reranker) is a flexible front-end designed to supply the CGM with the most relevant context, and its components can be adapted or simplified depending on the task's complexity.

To clarify how this works in practice for our two experimental tasks:

1. For Complex Issue Fixing (Section 5.1): This task starts with an ambiguous natural language query (an issue report). Therefore, we employ the full Graph RAG pipeline (Rewriter, Retriever, Reranker) to progressively refine the query and localize the most relevant subgraph to feed into the CGM. The CGM's role here is purely as the final Reader/Generator.
2. For Code Completion (Section 5.2): This task is simpler as the location for code generation is already known. Consequently, the full, sophisticated RAG pipeline is unnecessary. We instead use a simplified retrieval strategy, as explicitly described in Section 5.2 (Lines 334-336) and Appendix C.6.1. Specifically, we directly build the input subgraph by retrieving the one-hop neighbors of the target function from the code graph. This subgraph, along with the incomplete file, is then fed into the CGM, which acts solely as the generator.

To ensure our comparisons in the code completion experiments were fair and informative, we designed them as follows:

1. Comparison against other LLMs (Table 3): To isolate the power of the CGM as a generator, we fixed the retrieval method for all models. We used the same one-hop graph retrieval for CGM and all baseline LLMs (Mixtral, DeepSeek, etc.). For the baseline models that cannot process graphs, we flattened the retrieved subgraph into text. The results, which show CGM's superior performance on ComplexCodeEval, demonstrate its enhanced capability as a generator, even when the retrieval strategy is held constant.
2. Comparison against other RAG methods (Table 4): To assess our end-to-end performance, we compared our simplified pipeline (one-hop graph retrieval + CGM generator) against other specialized RAG baselines (e.g., BM25, R2C2, RepoFuse). As detailed in Section 5.2 (Lines 342-347) and Appendix C.6.4, each baseline uses its own distinct retrieval mechanism. The results show that our graph-based approach typically outperforms these methods, highlighting the effectiveness of our method.

To directly answer your final question: The CGM is exclusively used as the final generator in all experiments. It was never used for retrieval. The components of the Graph RAG framework are responsible for retrieval, and their application is adapted to the specific task. We will ensure this distinction is further clarified in the final version of the paper.

L1: The implications of open source models on the issue of automation

Our work is motivated by the significant challenges that the current reliance on proprietary, closed-source models poses to the automation of software engineering. We have explicitly discussed these implications in several key sections:

1. Problem Formulation (Introduction, Lines 33-36 & 85-87): We directly address the limitations of closed-source models, which currently dominate the field. We highlight that their use creates substantial barriers for the research and engineering communities, including:
   • Limited Accessibility & High Cost: Preventing widespread adoption and experimentation.
   • Lack of Customization: Prohibiting researchers and organizations from fine-tuning models on proprietary or domain-specific codebases.
   • Data Privacy & Security Risks: Making it infeasible for many companies to use these models with their sensitive, internal code repositories.
2. Core Research Question (Introduction, Lines 37-41): Based on these implications, we frame our entire investigation around the central question: "Can open-source LLMs be employed in an agentless manner to complete repository-level coding tasks?" This question positions our work as a direct response to the need for more accessible and transparent automation tools.
3. Contribution and Impact (Conclusion, Lines 376-377): Our conclusion reinforces this theme by stating that our work "establishes a new direction for developing powerful, transparent, and accessible tools for automated SE." By demonstrating that a fully open-source framework can achieve state-of-the-art results, we argue that our approach democratizes access to powerful SE automation, fosters a more transparent and extensible ecosystem, and provides a practical solution for real-world deployment where security and customization are paramount.

In summary, our paper is not merely an application of an open-source model; it is a dedicated effort to demonstrate its viability and to provide the community with a practical, powerful, and transparent pathway for future research in automated software engineering. We will ensure this narrative is further emphasized in the final version.

W1&Q1: only engineering contributions & detailed ablation study

First of all, we would like to clarify that, beyond engineering contributions, we propose a novel method to integrate the graph modality with the language modality (i.e., LLMs). Specifically,

(i) Semantic Integration: Node attributes (containing code or comments) are first encoded by a pretrained text encoder and then mapped to the LLM's input space via an adapter, enabling the model to understand the semantic information of all nodes.

(ii) Structural Integration: The graph structure is incorporated into the LLM through the attention mask, allowing direct message passing only between neighboring nodes in each layer of the LLM, similar to spatial Graph Neural Networks (GNNs).

Moreover, we have conducted comprehensive ablation studies in Appendix C.7, listed as follows:

• Ablation of the high-level Graph RAG Framework: Verifying the contribution of each pipeline component.
  • Rewriter: Removing it dropped performance by 8.33%, confirming its value in clarifying issue descriptions.
  • Retriever: Its removal caused an 11.33% performance drop, highlighting its role in finding relevant subgraphs.
  • Reranker: Proved the most critical component; its removal led to a 24.67% performance drop.
  • Full RAG Pipeline (w/o R3): Removing all three modules caused a 33.33% drop, showing the RAG pipeline is essential to filter noise.
• Ablation of the Core CGM Reader: Analyzing the model's architecture, training, and generalization.
  • Model Architecture:
    • Semantic Integration: Progressively fine-tuning the model's Adapter, Decoder, and Encoder showed that training all components is essential for maximum performance.
    • Structural Integration: Replacing the graph-aware attention mask with the original causal mask hurt performance by 8.61% (Java) and 5.56% (Python).
    • Graph Modality: Replacing the graph modality input with pure text input (by flattening the text-rich graph as in RepoGraph and GraphCoder) resulted in a 37.67% performance decrease, validating CGM's superior design.
  • Training Strategy:
    • Pre-training Task: Removing the subgraph reconstruction task reduced performance by 7.65%, confirming its importance.
  • Generalization:
    • Backbone Models: CGM’s performance scales with the underlying model's coding ability across Llama3.1-70B-Instruct, Qwen2.5-Coder-32B/7B-Instruct, and Qwen2.5-72B, consistently enhancing each backbone.

Due to space constraints in the initial version, we were only able to include the most essential ablation findings in Section 5.3 of the main paper. For the camera-ready version (which allows one additional page), we will move more comprehensive ablation results and analyses into the main content, as you suggested.

W2&Q2: discussion and comparison with RepoGraph and GraphCoder

We would like to clarify that we have discussed the differences between our proposed CGM and previous graph-based methods such as RepoGraph [45] and GraphCoder [49] in Section 2 (Lines 141–147):

Recent research has also focused on enhancing code understanding by incorporating structural information through graph-enhanced repository modeling [45,48,49]. However, even when graph structures are used during retrieval, existing methods typically flatten the retrieved code snippets into linear text sequences for downstream model prompting. This flattening process fails to preserve the inherent heterogeneity between graph and text modalities. As a remedy, we propose the CGM that explicitly aligns these two distinct modalities, enabling better preservation and utilization of structural information throughout the entire process.

To address performance comparison, RepoGraph achieves 29.67% on SWE-bench Lite (using GPT-4o as its base model), as reported in their paper. Our CGM attains 43.00% on the same benchmark using the open-source Qwen2.5-72B, demonstrating a significant improvement. We will update Table 1 and the corresponding discussion to include these direct comparisons for greater clarity.

To summarize the strengths of CGM over RepoGraph and GraphCoder in two sentences:
CGM enables decoder-only LLMs to explicitly understand and attend over graph structures, going beyond simple textual flattening of code graphs. Additionally, by compressing node text into compact embeddings, CGM can efficiently encode much larger repository graphs than previous approaches, leading to superior performance and scalability.

Finally, as further evidence, our ablation study (Lines 810–814) shows that flattening the graph into plain text for LLM input leads to a 37.67% performance decrease, underscoring the importance of our explicit graph integration.

W3 & L1: effectiveness of the achitecture itself & more backbones

To clarify, Table 1(a) shows that even when using the same Qwen2.5-72B backbone, methods such as SWE-Fixer and Lingma SWE-GPT achieve issue resolution rates of only 24.67% and 22.00% respectively on SWE-bench Lite, while our proposed CGM achieves a significantly higher 43.00%. This large margin demonstrates that the gains stem primarily from our architecture, not just the choice of backbone.

Furthermore, as discussed in Appendix C.8, we have also trained CGM on other LLM backbones, including Llama3.1-70B-Instruct, Qwen2.5-Coder-32B-Instruct, and Qwen2.5-Coder-7B-Instruct, to test the generalizability of our approach. Notably, CGM improves the performance of Llama3.1-70B-Instruct from 7% (as reported in Lingma SWE-GPT [1]) to 25.33% on SWE-bench Lite—an absolute improvement of 18.33%. These results further confirm that the effectiveness of CGM comes from its architectural innovations, independent of the underlying LLM.

W4: justification for use against agent-based methods

We would like to clarify the positioning and advantages of our approach compared to both open-source and closed-source agent-based methods.

For agent-based methods using open-source models, their performance is generally lower than CGM’s. As shown in Table 1(a) on SWE-bench Lite, CGM outperforms the strongest open-source agent-based baseline (Moatless+DeepSeek-V3) by 12.33%, despite DeepSeek-V3 itself being a stronger model than Qwen2.5-72B, which we use as our backbone. This highlights the advantage of our architecture over even the best open-source agent frameworks (see Lines 298–303, Page 8).

For agents based on closed-source model, the reliance on closed-source models creates substantial barriers for the broader SE community [13,14], including limited accessibility, inability to enhance or customize models for specific tasks, and serious security concerns regarding the privacy of sensitive code repositories when interacting with external API services. This point is mentioned in Lines 33-36. A more comprehensive discussion regarding agent-based methods and their practical challenges and limitations is provided in Lines 112-128 (Agent-drive Methods for SE in Section 2).

At the time of submission, our method ranks eighth overall on SWE-bench Lite, making it highly competitive even compared to the latest agent-based approaches. Importantly, CGM delivers these results using a fully open-source, agentless workflow, ensuring transparency, extensibility, and real-world deployability without the constraints of commercial API access. We believe this is meaningful for both AI and SE communities. Furthermore, the CGM framework is inherently flexible and can be further improved as stronger encoders, adapters, and LLM backbones become available, which is likely to further narrow (or even close) the gap to the very top results dominated by proprietary agents.

L2: broader impact

Following your suggestion, we will add the following discussion to the paper:

This research introduces a powerful new paradigm for reasoning about complex, interconnected information by deeply integrating text-rich graph structures directly into Large Language Models. Instead of perceiving a software repository as a flat, linear sequence of text, our approach endows the LLM with an explicit, structured understanding of the codebase's architecture. This is crucial for advancing AI in software engineering, as the model can now explicitly reason about the intricate web of dependencies—how functions call each other, classes inherit properties, and files are interconnected. Consequently, the model can better anticipate the downstream impact of a proposed code change, a reasoning ability previously exclusive to experienced human developers. Furthermore, the code graph itself serves as a valuable asset for human oversight, providing a clear visualization that helps developers grasp complex repository structures and validate the model’s logic.

The implications of this approach, however, extend far beyond software engineering, offering a blueprint for enhancing LLM reasoning in any domain that can be represented as a text-rich graph. In academic research, for instance, a citation network where papers are nodes and citations are edges could be ingested. This would allow a model to not only summarize a paper's content but also understand its intellectual lineage, identify seminal works, and even forecast emerging research trends by analyzing its structural position within the scientific community. Similarly, in e-commerce, a graph of products and user interactions would enable hyper-personalized recommendation engines that understand the nuanced relationships between items based on both their descriptions and their connections in purchasing patterns. By fusing the structural context of a graph with the deep semantic understanding of an LLM, this methodology unlocks a new tier of contextual intelligence applicable across countless fields.

3. Experimental Design and Validation

Our experiments were designed to systematically validate our hypothesis and answer our research question.

• Primary Evaluation: We tested CGM on challenging repository-level tasks, including Issue Fixing (Sec 5.1) and Code Completion (Sec 5.2), using standard benchmarks. As shown in Tables 1, 2, 3, and 4, CGM significantly outperforms existing open-source methods, demonstrating that our graph-integrated architecture is highly effective in practice.
• Ablation Studies: To isolate the impact of our novel components, we conducted extensive ablation studies (Section 5.3 and Appendix C.7). These studies dissect the sources of our performance gains and confirm that our core architectural innovations are crucial. For example, we showed that removing the graph-aware attention mask or disabling the fine-tuned semantic integration module significantly degrades performance. This provides strong evidence that the architectural integration itself, not just the surrounding RAG framework, is the primary driver of our model's success.

4. Conclusion

Our conclusion in Section 6 directly answers our initial research question. We confirm that by architecturally enhancing an LLM to be graph-aware, it is indeed possible for an open-source, agentless model to serve as a viable and highly competitive solution for repository-level tasks. Our work demonstrates that moving beyond text-only representations and toward true graph-language models is a powerful and promising direction for the future of automated software engineering.

We hope this detailed explanation clarifies the scientific contributions of our paper. Thank you again for your feedback.

2. Our Novelty: A Principled, Architectural Solution

To investigate this question, we proposed the Code Graph Model (CGM). We argue this is a novel architectural contribution, not just an engineering pipeline. Our approach is conceptually analogous to how Vision-Language Models (VLMs) integrate distinct modalities (e.g., images and text) into a unified architecture. In our work, we replace the vision modality with the graph modality to represent a code repository, which required us to develop specialized techniques for this unique data type.

• Why Code Graph? The Need for a Structured Representation. Before integration, a repository must be properly represented. We use a code graph (Section 3) because it explicitly captures the structural information that is critical for complex coding tasks. The graph's edges represent both hierarchical (contains) and reference (calls, imports) dependencies, mirroring how a developer examines a function's context—its outer class, its callers, and its callees—before editing the function itself. The graph nodes, in turn, hold the semantic information (the source code itself).
• Why CGM? Bridging the Modality Gap. Simply creating a graph is insufficient; the LLM must be able to reason over it. Existing methods that "flatten" the graph into text inevitably lose the rich structural heterogeneity and struggle with the context length limitations of LLMs. Our CGM architecture bridges this modality gap through two key innovations:
  1. Semantic Integration via Node-Level Compression: A primary challenge is fitting the rich content of a repository graph into the finite context window of an LLM. If we were to simply feed the raw source code from each graph node into the LLM decoder, the text would be tokenized into a large number of tokens, quickly exhausting the model's context window and limiting its view to only a small fraction of the repository. Our semantic integration module addresses this directly. It encodes the entire source text of a graph node (which can be hundreds of tokens long) into a single, compact node token via a specialized encoder and adapter. This compression is a key part of our design, as it effectively expands the LLM decoder's context length by orders of magnitude. It allows the model to simultaneously consider the semantic information from a vast number of code entities (nodes), enabling it to reason over a much larger and more complete subgraph than would be possible with raw text.
  2. Structural Integration via Graph-Aware Attention: We integrate the graph's topology directly into the LLM by replacing the standard causal attention mask with a graph-aware attention mask derived from the repository's adjacency matrix. This forces the model's attention mechanism to operate along the actual dependencies of the code graph, ensuring that information flows only between neighboring nodes. This mimics the message-passing of a Graph Neural Network (GNN) within the Transformer architecture, enabling genuine structural reasoning. The benefit of this becomes clear in a practical task. When CGM is tasked with modifying a specific function, the generation process for the new code attends to the corresponding "node token" for that function. Crucially, due to the graph-aware attention mechanism, this single node token has already aggregated contextual information from its neighbors (e.g., its callers, callees, and parent class) during the LLM's forward pass. This means that the model's prediction for the very next token is implicitly conditioned on the function's entire structural context. In essence, the model modifies the function with direct reference to its dependencies, which explains the performance improvements we observe on complex issue-fixing benchmarks like SWE-bench Lite, where understanding such relationships is paramount.

By integrating the graph modality and considering its special properties (like the adjacency matrix) in our model design, we believe our work is a significant step beyond a mere engineering solution.

We thank the reviewer for their instant feedback and the opportunity to provide further clarification.

Q1: Move ablation study to main content

We completely agree. The camera-ready version allows an extra page, and we will use this space to expand the description of our ablation study.

Q2: Our research question

This is a question that requires a detailed response, and we appreciate the opportunity to elaborate on the core research question that drives our work, the principled design of our method, and how our experiments validate our scientific claims.

1. The Core Research Question

Our paper's central research question is: "Can open-source LLMs be employed in an agentless manner to effectively complete repository-level coding tasks?" (Lines 37-38).

The fundamental challenge (Lines 42-45) we address is the mismatch between the sequential nature of LLMs and the complex, non-linear structure of code repositories. While human programmers rely on understanding hierarchical and reference dependencies (e.g., callers, callees, inheritance) to navigate code, LLMs traditionally struggle with this relational information. We posit that enabling an LLM to natively comprehend this structure is key to unlocking its full potential for software engineering.

Dear Reviewer 2ymT,

Thank you again for the valuable discussion! We just wanted to check if it helped to address the further concerns you raised.

We appreciate all your guidance.

Best regards,

Authors

W1&Q1: analysis of TTS performance

Thanks for your insightful suggestion! We have evaluated CGM's performance from Pass@1 to Pass@3 on both SWE-bench Lite and Verified dataset. Our results below show a clear and positive trend: multiple sampling boosts the resolution rate by approximately 2-3% on both datasets compared to single sampling (Pass@1).

This confirms CGM's performance scales with multiple samples. We will add this analysis to the final version.

W2: severe performance drop for small models

First, we want to clarify that the comparison between our 4% result and the ~20% from Seed-Coder is indirect. Seed-Coder's 19.2% result [R1] was achieved on the SWE-bench Verified dataset, which is known to yield higher scores than the more challenging SWE-bench Lite dataset used for our Table 12 ablation.

To create a direct and meaningful comparison, we established two crucial baselines for the Qwen2.5-Coder-7B model in an agentless setting:

1. Performance on Verified: The Seed-Coder report itself shows that the base Qwen2.5-Coder-7B model, when used with the agentless workflow from [2], scores only 4.2% on SWE-bench Verified.
2. Performance on Lite: We further replicate this exact agentless workflow [2] with the Qwen2.5-Coder-7B model and tested it directly on SWE-bench Lite, achieving a resolution rate of 3.67%.

In this direct, like-for-like comparison on SWE-bench Lite, our CGM with the same backbone achieves 4.0%, showing a slight performance improvement over this SOTA agentless baseline.

These results strongly suggest the primary limitation is the intrinsic capability of the Qwen2.5-Coder-7B model itself, limiting the performance across all frameworks built upon it. Moreover, our findings highlight that the CGM architecture benefits from stronger LLM decoders. As shown in Table 12, scaling the backbone from Qwen2.5-Coder-7B to Qwen2.5-72B-Instruct substantially increases CGM’s resolution rate from 4.0% to 43.0%.

W3: limited language paradigms

We agree that expanding our graph-based framework to additional paradigms—such as functional, scripting, or other complex programming styles—is an important direction for future research.

Our focus on Python and Java was determined primarily by the availability of high-quality benchmarks with testing environments. At the time of our experiments, the main multilingual benchmark initiative was an early release of what has since become Multi-SWE-bench [2]. Before its formal release in April 2025, only the Java component (SWE-bench-java Verified [1]) was accessible for comprehensive evaluation, which is why it served as the basis for our Java experiments. The full Multi-SWE-bench, with expanded language and paradigm coverage, was released too close to the submission deadline to allow for the extensive data collection and model training required for robust experimentation.

We have acknowledged this object-oriented focus as a limitation in Appendix E and will expand this discussion in the final version. We are eager to extend CGM to cover a broader range of languages and paradigms by using the new Multi-SWE-bench in our future work.

W4: long execution time

We would like to clarify that our current graph construction tool is a prototype, and there are several clear avenues for optimization. For example, we can parallelize the code parsing phase—followed by a final, serialized step for node relation establishment—to significantly accelerate graph generation. Moreover, after initial construction, the graph can be maintained through lightweight, incremental updates: when code is modified, affected nodes and edges are removed, and only the changed code is re-parsed and reinserted.

With these engineering improvements, we anticipate reducing graph generation time to about 10 seconds for small projects (under 100K lines of code) and 30 seconds for large projects (over 100K LOC).

According to SWE-Gym [44], agent-based methods require at least 20 interaction turns for comparable performance, whereas our approach uses fewer than 5 LLM calls. Therefore, its overall runtime is substantially shorter, even accounting for the optimized graph construction and intermediate processing.

We will add this point in Appendix B.

W5&Q2: Bad case analysis

Thanks for your insightful input! Here we provide the evaluation report (generated by the official SWE-bench toolkit) for CGM-SWE-PY on SWE-bench Lite, followed by an error analysis to diagnose CGM's limitations..

Evaluation Report

The toolkit shows CGM achieves an overall 43.00% (129/300) resolution rate. The detailed results by repository (e.g., performance on django or sympy) and year (year of issue origin) are below.

CGM's performance is stable over time, indicating no data contamination, while its varied success across repositories suggests the potential influence of project-specific characteristics.

Error Analysis

Firstly, the vast majority (~80%) of CGM's failures are unresolved cases (i.e., executable but incorrect), not execution errors, highlighting its high fidelity in generating syntactically valid code, even when the semantic logic for the fix is not perfect.

Then we manually inspect 33 failure cases (20% of total, 6 from execution errors and 27 from unresolved ones) to profile the failure patterns.

1. The main reasons for execution errors are (1) being misled by complicated issue (60%) and (2) occasional mistakes in syntactic generation (40%). For example, in the 'django__django-12113', model directly copies a diff snippet from the issue description; breaking down issue into clearer instructions could alleviate this. As for the second, we observe occasional missing functionality like a missing return clause in 'sympy__sympy-13043'. The appearance of such errors, while infrequent, is a known characteristic of code LLMs [R3].
2. As a more major error, the unresolved ones are mainly caused by: (1) limited reasoning ability for complex issues (55%), (2) knowledge gap (26%), and (3) cascading errors from the graph RAG module (19%).
   1. Limited Reasoning: This is demonstrated in 'scikit-learn__scikit-learn-11040', where CGM does locate and fix the user-reported vulnerable class, NearestNeighbors. However, the architecturally superior solution, provided by the golden patch, was to fix its parent class, KNeighborsMixin, from which NearestNeighbors inherits. This distinction is not trivial. In fact, in the code graph, there exists an edge connecting NearestNeighbors with KNeighborsMixin. Despite it, the current LLM decoder in CGM fails to leverage the edge to modify KNeighborsMixin instead of NearestNeighbors.
   2. Knowledge gap: This is exemplified by LLM's unawareness of the current implementation of third-party packages. In 'scikit-learn__scikit-learn-10949', model attempts an unsupported operation on a NumPy array, which could be mitigated by including such details directly in the code graph.
   3. Cascading errors: These errors occur when necessary files have not been retrieved by the previous RAG module, thus leading to the failure of CGM. Improving the Recall of the GraphRAG module can further improve CGM's final performance.

This analysis will be added to the final paper.

Q3: java is worse than python

While the performance of CGM-Multi is lower on the Java benchmark compared to the Python benchmark, we believe this reflects the benchmark's higher inherent difficulty, rather than a specific weakness of our model in processing Java.

This difficulty gap is not unique to our method; it is a consistent trend across other SOTA models in the field. To illustrate this, we can look at the performance of a powerful baseline from the official leaderboards:

• On the Python-centric SWE-bench Lite, SWE-Agent + GPT-4o achieves a resolution rate of 32.00% (as seen in Table 1a).
• On the SWE-bench-java Verified benchmark, the same SWE-Agent + GPT-4o system's performance drops sharply to just 6.59% (as seen in Table 2).

This demonstrates that the Java benchmark poses a substantially greater challenge for current methods. In this difficult context, our CGM-Multi achieves a 14.29% resolution rate, more than doubling the performance of the SWE-Agent+GPT-4o baseline. Therefore, while lower than our Python results, our Java performance is SOTA and significantly outperforms other leading methods on that specific, more difficult benchmark.

Q4: performance vs data amount

Thanks for pointing this out! To analyze the influence of training data volume, we have trained CGM on subsets of the training data. The results below show a clear positive trend:

This demonstrates CGM's potential to improve further as more issue-fix pairs are available.

[R1] Seed-Coder: Let the Code Model Curate Data for Itself

[R2] Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

[R3] LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation

Dear Reviewer 7ALZ,

Thank you again for your valuable feedback. We have submitted our rebuttal and hope it addresses your comments.

With the discussion period ending soon, we just wanted to send a gentle reminder. We'd be grateful for a chance to hear your thoughts.

Best regards,

Authors

Thank you for your positive feedback! We will incorporate these parts into the revised paper.