HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale

Thank you for the thoughtful and detailed feedback on our manuscript. We appreciate your engagement with our work and the opportunity to clarify and strengthen our contributions. Below, we provide responses to the key points raised.

Comment 1:

"The idea of using the same base-LLM to act as different subagents has been previously explored before [1,2].
How are the subagents specialized? Is the difference primarily from using different input contexts / system prompts, tools, etc.?"

We acknowledge the reviewer’s observation regarding prior work exploring the use of the same base LLM for different subagents. While this concept is not entirely novel, HyperAgent distinguishes itself through several key aspects:

1. Dynamic Orchestration:
   Unlike prior work where subagents may operate in a rigid sequence, HyperAgent’s Planner dynamically decides which subagent to invoke based on the task requirements. This adaptability is key to handling complex workflows. Many additional details about how we enhance reasoning capabilities, save costs in the key component (Planner), and implement our other novelties are included in our response to Reviewer Wb3u.

2. System Prompts and Contextual Specialization:
   Yes, each subagent in HyperAgent is specialized with role-specific prompts and tools. These prompts are carefully designed to align with the agent’s specialized tasks. For example:

   • The Navigator focuses on exploring the codebase, retrieving relevant files, or locating dependencies.
   • The Editor performs patch generation and iterative refinement, leveraging repair-specific tools.
   • The Executor handles runtime validation and test execution.

   This role differentiation allows HyperAgent to effectively decompose tasks into smaller, manageable units, enhancing overall system efficiency and performance.

3. Tool Integration:
   Subagents interact with specialized tools that align with their roles, such as LSP-based tools for the Navigator and IDE-inspired editing features for the Editor. This modular integration ensures that each subagent operates with optimized resources for its respective function.

Comment 2:

"The paper claims that the final agent is generalist and applicable across diverse tasks. However, it’s not clear if the context and implementation for each task is specialized or not?
For instance, when solving the SWE-Bench-like tasks, the original SWE-Agent implementation provides task-specific instructions/tips (e.g., reproducing the issue) in the input context. Is a similar strategy also applied for HyperAgent?"

HyperAgent is designed as a generalist system capable of addressing diverse software engineering tasks, while SWE-Agent’s approach could be considered more specialized. Here’s how they differ:

Dynamic Adaptation vs. Specialist Design:

• SWE-Agent relies on task-specific strategies tailored for GitHub issue resolution, such as pre-designed demonstrations that provide explicit instructions for reproducing issues or resolving them (as detailed in SWE-Agent repository). This approach makes SWE-Agent highly effective for the GitHub issue task but also aligns it more closely with a specialist system, as it is designed specifically for a narrow set of tasks.

• In contrast, HyperAgent dynamically adapts to tasks through its Planner, which interprets task prompts and constructs high-level strategies on the fly. For example, when solving SWE-Bench-like tasks, the Planner may decide to reproduce an issue based on dynamically extracted error traces, without relying on hardcoded, task-specific instructions. While integrating task-specific demonstrations like SWE-Agent could improve performance, HyperAgent’s focus remains on maintaining flexibility and generalizability across tasks.

Generalist Framework with Reusable Templates:

HyperAgent utilizes flexible task templates that abstract common software engineering challenges into reusable structures, such as patch generation or prediction tasks. These templates are populated with task-specific details (e.g., error traces for Defects4J or GitHub issues for SWE-Bench), enabling efficient adaptation to diverse tasks while minimizing task-specific design. This framework allows HyperAgent to handle new tasks with minimal reconfiguration, highlighting its generalist nature.

By dynamically adapting to task requirements and leveraging reusable structures, HyperAgent demonstrates a generalist approach, in contrast to SWE-Agent’s specialist design, which is tightly coupled to the GitHub issue resolution task.

Comment 5:

"How are the ablation studies performed in Sec. 6? For instance, w/o navigator which agent is used for code navigation: the planner agent or any other subagent?"

In Section 6, we evaluated the system’s behavior when individual agents were removed, and their tasks were reassigned to other components. When an agent is removed, the Planner assumes the role of the missing agent by directly invoking tools to perform the tasks originally handled by the removed agent. For instance, without the Navigator, the Planner directly uses navigation tools to locate relevant files or dependencies. This increases context length and introduces noisy information, leading to performance degradation (as noted in our response to Reviewer Wb3u).

Comment 6:

"Are there some in-context examples teaching which subagent should be called by the planner agent? If not, is it mostly reliant on the function-calling capability of the main planner agent?"

We indeed provided some in-context examples for the Planner to demonstrate how to trigger and request specific subagents for their respective tasks. These examples significantly improved the Planner’s performance. Below is an example of one such in-context demonstration:

Example:

Thought:
The issue is related to the TimeSeries object in the astropy.timeseries module. When a required column is removed from the TimeSeries object, the exception message is misleading. The expected behavior is to raise an exception that informs the user that required columns are missing. However, the actual exception message is confusing and does not clearly indicate the problem. To resolve this issue, I will first try to reproduce the bug by running the provided code snippet. Then, I will investigate the code in the TimeSeries class to see where the exception is being raised and why the message is misleading.

Agent Name: Executor
Request:
Reproduce the bug by running the provided code snippet and verify that the exception message is indeed misleading. Please run the following code and report the output:

from astropy.time import Time
from astropy.timeseries import TimeSeries
import numpy as np

time = Time(np.arange(100000, 100003), format='jd')
ts = TimeSeries(time=time, data={"flux": [99.9, 99.8, 99.7]})
ts._required_columns = ["time", "flux"]
try:
    ts.remove_column("flux")
except ValueError as e:
    print(e)

Context:

The TimeSeries class in the astropy.timeseries module is used for handling time-series data. It has a set of _required_columns that must always be present in the data. If a required column is removed, the system is expected to raise a clear exception informing the user about the missing columns. However, the current implementation raises a confusing or irrelevant exception message, which can mislead users and make debugging harder.

We appreciate the time and effort you have put into reviewing our manuscript and providing feedback. Below, we address your comments and concerns in detail:

1. Clarification on ACR

"Compared to previous work, such as ACR and SWE-Agent, the innovation in this paper is primarily the addition of an Execution Agent component; the other agents are almost identical to those in previous work."

Could you clarify what “ACR” refers to? If it is AutoCodeRover (ACR), we would like to highlight that both AutoCodeRover and SWE-Agent are not multi-agent frameworks. Instead, they follow single-agent or pipeline designs. Here are some distinctions and novelties of HyperAgent compared to these systems:

• Freedom in Execution:

  • AutoCodeRover relies on rigid sequences of phases that are fixed for each task. If one phase fails (e.g., localization), the task cannot be completed.
  • In contrast, HyperAgent’s Planner dynamically adapts to the task context, allowing it to revisit phases, correct mistakes, and explore alternative strategies.

• Faster Execution:

  • HyperAgent leverages parallelism and asynchronous message queues, significantly reducing execution times on large tasks compared to AutoCodeRover.

• Cost Efficiency:

  • SWE-Agent incurs prohibitively high costs due to long context windows and noisy interactions. HyperAgent achieves similar or better performance on benchmarks with significantly lower API costs.

2. Performance Improvements

"The HyperAgent proposed in this paper does not show significant improvements over previous methods in the final experimental results."

We respectfully disagree with this assessment and would like to provide clarification. HyperAgent demonstrates substantial improvements over prior methods across multiple benchmarks, as detailed below:

• RepoExec:

  • HyperAgent achieves state-of-the-art results, outperforming all existing baselines on repository-level code generation.

• Defects4J:

  • HyperAgent surpasses specialized baselines designed specifically for bug repair tasks, showcasing its generalizability and scalability.

• SWE-Bench:

  • While HyperAgent performs comparably on SWE-Bench Lite and Verified, its broader applicability and modularity make it a more versatile solution across tasks and programming languages.

3. Fault Localization Comparison with SWE-Agent

"The HyperAgent proposed in this paper does not compare its fault localization performance against SWE-Agent."

SWE-Agent does not natively support Java, making it incompatible with benchmarks like Defects4J. In contrast, HyperAgent’s modular design allows easy adaptation to new programming languages by swapping or extending its language server backend.

This adaptability is a key advantage of HyperAgent over SWE-Agent, demonstrating its versatility in diverse programming environments. We will clarify this distinction in the manuscript to avoid any confusion.

4. Executor Design and Unit Tests

"For the Executor, how does it find appropriate unit tests to execute? If defect fixing requires introducing unit tests that do not currently exist in the project, how does this agent function in such scenarios?"

The Executor is designed to handle a wide range of tasks beyond simply executing unit tests:

• Unit Test Execution:

  • The Executor identifies existing test cases based on the Planner’s instructions, using navigation tools to locate test files in the codebase.

• Command Execution:

  • The Executor can execute complex commands, such as build scripts, installation processes, or testing pipelines.
  • In case of failures, the Executor can debug the issue by iterating with the Planner.

• Test Creation:

  • When new unit tests are needed (e.g., for newly added features), the Executor can create and execute test cases based on the Planner’s instructions.

We will expand the manuscript to provide more details about the Executor’s functionality and its ability to handle complex scenarios.

5. Performance Comparison on SWE-Agent

"In Section 5.1, how does HyperAgent perform compared to other models on SWE-Agent?"

Could you clarify this question? SWE-Agent is a system, not a benchmark.
If you are referring to SWE-Bench, we have already compared HyperAgent to multiple baselines (e.g., HyperAgent-Full and HyperAgent-Lite configurations) in Section 5.1.

6. Comparison on Defects4J

"In Section 5.3, how do other open-source agents in Section 5.1 perform on this dataset?"

Could you clarify this question as well?
We have compared all known agent systems on Defects4J at the time of writing this paper. If you are referring to SWE-Agent specifically, it does not support Java, making it incompatible with Defects4J.

This limitation highlights HyperAgent’s adaptability, which supports both Python (SWE-Bench) and Java (Defects4J) out of the box.

We hope these detailed responses address your concerns and clarify HyperAgent’s contributions and distinctions. We are committed to refining the manuscript further to incorporate these points and improve its clarity.

7. Additional Clarifications

Specialization of Subagents

- Subagents are specialized using tailored prompts, tools, and reasoning capabilities. For example, the Navigator uses language servers for file retrieval, while the Editor integrates IDE-like editing tools to handle patch generation and refinement.

We hope these responses address your concerns and clarify the contributions and novelties of our work. We will revise the manuscript to incorporate these points and ensure that HyperAgent’s design and results are clearly presented. Thank you again for your valuable feedback.

We sincerely appreciate your thoughtful and detailed feedback on our manuscript. Your insights have been invaluable in helping us clarify our contributions and strengthen the presentation of our work. Below, we address your concerns point by point.

Comment:

"It’s not quite clear what distinguishes HyperAgent from prior work. The methodology feels like a slight adaptation of existing multi-agent systems (e.g., MetaGPT) for the SWE-bench benchmark."

HyperAgent significantly diverges from existing multi-agent systems through its design philosophy and capabilities, addressing tasks that require dynamic reasoning, scalability, and specialized workflows.

1. Scope and Workflow Design:

Current multi-agent systems like MetaGPT Primarily focuses on constructing entire software projects from scratch (e.g., building a simple program or repository). Its rigid waterfall-like workflow is optimized for end-to-end software construction, following predefined Standard Operating Procedures (SOPs) without dynamic adaptability. In contrast, HyperAgent focuses on solving a broader range of challenging and real-world software engineering (SE) tasks. These include bug localization, fault repair, and repository-level code generation—tasks that require dynamic reasoning, fine-grained localization, iterative editing, and scalable execution. This requires a planner-centered framework, not just a procedural SOP-following system. Unlike MetaGPT, HyperAgent’s Planner is dynamically adaptive, orchestrating agents based on task requirements, parallelizing subtasks when possible, and revisiting earlier phases (e.g., re-localization if editing fails). This makes HyperAgent more suitable for tasks requiring iterative problem-solving, rather than rigid task decomposition.

2. Agent Design for Specialized SE Tasks:

HyperAgent introduces specialized tools and base LLM assignments tailored for complex SE tasks.

• Unlike MetaGPT, which relies on prompt customization, HyperAgent assigns specialized roles (e.g., navigation, editing, execution) and optimizes tools and suggests suitable LLMs for these roles, ensuring scalability, precision and cost saving in solving SE challenges.

3. Scalability Through Parallel Execution:

HyperAgent implements asynchronous communication and parallel task execution via a message queue system, a capability absent in MetaGPT and SWE-Agent.

• For example, in tasks like deploying a web app, HyperAgent can handle subtasks such as deployment scripts, CI/CD configurations, and documentation preparation simultaneously, reducing overall execution time.
• In SWE-Bench, this scalability is evident: HyperAgent achieves time efficiency (106–320 seconds vs. AutoCodeRover’s 720 seconds).

4. Planner-Centered Reasoning for Complex SE Tasks:

HyperAgent’s Planner surpasses MetaGPT’s static Manager by actively monitoring task progress, adjusting strategies, and revisiting prior phases as needed.
Key design features include:

• Minimized Contextual Noise: Child agents transfer only essential information to the Planner, avoiding noisy contexts that degrade LLM reasoning capabilities, as noted in [1].
• Format-Free Reasoning: The Planner operates without rigid agentic formats, enabling better reasoning and problem-solving, as supported by [2].

In summary, HyperAgent is not a slight adaptation of MetaGPT but a dynamically adaptive framework built for real-world, complex SE tasks, incorporating specialized agents, parallel execution, and iterative problem-solving strategies.

[1] Fraga, Natanael. "Challenging LLMs Beyond Information Retrieval: Reasoning Degradation with Long Context Windows." (2024).

[2] Tam, Zhi Rui, et al. "Let me speak freely? a study on the impact of format restrictions on performance of large language models." arXiv preprint arXiv:2408.02442 (2024).

Comment:

"Line 42: 'Their claim of addressing general software engineering tasks is overstated.' Why is this the case? Given that the downstream tasks you evaluate on are all evaluated on Python or Java, how does HyperAgent resolve the shortcomings of more versatility across tasks, languages, and dev. scenarios?"

We acknowledge that our evaluation focuses on Python (SWE-Bench, RepoExec) and Java (Defects4J), but HyperAgent is designed as a flexible and extensible framework capable of supporting additional languages and scenarios in the future.

Limitations of Current Evaluation:

• Demonstrating capability across all programming languages is beyond the scope of this work, as it would require more engineering efforts to evaluate on other repository-level tasks (these kinds of benchmarks are still rare today).
• Current benchmarks, such as Python-based SWE-Bench and RepoExec, are widely adopted in the research community due to Python’s popularity.

Demonstration of Generalization:

• We have demonstrated HyperAgent’s adaptability by extending its capabilities from Python (SWE-Bench, RepoExec) to Java (Defects4J).
• This showcases its potential for generalization across different programming languages, even within the scope of our current evaluation.
• HyperAgent’s modular design, leveraging language server-based tools, makes it straightforward to expand support to additional languages.

Comment:

"Line 99 - How are your three advantages made clear in your evaluation? Aside from higher performance, I do not see why existing works are less generalizable than HyperAgent, or why these agents are more efficient or scalable."

HyperAgent’s flexibility, efficiency, and scalability are reflected in its design and demonstrated in its evaluation, even if not fully elaborated.

1. Generalizability:

• Flexibility Across Tasks: HyperAgent seamlessly handles fault localization, repository-level code generation, and bug repair, unlike task-specific systems such as AutoCodeRover and Agentless.
• Cross-Language Support: Extending from Python to Java in our evaluation (Defects4J) demonstrates HyperAgent’s potential for generalization across languages.

2. Efficiency:

• Streamlined Communication: HyperAgent uses concise interactions between agents and the Planner, reducing the computational overhead seen in prior frameworks like SWE-Agent.
• Example: In SWE-Bench, HyperAgent efficiently solves tasks with lower computational overhead, aided by its modular and lightweight agent design.

3. Scalability:

• Parallelism: HyperAgent employs multi-agent parallelism, where subtasks (e.g., navigation, editing) can be executed concurrently.
• Example: In large-scale tasks, HyperAgent significantly reduces execution time by delegating work across multiple agents, as seen in the faster resolution of SWE-Bench tasks compared to sequential architectures like SWE-Agent.

Comment:

“Line 116 - What is SWE-bench-Python, RepoExec-Python, and Defects4J-Java? Are these variants of the original benchmark?”

No, these are not variants of the original benchmarks. The format BenchmarkName-Language is our way of specifying the programming language associated with each benchmark, but we understand that this may have caused some confusion. For example, SWE-Bench focuses on resolving GitHub issues for Python projects, so we refer to it as SWE-Bench-Python [3]. Similarly, RepoExec [4] is a repository-level code completion benchmark for Python repositories, so we denote it as RepoExec-Python. Defects4J is a widely used benchmark for bug detection and program repair for Java programs, so we refer it to Defects4J-Java [5].

We sincerely apologize for any confusion caused by this naming convention and will make sure to clarify this more explicitly in the paper. This notation was intended to highlight the range of software engineering tasks HyperAgent can handle, demonstrating its capability to work at a repository-level scale across different programming languages.

[3] Jimenez, Carlos E., et al. "SWE-bench: Can Language Models Resolve Real-world Github Issues?." The Twelfth International Conference on Learning Representations.

[4] Hai, Nam Le, Dung Manh Nguyen, and Nghi DQ Bui. "REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark." arXiv preprint arXiv:2406.11927 (2024).

[5] Just, René, Darioush Jalali, and Michael D. Ernst. "Defects4J: A database of existing faults to enable controlled testing studies for Java programs." Proceedings of the 2014 international symposium on software testing and analysis. 2014.

Comment:

“Line 118 - SWE-agent and OpenDevin were both evaluated on non SWE-bench tasks, including HumanEval, HumanEvalFix, Bird, BioCoder, WebArena, and much more. The claim that this is the first system to work on many SE tasks feels like an over-claim.”

Thank you for your comment. Our work specifically targets repository-level software engineering tasks, which we believe are essential for addressing real-world programming challenges. These tasks involve fixing bugs or completing functions in actual software repositories and demand navigating and modifying interconnected codebases, resolving cross-file dependencies, and running comprehensive tests. Such tasks are significantly more complex and realistic than those defined in the benchmarks you mentioned, which are primarily designed to evaluate isolated capabilities of LLMs.

For instance, benchmarks like HumanEval and HumanEvalFix are valuable for assessing the ability of models to generate code snippets but do not represent the intricacies of software engineering. Real-world software engineering rarely involves single-file or self-contained problems; instead, it requires iterative problem-solving across multiple files, maintaining coherence throughout the codebase, and adapting to the context of large, interconnected systems.

Repository-level benchmarks, such as RepoExec and Defects4J, better reflect these real-world scenarios. They simulate workflows that developers encounter daily, requiring deep interactions with the structure and context of repositories. In contrast, benchmarks like HumanEval, Bird, BioCoder, and WebArena focus on simpler, narrowly scoped tasks, such as generating code from concise prompts. While these benchmarks are useful for evaluating fundamental capabilities, they fail to capture the full spectrum of challenges inherent to real-world software engineering.

Our work bridges this gap by demonstrating strong performance on repository-level tasks, showcasing HyperAgent’s ability to handle complex, real-world scenarios. These benchmarks push systems beyond generating isolated code, instead focusing on solving end-to-end tasks at scale. This distinction highlights the novelty of our approach and sets it apart from systems evaluated on benchmarks that do not reflect the complexities of real-world software engineering.

We will revise our manuscript to clarify this distinction and contextualize our contributions more effectively, emphasizing the unique challenges addressed by HyperAgent and its capabilities in tackling interaction-heavy, repository-level software engineering tasks.

We appreciate the thoughtful feedback and the opportunity to address the concerns raised. Below, we provide detailed responses to each point.

Comparison with CodeR and MASAI

While CodeR also employs a multi-agent framework, there are significant differences in design and execution between CodeR and HyperAgent.

1. Agent Coordination:

• CodeR relies on a rigid task graph to enforce agent interactions, which constrains its adaptability to evolving or non-standard tasks.
• MASAI employs a fixed sub-agent sequence tailored specifically for GitHub issue resolution, with components like a Ranker explicitly designed for patch proposals.
  • These fixed designs limit the flexibility and applicability of both frameworks to a broader range of software engineering tasks.
• Other Approaches: Systems like AutoCodeRover and Aider employ rigid, single-phase flows that limit adaptability.
  • For example, if the localization phase in AutoCodeRover fails to identify correct buggy locations, the entire patching process fails, as it cannot revisit the localization phase.

In contrast, HyperAgent is inspired by the natural workflows of software developers, enabling flexibility and adaptability:

• HyperAgent encompasses four key stages:
  1. Analysis and Planning,
  2. Feature Localization and Knowledge Grounding,
  3. Code Editing, and
  4. Execution.
• This design allows HyperAgent to adapt easily to diverse software engineering tasks without requiring significant redesign.
• Dynamic Orchestration:
  • HyperAgent’s Planner dynamically orchestrates agent interactions in real time, tailoring strategies to specific problems while maximizing reasoning capabilities.
  • For example, if an edit fails to resolve an issue, the Planner can:
    • Switch back to the localization phase using the Navigator agent, or
    • Continue refining the edit made by the Editor.
  • This dynamic adaptability distinguishes HyperAgent from the rigid approaches of CodeR, MASAI, AutoCodeRover, Agentless, or Aider.

2. Planner as a Central Component:

• While CodeR’s Manager handles user interaction and plan selection, HyperAgent’s Planner is integral to problem-solving.
• The Planner performs the following:
  • Generates high-level strategies to resolve core problems (e.g., identifying root causes in GitHub issue resolution).
  • Determines optimal subgoals.
  • Selects appropriate agents for execution.
  • Interprets the outcomes of agent actions.

Optimized Communication Flow: To ensure optimal performance, we designed the communication flow between the Planner and child agents with care:

1. Minimized Contextual Noise:
   • Child agents transfer only essential grounding information to the Planner, avoiding noisy or lengthy contexts that could degrade LLM reasoning capabilities (as noted in [1]).
2. Format-Free Reasoning:
   • The Planner operates without rigid agentic formats, enabling better reasoning and problem-solving (as supported in [2]).

This architecture enhances HyperAgent’s adaptability, efficiency, and generalizability across tasks beyond GitHub issue resolution.

3. Fault Localization:

• CodeR relies on BM25 and test coverage for fault localization, which:
  • Requires repository-specific configurations.
  • Reduces generalizability.
  • Demands additional engineering efforts to adapt to new programming languages or scenarios.

HyperAgent abstracts navigation tasks into Navigator agent with modular tools with natural interfaces:

• Tools like language servers execute requests under the hood, making adaptation to new repositories straightforward.
• This abstraction reduces engineering overhead and ensures compatibility with diverse programming languages and scenarios.

We hope this detailed response clarifies how HyperAgent’s dynamic, modular, and adaptive design distinguishes it from prior frameworks like CodeR, MASAI, and other approaches. We are committed to enhancing the manuscript further to address these key differences and contributions.

[1] Fraga, Natanael. "Challenging LLMs Beyond Information Retrieval: Reasoning Degradation with Long Context Windows." (2024).

[2] Tam, Zhi Rui, et al. "Let me speak freely? a study on the impact of format restrictions on performance of large language models." arXiv preprint arXiv:2408.02442 (2024).

Comment:

"The paper also does not provide sufficient evidence as to why a multi-stage approach in general is more "generalisable"."

In our work, generalizability refers to HyperAgent’s ability to adapt seamlessly to a wide range of software engineering (SE) tasks and programming languages without requiring significant architectural changes. Here’s how HyperAgent achieves this:

1. Modularity and Specialized Agents:

• HyperAgent comprises specialized agents (Navigator, Editor, Executor, and Planner), each tailored to perform distinct SE functions.
• This modularity allows us to:
  • Easily introduce new agents.
  • Modify existing agents to accommodate additional tasks or programming languages.
• Dynamic Planning:
  • HyperAgent’s Planner provides maximized reasoning capability and dynamically orchestrates tasks, as detailed in the Comparison with CodeR and MASAI section above.

2. Programming Language-Agnostic Tool Interfaces:

• HyperAgent leverages language servers that provide standardized interfaces for various programming languages.
• By abstracting tool interactions through these servers, HyperAgent can support multiple languages (e.g., Python, Java) with minimal adjustments.
  • Example: Integrating a new language involves connecting to its corresponding language server, which handles tasks like:
    • Linting.
    • Code search.
    • Definition lookups.

3. Ease of Adaptation to New Tasks:

we can introduce new SE tasks into HyperAgent involving modifying or creating task-specific prompt templates, rather than overhauling the entire system architecture.

4. Comprehensive Evaluation Across Diverse Tasks:

• HyperAgent has been rigorously evaluated across a variety of software engineering tasks, achieving state-of-the-art results:
  • RepoExec: Leading performance for code generation.
  • Defects4J: Demonstrated superior bug-fixing capabilities.
  • SWE-Bench: Comparable performance to specialized baselines like AutoCodeRover and Agentless.

In summary, HyperAgent’s modular design, language-agnostic interfaces, and task-specific adaptability enable it to handle varied SE tasks effectively, demonstrating its generalizability across tasks, languages, and scenarios. We admitted that we did use out dated performance reported in Agentless paper at the time we wrote this paper, we will update this entry in manuscript.

Thank-you to the authors for the response and clarifying how this work varies from similar prior approaches. However, given that this work's performance on swebench is comparable or lower than both structured and unstrctured approaches, it is still unclear whether the central claim of this paper that "flexibility and adaptability" of such an approach leads to either concrete improvements in task performance or better generalizability (without comparisons on other benchmarks with other agentic approaches).

Thank you for your thoughtful follow-up question. We appreciate the opportunity to clarify further and address the concerns raised. Below, we provide detailed reasoning and evidence to support the central claims of flexibility, adaptability, and generalizability in HyperAgent.

Performance on SWE-Bench

While HyperAgent’s performance on SWE-Bench is comparable to other structured and unstructured approaches, we would like to emphasize several key points:

1. Task Complexity and Stochasticity
   SWE-Bench is an inherently challenging benchmark due to its complex and stochastic problem-solving dynamics with some unreliable test cases, compounded by the high cost of evaluation and variability in LLM performance, which can lead to unstable statistics during execution. Slight differences in task resolution rates (e.g., between Lite versions of HyperAgent and other baselines) can often be attributed to these factors, making it challenging to draw definitive conclusions from minor performance differences.

2. Comparable Results with Cost Efficiency
   Even on SWE-Bench, HyperAgent demonstrates comparable performance with modular Lite configurations, offering cost savings and enabling real-world deployment flexibility.

Generalizability Beyond SWE-Bench

HyperAgent’s core advantage lies in its flexibility and adaptability, as demonstrated on other benchmarks not only SWE-Bench where it outperforms specialized systems:

1. Program Repair (Defects4J)

   • HyperAgent achieved 25% accuracy on program repair tasks, outperforming the specialized RepairAgent system, which employs a task graph-based approach similar to CodeR, achieving only 20.5%.

2. Bug Localization

   • HyperAgent achieved 59.70% localization accuracy, surpassing AutoFL (51%), an agentic system with a fixed architecture similar to Agentless.

Challenges in Benchmark Comparisons

We acknowledge the reviewer’s concern about comparisons with other agentic systems. However, we hope the following points provide sufficient context:

1. Limited Availability of Source Code

   • Systems like CodeR and MASAI have not released their source code, making direct benchmarking infeasible.
   • This limitation prevents us from directly testing HyperAgent against these systems on broader benchmarks.

2. Cross-Language Benchmarking Challenges

   • SWE-Agent and Agentless are Python-focused systems and do not natively support Java benchmarks like Defects4J.
   • Adapting these systems would require significant engineering efforts to address cross-language challenges (e.g., fault localization or repair in Java). In contrast, HyperAgent easily adapts to new programming languages through its language server interface, showcasing its flexibility.

Dear Reviewer,

We hope this message finds you well. This is a gentle reminder that the rebuttal deadline is today, December 2. We wanted to check if our response has sufficiently addressed your concerns. Please don’t hesitate to reach out if there are any remaining questions or points you’d like us to clarify or discuss further.

Best regards, The Authors

Comment 6:

"Can the authors include a discussion about LSPToolKit and if there are specific design choices that were made to better align it as a tool for LLMs? Specifically, can the authors compare with some of the existing approaches that use LSP to build tools for LLMs? This could be instructive for devtool builders."

We very appreciate the reviewer’s interest in LSPToolKit, as its development was a significant focus of our work earlier this year and even open-sourced, even preceding SWE-Agent. While many details about LSPToolKit are provided in Appendix A.3, we are happy to summarize some of the key design decisions that make it particularly effective as a tool for LLMs. To illustrate, let us take the example of the go_to_definition tool. In standard IDE functionality, go_to_definition requires the cursor position and the file containing the target word (e.g., a method call). Initially, we tasked the LLM with identifying the cursor position by providing the document content annotated with line numbers (similar to IDEs like VSCode). However, we observed that the LLM often struggled with accurately identifying the cursor position, particularly the line and column numbers. In subsequent iterations, we optimized this process by simplifying the input requirements. Instead of asking the LLM for both the cursor position and line number, we only requested the target word and the associated line number. This change significantly improved reliability, as the column number could be easily inferred through an exact match of the word within the specified line. Later, we further enhanced the tool by adding a lightweight line-search functionality based on the target word, which mitigates issues of LLM hallucination regarding line numbers. These iterative refinements showcase our emphasis on minimizing ambiguity and leveraging simple, deterministic mechanisms to reduce reliance on LLM reasoning for tool-specific functionalities. Compared to existing approaches that integrate Language Server Protocol (LSP) features with LLMs, LSPToolKit focuses on designing robust interfaces that reduce error-prone LLM decisions while maximizing alignment with standard IDE workflows. This philosophy ensures greater reliability and efficiency, particularly when handling complex real-world codebases. We agree that expanding on these comparisons and highlighting the lessons learned could be valuable for devtool builders. In future iterations, we will incorporate these insights into the main text to make the contributions of LSPToolKit clearer and more instructive.

Thank you for your thorough review and insightful feedback on our manuscript. We appreciate the opportunity to address your concerns and provide clarifications.

Effectiveness of Multi-Agent Setup:

We acknowledge the comparison with Agentless, which achieved notable results on SWE-Bench-Verified with a cost-effective approach. Our design of HyperAgent aims to balance generalizability across diverse software engineering tasks with performance efficiency. While Agentless demonstrates impressive results on specific benchmarks, its approach may not generalize as effectively to tasks requiring complex interactions, such as repository-level code generation or intricate bug localization.

HyperAgent’s multi-agent architecture is designed to handle such complexities by delegating specialized tasks to dedicated agents, thereby enhancing adaptability across various scenarios. Additionally, we achieved state-of-the-art (SoTA) results on other benchmarks, even when compared to specialized and complex frameworks like RepairAgent on Defects4J.

Comparison to ChatRepair:

ChatRepair follows a fundamentally different evaluation procedure. It provides ChatGPT with “relevant information about the initial test failures as well as earlier patch attempts,” explicitly exposing the failure location to the model and requiring only patch generation.

In contrast:

• HyperAgent and other baselines (e.g., RepairAgent, SelfAPR) operate fully autonomously, starting from an error trace without prior localization information.
• This makes the task significantly more challenging and emphasizes end-to-end problem-solving capabilities.

While ChatRepair’s results are impressive, they do not reflect the full pipeline of bug resolution, making it unsuitable for direct comparison to HyperAgent. We acknowledge that this distinction could be clarified further in the manuscript, and we will update it to include these points.

Comparison to Agentless on SWE-Bench-Verified:

HyperAgent is already compared to Agentless on SWE-Bench-Verified, as shown in Table 1. While Agentless achieves a slightly higher task resolution rate (33.20% vs. 33.00% for HyperAgent-Full-1), HyperAgent provides a modular architecture suitable for more complex tasks beyond SWE-Bench, which Agentless is not designed to handle.

Question:

"The result and analysis shows the developed system works well, how ever it is not clear how generalizability and scalability is presented in the paper. It is recommend that authors clearly and explicitly explain how you achieved generality, efficiency, and scalability, what experiments you conducted, and how you evaluated the result section. Or/and add section on generalizability and scalability experiments, because as for "scalability"..."

The reviewer rightly pointed out the need for explicit validation of these properties. Our evaluations (e.g., RepoExec and Defects4J benchmarks) implicitly address generality and efficiency

Generality:

HyperAgent’s adaptability across diverse benchmarks, such as RepoExec and Defects4J, demonstrates its generality. It seamlessly handles tasks in multiple programming languages (e.g., Python and Java) with minimal effort, requiring only modifications to the task prompt template. Unlike systems like AutoCodeRover [3], which relies on language-specific tools tailored for Python, HyperAgent abstracts functional tool designs through a natural interface between child agents and a language server. These language servers, which handle features like linting, “go to definition,” and code search, can be easily replaced or extended to support new programming languages. This middleware abstracts the complexities of interacting with LSP backends, allowing the LLM to work with multiple programming languages without needing to comprehend the unique protocols of each (Appendix A.3.1). Since most languages have open-source language server implementations, integrating additional languages is straightforward.

In contrast to task-specific designs like SWE-Agent [4], which rigidly incorporates issue reproduction before bug fixing, or Agentless [5] AutoCodeRover [3], CodeR [6], which all follow a fixed sequence of execution steps (e.g., localization, modification, testing), HyperAgent dynamically achieves high-level strategies through its intelligent Planner. The Planner leverages dynamic planning with specialized role prompts and operates with reduced cognitive overload. This is achieved by limiting its interactions to child agents rather than directly engaging with tools, which often produce lengthy outputs. This design ensures flexibility and efficiency in reasoning. Design details can be summarized as following points:

• Reduced Contextual Noise: Child agents share only the critical grounding information with the Planner, eliminating unnecessary or overly lengthy inputs that could impair the LLM’s reasoning abilities, as highlighted in [7].
• Unconstrained Reasoning: The Planner functions without the limitations of rigid agentic formats, enhancing its reasoning and problem-solving capabilities, as supported by [8].

Moreover, adapting HyperAgent to new software engineering tasks is streamlined through the modification of task templates (as described in Section 3.4). By classifying tasks into two broad categories—Patch Tasks (requiring code edits) and Prediction Tasks (not requiring edits)—HyperAgent provides a reusable framework for most SE challenges.

Efficiency:

HyperAgent demonstrates significant efficiency improvements over state-of-the-art (SOTA) methods. Metrics such as reduced average execution time and token costs (as shown in Tables 1 and 2) highlight its ability to achieve competitive results with lower computational overhead. These results are further supported by the system’s modular architecture, which optimally distributes tasks among specialized agents.

Scalability:

Experiments with varying repository sizes and evaluation instances are done in parallel manner, as shown in fault localization and code generation tasks, illustrate scalability with asynchronous message queues. We will add these specific results in a dedicated subsection for clarity.

[3] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1592–1604. https://doi.org/10.1145/3650212.3680384

[4] Yang, John, et al. "Swe-agent: Agent-computer interfaces enable automated software engineering." arXiv preprint arXiv:2405.15793 (2024).

[5] Xia, Chunqiu Steven, et al. "Agentless: Demystifying LLM-based Software Engineering Agents." CoRR (2024).

[6] Chen, Dong, et al. "CodeR: Issue Resolving with Multi-Agent and Task Graphs." arXiv preprint arXiv:2406.01304 (2024).

[7] Fraga, Natanael. "Challenging LLMs Beyond Information Retrieval: Reasoning Degradation with Long Context Windows." (2024).

[8] Tam, Zhi Rui, et al. "Let me speak freely? a study on the impact of format restrictions on performance of large language models." arXiv preprint arXiv:2408.02442 (2024).

We thank the reviewers for their constructive feedback and insightful questions, which have helped us identify areas to clarify and improve. Below, we address the key concerns raised:

Implementation Details in Section 4

We acknowledge that Section 4 would benefit from additional elaboration to provide a more comprehensive architectural overview. While it currently summarizes agent roles, tools, and configurations, it lacks sufficient detail regarding the system’s underlying design. To address this, we propose the following clarifications:

1. Detailed Architecture:

The implementation details of HyperAgent are outlined in Appendix A.7, which includes agent-specific prompt templates. All agents, except the Planner, follow a ReAct-like design [1]. In each iteration, an agent performs the following sequence:

• Thought Generation:
  The agent generates a thought string, representing its reasoning process for the current task (e.g., “Determine which function contains the bug”).
• Action Generation:
  The agent produces an action string, implemented as Python code [2]. This code serves as the unified action space, with tools (e.g., navigation or editing functions) predefined as built-in Python functions that the agent can invoke.
• Observation:
  The agent receives an observation string, capturing the execution results of the generated Python code (e.g., the output of a code search or edit operation).

Detailed examples of agent trajectories, including interactions and tool usage, are provided in Section A.3.

2. Planner’s Role:

The Planner operates differently, serving as the central coordinator. In each iteration, it generates:

• A thought string, dynamically outlining its strategy to resolve the current task.
• A request to a child agent (e.g., Navigator, Editor, or Executor), specifying the action required.
• A context, which includes:
  • Objective Details: Information necessary for the child agent to complete the specific task.
  • Global Background: Broader context required for successful task execution (e.g., dependencies or impacted files).

Example:
If the Planner instructs the Editor to modify a Python file to add a new feature, the context might include details on the impacted files or functions required for dependency resolution. Once the child agent completes its task, it sends a summary of its execution back to the Planner.

3. Agent Communication:

The communication framework, detailed in Section 3.2 (“Agent Communication and Scalability”), ensures smooth interactions between the Planner and child agents:

• It employs an asynchronous message queue system to handle requests and responses, enabling dynamic task coordination and efficient resource allocation.

4. Scalability and Message Queue Role:

The role of the message queue in enabling scalability was briefly outlined in Section 3.2 but requires clearer linkage to task complexity. We emphasize:

• Parallelism:
  • Multiple Navigator or Executor instances can run concurrently, handling subtasks independently.
  • This design significantly reduces bottlenecks in large codebases or multiple user request scenarios (e.g., real deployment).
• Resource Allocation:
  • The message queue dynamically distributes subtasks, allowing efficient scaling with minimal overhead.
• Retries and failure determination: Each tool of child agent explicitly reports errors (e.g., syntax issues or failed edits). These errors are summarized in the agent trajectory and sent back to the Planner. Then Planner will decide whether we should retry this request, if yes, this will be inserted into the queue.

Demonstrated Efficiency:

The scalability and efficiency of these design details are reflected in the low average task execution times in SWE-Bench (Section 5.1.2):

• Full-1 Configuration (Claude 3 Sonnet): 320 seconds.
• Lite Version: 106 seconds.
• Compared to AutoCodeRover’s 720 seconds, HyperAgent achieves 2–6x time efficiency.

We will expand Section 3.2 with concrete examples to better demonstrate these capabilities.

[1] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." The Eleventh International Conference on Learning Representations.

[2] Wang, Xingyao, et al. "Executable Code Actions Elicit Better LLM Agents." Forty-first International Conference on Machine Learning.

Dear Reviewer,

We hope this message finds you well. This is a gentle reminder that the rebuttal deadline is tomorrow, November 26. We wanted to check if our response has sufficiently addressed your concerns. Please don’t hesitate to reach out if there are any remaining questions or points you’d like us to clarify or discuss further.

Best regards, The Authors