OpenHands: An Open Platform for AI Software Developers as Generalist Agents

We appreciate the reviewer’s thoughtful feedback and are pleased that the reviewer find our work well-motivated, well-written, and useful to the community.

The paper is an announcement of a software platform, more than it is a description of a contribution to a research area. Normally I'd expect a paper to look like a research question, with some motivation, experimental design, and results…. I'm unsure whether this kind of project-announcement is a fit for the conference…. and then uncritically showcases performance on a broad range of existing benchmarks.

According to ICLR 2025 call for paper (https://iclr.cc/Conferences/2025/CallForPapers), OpenHands primarily falls under the category of “infrastructure, software libraries, hardware, etc.” where we consider the primary contribution be the infrastructure that supports development of generalist software (multi-)agent systems with runtime environment, interface, and evaluation. The demonstrated performance across a broad range of benchmarks covering different domains (e.g., software, browsing, general assistance) shows OpenHands’ ability to solve a wide range of tasks without changing the infra as well as the agent, and supports the potential of OpenHands as an infrastructure to support development of generalist digital agents.

Why do the (e.g.) GPQA results not compare like-for-like LLMs? I would want to see whether a CodeActAgent powered by a particular LLM was better or worse than the bare LLM, but the table compares (Llama 3 70b chat, GPT 3.5 turbo 16k, GPT 4) with (gpt-4o-mini-2024-07-18 , gpt-4o-2024-05-13 , claude-3-5-sonnet), so it's unclear how much of the difference is due to the scaffold and what is due to the choice of LLM. This is also true for many results in Tables 4 onwards, perhaps an unfortunate downside of comparing to results from the literature, but this limitation precluding rigorous comparison should at least be acknowledged in the text.

Thanks for pointing this out, we acknowledge this limitation in our current evaluation and will update our next revision accordingly. We are actively working on expanding results for our evaluation benchmarks to include more LLM backbones. Nevertheless, we believe that our existing result does serve the purpose of demonstrating the generalist capabilities of OpenHands: it is able to solve a wide range of tasks, that used to require custom prompting, with the SAME agent scaffold and system message and achieve non-trivial performance compared to prior work that applies custom prompting.

A large fraction of the paper comprises short summaries of many different benchmarks used to evaluate the codebase. Since these summaries are not original work, I question whether so much of the paper should be dedicated to them…

We appreciate the reviewer’s feedback and will be sure to include a more extensive discussion on the results in the next revision. We spend a lot of effort trying to cut down the summaries for original work, however, we re-adapt some settings about these existing benchmarks to bring them into OpenHands evaluation harness and we feel it is still necessary to include these in the paper.

L305 - I'm uncertain what is meant by "we compare OpenHands to open-source reproducible baselines that do not perform manual prompt engineering specifically based on the benchmark content"

We consider the following criteria for including baseline for browsing benchmarks:

• The baseline needs to have open source reproducible code so we can verify its result. There were some WebArena numbers publicized only on Twitter, without actual open-source code to reproduce the result.
• We also disqualified baselines that perform manual prompt engineering on the testset of a benchmark (e.g., SteP uses handcrafted "routines" or "workflow" library specifically designed for WebArena websites)

I welcome the pointer to the Call for Papers, and specifically the pointer to the category of “infrastructure, software libraries, hardware, etc.” I agree that OpenHands falls into this category, and so my main concern is allayed.

The clarification around L305 is helpful, thank you. Perhaps this is worth including in the next revision.

The 2% -> 36% is surprising enough that the explanation you give about irrelevancies in long prompts distracting weaker models (very clear now, thank you) warrants inclusion in the next version, as perhaps does the points about CAA sometimes passing an inexact task to BA.

For the other points, I welcome the next revision.

It seems surprising that OH CodeActAgent w. GPT-3.5 would get 2% on ToolQA, while ReAct (perhaps the most minimal form of agent scaffolding) would get 36%. Do you have any idea what's behind this performance difference?

We are using the exact SAME system prompt to evaluate all the tasks for CodeActAgent (the generalist) to make sure it general enough to be able to handle a wide range of tasks. This includes distractors to weaker models (e.g., instruction about how to use browser is typically not relevant to solve a ToolQA problem), hence we are seeing a lower performance on weaker models like GPT-3.5 where we observe they would be overwhelmed by the system message and get stuck into loops. On the contrary, stronger models like gpt-4o demonstrate reasonable performance.

Also surprising is that CodeActAgent delegating to BrowsingAgent does very slightly worse than straightforwardly using BA directly. Could you discuss this?

We measure performance of the browsing delegation by asking CodeActAgent to produce the correct agent delegation action and evaluate the correctness of the delegation request. In some cases, CodeActAgent would not be able to pass the exact browsing task to BrowserAgent, which results in the minor performance degradation.

We appreciate the reviewer’s thoughtful feedback and are pleased that the reviewer finds our work sound, well-written, and valuable to the community.

While the value of OpenHands is apparent, the research question of this paper is unclear. If it is that they can develop a general agent that performs better across a range of tasks than any existing agent does, it is my judgement that the results and analysis do not explore this sufficiently to warrant a research contribution. To do so, discussion and ablations should be included for the design decisions of OH Browsing Agent v*, OH CodeAgent v*, OH GPTSwarm v*, etc. To then isolate the effect of those decisions, the choice of backend model should be held consistent to those of the baselines against which the OH agents are compared. Error analysis of results comparing OH to baselines would be another welcome contribution that has been omitted from this paper.

Based on the ICLR 2025 call for papers (https://iclr.cc/Conferences/2025/CallForPapers), we consider OpenHands primarily falls under the category of “infrastructure, software libraries, hardware, etc,” where we consider the primary contribution be the infrastructure that supports development of generalist software (multi-)agent systems with runtime environment, interface, and evaluation. While analyzing agent design decisions would be valuable, we have limited space in the paper and we barely fit the description of all existing OpenHands components and evaluations with current page limits. Therefore, we would refer to the individual agent papers for detailed analysis (e.g., CodeAct, GPTSwarm) and focus on infrastructure that enables such agentic research and evaluation in this paper.

Can we further motivate why we want a generalist agent? If we have specialized agents for different tasks, can we just have a coordinator deploy specialized agents where they perform best?

While having a “specialized agent” + multi-agent system could be good for individual task performance (and maybe cost, since specialized agent could use specialized model which can be made small and efficient), they are not without practical drawbacks:

• Real-world tasks are not always perfectly decomposable into multi-agent + specialized agent setting: For example, for coding tasks, a multi-agent system CodeR [1] has five specialized agents (Manager, Reproducer, Fault Localizer, Editor, Verifier). For instance, what if the verifier, which is responsible of outputting yes/no decision of whether the issue is resolved or not, wants to perform file localization to be sure that it has verified the answer properly? Because these agents are completely separate, the verifier would not have access to the tools necessary to do its job.

• Challenging to preserve context in communication: Multi-agent systems typically pass information between multiple agents, but this can be a source of information loss. Following the above CodeR example, if the fault localizer passes only a summary of its work on to the further agents, it often results in the loss of important contextual information that could be useful to the downstream agents for their jobs.

• Maintainability: Finally, each of these specialized agents typically has its own separate code base, or at least a separate prompt. Because of this, multi-agent systems can have larger and more complex codebases. And the codebase could be constantly growing as we require more capability from the agent – making maintenance hard and eventually causing reliability issues with the codebase.

Due to the above potential drawbacks, we are motivated to make our infrastructure general enough to support developing and researching both generalist and specialist agents so we can further explore both the advantages and disadvantages of these directions going forward.

[1] CodeR: Issue Resolving with Multi-Agent and Task Graphs

We appreciate the reviewer’s detailed feedback and are pleased that the reviewer finds our work solid, well-written, and useful to the community.

…Some features, like code execution in a sandbox and web browsing agents, are present in other platforms. The paper could better articulate how OpenHands distinguishes itself from similar frameworks like LangChain, DSPy, or AutoGen… Distinction from Existing Platforms: How does OpenHands fundamentally differ from other agent development platforms like LangGraph, CrewAI and DSPy for multi-agent collaboration or BrowserGym for benchmark evaluation? ….. Specifically, what makes the framework the best solution to develop an agent when one approaches such a task?..... Distinction from Existing Platforms: How does OpenHands fundamentally differ from other agent development platforms like LangGraph, CrewAI and DSPy for multi-agent collaboration or BrowserGym for benchmark evaluation?

We describe the differences between OpenHands and some popular existing agent frameworks in Table 1. We did not include DSPy as it is focusing on end-to-end prompt optimization and less relevant as an “agent framework”. We will include CrewAI and BrowserGym in the next revision. Here’s a high-level overview of the differences between OpenHands and CrewAI & BrowserGym

• CrewAI largely focuses on orchestrating multi-agent communications, but lacks features on the sandboxed terminal, browser, and file system compared to OpenHands - which we will explain more in the following paragraph. It also doesn’t include a comprehensive evaluation harness to support development.
• BrowserGym, on the other hand, is a platform specifically designed for browsing agents - it lacks the feature of sandboxed code execution and file system, and contains benchmarks only for web browsing, whereas OpenHands aims to build a general software engineering agent that’s capable of doing much more than browsing.

That is, existing frameworks mostly cover a subset of OpenHands features, whereas OpenHands smoothly integrates the entire workflow of AI agent development and deployment:

(development & deployment) Sandboxed Runtime Environment that can build on arbitrary docker images: Different from most agent frameworks that don’t provide this feature, OpenHands provides a complete integrated runtime environment, which is non-trivial to implement and scale. Compared to existing frameworks:

• The OpenHands runtime environment automatically builds supports for the terminal, file system and editor and browser automatically on top of arbitrary docker images that the user provides - this allows OpenHands to run on any user-provided repository-specific development image, which other existing frameworks don’t support to the best of our knowledge.
• LangChain/LangGraph has only very limited support for runtime (e.g., only supports code execution - but not bash-like terminal commands), and most of this support requires using third-party non-open-source solutions (https://python.langchain.com/docs/integrations/tools/#code-interpreter).
• AutoGen does have limited support for python, bash execution. But different from the OpenHands implementation, to the best of our knowledge, AutoGen’s python & bash execution based on docker exec is not stateful: If the agent changed to a directory from a previous action (e.g., cd aaa/), the next bash action will NOT be executed in the directory aaa/ - this is non-trivial to implement and can be a major confusion point for an agent if not implemented. To the best of our knowledge, OpenHands is the only one comprehensive open-source agent framework that supports stateful bash command execution via pexcept.
• CrewAI has limited support for sandboxed code execution (https://github.com/crewAIInc/crewAI-tools/blob/afa7e6a64347e09003088c400afcf4503433f2a2/crewai_tools/tools/code_interpreter_tool/code_interpreter_tool.py#L107), but it is restricted to stateless Python code, whereas OpenHands support both stateful bash & python (jupyter) execution within sandboxed runtime.
• AutoGen and CrewAI also have limited support for browsing (https://microsoft.github.io/autogen/0.2/docs/reference/browser_utils/abstract_markdown_browser, https://github.com/crewAIInc/crewAI-tools/blob/afa7e6a64347e09003088c400afcf4503433f2a2/crewai_tools/tools/browserbase_load_tool/browserbase_load_tool.py#L47-L50), however, they only support one feature which is “getting the content of a webpage in a markdown format.” On the other hand, OpenHands sits on top of BrowserGym, not only supports getting markdown format, also allows the agent to interact with the web page directly by observing screenshots, clicking buttons, inputting text, etc.

We appreciate the reviewer’s thoughtful feedback and are pleased that the reviewer found our work well-motivated, well-written, and useful to the community.

The authors mention that event streaming enables human intervention in agent tasks, which I believe is a crucial step for calibrating agent outputs. I wonder whether human intervention is enabled during the evaluation of agents and is there a mechanism to determine whether human intervention is required?

During evaluation, following the existing benchmark setting, human intervention is disabled to ensure consistent and reproducible results. For example, the original SWE-Bench setup does not consider human inputs during evaluation, hence we prompt the agent to not interact with humans while solving a given SWE-Bench task. When it did try to interact with the user, we provided pre-determined feedback “Please continue working on the task on whatever approach you think is suitable. If you think you have solved the task, please first send your answer to user through message and then finish the interaction.” – this prompts the agent to either continue to solve the problem without human intervention, or finish the interaction.

Regarding a mechanism to determine whether human intervention is required, typically the agent (and the underlying foundation model) is responsible for making such decisions (i.e., it only generates text without action for the next step). Based on our observation, it typically happens when the agent finishes the task, or gets stuck into some problems it cannot resolve.

Regarding the multi-agent delegation feature, I'm unclear on how agents collaborate. Is there an orchestrator agent responsible for task delegation, or is this managed through event streaming? If it's the latter, how are tasks assigned and dispatched, and where do agents deliver their outputs?

In OpenHands, agent collaboration is implemented through an event streaming mechanism rather than a single orchestrator agent. When an agent needs to delegate a task, it submits an AgentDelegateAction to the event stream. A central controller then dispatches this request to the appropriate child agent and manages the return of results through the same event stream. Child agents are typically specialized (e.g., for browsing) and receive only their specific tasks and necessary context. This design ensures a clear separation of responsibilities while enabling flexible collaboration patterns. For example, our DelegatorAgent in our agenthub is an orchestrator agent that coordinates between three specialized agents for repository analysis, coding, and verification, while our ManagerAgent maintains a catalog of micro-agents defined in markdown and delegates tasks based on their capabilities. The ManagerAgent analyzes each micro-agent's description and constraints to select the most suitable one for a given task or rejects the task if no micro-agent is capable of handling it.