OpenAgents: An Open Platform for Language Agents in the Wild

Thank you for your positive review and constructive feedback. We appreciate your recognition of the significance and quality of our work on OpenAgents. We have addressed your concerns and provided further clarification on the points you raised.

W1: Emphasizing the difference between OpenAgents and ChatGPT Plus:

A: Thank you for highlighting this important point. While we don’t have detailed knowledge of ChatGPT Plus’s internals, we emphasize the unique aspects of OpenAgents, including its open-source nature, modular architecture, and wide API and tool integration. Unlike ChatGPT Plus, OpenAgents offers customizable interfaces for different user types and supports the integration of external agents and plugins, enhancing flexibility and extensibility.

W2: Functionality based on code or APIs:

A: Thank you for pointing out a common issue with current API-calling-based agents. Although OpenAgents straddles both approaches with our WebAgent, we believe these two methods address different angles. The performance of advanced models like GPT-4 on benchmarks like WebArena and OSWorld is only around 10%, which is not practical for the current users and developers. In comparison, the API-based modeling approach is more easily implemented by current language models, making it a tradeoff in the application.

Q1: Comparison with LangChain and integration possibilities:

A: LangChain aims to facilitate the construction of proof-of-concept language agents while neglecting non-expert user access to agents and paying little attention to application-level designs. We have integrated these aspects into the backend logic, frontend logic, and user interface, building a product-grade platform, and uncovered many issues that only emerge in real-world applications, such as the need for handling long text efficiently, and the expansion and maintenance of real tools, etc. OpenAgents is designed to be compatible with LangChain. Users can integrate the two platforms easily to further enhance functionality.

Q2: Prompting strategies in OpenAgents:

A: Prompting is indeed a critical part of OpenAgents. Our platform supports a diverse set of prompting strategies, including templates, chaining, and context-aware prompting. We have designed OpenAgents to be user-friendly, allowing users to add and customize their own prompting strategies easily. This flexibility ensures that users can tailor the platform to their specific needs and preferences.

Thank you for your thoughtful review and appreciation of our work. We are glad you highlighted the importance of our studied topic, the quality of the illustrations, and the provision of an open-source codebase. We have addressed the specific points you mentioned and will update the paper accordingly if it is accepted.

W1: Writing Style & Technical Depth:

A: Thank you for highlighting the need for more detailed technical explanations. We have expanded several subsections to provide deeper insights into our method. Specifically, in Section A.1.2, we elaborate on the challenges of strategic data storage, detailing our hierarchical storage strategy (local memory -> Redis -> persistent database) with illustrations. We explain which variables are stored at each level and the benefits of this approach. Additionally, we provide more details on grounding sources in Section A.2.1 and auto plugin selection in Section A.2.2. We will update the paper accordingly if accepted.

W2: No Experiments & Analysis:

A: We agree that quantitative verification is essential for demonstrating the effectiveness of OpenAgents. In the revised manuscript, we will include human-in-the-loop experiments: 1) user satisfaction surveys, 2) average time to complete selected tasks, and 3) failed case studies. Evaluating human-agent interaction is challenging, but we have made significant efforts to improve platform quality. We have established Discord and Slack channels with over 6,000 users and addressed 87 issues and engaged 16 new contributors. Despite the lack of reliable automatic evaluation, we are committed to enhancing OpenAgents based on user feedback.

W3: Preciseness & Timeliness:

A: Thank you for the observation. We have updated Table 1 with the latest data as of April 2024 to ensure our comparisons are both fair and current.

Thank you for your thoughtful review and for recognizing the broad utility and detailed exploration presented in our OpenAgents platform. We appreciate your constructive and detailed feedback.

W1: Gathering feedback from actual users

A: We agree that user feedback is crucial for evaluating the platform's effectiveness. Although our main focus is the design philosophy and technical details of OpenAgents, we discuss the need for new language agent metrics and in-the-wild evaluation in Sec. 5 and 6. Evaluating human-agent interaction is challenging, but we have made significant efforts to improve platform quality. While automatic evaluation isn't reliable yet, we are committed to enhancing OpenAgents based on user feedback.

W2: Performance of the auto-selection feature for plugins

A: The goal of the OpenAgents paper is on how features like auto tool selection significantly enhance user experience compared to ChatGPT Plus, rather than examining the capabilities of individual components in detail, which is beyond the scope of this paper.

W3: Novel contributions in agents' design

A: While the primary focus of our work is to create an accessible and versatile platform, we also highlight our technical contributions in two key areas.

1. System design: For Data & Plugin Agents, we follow the ReAct agent definition but introduce techniques such as adaptive data mapping, data grounding, and auto tool selection/tool scaling to transition from proof-of-concept to real-world applications.
2. Web Agent: We use a Chrome extension for the Web Agent, enabling dialog-based assistance to help users complete tasks through interactive actions. We will revise the manuscript to detail our agents' unique design aspects.

W4: Chat API service and handling large volumes of user queries

A: We appreciate your concern regarding the chat API service. In our released code, we expose the chat/ endpoint API on the server side for handling large concurrent user queries. Users can directly query this API in their programs/applications for concurrent access instead of on the web demo.

Q1: Missing Plugins

A: We previously uploaded 16 best-performing tools in our user test as some other public tools suffered from connection or maintenance issues. We have been cleaning and validating the rest tools this month and have uploaded all 200 API tools these days.