Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence

Thank you for your thoughtful and detailed review, and for recognizing the motivation, design, and contributions of our IoA framework. Below, we address your questions and concerns in detail.

1. Theoretical Analysis of Properties (Convergence, Scalability)

Thank you for raising this important point. Still we would like to say that LLM-based agent systems represent a highly empirical research area, and sometimes the primary goal is to develop systems that work effectively in practice, as the field is still in its infancy. While theoretical guarantees such as convergence are desirable, they are inherently challenging to formulate for LLM-based systems. The use of language as the primary medium introduces complexities that are indifferentiable, making it difficult to apply theoretical frameworks like those used in traditional RL research.

In multi-agent systems, these challenges are further compounded by the dynamics of inter-agent communication and collaboration, where properties like convergence are particularly hard to predict or measure. Despite these hurdles, we have made an effort to justify IoA's scalability from the perspective of communication topology. Specifically, the sparse communication structure introduced by the nested team formation mechanism significantly reduces the number of communication edges compared to fully connected architectures. This sparsity is a promising direction for improving the scalability of multi-agent systems, and our empirical results already demonstrate its potential.

We fully acknowledge the importance of further theoretical analysis and recognize this as a shared challenge within the LLM-based multi-agent research community. We will continue exploring ways to incorporate theoretical insights into future iterations of IoA. Thank you for your thoughtful advice, which has helped us reflect on these critical aspects of our framework.

2. Analysis of Latency, Resource Usage, and Communication Overhead

Thank you for pointing this out, and we appreciate your careful reading. As noted, Appendix E includes preliminary analysis of costs. As the communication overhead is largely associated with the number of generated tokens and input tokens, the cost can also partially serve as a proxy of communication overhead. On average, IoA is approximately 2 to 3 times slower than standalone agents due to the added complexity of agent collaboration. Still, as mentioned in our response to reviewer 78P6’s first question, we have a large room for optimization. In addition, directly quantifying network latency in distributed settings remains challenging due to varying device configurations and network conditions.

As a prototype implementation of the broader IoA concept, our current focus is on demonstrating performance effectiveness and validating the benefits of connecting heterogeneous, distributed agents. We recognize the importance of analyzing additional metrics, such as latency and real-world operating performance. We will address these aspects in future work with more advanced and comprehensive designs.

3. Potential Failure Modes, Mitigation Strategies, Security, Deadlock, and Resource Allocation

Thank you for suggesting a more detailed discussion of failure modes and related aspects. In scenarios with unreliable networks or agent failures, IoA employs the following strategies:

• Retry Mechanisms: Agents automatically retry communication with the server or tools when initial attempts fail.
• Autonomous Error Handling: Our experiments demonstrate that agents can handle task failures autonomously, even without explicit instructions. An example of this process is illustrated in the anonymous link https://anonymous.4open.science/r/ioa-rebuttal-606B/complete_trajectory_GAIA.pdf, where a WebBrowserAgent collaborates with a WikidataAgent. When the WikidataAgent fails due to tool limitations, the agents decide to let the WebBrowserAgent attempt the task, ultimately succeeding.

Regarding security and deadlock, we acknowledge that these aspects are currently addressed only at a basic level. For example, simple timeout-based retries are employed to handle potential deadlocks. Additionally, IoA does not impose explicit assumptions on the resources available to each agent. While these are important considerations, our current focus has been on demonstrating task performance effectiveness as a foundational exploration of the Internet of Agents concept.

We agree that IoA's robustness can be significantly enhanced with more advanced error recovery, fault tolerance, and security mechanisms. These, along with strategies for addressing deadlocks, resource allocation, and behavior monitoring, are essential for the “ultimate” version of IoA and will be explored in future iterations. To address your feedback, we will add a section in the appendix outlining these considerations and potential enhancements to improve the system's overall robustness and reliability.

Thank you for engaging with my review! I'm very happy to maintain my score and hope to see this paper at the conference!

Thank you for your thoughtful review and for highlighting the novelty, rigorous design, and comprehensive evaluation of our IoA framework. We are especially pleased that you found the architecture well-motivated and the empirical results compelling. Below, we address your concerns.

1. The Cost of IoA Compared to Other Frameworks

We completely understand your concern regarding the higher costs associated with IoA compared to other frameworks such as AutoGPT and OpenInterpreter. We have identified three primary factors contributing to this:

1. Unoptimized Prompts: In our experiments, we prioritized clarity and robustness over brevity in prompt design. This has resulted in longer input prompts, increasing the input token cost per LLM call.
2. Repeated Responses: LLMs occasionally generate redundant responses, repeating prior outputs without contributing new information.
3. Redundant Communication Patterns: The communication patterns between agents currently mimic human conversational norms, which often include greetings, verbose explanations, and redundant grammar. While this enhances readability, much of this content is unnecessary for task performance and could potentially be omitted or simplified.

To address these cost concerns, we believe there is substantial room for optimization:

• Optimizing Prompts: By making prompts more concise and context-aware, we can significantly reduce input token costs, especially since multiple LLM calls occur during task execution. Efforts to streamline prompts are already underway.
• Leveraging Advanced Models: As LLMs like GPT-4 and subsequent iterations improve foundational capabilities and alignment, issues such as repeated responses are expected to diminish, leading to lower output token costs.
• Improving Communication Efficiency: Recent work on enhancing agent communication efficiency, such as sparse communication protocols and compact message representations [1,2,3], can be integrated into IoA. These methods prioritize semantic content over verbose dialogue, substantially reducing token usage.

We believe that IoA serves as an important prototype for LLM-based MAS and that further optimizations will make it more cost-effective while maintaining its performance advantages. We are encouraged by your recognition of our efforts and remain committed to exploring these optimizations in future work.

2. Scalability with Increasing Number of Agents

Scalability is a core consideration in IoA’s design. Empirical results from GAIA and RocoBench already demonstrate IoA’s effectiveness with teams of varying sizes. However, the current agent benchmarks rarely require coordination among more than five agents for task-solving, making it challenging totest IoA's performance in larger-scale scenarios.

We believe this reflects a broader challenge in the development of multi-agent benchmarks: the lack of benchmarks that simultaneously require significant collaboration, involve complex problem-solving, and demand scalability to larger agent groups. Existing benchmarks primarily focus on small-scale tasks, which, while valuable, do not fully capture the potential of distributed multi-agent systems like IoA. When suitable and sufficiently complex benchmarks for larger groups of agents are developed, we are eager to extend IoA’s evaluation to those scenarios. In the meantime, our experiments have robustly demonstrated IoA’s effectiveness in not-that-large scale, highlighting its ability to facilitate collaboration and enhance performance. These results provide a strong foundation for scaling IoA to larger teams in future work, and our work may open a new research direction for researchers, providing a possible effective way for scaling MAS.

We hope these responses address your concerns and clarify the cost-related and scalability aspects of IoA. If our explanations alleviate your concerns, we kindly request you to maintain your positive evaluation. Thank you again for your constructive feedback and support, which will guide our future optimizations and research.

References

[1] Li, Yunxuan, et al. "Improving Multi-Agent Debate with Sparse Communication Topology." arXiv preprint arXiv:2406.11776 (2024).

[2] Chen, Weize, et al. "Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication." arXiv preprint arXiv:2402.18439 (2024).

[3] Chen, Weize, et al. "Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System." arXiv preprint arXiv:2410.08115 (2024).

Thank you for your detailed review and for highlighting the exciting potential of our IoA framework. We appreciate your recognition of the novelty in designing a “language” for agent collaboration and the significant questions this work addresses. Below, we respond to your concerns and clarify key aspects of our work.

1. IoA only as a Proof of Concept

We agree that creating a fully functional Internet of Agents is an ambitious and extensive project. In this paper, we present IoA as a proof of concept, focusing on demonstrating its potential to connect heterogeneous and distributed agents. To validate its effectiveness, we evaluated IoA on diverse tasks, including the GAIA benchmark (complex real-world tasks), RocoBench (embodied agent collaboration), and QA with RAG across various datasets. The variety of benchmarks illustrates IoA’s potential in different contexts. While IoA remains a prototype, we believe showing its promise across diverse areas is an important step forward for the field.

2. Figure 1 Is Too Cluttered

Thank you for pointing this out. We have redrawn Figure 1 with a simpler and cleaner design using basic arrows and boxes to illustrate the layered architecture and component interactions clearly. The updated figure is accessible via the following anonymous link: https://anonymous.4open.science/r/ioa-rebuttal-606B/architecture.pdf. We will update the submission to include this revised figure, ensuring it enhances the clarity of our presentation. Thank you for your advice.

3. Example of the Task Workflow

We agree that providing a concrete example of the workflow would enhance the readers' understanding of the system. While Appendix B includes a brief example, we have now collected a complete interaction trajectory from IoA to demonstrate its workflow in more detail. These trajectories are visualized in new figures, available at https://anonymous.4open.science/r/ioa-rebuttal-606B (files labeled complete_trajectory_*.pdf). We will include these figures in the appendix and reference them in the main text to improve clarity and accessibility.

4. Selection of Benchmarks

The benchmarks were chosen to cover diverse areas and demonstrate IoA’s versatility:

1. GAIA: A popular, comprehensive benchmark for testing agents on complex real-world tasks, suitable for evaluating IoA’s potential for heterogeneous agent collaboration.
2. RocoBench: Designed for embodied agents in distributed environments, aligning with IoA’s distributed architecture.
3. QA with RAG: Reflecting real-world scenarios where distributed agents possess distinct knowledge, requiring collaborative information exchange to improve question answering.

Our goal was to showcase IoA’s effectiveness and potential across diverse task categories, demonstrating its capability to address different collaboration challenges.

5. Task Breakdown and Workflow of the FSM

IoA does not explicitly define a separate stage for task decomposition. Instead, as described in Section 2.3.3 and Figure 3, the discussion state is where agents autonomously analyze and decompose tasks into subtasks, leveraging their understanding of teammates’ capabilities. Our experiments show that agents transition from discussion to task assignment stages effectively, with minimal need for external intervention. This behavior highlights the strength of IoA’s flexible design, and shows that current LLMs are capable of adapting their behavior to our IoA design.

6. Components Required for Participation in IoA

We use one LLM to handle most of the framework’s mechanisms, including state transition, task assignment, summarizing execution results, etc. Our experiments demonstrate that SoTA LLMs such as GPT-4 and GPT-3.5-turbo, even with simple prompts, are effective in IoA. Key required abilities for an LLM in IoA may include:

1. Task decomposition and assignment based on teammate capabilities.
2. Understanding IoA’s framework and mechanisms for state transition.
3. Correctly interpreting and summarizing tasks and results.

As LLM technology advances, we anticipate broader accessibility and higher performance for IoA-compatible agents.

Thank you for your thoughtful and detailed review, and for recognizing the clarity, organization, and contributions of our work. We are delighted that you found the IoA framework, experimental design, and results compelling. Below, we address your comments and suggestions in detail.

1. Lack of Representative Examples in the Constructed Benchmark

The figures are available at the anonymous link: https://anonymous.4open.science/r/ioa-rebuttal-606B. Files labeled responses_*.pdf include representative instructions from the open-ended instruction benchmark and the corresponding responses from various frameworks. Additionally, files labeled complete_trajectory_*.pdf present complete communication trajectories from GAIA and the open-ended instruction benchmark, offering a detailed view of IoA's workflow and performance. We hope these additions address your concerns effectively.

2. Discussion on Potential Biases in Using GPT-4 for Evaluation

We acknowledge the potential biases and limitations associated with using GPT-4 as an evaluation metric. While GPT-4 has demonstrated strong alignment with human judgment in prior works [1], overconfidence and subtle biases can occasionally affect its scoring. To mitigate these issues:

1. Robust Evaluation Protocols: We adopt the protocol from [1], alternating the order of responses during evaluation to minimize order-induced biases.
2. Bias Mitigation through Diversity: The instructions in our benchmark span diverse categories, reducing the potential for category-specific biases in GPT-4’s scoring.

While we considered alternative approaches, such as human evaluation or using multiple LLM judges, these introduce scalability challenges. GPT-4 remains a practical choice for large-scale evaluation, and its agreement with human evaluators in prior works supports its reliability. Also, it is widely adopted by current LLM judgment, e.g., MT-Bench [1], Arena-Hard [2], Alpaca-Eval [3,4].

We will add a discussion in the appendix explicitly outlining the potential limitations of GPT-4 in the evaluation and our strategies to address them.

We hope these responses address your concerns and further enhance the clarity and rigor of our work. If these clarifications alleviate your concerns, we kindly request you to consider maintaining your positive evaluation. Thank you again for your constructive feedback and insightful questions.

[1] Zheng, L., et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." Advances in Neural Information Processing Systems 36 (2023): 46595-46623.

[2] Li, Tianle, et al. "From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline." arXiv preprint arXiv:2406.11939 (2024).

[3] Li, Xuechen, et al. "Alpacaeval: An automatic evaluator of instruction-following models." 2023,

[4] Dubois, Yann, et al. "Length-controlled alpacaeval: A simple way to debias automatic evaluators." arXiv preprint arXiv:2404.04475 (2024).

Thank you for your thoughtful review and for recognizing the innovative aspects of our framework. We are delighted that you found the strengths of IoA compelling, and we appreciate your detailed feedback, which provides valuable directions for enhancing our work. Below, we address each of your comments in detail.

1. Lack of Detailed Evaluation on Real-World Applications

Thank you for raising this important point. We completely agree that real-world application evaluations would further strengthen our claims. But we have to say current benchmarks for multi-agent systems lack the complexity and metrics suitable for benchmarking of distributed, heterogeneous agents. Therefore, in our paper, we utilized GAIA and embodied task benchmarks like RocoBench, which simulate real-world conditions requiring agent cooperation and diverse capabilities. While GAIA might not intrinsically require distributed agents, it effectively tests coordination and problem-solving capabilities in complex tasks.

Actually, we have conducted experiments deploying IoA to smart home devices, allowing agents to manage daily routines across distributed devices (e.g., adjusting lighting via a home assistant based on notifications from a portable device agent). Although these experiments show promising results, they are challenging to quantize and standardize for publication, we choose to not present them in our paper. Constructing a real-world benchmark of multi-agent is challenging, and constructing such benchmark could itself be a solid work. We temporarily leave it for future work.

2. Scalability and Performance Analysis

Scalability is indeed critical for IoA. As highlighted in Appendix E (Table 6), we report costs, which partially reflect IoA's computational overhead as it is calculated with the number of input tokens and output tokens. Directly quantifying network latency in distributed settings is complex due to varying device configurations and network conditions. Nonetheless, our architectural design tried to enhance the scalability through features like nested team formation, which reduces communication overhead by segmenting tasks into smaller, manageable group chats. We will mention the computational overhead and the designs that tried to address the scalability concerns explicitly in the main text.

3. Scalability of FSM

We address your concerns of FSM proposed in the weakness and question here. The FSM is designed to operate efficiently within small group chats. For scenarios with a large number of agents, our nested team formation mechanism creates hierarchical subgroups. This minimizes the complexity and ensures FSM efficiency within each subgroup. Recent work has also shown the effectiveness and efficiency of such sparse communication topology [1]. We will expand on this mechanism in the paper to clarify how scalability is achieved through decentralized team formation.

4. Limited Insight into Integration of Third-Party Agents

We appreciate this observation and will add implementation details to the appendix for transparency. These are mostly engineering work so we did not include them in the initial submission. Basically, each third-party agent is wrapped as a Docker container with an adapter that unifies its input/output interfaces, facilitating seamless integration. For example, AutoGPT's integration involved invoking its steps method repeatedly when implementing the run API of its adapter. These adapters handle agent-specific design while presenting a consistent interface to the IoA framework. We will also ensure this process is well-commented in our open-source repository upon its release.

5. Limited Novel Research Contributions

While IoA emphasizes architecture and engineering, we believe it addresses a fundamental gap in LLM-based MAS by enabling heterogeneous, distributed agent collaboration. This aligns with the ICLR track that we submit to, which focuses on "applications to language and other modalities." Furthermore, our extensive benchmarks and novel features, such as the layered architecture and dynamic team formation, establish IoA's practical value. We argue that “agent” is itself a rather empirical research area, and the effectiveness can sometimes be important, as agent should be “useful”. The presented results underscore IoA’s potential for advancing both empirical research and real-world deployment.