AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant

1. I strongly suggest that the authors include at least one additional benchmark. Since the current OSWorld benchmark is relatively new, achieving good results on it may not be fully convincing. Most importantly, it seems that there are many similar benchmarks within the same scope, and incorporating several of them would provide a more comprehensive evaluation.

2. Building on the first weakness, it would be helpful if the authors could conduct experiments that explore the generalizability of the model across different benchmarks.

3. Is the main contribution of this paper training a model that orchestrates different agents, and prior to this, do you also introduce agents in the AgentStore? If so, I believe the most relevant baselines would be models that train to select APIs or tools, which can be analogous to selecting agents (as they function similarly). It would be beneficial to compare your results with these existing methods.

4. Expanding on the 3, I suggest supplementing the experiments using alternative methods to orchestrate the agents within AgentStore. For example, you could compare against RL-based approaches such as [1] GPTSwarm, which orchestrates agents using graphs, or model-based methods like [2] Toolformer, which selects tools from a trained model, and [3] LangChain's ICL-based tool-calling agent.

5. The time or cost analysis of training and inference is missing and would provide valuable insights.

References:

[1] GPTSwarm: Language Agents as Optimizable Graphs." ICML 2024

[2] Toolformer: Language models can teach themselves to use tools." NeurIPS 2024

[3] LangChain: https://python.langchain.com/v0.1/docs/modules/agents/agent_types/tool_calling/

1. The authors claim that their methods "double the performance of previous systems". However, this comparison is not entirely fair, as their approach employs a significantly larger number of agents and incurs substantially higher memory and costs. The paper does not address these additional costs, nor does it include experiments comparing baselines that utilize multiple agents, which would provide a more accurate comparison with the proposed method. I suggest that the authors test multi-agent baselines that use the same group of agents mentioned in the paper.

2. While the authors describe their AgentStore as a "generalist" assistant, the evaluation lacks sufficient breadth. The method could be tested on one additional benchmark such as WebArena or Mind2Web to demonstrate generalizability. Both APPAgent and OSWorld-Multi involve fewer than 100 tasks, which is a relatively small number and could allow for manual tuning of the agents to game the evaluation system.

3. The presentation of the paper lacks rigor. The introduction uses overly fancy language and falls short of the scientific rigor expected, including imprecise terms such as "stunning results." Additionally, in Figure 2, the "AgentPool" is illustrated with agents like Sheet Agent, Slide Agent, Web Agent, etc., which are not clearly defined in the paper. Please provide an explanation of what each of these agents is and how they are built in the main text or appendix, or revise the figure to present a more accurate representation.

4. The related work section is not comprehensive, particularly regarding multi-agent systems. The authors state that previous works "use a fixed number of agents with predefined roles" and that "their agents are usually homogeneous," but this is inaccurate for many studies, such as "Internet of Agents" and "AutoGen". A review of classical papers in multi-agent systems would also reveal that many incorporate heterogeneous agents, a discussion that the authors have entirely overlooked.

The paper may overstate the difficulty of agent selection and potentially understates how much the success depends on designing customized agents for the applications specifically in the benchmark. While MetaAgent with AgentToken is presented as a main contribution, the paper does not conclusively demonstrate its superiority over an ICL baseline.

The reported 49.63% accuracy for ICL with GPT-4o in agent routing seems unusually low. This appears to stem from implementing a simplistic ICL baseline that inefficiently includes entire agent documents in the prompt (as shown in the appendix, though the baseline implementation should be better detailed in the main text). A fair comparison would require:

• Testing ICL with more concise capability descriptions
• Including few-shot examples
• Providing concrete examples demonstrating where ICL fails compared to AgentToken

The system's current implementation raises significant scalability concerns:

• Custom agents and documentation were developed specifically for each app in the OSWorld dataset
• Scaling requires substantial manual effort for each new application:
  • Collecting demonstrations
  • Implementing new agents
  • Writing documentation
• The strong performance appears largely attributable to carefully engineered custom agents rather than a scalable automated approach
• True scalability would require automated agent generation

The paper lacks sufficient detail on the nearly twenty custom different agents (excluding existing ones e.g., Friday) used in the system. Without these details, it is difficult to assess the effectiveness of the approach. Most concerning is the unclear origin of training demonstrations and their potential overlap with OSWorld test tasks. The paper should:

• Specify the source of demonstrations
• Detail measures taken to prevent data leakage between training and test sets
• Discuss how generalization is ensured

There are spelling and grammar errors.

• In Figure 1,”SildeAgent specialize…” should be “SlideAgent specializes…”,
• In Figure 1, “are required to collaborate system-wide” should be “are required to collaborate on system-wide…”
• In the prompts in the Appendix, “Demostation” should be “Demonstration”
• In the prompts in the Appendix, “Templete” should be “Template”

1. Vague Agent Descriptions: The descriptions of agents in the AgentPool are too insufficient to understand what each agent actually are. The only available information seems to be from Table 6, which provides only names for many agents. What distinguishes each agent, such as SheetAgent from SlideAgent? Is it simply their prompts, or are there other differences?
2. Over-engineered, Overfitting to OSWorld: Many agents in Table 6 appear optimized for tasks specific to OSWorld, raising doubts about their general applicability. Evaluating the system against broader benchmarks, like GAIA or SWE-Bench, would strengthen the claim of generalist capabilities.
3. Scalability Concerns: the claimed scalability of this system is unclear. Will there be contributors create specialized agents? And can this platform effectively integrate diverse agents? Table 6 shows that 18 of the 20 agents are authored by the team. So it's unclear if this system design can effectively integrate diverse agents found in the wild.
4. Missing Key Baselines in AgentToken: The study only presents AgentToken training with tunable embedding layers. It would be valuable to compare performance and efficiency when the entire model is tunable to understand the trade-offs better.