AIOS: LLM Agent Operating System

Comparison with running with traditional agent frameworks

We would like to clarify that our evaluation indeed includes comparison with classical frameworks. Specifically, we implemented agents using frameworks, i.e., Autogen, Open-Interpreter and MetaGPT. The "w/o AIOS" baseline directly uses the concurrent execution mechanisms provided by these frameworks themselves. This ensures our comparison is straightforward to reflect performance improvements of using AIOS over the standard concurrent execution patterns used in established agent frameworks.

More ablation studies

We appreciate this important question about the individual contributions of AIOS modules. Each module in AIOS serves a critical role analogous to traditional OS components, specifically adapted for agent environments. We conducted ablation studies examining the impact of removing key modules:

The ablation results show that each module contributes meaningfully to system performance. The Memory Manager improves throughput by reducing redundant conversation history loading. The Tool Manager provides the largest benefit by preventing error-recovery cycles when multiple agents access tools concurrently. The Storage Manager primarily affects persistent data operations.

Details of experimental setting regarding w and w/o AIOS

To ensure fair comparison, we use identical prompts and the same LLM backend infrastructure when executing agents with and without AIOS, with the key difference being the scheduling strategy. In the baseline w/o AIOS, LLM service calls are managed by the native concurrent mechanism provided by the agent frameworks, competing for GPU resources without coordination. In contrast, AIOS employs the scheduling that pre-allocates GPU resources to a limited number of concurrent requests, preventing resource contention and OOM conditions.

Design rationale of context switch

We acknowledge that context switching introduces computational overhead, which is an inherent trade-off in any multi-tasking system. AIOS mitigates this overhead by caching logits at the last layer, allowing generation to resume from cached logits rather than restarting the entire computation. This design choice ensures our optimization is orthogonal to KV cache reuse approaches like vLLM[1] and SGLang[2], enabling AIOS to leverage their advantages and work jointly with existing optimizations. While context switching does introduce latency overhead similar to traditional OS context switches, this cost is justified by preventing system-wide blocking scenarios where long-running requests monopolize resources. Our experimental results demonstrate that despite the context switching overhead, AIOS still achieves superior throughput and latency compared to systems without scheduling, confirming that switching costs are acceptable.

[1] Kwon et al. Efficient memory management for large language model serving with paged attention. SOSP 2023.

[2] Zheng, Lianmin, et al. Sglang: Efficient execution of structured language model programs. NeurIPS 2024.

LLM-nuanced scheduling design

Our original design philosophy prioritized architectural simplicity to demonstrate the core AIOS framework using standard FIFO and Round Robin algorithms. However, AIOS's modular architecture can easily enable it to incorporate more sophisticated LLM-nuanced scheduling modules. To address the compability concerns of LLM-nuanced scheduling in AIOS, we design a latency-aware scheduler that leverages historical query statistics, specifically token usage patterns. When a new request arrives, the scheduler retrieves top-k similar requests based on cosine similarity and averages their token usage to estimate the computational latency for the current request. The scheduler then prioritizes requests with lower expected latency and assign these requests to the more idle LLM core by tracking the number of in-process requests on each LLM core.

We deploy two identical LLM instances (Llama-3.1-8b) in our evaluation, where the latency-aware analyzes incoming requests based on estimated token usage, with a maximum of 250 agents running concurrently in the system. This setup allows the scheduler to distribute load more effectively. Specifically, we use the HumanEval benchmark and sample 50% of the questions as historical data to store in a vector database and use the rest for evaluation and set top-k as top-5.

The latency-aware scheduling achieves 15% additional throughput improvement and 15% latency reduction compared to basic AIOS scheduling. This demonstrates that AIOS can effectively integrate LLM-specific optimizations.

Discussion of Robustness/Compability Issues

To demonstrate AIOS's compatibility across different LLM architectures, we conducted additional experiments with three diverse models: GPT-4o-mini (closed-source API), Gemini-1.5-flash (closed-source API), and Llama-3.1-8b (open-source). Since closed-source LLMs cannot save logits for context switching, we adopt streaming APIs and save intermediate generated texts instead to enable snapshot and restore functionalities.

The results show that AIOS maintains consistent performance improvement with hybrid LLM cores, demonstrating that our design can easily be extended to combine both open-source and closed-source models through standardized API interfaces.

Regarding robustness, AIOS provides several defensive mechanisms including input validation for all agent requests and comprehensive error handling at each processing stage. More discussions about the security mechanisms will be included in the discussions in our revised version. While we acknowledge that sophisticated recovery mechanisms for enhanced fault tolerance are important, we still would like to focus on the overall AIOS architecture in this paper and would consider more complex security features and recovery mechanisms as important future work.

Summarize implementation details in main text

Thank you for your recognition of our extensive implementation details. We will summarize them accordingly in the main text in the revised version.

Overhead balancing between different LLM cores

We conduct further evaluations of our architecture with multiple LLM cores on multiple GPUs to demonstrate AIOS's load balancing capabilities. The experimental setup is as follows: Each GPU hosts one LLM instance (core) running Llama-3.1-8b, with a maximum of 250 agents running concurrently in the system. The rest of the experimental settings remain consistent with our ablation study in Appendix D.1.

To handle overhead balancing across multiple LLM cores, we implement a simple yet effective round-robin strategy that monitors the number of in-process requests for each LLM core and dynamically assigns incoming requests to the least-loaded core. This approach ensures balanced resource utilization and prevents any single core from becoming a bottleneck.

The results demonstrate AIOS's effective overhead balancing across multiple LLM cores. As the number of GPUs increases, AIOS consistently achieves better throughput (2× improvement on 2 GPUs, 2.1× on 4 GPUs) and lower latency (71% reduction on both configurations) compared to baseline approaches. This consistent performance advantage across different scales shows that AIOS's load balancing mechanism successfully distributes computational overhead across multiple LLM cores, thus benefiting efficient execution of agents in complex multi-LLM deployment scenarios.

Discussion of limitations on evaluations of existing benchmarks in continous agent-use scenarios

While we have adapted existing benchmarks to simulate concurrent agent usage, we acknowledge that these benchmarks cannot fully capture long-term or intensive, continuous multi-agent use patterns. We appreciate you highlighting this limitation of current evaluations. However, we think benchmarking long-term agent-use is not trivial and we plan to include this discussion in our revised version to encourage community efforts to developing benchmarks towards continuous agent-use scenarios.

Discussion of real-world deployment and user studies

We agree that real-world user studies are crucial for validating AIOS's effectiveness. We have been developing a deployed version of AIOS and are conducting early-stage real-world testing. However, this deployment remains in its initial phase and requires more extensive evaluation before drawing meaningful conclusions about production performance. We would like to include this in the discussion of our revised version and consider comprehensive production-use evaluation and user studies as our important future work to provide more valuable insights.

Details of experimental setting regarding w and w/o AIOS

We appreciate this important clarification request. The comparison setting is as below: Same prompts: Both AIOS and the baseline w/o AIOS use identical prompts to ensure fair comparison. Same backend for LM inference: Both configurations use the identical LLM backend infrastructure to isolate the impact of scheduling strategy. Different scheduling methods: The "w/o AIOS" baseline relies on OS-level scheduling, where each LLM query can preemptively access GPU resources without coordination. In contrast, AIOS's scheduling mechanism is to pre-allocate GPU resources to a limited number of concurrent LLM queries to prevent recovery overheads from OOM.

Functionalities of other modules in AIOS

We appreciate this important question about the individual contributions of AIOS modules. While our paper focused primarily on scheduler ablation, each module in AIOS serves a critical role analogous to traditional OS components, specifically adapted for agent environments. We conducted additional ablation studies examining the impact of removing key modules. The setting is the same as introduced in ablation study of scheduler in Appendix D.1 of our paper.

Through the ablation studies, we find that memory manager helps improve throughputs due to its reduction of the length conversation histories. Storage manager works similarly as memory manager but it can only affect persistent data handling scenarios. The tool manager contributes as it prevents the error-then-recover process when multiple agents attempt concurrent tool usage. These modules work jointly to improve the execution efficiency of agents on AIOS.

Scheduling effectiveness when scaling up

Thank you for your insightful question, we would like to clarify that even at 100% GPU utilization, scheduling strategy remains critical for the following reasons: GPU Memory Management Overhead: Poor scheduling causes frequent memory allocation/deallocation as agents compete for GPU memory. OOM Prevention: At high agent counts, uncoordinated GPU access frequently triggers OOM conditions, forcing expensive garbage collection and memory reorganization that increases execution time despite high utilization. However, AIOS adopts the pre-allocation strategy to assign GPU access to a limited number of concurrent requests. The key insight is that preventing failures is exponentially more efficient than recovering from them. As agent count increases, AIOS prevents the exponential degradation that occurs when resource contention triggers system-level failures and this explains why the performance gap grows larger rather than smaller at high utilization.

Evaluations using multiple GPUs & LLMs

We conduct further evaluations of our architecture with multiple LLM cores on multiple GPUs and the setup is as below: Each GPU hosts one LLM instance(core) (Llama-3.1-8b), and there can be a maximum of 250 agents running concurrently in the system. The rest of the experimental settings are kept the same as introduced in our ablation study in Appendix D.1. To handle load balancing across multiple LLM cores, we adopt a simple round-robin strategy that counts the number of in-process requests for each LLM core and assigns incoming requests to the least-loaded LLM core.

The results show that as number of GPU increases, AIOS can also achieve better throughputs and lower latency, which suggests the effectiveness of AIOS in more complex multi-LLM and multi-GPU scenarios.