Transcending Cost-Quality Tradeoff in Agent Serving via Session-Awareness

Thank you for your detailed review and feedback.

[Weakness 1, Question 1, Limitations 1] Have you evaluated AGSERVE on any non-programming agent workloads to assess generality?

Thank you for your question. We implement four agents, AlfWrold (simulated household jobs / embodied agents), Card Game (card games/ game playing), Knowledge Graph(knowledge graph / Q&A dialogue) , Mind2Web (web browsing / planning) on top of AgServe, spanning across a wide range of agent application and request patterns. As shown in Figure 8, AgServe consistently outperforms baselines across all four domains by pushing the cost-quality frontier. We will clarify this generality and add a forward reference to Appendix D.1 at the beginning of Section 7 in the revision.

[Weakness 2, Question 2] Can you provide false-positive/false-negative curves for the R-Judge, and quantify migration overhead when misclassifications occur?

Thanks for your question. Sure! Here is some data points on the False-Positive/False-Negative curve. We’ll add the curve to our revision. False positive means that the R-Judge mistakenly classifies a satisfactory result as unsatisfactory, while false negative menas that the R-Judge takes an unsatisfactory one in the ground truth as satisfactory. To reduce the early-migration cost of false positive, AgServe adopts the retry strategy and assesses the response again prior to migration, as described in Section 5.2.

We also conduct a study on the cost of early migration in Appendix D.7. An ideal human judge with no overhead and perfect accuracy can further reduce the cost by 14%, yet only improves the performance by 2.4%. However, human-as-a-judge introduce second-level latency in reality and raises the cost significantly.

[Weakness 3, Question 3] What is the worst-case decision latency of the ECE algorithm as agent count and cache size scale?

Thanks for pointing this out. Each SAS runs its ECE algorithm independently. The theoretical algorithm complexity is O(k*n*W). We optimize the large constant W (knapsack size) with a Python dictionary in our implementation. Due to the long context of each session, the number of session cache maintained in the context is limited. As a result, even in worst-case conditions, the observed latency per ECE decision remains under 0.1 milliseconds, which we find negligible compared to typical inference latency. We will clarify this in the final version and include this worst-case estimate.

I appreciate the clarification and new experiments. I have no further comments.

Thanks for your feedback, which helps us improve the quality of our manuscript.

[Weakness 1, Question 1] （1）I am having doubts that the mentioned training of Q-Judge / R-Judge is sufficient to accomplish their designed roles, by only training on Chatbot-Arena for such little time, also the only training on this dataset seems limited, leaving doubt that whether the trained Q-Judge and R-Judge is able to handle all kinds of tasks, or it would require task specific training. （2） I am curious what the Q-Judge / R-Judge are practice? Are they also LLMs? How can you ensure these instances fulfill the designated tasks? E.g., the Q-Judge should select the most cost-efficient model for the session by assessing the task’s difficulty, but how can you be sure it chose the most cost-efficient model? Do you need to train them or provide additional context? Similarly, the R-Judge should evaluate the reasoning quality, how do decide the consistency in evaluation and how do you set the standard / threshold?

Thanks for your insightful question. We would like to address your concerns twofold.

1. Q-Judge and R-Judge are lightweight classifiers built upon BERT and DistilBERT, respectively. Both judges are trained on customized Chatbot-Arena datasets. Q-Judge learns from RouteLLM [35], which has demonstrated robust performance across a wide range of tasks in routing scenarios without task-specific tuning. To minimize unnecessary use of powerful models ("overkill"), Q-Judge employs a novel cost-aware loss that encourages routing to cheaper models when appropriate. R-Judge focuses on evaluating reasoning quality by detecting whether the response addresses the original task, avoids redundant reasoning steps, and maintains logical coherence.
2. We agree that generalization is an important concern. In Section 7.4.2, we present a micro-benchmark evaluating the two judges across four implemented agents. Q-Judge agrees with human preferences in over 70% of cases, and R-Judge achieves 90.5% recall on detecting low-quality reasoning. These results suggest that despite being trained on a single dataset, the judges generalize well in practice. We will clarify this further in the revision.

[Question 2] The author mentioned the customized CUDA kernels in chapter 7, what exactly does the kernels do and why do you need them? There is no explanation of that.

Sorry for the confusion. The customized CUDA kernel is designed to batch tokens from multiple sessions and perform in-place calibration of positional embeddings, as described in Section 4. This operation allows us to better utilize GPU memory and reduce kernel launch overhead in high-throughput settings. We will add a detailed explanation of the kernel’s role and its benefits in the revision.

[Question 3] The ECE eviction policy seems reasonable, but it comes with additionally computation overhead, because it needs to make prediction? I believe LRU has shown robustness in large scale caching. Whether ECE is truly beneficial needs more large scaled validations.

Thanks for your question. We acknowledge the concern about potential overhead. In practice, the ECE policy incurs only about 0.1 milliseconds per eviction decision, which we find negligible compared to overall serving latency of 2 seconds. To achieve this, AgServe maintains lightweight metadata for each session (such as prior arrival times and interval) and leverages statistical estimation to predict the expected time of next arrival (ETA). While LRU is very useful in general LLM serving and widely used by many frameworks, it may evict the session cache just before reuse in agent serving, as depicted in Figure 7. To tackle this problem, we introduce estimated-time-of-arrival to agent serving for the first time and proposes ECE. ECE mainly focus on the KV cache of tokens that are not shared by other sessions (i.e. task-specific template and truncatable). The eviction decision of such content is handled entirely within each node. Therefore, the scale of nodes will not reduce ECE’s effectiveness. We will include this clarification and the runtime overhead measurement in the revised manuscript.

[Limitations 1] Currently they keeps on mentioning new concepts, without previous definition or explanation, making the reading difficult. For example, in the beginning of section 7, the author mentioned different agens (AW, CG, KG, M2W), but you have to search through the whole paper and find the definition in the Appendix, or in the ECE Eviction Policy section, it states the formula for calculating the eviction cost, and the P is also not defined. There are other examples like this.

Thank you for pointing this out. We will revise the manuscript to ensure all technical terms and abbreviations are clearly defined when first introduced. Specifically, we will add a forward reference to Appendix D.1 in Section 7 for the agent definitions (AW, CG, KG, M2W). Additionally, we will clarify that P in the eviction cost formula denotes the penalty of recomputation—i.e., the ratio between prefill and decode latency per token. A thorough pass will be conducted to ensure that all such symbols are properly introduced and explained.

[Limitations 2] The strictness of R-Judge theta seems to be an important hyperparamter, as it greatly controls the decision of R-Judge, but how it is obtained and quantified seems unclear.

We appreciate your observation. The strictness parameter $\theta$ in R-Judge controls the pass threshold for response quality. A value of 0 means no response will pass the quality check, while 1 accepts all responses. We train the Q-Judge for strictness of 0.5. Users can adjust this threshold based on their own quality requirements or workload characteristics. We will include an explanation of this hyperparameter and its practical implications in the revision.

[Limitations 3] Model allocation seems to be too simplified, the RS profiles the nunimum number of GPUs required, but in practise, increasing the number of GPUs will lead to faster inference, this trade-off should be considered when allocating memory

Thanks for your insight. While RS profiles the minimum number of GPUs required to fit a model, it does not restrict allocation to the minimum. AgServe supports tensor parallelism, and RS can assign more GPUs than the minimum when available. For example, if one GPU suffices but six GPUs are idle, RS will spawn two SAS instances using 4 and 2 GPUs respectively, allowing faster inference and flexible resource adjustments in the meantime. We will clarify this design in the revision, and will support more types of parallelism (e.g. pipeline parallelism) in the future.

Thanks for your detailed review, which helps us improve our quality of manuscript.

[Weakness 1] The evaluation primarily focuses on four specific agent implementations . While these cover different interaction patterns, the generalizability of AGSERVE's benefits across a wider range of LLM agent applications is not fully explored.

Thanks for your advice. The four agents cover multiple interaction patterns, showing that AgServe improves agent serving efficiency with our proposed techniques based on agent serving behaviors. We’ll clarify AgServe’s generalization to agent serving scenarios in our revision, and expand agent implementation in the future.

[Weakness 2] The system's performance and optimizations might be less effective or require significant re-tuning for other LLM architectures or model families that are not from the same family and do not share the same tokens and dictionary .

Thanks for your question. Generally speaking, LLMs with different dictionaries in the cascade will not affect AgServe’s performance.

1. For generalizabilty, AgServe does not reuse tokenization result when performing service upgrade. Therefore, we expect little performacne divergence between heterogenous models and the current setup.
2. Our profile on Llama-8B shows that tokenization takes only 1~2 milliseconds, while generation takes 2 seconds per round. Therefore, we believe the gain from dictionary reuse is limited.

We‘ll clarify it in Appendix C.

[Weakness 3] The simplification of VM rentals might not fully capture the true cost-efficiency advantages or disadvantages in all deployment scenarios, potentially skewing the cost-quality comparison.

Thanks for your question. AgServe keeps most lifetime of sessions on locally-deployed models, and only upgrades to GPT-4o when necessary. Therefore, lower rental price is beneficial for AgServe. The table below shows the performance of AgServe for Figure 8(a) if rental is 30% or 60% lower. We’ll add these statistics to Appendix D.

[Weakness 4] It's unclear if the "human-selected preferences" for R-Judge fully capture the nuances of agent task success or are prone to subjective biases.

Thanks for your question. To minimize the bias, the annotators were shown the observation and LLM’s response round by round, with no previous knowledge of the session’s final result or the model size. We also utilize the voting mechanism and each label is agreed by three annotators at least. We’ll clarify it in our revision.

[Weakness 5] Some aspects of AGSERVE, like KV cache management and model cascading, could benefit from a more in-depth comparison with the very latest developments in those specific areas, beyond just general LLM serving systems. For instance, the discussion on cache reuse mentions CacheBlend , but a detailed comparative analysis of their respective strengths for agent serving is limited.

Thanks for your advice. As we discussed in Appendix F, CacheBlend is orthogonal to AgServe. CacheBlend selectively updates the KV cache for pre-computed RAG contents, and is applicable to AgServe. We will discuss CacheBlend in the main text in the revised version, if the space permits.

[Question 1] What are the specific heuristics or machine learning models used to predict the Estimated Time of Arrival (ETA) for sessions, which is crucial for the ECE eviction policy, and how robust are these predictions to unexpected agent behaviors or workload shifts?

Thanks for your insight. AgServe predicts ETA in the following manner. First, AgServe maintains the metadata of each session. When a session starts, AgServe initialize the metadata by the agent type. Throughout the session, AgServe updates the metadata with the last several rounds’ round duration, and last arrival time data. AgServe will use the average round duration to calculate the ETA. The ETA grows if the next request has not arrived at the original ETA time, and finally reaches infinite in case of unexpected session interruption. The ETA prediction will not be drastically affected by the unexpected behaviors, since it keeps track of multiple rounds of history data. It is robust to workload shift, since each session is treated independently.

[Question 2] Beyond the satisfactory rate observation, can a more quantitative characterization of the increasing task difficulty for LLMs as context grows be provided, perhaps in terms of specific metrics or a more detailed analysis of failure modes at later stages of agent sessions?

Thanks for your question. Task difficulty is a subjective metric, which is hard to quantify. We discussed different types of quality issues in Section 5.2 (L139), which can be considered as different modes of failures. Taking Alfworld as an example, here is a detailed analysis of agent outcomes for 20 AW cases. There is an increasing trend for invalid action or format in the later stages and smaller models. We’ll add more discussions on failure modes in our revision of Section 2.2.

[Question 3] Given that the paper "disregard[s] the sessions that trigger rule violations" in Sec 7.4.3, how frequently do such violations occur in practice, what are their root causes, and how does AGSERVE handle them to maintain overall system stability and performance?

Thanks for your question. These violations mainly refer to the violation of preset agent-specific rules or action format. This will make agent exit prior to task completion or reaching round limit. Such violation takes place in 6% of sessions. This probability is similar for FS1, FS2 and AgServe. This is mainly due to the LLM’s limited capability in long-context scenarios. In its normal operation, AgServe will choose to upgrade the model to GPT-4o.

Thank you for your postive appraisal of our work!

[Weakness 1] The assumption of ece is uniform/burst, which may be vulnerable to the variability as noted in [1].

Thanks for bringing [1] to our attention. ECE relies on the assumption that the estimated time of arrival (ETA) for the next request in a session is independently predictable. This assumption holds well in practice for agent workloads.

Interestingly, as a contemporaneous work, [1] investigates multi-turn request traces (particularly agent workloads as cited by their [55, 57]) and arrives at a similar conclusion. As shown in their Section 3.3, the probability of issuing the next turn request is predictable given a workload category and time interval, which supports our design. We will explicitly discuss and cite [1] in Section 2.2 of our revision to acknowledge this connection.

[Weakness 2] The low overhead of QMM may not work under accumulated long contexts.

Thanks for your question. The two judges’ context window is 512, limited by their backbone (i.e. BERT, DistilBERT). Q-Judge's input of $t$ and $p$ is usually long enough to saturate the window. As for R-Judge, we feed as much information as possible in $e$ to fill the window up, since longer context facilitates R-Judge to capture the LLM’s repetitive behavior if applicable. As a result, AgServe almost always utilize full context window of both judges in our evaluation. Despite that, we witness no significant overhead, since QMM overlaps with observation generation. We will clarify it in Section 7.4.2 of our revised manuscript.

[Weakness 3, Question 2] Could the authors provide more cascade performance with more levels/non-Llama models, and potentially extended to large-scale serving involving more than two nodes?

Thanks for your question. We will address your concern in three parts.

1. AgServe selects the three-tier cascade due to the capability gap of LLMs and the limited testbed size, justified in Appendix C (L639). We will add reference to it in the main body.
2. Llama structure is adopted by many other LLMs, such as the Qwen family. Therefore, these models benefit the same from SAS's technology just like Llama models. Moreover, AgServe does not take advantage from tokenization reuse, as tokenization only takes negligible time of 1~2 milliseconds compared to the 2 second decoding duration for Llama-8B. Therefore, heterogenous models will not affect AgServe's performance.
3. Since each SAS processes its sessions independently in AgServe, we expect little performance divergence for larger testbeds. We are working on finding a larger testbed, and we expect to evaluate on a larger scale in the revised version.

[Question 1] The authors mentioned the long-input scenarios in sec 7.2. Could the authors elaborate a bit more on how to handle the straggler settings with the four agents?

Thanks for your question. We would like to explain why we claim that RouteLLM can not tackle the long-input scenarios. RouteLLM is tailored for short-input applications (e.g. MMLU[1] and GSM8K[2]), where average input is less than 100 tokens. However, in general, the input for agent serving exceeds thousand tokens due to the long few-shot examples and tool instructions. To tackle this problem, AgServe's QMM selectively picks the information that are necessary for quality assessment. This enables QMM to perform well in the long-input agent serving scenarios. We will clarify it in the revision.

[1] D.Hendrycks, et al. Measuring massive multitask language understanding. ICLR 2020.

[2] K.Cobbe, et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.