We thank you for the insightful and helpful feedback! We would like to address your questions in the response below.

Q1: On the generalization of the length predictor across different datasets and whether a separate predictor is needed for each.

A1: To clarify, in our current experiments, we train a separate predictor for each dataset, which is a fast process (approx. 10 minutes). To further investigate its cross-dataset generalization capability (as suggested by the Reviewer), we conducted a new experiment by training a predictor exclusively on one dataset and evaluating it on a different dataset. Note that these two datasets have different distributions (see Figure 7 in our paper).

The results are summarized in Table 3 shown below. We report Kendall's Tau (↑) , Off-2 Accuracy (↑) and RMSE (↓), where arrows indicate the direction for better performance. Definitions of the metrics are in Appendix A.4 in our paper.

Table 3: generalization study of the length predictor

As shown in Table 3, the predictor experiences a performance degradation when evaluated on an out-of-distribution dataset, which can be attributed to the significant distribution shift between the two datasets (as shown in Figure 7 in our paper). For example, when trained on ShareGPT and tested on LMSYS, Kendall's Tau drops from 0.56 to 0.37, and Off-2 Accuracy decreases from 0.80 to 0.67. However, the predictor still retains significant predictive capability, demonstrating a certain level of generalization. For instance, the Kendall's Tau remains well above random, indicating it can still effectively rank requests by length.

Furthermore, in a real-world production environment, this distribution shift can be effectively managed by periodically retraining the predictor on recent historical data, a standard practice to maintain model performance $1, 2$ . We will add a detailed discussion of this new experiment to the Appendix.

 

Q2.1: On the overall novelty of our work beyond length prediction, given that prior work uses similar methods for homogeneous SLOs.

A2.1: We would like to argue that the novelty of our paper goes well beyond a length-prediction work with minor enhancements. Our main contribution lies in a novel scheduling system designed for a new and challenging problem: serving LLM requests with heterogeneous, multi-metric SLOs (both TTFT and TPOT). To the best of our knowledge, no prior work has holistically addressed this scenario. While the sequence predictor plays an important, supportive role, the core contribution lies in our system-level co-design of the TTFT Guard and, most significantly, the TPOT Guard.

• For the TPOT Guard, we introduce two key mechanisms. First, inspired by CPU scheduling, we designed a credit-based batching system that fairly allocates processing opportunities according to each request's specific TPOT SLO. Furthermore, inspired by load control in cloud computing, we developed a VBS-based admission control mechanism. This is crucial for preventing the cascading SLO violations common in prior systems under high load.
• For the TTFT Guard, we use a Least-Deadline-First (LDF) strategy, which is simple but highly effective for this SLO dimension. Additionally, we implement the Unattainable SLO Rejection strategy to proactively reject requests that will inevitably miss their deadlines under continuous high load scenarios. Also, this could be efficient for mitigating the starvation problem common in scheduling system (See our response to R2, Q2).

Both guards are guided by dedicated analytic models to make informed, real-time decisions. We believe this integrated approach represents a substantial and novel contribution to the field.

Q2.2: On the specific contributions and novelty of our length predictor.

A2.2: Regarding the sequence predictor itself, while we share a similar architecture with prior work like S3, our approach is far from a naive application. We observed that the binning strategies used in previous work were suboptimal for real-world data distributions. To address this, we conducted a comprehensive experimental study (detailed in Appendix A.4 in our paper) to determine the most effective binning strategy, a new contribution that improves prior approaches.

Q2.3: On the source of performance gain over the S3 baseline and whether it's mainly due to the predictor.

A2.3: To clarify, for a fair comparison, our S3 baseline was implemented using our improved predictor. Hence, the performance improvement in our results (Figure 4 in our paper) stems from our overall scheduling system (TTFT/TPOT Guards) over S3's simple Shortest-Job-First strategy for the complex, heterogeneous SLO problem. This is further validated by our ablation study (Figure 6 in our paper), which clearly shows each component of SCORPIO makes a substantial contribution to the final performance.

 

Q3: On why SCORPIO sometimes yields lower goodput at lower request rates.

A3: This is an important trade-off that we also observed and discussed in Section 4.2. A key contributing factor to this performance characteristic is the resource contention between our sequence length predictor and the main LLM server when they are co-located. Our study in Appendix A.5 confirms this, showing that this performance degradation can be mitigated by deploying the predictor on a separate, low-cost GPU.

 

Reference:

$1$ Zhang, Haoran, et al. "Why did the model fail?" attributing model performance changes to distribution shifts. ICML. 2023.

$2$ Chen C, Li R, Hu Y, et al. Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue. ACL. 2024.

Dear Reviewer EdBo,

Thank you for your response. We are glad that we could address most of your concerns and appreciate your thoughtful review. We will incorporate the key discussions and results into our final revised manuscript.

Best Regards,

Paper 13264 Authors

We thank you for the insightful and helpful feedback! We would like to address your questions in the response below.

Q1: On the basis of setting experimental SLOs, suggesting a search for more authoritative guidelines.

A1: We apologize that our rationale for the SLO settings was not sufficiently clear. Our SLOs were, in fact, carefully chosen to be representative of realistic use cases, guided by prior work. Specifically, the TPOT requirements for interactive tasks (Categories 1, 4, 5) were inspired by the performance benchmarks in $1$ (reference $11$ in our paper). Regarding TTFT, while paper $1$ does not explicitly define its values for these cases, we set TTFTs to reflect typical user expectations (e.g., a tight TTFT for an interactive chatbot). Similarly, our summarization task (Category 6) was informed by $2$ (reference $12$ in our paper). The remaining categories (2 and 3) were intentionally designed to introduce diversity in SLO profiles. We will revise the manuscript to explicitly provide this detailed justification, clarifying how our settings were grounded in and inspired by existing literature to ensure the soundness of our experimental design.

 

Q2: On the potential for request starvation under high load and the lack of explicit discussion or mitigation strategies.

A2: We agree that request starvation is a critical aspect, and more discussion is needed. We will clarify this in our revision by discussing the potential for starvation in both the TTFT and TPOT guards and how our proposed mechanisms address them.

• Regarding the TTFT Guard: Our Least-Deadline-First (LDF) reordering inherently prevents starvation. A request with a looser TTFT SLO is not perpetually delayed; as time passes and its deadline approaches, its priority dynamically increases, eventually moving it to the front of the queue to be served before its deadline expires. In cases of continuous high load where even this dynamic prioritization is insufficient, our Unattainable SLO Rejection strategy (Section 3.3, second paragraph) acts as a final backstop, ensuring a bounded wait by proactively rejecting requests that will inevitably miss their deadlines rather than allowing them to starve. Furthermore, as discussed in the paper (Section 3.3, second paragraph), these rejected requests can be handled with alternative approaches common in cloud computing, such as migration to other nodes or elastic scaling $3, 4, 5$ .
• Regarding the TPOT Guard: Starvation is prevented by a two-level mechanism. Our VBS-based Admission Control carefully manages the volume of concurrent requests, working in concert with the Credit-based Batching mechanism. This synergy ensures that all admitted requests are served harmoniously, each adhering to its specific TPOT SLO. While some requests are inevitably rejected under high load, this is a deliberate design choice that aligns with our core focus: maximizing the serving quality and goodput for admitted requests, rather than attempting to serve all requests under limited resources.

In summary, SCORPIO is designed with mechanisms that explicitly prevent starvation through dynamic prioritization and proactive rejection. We believe that further research exploring the nuanced balance between goodput, SLO attainment, and request fairness is a promising and practical direction, which we will leave for our future work.

 

References

$1$ Li Z, Chen Z, Delacourt R, et al. Adaserve: Slo-customized llm serving with fine-grained speculative decoding. arXiv. 2025.

$2$ Zhong Y, Liu S, Chen J, et al. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model servin, OSDI, 2024.

$3$ Suresh Chandra Moharana, Bishwabara Panda, Manoj Kumar Mishra, et al. Load balancing in virtualized environments using virtual machine migration: A comprehensive survey. International Journal of Knowledge-based and Intelligent Engineering Systems, 2021.

$4$ Chen Y, Peng Y, Bao Y, et al. Elastic parameter server load distribution in deep learning clusters. ACM Symposium on Cloud Computing, 2020.

$5$ Gu D, Zhao Y, Zhong Y, et al. Elasticflow: An elastic serverless training platform for distributed deep learning, ASPLOS, 2023.

We thank the reviewer for this insightful suggestion.

Q1: On the framework's applicability to iconic LLM serving scenarios with long inputs or long outputs (e.g., reasoning).

A1: We agree that assessing our framework in "long output" (e.g., reasoning) and "long input" (e.g., document analysis) scenarios is crucial for evaluating its practical utility. To address this, we have conducted two new sets of experiments.

• For the long output scenario, we evaluated our method on the opencoder-reasoning benchmark $1$ . We curated a subset of requests with lengthy generated responses (1K-2K tokens), which are significantly longer than datasets used in our main evaluation (see Figure 7 in our paper).
• For the long input scenario, we used the dataset published in $2$ to simulate long-input tasks, selecting prompts with lengths between 1K and 2K tokens.

The results for SLO adherence are presented below in Table 1 and Table 2. They show that our solution, SCORPIO, consistently outperforms all baselines in both scenarios, maintaining significantly higher SLO adherence, especially as the system load (QPS) increases. For example, at a QPS of 5, Scorpio yields up to 43%-50% higher SLO adherence than baselines in the long output scenario, and 34%-38% higher in the long input scenario. This confirms that our scheduling mechanisms are robust and effective even for these challenging scenarios, which is consistent with the findings in our paper.

Table 1: SLO Adherence vs. QPS (Long Output Scenario)

 

Table 2: SLO Adherence vs. QPS (Long Input Scenario):

We will add the full results (e.g., goodput) and a detailed analysis to the appendix of the final manuscript.

 

Q2: On a potential anonymity issue in the supplementary materials.

A2: We assure the Area Chair that we hold the integrity of the double-blind review process in the highest regard. This was a genuine technical oversight. The name appeared in auto-generated dependency files (.dep) created by the vLLM framework's build system, which unfortunately embeds the absolute file paths of the build environment. We are deeply sorry for this mistake and assure the AC and reviewers that this was an honest error and not an attempt to circumvent the rules.

 

References

$1$ Ahmad W U, Narenthiran S, Majumdar S, et al. Opencodereasoning: Advancing data distillation for competitive coding, arXiv preprint. 2025.

$2$ Cohan A, Dernoncourt F, Kim D S, et al. A discourse-aware attention model for abstractive summarization of long documents. NAACL. 2018.

Dear Reviewer xrDr,

Thank you for your feedback. We appreciate your valuable suggestions throughout the review process and will ensure all discussed improvements are incorporated in our final manuscript.

Best Regards,

Paper 13264 Authors