HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location

We thank the reviewer for their valuable feedback and for recognizing our work as an "interesting and important research direction" with a "relatively comprehensive" experimental evaluation. We believe the concerns raised are primarily due to misunderstandings, which we have addressed point-by-point below. We have also included new experimental results on additional hardware to further strengthen our claims of generalizability.

Concern 1: Limited Evaluation Scope and Generalizability

Does HyGen work equally well across different model architectures... and sizes?

No discussion of how the system performs under different hardware configurations...

HyGen is generalizable across diverse models and hardware, as demonstrated in the paper and confirmed with new experiments.

1. Paper's Evidence: Our evaluation already included diverse models (Llama2-7B, Qwen-14B, Yi-34B, Mistral-7B) on two distinct hardware platforms (4x A100 & 4x A40 GPUs), as detailed in Sec. 5.1 & 5.4. The consistent results in our figures validate HyGen's robustness.

2. New Results: We conducted new tests on an NVIDIA A5000 GPU (24 GB VRAM) with a Sheared-LLaMA-2.7B model. The results below confirm HyGen achieves significant throughput gains (up to 2.18× offline and 1.30× total), reinforcing its adaptability.

Concern 2: Overhead and Complexity Analysis

The paper introduces a latency predictor and SLO-aware profiler but doesn't appear to quantify their computational overhead.

...how long does it take for the predictor to become accurate? What happens during this warm-up period?

HyGen has a low-cost, one-time bootstrapping and negligible runtime overhead.

1. Bootstrapping: There is no "warm-up" period. The process is a one-time, pre-deployment step that is extremely lightweight, requiring only ~15ms on a CPU for over 80,000 samples (Line 299).
2. Runtime Overhead: We measured prediction overhead just ~80μs per scheduling iteration on a CPU, three orders of magnitude smaller than a typical chunked batch step. This overhead can be fully hidden by asynchronous scheduling (e.g., in recent of vLLM/SGLang versions), where scheduler logic runs on the CPU concurrently with GPU execution. We will clarify this in the final version. We included algorithm complexity analysis to analyze its negligible overhead in Appendix A.4 too.

No discussion of deployment complexity or operational overhead. How difficult is it to tune the system parameters in practice?

Missing analysis of how the system handles model updates or version changes, which are common in production environments

HyGen is designed for simple deployment and minimal operational overhead.

1. Automatic Tuning & Trivial Updates: Deployment is straightforward. The key parameter (latency budget) is set automatically by a one-time profiler (Sec. 4.2), requiring no manual tuning. For model updates, this lightweight profiling is simply re-run. Training on 80k+ samples takes only ~15 ms on a CPU (Line 299), making adaptation cost trivial.

2. Zero-Overhead Runtime Adaptation: HyGen adapts to workload shifts instantly. The switch from offline to online processing completes "within a single decoding step" (Lines 628–632), due to our asynchronous design. Fig. 14 demonstrates the system's robustness to prediction errors and workload variations, maintaining SLOs without reconfiguration.

Concern 3: Workload Assumptions and Limitations

The system appears to assume predictable workload patterns. How does it handle sudden spikes or highly variable request patterns?

HyGen is expressly designed for unpredictable, bursty workloads.

1. We evaluate our system using two public, bursty production traces: the Microsoft Azure trace (Patel et al.; Sec. 3.2, Fig. 1) and the Kimi Mooncake trace (App. B, Fig. 13 & 16).
2. Fig. 8 shows HyGen dynamically throttling offline throughput in direct response to unpredictable online bursts.
3. The preemption mechanism is built precisely for this purpose (Sec. 4, App. D).

No discussion of handling failures or degraded performance scenarios. What happens when predictions are wrong?

HyGen is inherently robust to prediction errors and system degradation via its closed-loop design.

1. Prediction Errors: The system is not brittle. The profiler establishes a conservative latency budget (Alg. 1, Lines 15–16). Fig. 14 shows that even with predictor error (MAPE) >20%, HyGen maintains high throughput while strictly meeting the P99 SLO.
2. System Degradation: For issues like thermal throttling, HyGen adapts automatically without any tuning. If system performance degrades, batch latencies increase, automatically reducing the available latency budget for the next scheduling iteration and ensuring SLOs are always met.

Concern 4: Comparison Methodology Concerns

Lack baseline fairness. Were the baselines properly optimized?

Missing comparison with other state-of-the-art hybrid serving systems beyond "online and hybrid serving baselines"

Our baselines are SOTA and meticulously designed for a rigorous and fair evaluation. While other hybrid systems (e.g., MuxServe, Punica) solve orthogonal problems, our comparison focuses on single-model, online/offline co-location:

1. SOTA Foundation: We chose Sarathi (OSDI '24) as our foundation. Sarathi's chunked prefill is used to support numerous recent LLM advancements (e.g., EMNLP '24 Memorize Step by Step, ASPLOS '25 POD-Attention, SwiftKV, PrefillOnly), making it a widely-used SOTA system.
2. Optimized "Ceiling" Baseline: We compared against a fully-optimized pure-offline system (Sarathi-offline) representing max throughput (Fig. 4). HyGen reaches 84.3% of this theoretical ceiling while serving an online workload under strict SLOs.
3. Fair Isolation Baselines: To ensure a fair, apples-to-apples comparison, we enhanced Sarathi with our own online-first preemption mechanism, thus isolating the benefit of our SLO-aware scheduler. To ablate our dynamic scheduling algorithm, we created HyGen* that uses a static offline rate. The large performance gap between HyGen and HyGen* (Fig. 4 & 9) cleanly quantifies the gains from our dynamic, SLO-aware approach alone.

Specific Questions

How did the authors obtain the request rate data for Microsoft Azure's LLM service?

As cited in Line 106, this trace is publicly available (Patel et al., ISCA 2024). Our use of this trace is detailed in Sec. 5.1 (Line 222).

The paper's organizational structure and presentation need significant optimization... the explanation of the proposed framework and Algorithm 1 in Section 4 is overly vague... content from Appendix A.1 and A.2 should be moved into the main body.

Thank you for the feedback. We agree Section 4 can be clearer. As moving the full pseudocode would exceed the page limit, in the final version, we will enhance Sec. 4 by integrating the key architectural details from Appendix A.1. We will add a description of our asynchronous, dual-queue scheduling workflow to the main text, clarifying how Alg. 1 is invoked and providing necessary system-level context.

The architectural diagram (Figure 2) requires more detailed explanations. For instance, the diagram mentions "check violation" under "Section 4.2," but there is no corresponding explanation.

The "check violation" block refers to our core SLO compliance logic. The system predicts a potential batch's latency and checks if it violates the pre-set budget. This is the central idea of Sec. 4.2. We will add this explanation to the main text and Fig. 2's caption.

What are the specific novelties or differences of the "Prefix Sharing Maximization Strategy"...?

We clarify that our novelty is not the Prefix Sharing Maximization (PSM) concept itself, which we cited as prior work (Line 189). Our contributions are the SLO-aware application and extension of PSM, which again highlights the effectiveness of our solution to augment existing deployments:

1. Our system integrates PSM to maximize throughput for offline tasks using only the residual GPU capacity available after satisfying strict online SLOs.
2. Our extended policy (Lines 207-213) introduces a fairness mechanism to mitigate request starvation, a critical issue for production deployments that is unaddressed by naive PSM.

Please provide a specific and clear explanation for the phrase "at a specific offline QPS" in Line 240.

This describes our HyGen* baseline. We profile the max fixed, constant offline QPS the system sustains without violating online SLOs. HyGen* serves requests at this static rate, contrasting with the full HyGen system, which dynamically adjusts its offline rate. We will rephrase for clarity.

We sincerely thank the reviewer for their positive assessment and insightful questions. We are encouraged that the reviewer finds our work "well-written", our motivation "clear and well-conveyed," and recognizes the potential impact of our research. Below we address the questions and concerns point-by-point.

Concern 1: Latency Predictor Effectiveness

The key to jointly handling online and offline requests lie in accurately estimating the latency for each request, but there is no experimental evaluation of the effectiveness linear regression model for latency prediction, also whether more advanced modelling can improve its robustness.

Our latency predictor is highly accurate, as proven in our evaluation, and the system is robust even to significant prediction errors.

1. High Accuracy Proven: As evaluated in Fig. 5 (Sec. 5.3), our predictor is highly effective. The results show a mean absolute percentage error (MAPE) of only 1.78% and 1.07% on a mixed production workload.
2. Robustness to Inaccuracy: While more complex models are possible, our system is not brittle. As shown in Fig. 14, HyGen strictly meets P99 SLOs even when we artificially increase the predictor's error to over 20%. This demonstrates that extreme accuracy is not required, making our lightweight linear model a practical and sufficient choice under runtime dynamics.

Concern 2: Modeling of Batch Execution Time

In section 4.2, when calculating the batch execution time, why do you add the quadratic scaling term, when the decode stage shows linear scaling?

The quadratic term is essential for accurately modeling mixed batches containing requests in the prefill stage, which has quadratic complexity. While the decode stage is linear, a batch often contains a mix of requests. As stated in Sec 4.2 (Lines 156-158) and established in prior work [1,2,3], the prefill stage's attention mechanism has quadratic complexity w.r.t. input length. Our model (Eq. 1) includes the quadratic term S²p to correctly account for these prefill requests.

[1] Attention Is All You Need. NeurIPS '17. [2] On The Computational Complexity of Self-Attention. ALT '23. [3] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse. NeurIPS '24.

Concern 3: Novelty of Prefix Sharing Strategy

Is the prefix sharing maximization strategy similar to the one used in the SGLang framework’s radix tree structure for prefix caching? Have you compared using block-level prefix caching, similar to vLLM? It would be interesting to have some ablation studies of using different prefix caching strategies.

Our novelty is not the caching mechanism itself, but its SLO-aware application and fairness-aware extension in a hybrid serving context. Our scheduling policy is orthogonal and complementary to low-level caching architectures like SGLang's RadixAttention or vLLM's Automated Prefix Caching. Our contributions are:

1. SLO-Aware Co-location: We are the first to apply a PSM-style strategy to opportunistically schedule offline requests using only the residual capacity from a primary, SLO-bound online workload.
2. Fairness Extension: We introduce a practical extension (Lines 207-213) that incorporates request freshness to mitigate the starvation problem inherent in naive PSM, making it deployable in production.

We agree that an ablation study on the interplay between our policy and different caching implementations would be a fascinating follow-up.

Concern 4: Clarification on Performance Gains

In the experiments, I am wondering why HyGen can improve over the baselines in pure online and pure offline settings? What are the key designs that achieve that?

To clarify, HyGen's significant throughput gain is achieved in a hybrid setting over a pure online baseline, demonstrating the value of co-location, not by outperforming specialized pure-mode systems. You are correct that a specialized system is optimal for a single task.

1. The Sarathi (pure online) and Sarathi-offline baselines in Fig. 4 represent the performance upper bounds for their respective tasks.
2. HyGen's goal is to productively utilize the idle resources of the pure online system. The reported gain (3.87-5.84×) is achieved by our hybrid system over the Sarathi pure-online baseline, demonstrating the value of our approach.

Concern 5: Scalability and Experiment Design

The work has the goal of improving the serving of LLMs in large clusters, but the scale of the experiments seem quite limited (with only two nodes with 4 GPUs each)

Our experimental scale is appropriate because HyGen is designed as an instance-level scheduler, a standard architecture for large-scale serving, and our results already prove its effectiveness in complex, distributed settings.

1. Scalable by Design (Architecture): HyGen operates as a modular, instance-level scheduler, intended to run on each node of a large cluster. As detailed in Sec. 5.1 (Line 227), a system-level router (e.g., Preble, as cited) distributes requests across these independent instances. This standard, decentralized architecture has no central bottleneck and scales horizontally simply by adding more nodes. Our experiments accurately model the workload and deployment complexity any single instance would face in such a deployment.

2. Proven Effectiveness (Evidence): Our evaluation already provides strong evidence for this scalability. The experiments using Tensor and Pipeline Parallelism (Sec. 5.4, Fig. 9) demonstrate that HyGen effectively manages the complex scheduling logic required in multi-GPU environments. This capability is foundational for and directly applicable to multi-replica production scenarios.

We sincerely thank the reviewer for recognizing our method as novel, our results as strong, and our evaluation as comprehensive. We appreciate the insightful questions and address them below.

Concern 1: Overhead of Workload Shifting

This paper does not consider the overhead of shift between different workloads.

The authors might strengthen their evaluation by evaluating the impact of workload shifting on throughput.

HyGen is designed for minimal overhead during workload shifts, at both the micro (task-switching) and macro (workload-adaptation) levels.

1. Zero Runtime Task-Switching Overhead: The switch from offline to online processing is nearly instantaneous, completing "within a single iteration step" (Lines 628-632). Our scheduling algorithm's low computational complexity, formally analyzed in Appendix A.4, ensures this preemption incurs negligible overhead.

2. Minimal Macro-Level Adaptation Overhead: HyGen adapts to new or changing workloads with extreme efficiency:

   • Inherent Robustness: The system is inherently robust to workload variations. Fig. 14 shows that HyGen's performance remains stable even when predictor accuracy varies, proving its resilience to distributional shifts without needing reconfiguration.
   • Trivial Retraining: For entirely new models or tasks, retraining our lightweight latency predictor is trivial. As stated on Line 299, training on over 80,000 samples takes only ~15ms on a CPU. This makes the one-time cost of adapting to a new workload practically zero.

Concern 2: Applicability to Multi-Replica Deployments

The method is designed to optimize workloads for a single model instance. However, real-world deployments often serve multiple model replicas simultaneously.

HyGen scales seamlessly to multi-model-replica deployments as it functions as a modular, instance-level scheduler managed by a system-level router.

1. Scalable by Design (Architecture): HyGen operates as a modular, instance-level scheduler, intended to run on each node of a large cluster. As detailed in Sec. 5.1 (Line 227), a system-level router (e.g., Preble, as cited) distributes requests across these independent instances. This standard, decentralized architecture has no central bottleneck and scales horizontally simply by adding more nodes. Our experiments accurately model the workload and deployment complexity any single instance would face in such a deployment.

2. Proven Effectiveness (Evidence): Our evaluation already provides strong evidence for this scalability. The experiments using Tensor and Pipeline Parallelism (Sec. 5.4, Fig. 9) demonstrate that HyGen effectively manages the complex scheduling logic required in multi-GPU environments. This capability is foundational for and directly applicable to multi-replica production scenarios.

Concern 3: Typo Correction

Typo: line 52: "while guaranteeing strict SLO compliance and ."

Thank you for catching this. We will correct it in the final version.

Concern 4: Code Availability

I understand that you might consider sharing the code after the paper is accepted. However, since ML sys papers are typically hard to evaluate without checking implementation details, would you be able to share the source code?

We agree on the importance of implementation details and have provided the AC with a private, anonymized link to our development codebase, per NeurIPS policy. We are enthusiastic about contributing our work to the community upon acceptance and believe a public release accompanying a peer-reviewed publication is the best way to ensure the work is presented completely and accurately. The publicized code will be thoroughly documented and well-structured, and we’re actively making efforts towards a public release version. We appreciate your understanding.