Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs

1. How the throughput h is profiled or defined?

We agree with the reviewer that, because $h$ is recursively defined, it is impractical to profile every possible configuration exhaustively. However, in practice, this can be solved by employing a one-time profiling strategy that captures the following components. (This approach is referred to the profiling method used in Vidur MLSys'24.)

• Inference prefilling latency: We profile the latency for a single transformer layer across varying tensor parallelism (TP) degrees, workload types, and GPU types.
• Inference decoding latency: We profile the decoding latency for a single transformer layer under similar variations in TP degrees, workload types, and GPU types.
• Pipeline communication latency: We measure the communication latency between different GPUs across various workload types.

Using these measurements, the per-request latency for any configuration is estimated by combining the TP costs (both communication and computation) of all layers—which may be served by different GPUs and at varying TP degrees—with the pipeline parallelism (PP) communication cost. (Note that our heuristics, as discussed in Section 4.3 and Appendix D, largely reduce the profiling space, e.g., TP is employed only intra-machine.)

Additionally, when estimating throughput, the prefill and decoding phases are treated separately: (1) The prefill phase is compute-bound, and its batched processing capacity is determined by the sum of the individual latencies; (2) the decoding phase is memory-bound, with its batched processing capability defined by a single latency value. This distinction has been validated in several studies (DistServe OSDI'24, Splitwise ISCA'24).

The table demonstrates examples of our cost estimation under long input and short output workload (i.e., workload 1 in Figure 4), the notation (2,4) indicates that the TP degree is 2 and the PP degree is 4. Although the estimations are not perfectly accurate, they are sufficiently reliable (estimation errors within 4%-7%) for selecting the optimal configurations.

We will integrate the details of the one-time profiling into the updated version of our paper.

2. Workload distribution absent from MILP formulation.

Thanks for the reviewers' detailed and in-depth understanding of our paper. Our workload assignment method ensures that every workload type is completely allocated across all configurations, as enforced by the constraint: $\sum_{c \in C} x_{c,w} = 1, \forall\,w$. Although these variables are normalized to sum to 1, they can be scaled by the actual workload counts (e.g., 500 requests for type 1300 requests for type 2) during implementation.

To avoid misunderstadning, we will revise the formulation in the updated version of our paper: "Let $f_w$ be the total number of requests for workload $w$. The time required on $c$ is given by $T_c = \sum_{w=1}^{W} \frac{x_{c,w} \cdot f_w}{y_c \cdot h_{c,w}}$."

3. Integrating more heterogeneous baseline.

We conduct additional experiments comparing our system with Helix (ASPLOS'25). Specifically, we compare our method against Helix under a price budget of $15 per hour on the AzureTrace dataset. While Helix optimizes heterogeneous LLM serving using max-flow and MILP scheduling, our method explicitly considers workload heterogeneity and GPU composition optimization, resulting in better cost efficiency. Experimental results show that our system outperforms Helix by 25–35%.

We will integrate the baseline into our updated paper.

4. Additional Suggestions.

Thanks for the reviewers' detailed review. We will solve the following mistakes and update our paper:

• We will update Figure 3, 4, 11, 12, 13 using "Throughput Per Dollar".
• We will explain that the x sticks represent P5 Latency, P10 Latency, P15 Latency, etc.
• • We will update Figure 4, 12, 13 using "Different Deployment Configurations" instead of "Different GPU types".
  • We will clarify that different bars within Figure 4 represent different deployment methods, and the optimal method varies according to workload, model type, and GPU type.
• We will correct m and M to w and W.
• We will check all the cited papers and update the references to the published ones, e.g., sarathi-serve (OSDI'24), HexGen (ICML'24), WildChat (ICLR'24) etc.

1. Workload spikes and dynamic GPU availability fluctuations.

Online rescheduling to adapt to workload changes and GPU drops is an interesting idea that can easily be integrated into our current solution. We introduce this approach and present some preliminary experimental results.

Solution: Online replanning. To address the reviewer's concern, we implement an online replanning mechanism analogous to the one proposed in DistServe (OSDI'24). Concretely, the system could (1) monitor the real-time composition of incoming workloads (mentioned in Section 3), (2) track GPU resource availability within the cloud environment, and (3) upon detecting a significant shift in workload distribution, (e.g., an increase in the proportion of certain workload types) the scheduling algorithm could be executed again, incorporating recent historical data to produce an updated serving plan.

Replanning results. (1) We test the workload surge in short output requests in AzureTrace. Before surge, the optimal GPU composition is {20%, 65%, 15%} for datacenter, workstation, and consumer GPUs, achieving 26.89 req/s. After workload change, the throughput degrades to 23.7 req/s. In this case, replanning (shifting allocation to {63%, 23%, 14%}) boosts throughput to 29.61 req/s. (2) We also test the case when GPU drop happens (4 H100s down), the throughput falls from 26.89 to 20.80 req/s. In this case, replanning raises throughput to 22.85 req/s. Additionally, the rescheduling and model-weight reloading phases can be completed within 1–2 minutes, which is significantly shorter than the hourly timescale of real-world workload changes.

We will integrate this discussion into Section 6 of our updated draft.

2. The trade-offs between prioritizing cost-efficiency and request latency.

Trade-offs. We acknowledge that there are trade-offs between optimizing cost-efficiency and latency. (1) Prioritizing cost-efficiency typically involves using fewer resources (i.e., lower budgets), which can lead to slightly higher response latencies. (2) In contrast, prioritizing latency often requires utilizing more resources (i.e., incurring higher costs).

Our focus. Despite the trade-offs, our work mainly focuses on the cost-efficiency for two reasons. (1) Some inference tasks do not require extremely low latency; meeting a predefined latency threshold (e.g., reading/speaking speed) is usually sufficient. And more importantly, (2) in resource-limited scenarios, where systems are naturally under-provisioned, emphasizing throughput can also indirectly improve latency by reducing queuing delays. Our experimental results in Figure 6 demonstrate that our method achieved the lowest P99 latency among all compared baselines.

We will integrate the above discussion into Section 6 of the updated draft.

3. Why adding budget constraints to existing LP solvers (e.g., Melange) is challenging?

Existing heterogeneous frameworks require substantial redesign of their scheduling algorithms or significant additional system development to achieve cost optimization comparable to our approach.

Compare with Melange. (1) Melange assigning each workload to a single GPU type, overlooking the impact of parallelism strategies on resource allocation. As shown in Section 3, tuning parallelism strategies is crucial, and incorporating this dimension significantly expands the search space and demands additional system development. (2) Melange assumes unlimited resources, yet in practice, GPU availability and budget constraints are real issues that necessitate evaluating workloads across multiple GPU types to find a globally optimal solution—further expanding the search space and rendering Melange’s approach impractical.

Compare with HexGen and Helix. Both approaches (1) fail to consider workload heterogeneity and the presence of mixed workloads during scheduling, and (2) their scheduling algorithms are designed for fixed heterogeneous cluster configurations, missing opportunities for further performance improvements achievable through GPU composition optimization.

4. Essential References Not Discussed.

QLM (SoCC 2024) focuses on SLO-aware serving and multi-node optimizations by refining request ordering; Sarathi Serve (OSDI 2024) optimizes batching through prefill chunking to mitigate interference between the prefill and decoding stages; and Vidur (MLSys 2024) develops an accurate simulator for deployment tuning.

Our work has a different goal—it is dedicated to achieving heterogeneous, cost-efficient serving in cloud environments. We will integrate the discussion of these references into Section 2 of the updated draft.

1. Specific shortcomings of existing methods.

Existing methods require a heavy redesign of the scheduling algorithms or demand significant additional system development to achieve similar cost optimization.

Compare with HexGen and Helix. Both approaches (1) fail to consider workload heterogeneity and the presence of mixed workloads during scheduling, and (2) their scheduling algorithms are designed for fixed heterogeneous cluster configurations, missing opportunities for GPU composition optimization.

Compare with Melange. Melange does not consider the impact of parallelism strategies or cloud resource constraints on deployment performance. Its method fails to utilize different parallelism strategies or different GPU types to handle various workloads.

2. Adapting to dynamic tasks and preemption uncertainty.

Online scheduling to adapt to workload changes and GPU drops is an interesting idea that can easily be integrated into our current solution:

Online replanning. We implement an online replanning mechanism analogous to the one proposed in DistServe (OSDI'24). Concretely, the system could (1) monitor the real-time composition of incoming workloads, (2) track GPU resource availability within the cloud environment, and (3) upon detecting a significant shift in workload distribution, the scheduling algorithm could be executed again, incorporating recent historical data to produce an updated serving plan.

Experimental Results. (1) We test the workload surge in short output requests in AzureTrace. Before surge, the optimal GPU composition is {20%, 65%, 15%} for datacenter, workstation, and consumer GPUs, achieving 26.89 req/s. After workload change, the throughput degrades to 23.7 req/s. In this case, replanning (shifting allocation to {63%, 23%, 14%}) boosts throughput to 29.61 req/s. (2) We also test the case when GPU drop happens (4 H100s down), the throughput falls from 26.89 to 20.80 req/s. In this case, replanning raises throughput to 22.85 req/s. Additionally, the rescheduling and model-weight reloading phases can be completed within 1–2 minutes, which is significantly shorter than the hourly timescale of real-world workload changes.

3. Vast.ai platform has small GPU availability.

GPU shortages are a well-documented challenge in cloud-based tasks (SkyPilot NSDI'23). This issue is not unique to Vast.ai—similar GPU availability limitations are observed across many widely-used platforms, including FluidStack, DataCrunch, and RunPod. Even major cloud providers, such as Google Cloud, face GPU quota constraints, as noted in Table 4 of Helix (ASPLOS'25), with quotas being limited and varying across regions.

4. Compare other scheduling methods.

Scheduling algorithm comparison. We further compare our method with a heuristic scheduling method by replacing the plan optimization component of MILP with population-based mutation and selection (i.e., genetic algorithm). In a 48-GPU cluster, the heuristic method requires 115 seconds to achieve comparable performance to our MILP formulation, which only requires 30 seconds.

End-to-end system comparison. We compared our method with HexGen in our paper, here, we further compare our system with Helix (ASPLOS'25), which optimizes heterogeneous LLM serving using max-flow and MILP scheduling, under a price budget of $15 per hour on the AzureTrace dataset. Our method explicitly considers workload heterogeneity and GPU composition optimization, resulting in greater cost efficiency.

5. Other factors that may affect the cost.

Network Latency impact. We have incorporated network latency considerations into our MILP formulation. For instance, (1) the GPU-to-GPU communication latency is used to determine the pipeline communication cost, and (2) tensor parallelism with high communication volumes is prioritized to use intra-machine networks.

Energy consumption. Our focus is optimizing the cost efficiency of cloud-based LLM serving; optimizing energy consumption is orthogonal to our paper’s scope. However, we acknowledge that energy consumption is a critical metric and could be considered as a future direction.

Computational complexity. Thank you for mentioning this. We will include a detailed discussion of computational complexity of the MILP formulation in Section 4.3 of the updated draft: “the theoretical worst-case time complexity scales as: O(poly(|C|,W,N)×2^|C|), the polynomial factor accounts for the overhead of processing each node in the search tree”.

6. Scheduling across data centers.

We have already used GPUs across data centers in our heterogeneous experiment setup, since Vast.ai provides different GPUs in different data centers (e.g., A40 and A100 GPUs reside in Australia and New Jersey, US).