Weakness

W1: The architectures of the target inference models are described only in text, which makes it challenging to grasp the inference pipeline and model architecture. An overview diagram would significantly aid comprehension of Lines 72–94.

A: Thank you for your great suggestion. Adding an architectural diagram of the MLLM inference pipeline would indeed help readers better understand the processing workflow. We will include this figure in the revised version.

W2: The evaluation focuses only on 7B and 11B models. It would be valuable to assess performance on larger model sizes for a more comprehensive understanding of scalability.

A: Supporting larger models brings our system closer to real-world deployment scenarios and provides a stronger test of its scalability. Serving such models requires substantially more GPU resources. For example, a 70B model typically needs two or even four GPUs for inference. In our decoupled-stage inference design, effectively leveraging ElasticMM’s elastic scheduling for such models necessitates at least two nodes. Therefore, deploying ElasticMM on larger clusters to support larger models is a key direction for our future work.

However, to preliminarily validate the scalability of our system, we conducted a preliminary multi-node experiment. we deployed two instances running the Llama3.2-Vision 11B model on two machines interconnected with 200GB/s InfiniBand. We migrated requests of varying sequence lengths from one instance to another and recorded the downtime during the migration, as shown in the below table.

Table 1. Downtime and overhead of KV cache migration on multi-node

The results show that even in inter-node scenarios, the overhead of KV cache migration remains low. This is not only far less than recomputation, but also lower than decode latency. This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy.

Minor:

• In Figure 4, the letter "s" in “Images” appears to have a smaller font size.
• Section 3.3 (Inference Optimization) may not logically belong under Section 3 (Elastic Multimodal Parallelism). I recommend either renaming Section 3 or moving Section 3.3 to a new standalone section.

A: Thank you for pointing this out. We acknowledge the inconsistency in the font in Figure 4 and will correct it in the revised version. Additionally, we agree that Section 3.3 does not logically align well with the rest of Section 3. To improve the structural clarity of the paper, we will reorganize the content and move it into a separate section in the revision.

Question

Q1: (Figure 1) What datasets are used in subfigures (b) and (c)?

A: The data shown in the figures are sourced from the publicly available ShareGPT-4o dataset. (dataset link) We will make sure to clearly indicate this in the revised version.

Q2: (Figure 2) Why does the image-text model exhibit dynamic request distribution?

A: This observation is based on traces from real-world production workloads. Today, many large model service providers use unified multimodal models to handle both text and multimodal requests. In practical user sessions, users often begin with pure text input and gradually introduce multimodal content such as images or PDFs, frequently followed by additional textual queries or descriptions. As a result, text-only requests tend to be more stable over time, whereas multimodal requests exhibit much greater temporal variability. This variability includes both clear periodic patterns (e.g., day-night cycles) and short-term spikes. Together, these behavior patterns lead to a highly dynamic request distribution over time for vision-language models.

Q3: (Figure 4) What do the dashed lines within the rectangles signify? What does the GPU icon specifically represent? How could existing methods be visualized within this figure for comparison?

A: In Figure 4, the dashed lines within each stage represent relative batch sizes. The decode stage shows more dashed lines than the prefill stage, indicating a larger batch size. This is because decode is primarily memory-bound, and increasing its batch size moderately does not significantly affect latency but improves overall throughput. Each GPU icon represents a single GPU instance. We agree that adding a comparison with existing methods would better highlight system differences. Due to a new policy introduced by the committee, we are currently unable to provide additional figures. However, we will include such a comparison diagram in the revised version of the paper.

Q4: Would it be possible to conduct ablation studies for each method proposed in Sections 3.1 and 3.2?

A: It is difficult to conduct separate, standard ablation studies for the methods proposed in Sections 3.1 and 3.2, because these methods are designed to function as a hierarchical and cooperative scheduling framework, rather than as independent components. After resource adjustment across modality groups via the strategies in Section 3.1, intra-group resources must be further scheduled dynamically using the methods in Section 3.2. These two components are tightly coupled—for instance, intra-group resource scarcity may trigger inter-group migration or preemption of GPU instances. Therefore, the two techniques are executed jointly and influence each other during runtime, making it difficult to perform standard ablation studies by simply "turning off" one of them in isolation.

Weaknesses

W1: Lack of baseline. This system only compares with vLLM and self-implemented vLLM-Decouple. However, in December 2024, a work related to MLLM serving was posted on arXiv (Efficiently Serving Large Multimodal Models Using EPD Disaggregation), which also involves issues such as the separation of different components. I suggest that the author include this work in the baseline to verify the advancement of the system.

A: Thank you for your suggestion. We agree that the original baseline was limited. We have incorporated two new baselines for a more comprehensive evaluation: EPDServe [1], as you suggested, and DistServe (OSDI 2024) [2].

• EPDServe is a recent work that was built on top of vLLM and introduces component separation and elastic scheduling optimizations specifically for multimodal LLMs. To ensure fairness in comparison, we have upgraded its vLLM version to match vLLM version.
• DistServe, another crucial baseline, proposes prefill/decode disaggregation and is known for its high throughput on general LLM workloads. We adapted DistServe to support multimodal LLMs by integrating a vision encoder module.

We evaluated all these systems on Input Latency and Maximum throughput using Llama3.2-Vision 11B model and ShareGPT-4o dataset. The results are illustrated as follows:

Table1. Norm input latency (s/token) with different request rate

Table2. Maximum throughput (req/min) meeting SLO

We can observe that ElasticMM significantly outperforms existing baselines by fundamentally addressing their architectural and scheduling limitations:

• Compared to DistServe, ElasticMM achieves 2.6× lower input latency and 2.3× higher throughput. DistServe adopts a decoupled architecture, but relies on static resource allocation, making it less adaptive to dynamic workloads and thus limiting throughput. In addition, it does not differentiate between text-only and multimodal requests, which further constrains efficiency under mixed-load scenarios.
• Compared to EPDServe, ElasticMM achieves 1.8× lower input latency and 1.4× higher throughput. EPDServe introduces elastic scheduling on top of separated inference stages, but still does not differentiate between request types. Specifically, it mixes text-only and multimodal requests in the same batch for backend LLM inference. For encoder-decoder models such as Llama3.2-Vision, this mixed batching strategy significantly increases inference latency due to computation graph heterogeneity, especially from cross-attention layers.

We will add these results and discussions into the revised paper.

W2: Scalability Validation. The paper's experiments are limited to a single-node environment, potentially constraining insights about performance in real-world large-scale deployments. Although the author stated that multi-machine experiments will be left for future work, as a serving task, I believe a single machine is not sufficient, especially since the author has also involved tech like elastic scaling that clearly require multi-machine validation. Considering that the author may have limitations in machine resources, different model, trace and machines can be used for small-scale experiments.

A: Thank you for pointing this out this. To address this, we conducted preliminary experiments to evaluate the scalability of our system in a multi-node environment below. Under the EMP paradigm, our architecture involves relatively frequent KV cache migrations, making inter-node communication latency a potential bottleneck. To assess this, we deployed two inference instances of the LLaMA-Vision 11B model on two machines connected via 200 GB/s InfiniBand, and migrated requests with varying sequence lengths between them. We measured the downtime during migration and compared it with both recomputation latency and single-step decode latency, as shown as the below table.

Table 3. Downtime and overhead of kv cache migration on multi-node

These results show that even in inter-node scenarios, the overhead of KV cache migration remains low. This is not only far less than recomputation, but also lower than a single decode step. In contrast, the recomputation cost increases significantly with sequence length, reaching up to 64× the migration time. For instance, recomputing the KV cache of a 10k token sequence takes 2.1 seconds, equivalent to stalling 47 decode iterations.

This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy. Extending our system to a larger number of nodes will be a crucial focus of our future work.

W3: Some statements in the paper are not very clear, for example, where does the workload in Fig. 2 come from? Is it an open-source trace or from the company's actual production trace? Please at least indicate the source.

A: The data for this figure is derived from the company's production traces, covering a 24-hour interval. We will supplement the source of this workload in the paper.

Reference：

[1]: Efficiently Serving Large Multimodal Models Using EPD Disaggregation. arxiv link

[2]: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv link

Question/Weakness

Q1: Multi-node scalability. Your experiments are limited to a single 8 × A800 box. Could you either (a) add preliminary multi-node results or (b) provide an analytic model/trace-driven simulation that shows EMP still improves TTFT and throughput when network latency and cross-node KV migration are present?

A: This is a good question regarding the scalability verification of ElasticMM. Firstly, our core scheduling algorithm, EMP, operates at the granularity of individual GPU instances, meaning there is no fundamental difference at the algorithmic level whether it is deployed across multiple nodes. However, the primary challenge with multi-node deployment is the additional latency caused by inter-node network bandwidth.

Therefore, we added a preliminary multi-node experiment: we deployed two instances running the Llama3.2-Vision 11B model on two machines interconnected with 200GB/s InfiniBand. We migrated requests of varying sequence lengths from one instance to another and recorded the downtime during the migration. We then compared this migration time with both recomputation time and single-decode latency.

Table 1. Downtime and overhead of KV cache migration on multi-node

The results show that even in inter-node scenarios, the overhead of KV cache migration remains low. This is not only far less than recomputation, but also lower than decode latency. In contrast, the recomputation cost increases significantly with sequence length, reaching up to 64× the migration time. For instance, recomputing the KV cache of a 10k token sequence takes 2.1 seconds, equivalent to stalling 47 decode iterations. This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy.

Q2: Baseline coverage. Please justify excluding SGLang, DeepSpeed-FastGen, Splitwise, DistServe and Mooncake, or add at least one of them to Table 3. If ElasticMM still shows ≥2 × improvement under identical settings, that would materially strengthen the claim of significance.

A: Thank you for your suggestion. Our original baselines were indeed limited. We have adopted your advice by adding a comparative experiment with DistServe [1]. Additionally, we also include EPDServe [2] (ICML 2025) as another newer baseline for comparison.

• DistServe serves as a crucial baseline because it adopts prefill/decode disaggregation and is recognized for its high throughput in general LLM workloads. To support evaluation on multimodal LLMs, we extended DistServe by adding a vision encoder module.
• EPDServe, a recent work, is built on top of vLLM and uniquely optimizes multimodal LLM inference by introducing component separation and elastic scheduling. To ensure fairness in comparison, we have upgraded its vLLM version to match ours.

We evaluated all systems on Input Latency and Maximum throughput using LLaMA-Vision 11B model and ShareGPT-4o dataset. The results are as follows:

Table 2. Norm input latency (s/token) with increased request rates

Table 3. Maximun throughput (req/min) meeting SLO scales

ElasticMM still outperforms the new baselines by addressing their architectural and scheduling limitations:

• Compared to DistServe, ElasticMM achieves 2.6× lower input latency and 2.3× higher throughput. DistServe adopts a decoupled architecture, but relies on static resource allocation, making it less adaptive to dynamic workloads and thus limiting throughput. Furthermore, it does not differentiate between text-only and multimodal requests, which further constrains efficiency under mixed-load scenarios.
• Compared to EPDServe, ElasticMM achieves 1.8× lower input latency and 1.4× higher throughput. EPDServe introduces elastic scheduling on top of separated inference stages, but still does not differentiate between request types. Specifically, it mixes text-only and multimodal requests in the same batch for backend LLM inference. For encoder-decoder models such as LLaMA-Vision, this mixed batching strategy significantly increases inference latency due to computation graph heterogeneity, especially from cross-attention layers.

These results clearly demonstrate the superiority of our proposed method. We will include these results and the corresponding discussion in the revised paper.

Reference：

[1]: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv link

[2]: Efficiently Serving Large Multimodal Models Using EPD Disaggregation. arxiv link

Weaknesses

W1: As the authors discussed in the Limitations section, the proposed method is only implemented and tested in a single-node environment. While multi-node model serving is not studied, the contribution of this work is non-trival. It would be great if how the proposed method could be extended to work in a multi-node model serving environment could be briefly discussed in a short future work section.

A: Thank you for the suggestion. We supplemented a preliminary experiment to demonstrate the capability of our proposed method in multi-node environment. We deployed two instances of the Llama3.2-Vision 11B model on two machines connected via 200 GB/s InfiniBand, and migrated requests with varying sequence lengths between them. We measured the downtime during migration and compared it with both recomputation latency and single-step decode latency, as shown in Table 1.

Table 1. Downtime and overhead of migration on multi-node

These results show that even in inter-node scenarios, the overhead of KV cache migration remains low. It is not only significantly less than recomputation, but also lower than a single decode step. This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy.

As suggested, we will further add a discussion on multi-node scalability in the future work section. Specifically, we would like to briefly discuss the challenges and potential solutions related to extending our system to multi-node environments. Under our EMP paradigm, the architecture involves relatively frequent KV cache migrations, which may make inter-node network latency a potential bottleneck. To mitigate this, we plan to use high-throughput communication technologies such as InfiniBand to maximize the benefits of elastic scheduling. Additionally, the scheduler can incorporate placement constraints to reduce latency. For example, by co-locating GPU instances handling the same inference stage on the same node. Furthermore, techniques such as asynchronous KV cache migration can help minimize the impact of migration on the inference process.

Quesiton

Q1: In Figure 2, the average time per output token is shown. For the proposed approach, it increases at a very low rate as request rate increases. While for vllm, the average time per output token increases sharply. I am wondering what is the reason for the sharp increase of average time per output token when vllm is used?

A: The significant difference in performance between vLLM and ElasticMM stems from their distinct architectural designs and scheduling strategies. Specifically,

1. vLLM adopts a tightly coupled architecture, where the inference pipeline stages—encode, prefill, and decode—are executed serially on the same hardware. This design leads to contention for compute resources and, more critically, causes the time-consuming encode stage to block subsequent inference steps. As a result of its coupled design, output latency increases sharply.

2. In contrast, ElasticMM employs elastic partitioned scheduling, which dynamically adjusts resource allocation between the prefill and decode stages. When the decode stage becomes resource-constrained due to large batch sizes, ElasticMM reallocates additional instances to this stage. Moreover, because different stages are isolated in execution, the decode stage is not interfered with by others. Consequently, as the request rate increases, output latency remains consistently low.

Q2: If model parallel is needed in order to serve the model, will things change? For example, if a very large vision encoder is used and a GPU only holds the vision encoder (pipeline parallel rank 0 only holds the vision encoder). Will the performance gain of the proposed method become less obvious?

A: As mentioned in your example, where the vision encoder is served on a dedicated GPU (or multiple GPUs), the performance benefit of our method remains unaffected. In our system, we also isolate the encode stage onto separate GPU instances, even when it is relatively lightweight. This is because the encode stage tends to be computationally intensive, and isolation helps prevent blocking the subsequent inference stages. This is a key optimizations proposed in our work. As shown in Figure 9, the ablation study confirms that this design significantly reduces input latency.

However, larger models do introduce additional complexity to our elastic scheduling algorithm. When an LLM backend needs to be deployed across multiple GPUs using TP or other parallelization methods, it introduces some additional constraints. For example, we only be able to use an even number of GPUs to execute a inference stage. Furthermore, larger models often require multi-node deployment, which introduces new system-level challenges. Maintaining the performance gains of our elastic scheduling under large models and large-scale GPU clusters remains an important direction for our future work.