Q1

It refers to the $q$-th parameter, i.e., $w_q$ denotes the $q$-th element of $\mathbf{W}$ after it is flattened.

Q2

In Appendix A.2, we implement MLGPTQ using Cholesky decomposition to increase speed and computational stability, similar to GPTQ's implementation (https://github.com/AutoGPTQ/AutoGPTQ/blob/v0.7.0/auto_gptq/quantization/gptq.py#L117). To address the comment, we record the condition numbers of the Hessian matrices for different layers, providing the mean and std-dev below. Results show that condition numbers are consistently within $10^3$, which is a reasonable value range in practice.

Q3

Our work centers on GPTQ due to its widespread use, but our solution can also adapt to AWQ. We presented the differences between MLAWQ and $AWQ_{tweaked}$ in Alg A of the one-page PDF (similar to Alg 1 of our manuscript). As introduced in Section 3.1, most quantization methods follow the Forward-Aggregate Info-Modify Weight-Quant paradigm. In essence, $AWQ_{tweaked}$ smooths outliers by multiplying weights with a smoothing factor, best_s, to minimize per-channel quant error:

• In Forward, the input is multiplied by the weights to create an unquantized monitor, guiding min error quantization (line 1).
• ​In Aggregate Info, the average of all samples and weights is calculated for each channel (lines 2&3) to determine smoothing factor s (line 7). $W$ and $X$ are smoothed to remove outliers (lines 8&9). Then, smoothed $W$ is pseudo-quantized (quantize-then-dequantize to simulate round loss) and compared to the unquantized monitor for quantization error (line 10). This process iterates over various ratios (line 6), selecting the factor with the smallest error as best_s (line 11). Then, this best_s is used to Modify the weight (line 12), followed by the Quant (line 13) process using the modified weight.

The drawbacks discussed in lines 146-154 of Section 3.1 also exist for $AWQ_{tweaked}$ in multi-task quantization:

• Forward (line 1): It can't pass LoRA adapters during activation distribution simulation, causing quantization bias during inference.
• Aggregate Info: It uses $X_{mean}=X.mean(0)$, a naive mixed average of multi-tasks' info. Since each task affects each channel differently, simply averaging blurs distributions, ignoring individual effects.

As explained in lines 155-172 of Section 3.1 and shown in Fig 2, MLGPTQ mainly improves the first three steps to tackle GPTQ's issues in multi-task scenarios. Similarly, we can fix AWQ's issues to create a better multi-task quantization algorithm MLAWQ:

• Forward: MLAWQ loads corresponding LoRA adapter for each task to participate in forward propagation, accurately simulating real activation distribution (line 1).
• Aggregate Info: Instead of mixing and averaging features of each column across all tasks to compute $s$, MLAWQ computes the average for each task separately to get $s_i$ (line 3&7). Then it calculates quantization error for each column rather than the entire matrix (line 10). If the $i$-th task results in the smallest quantization error for the $j$-th column, it sets best_s$[j]=s_{i}[j]$ (line 11). This approach allows optimal error minimization, showing each task's individual effect on different channels, enhancing Aggregate Info (lines 3&6-11), and improving the Modify Weight (line 12) and Quant (line 13) processes.

In summary, our work identifies common drawbacks of current single-task quantization methods in multi-task scenarios. By addressing these issues, we can develop more precise multi-task quantization algorithms. Due to time constraints, we couldn't complete the coding and experiments for MLAWQ during rebuttal. However, since the backbone algorithm choice is orthogonal to our work, and GPTQ is popular in LLM, we believe it doesn't diminish our work's significance.

Q4

As discussed in Section 3.3 and Global Responses, our multi-task scheduling strategy involves two key techniques: prediction-based SRTF and task grouping. Ablation studies in Figure 9 show that prediction-based SRTF and task grouping increase SLO Attainment by 2.27x and 1.16x, respectively. To address the comment, in Fig C of the PDF, we conducted two more experiments:

• We considered a variant of LoRA-Inlaid, enabling multi-task scheduling strategy while disabling multi-task quantization (i.e., served model not quantized), denoted as "Ours (w/o quant)." Results show "Ours (w/o quant)" increases SLO attainment by 16% compared to S-LoRA, showing the power of our scheduling strategy.
• We considered a variant of LoRA-Inlaid without output length prediction (instead, using averaged output length of all requests in the same task for sorting), denoted as "Ours (w/o prediction)." Results indicate 18% and 27% efficiency improvements ("Ours" vs. "Ours (w/o prediction)") for 7B and 13B models, respectively.

Q5

As discussed in Section 5 and Appendix F, our work does not consider task fairness currently and we leave it as a future work. When tasks have different priorities, it becomes a case of weighted fairness. A potential solution is using Weighted Fair Queueing (WFQ) [a]. For example, compute a weighted combination of input and output tokens for each task and scale it by the task's priority factor. This information can be seen as the service amount each task receives. Then, we can integrate WFQ to schedule different tasks.

[a] Parekh and Robert. A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single-Node Case.

Editorial Comments

Undefined Acronyms

There are three undefined acronyms in our abstract:

• LoRA, short for Low-Rank Adaptation, is one of the most widely used parameter-efficient fine-tuning techniques for LLMs.
• GPTQ and AWQ are state-of-the-art quantization algorithms for LLMs. GPTQ (Frantar et al., ICLR 2023) merges the name of GPT model family with "Q"uantization. AWQ (Lin et al., MLSys 2024) stands for Activation-aware Weight Quantization.

How is LoRA-Inlaid related to MLGPTQ?

LoRA-Inlaid consists of two major techniques: the multi-task quantization algorithm and the multi-task scheduling algorithm. We term the multi-task quantization algorithm as MLGPTQ (Multi-LoRA GPTQ). MLGPTQ supports joint quantization of multiple tasks and allows the quantized model to be shared across tasks. It also supports incremental quantization, facilitating dynamic task addition without impacting performance. We'll add the name of our MLGPTQ method to the abstract for clarity.

References

Thanks for pointing out the issue. We will update the references in our manuscript accordingly.

Outliers and Variability

Explanation for the Outliers in Fig 3

The outliers are due to the long-tail property of sequence lengths, where few sequences are significantly longer than others. This property is common in NLP datasets (e.g., see section 2 of this article: https://www.harmdevries.com/post/context-length/).

Scale of the y-axis in Fig 3

Thanks for the suggestion. We used the same y-axis value range in Fig 3 to clarify differences in sequence length distributions across tasks. For instance, it's clear that the table summary task has shorter output lengths than the code generation task. Thus, we are afraid that using a logarithmic scale or breaking the y-axis into two parts may not achieve clarity as expected.

To address the concern, we have redrawn Fig 3 as suggested, shown in Fig B of the PDF. We will update our manuscript if the reviewer prefers the new figure.

Variability in Fig 4

Thanks again for the suggestion. We present the mean and std-dev for each strategy in Fig 4 below. As discussed in Section 3.3, FIFO (S-LoRA) and Skip-join MLFQ (FastServe) lack consideration of tasks scheduled in each step, leading to significant variability. In contrast, our approach shows a smaller std-dev, so our strategy is more suitable for multi-task scheduling.

Malicious Task Detection

One of the most typical use cases of LoRA-Inlaid is the personalization of LLMs, where clients can upload their data to create personalized LoRA adapters using the same base model (or directly upload their self-tuned LoRA adapters). The server is responsible for serving requests from all these clients using the proposed LoRA-Inlaid system. Fortunately, these LoRA adapters are independently manufactured, so we can apply malicious detection to them individually. For instance, the server can prepare a rich set of evaluations to assess the security risks of each LoRA adapter, including violence, discrimination, unlawful responses, etc. If any LoRA adapters fail to pass the evaluation, the server can reject serving them.

Fairness Considerations

To measure fairness among tasks, we can compute a weighted combination of numbers of input and output tokens for each task. This is because the prefilling and decoding phases in LLM inference have different workload characteristics [a] (also why these tokens differ in online service pricing). Then, we can borrow the idea of Weighted Fair Queueing (WFQ) [b] for scheduling different tasks.

[a] Hu et al. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads.

[b] Parekh and Robert. A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single-Node Case.

Experimental Scope/Generalizability of Results

We evaluated our work's effectiveness with 9 datasets from 4 task types, as detailed in Appendix D of our manuscript. We believe these datasets are with significant diversity:

• We considered six languages (French, Czech, Indonesian, Vietnamese, Danish, and Swedish) for the machine translation task, covering various language families.
• Beyond texts, we also considered two other kinds of inputs (tables and codes) to evaluate our work over diverse tasks.

To address reviewer concerns, we added three datasets with distinct tasks to enhance experimental diversity: grade school math problems (GSM8K), medical QA (Medical_MMLU), and anomaly detection (malicious-600k). As shown in Table A of the one-page PDF, our work consistently outperforms baselines across all tasks. Thus, we believe our work is effective and robust across a wide range of tasks.

GSM8K: https://huggingface.co/datasets/openai/gsm8k

Medical_MMLU: https://huggingface.co/datasets/medalpaca/medical_meadow_mmmlu

malicious-600k: https://huggingface.co/datasets/bgspaditya/malicious-600k

Scalability and Real-Time Performance

In Table 2 of our manuscript, we evaluated the scalability of our work under different numbers of tasks. The results show a small decrease in LoRA-Inlaid throughput (less than 10%) when tasks increase from 2 to 100, even under three request rate levels. To address the comment, we further increase the number of tasks to 1000, and the throughputs under request rates of 5, 10, 20 are 3.42, 4.02, and 4.22 reqs/s, respectively, which are even better than that of S-LoRA serving only 2 tasks. Thus, we believe LoRA-Inlaid has a sound scalability to support both small and large scale workloads in real-world scenarios.

Details of the Incremental Quantization Process

Please refer to the Global Responses.

W2 & Q4 (first half)

Since divergent metrics (SacreBLEU and ROUGE-1) are used for different tasks, Fig 5 shows the relative performance to align the y-axis. We have presented the detailed results in Table A of the one-page PDF, showing that MLGPTQ consistently outperforms the baselines.

Q1

Thanks for pointing out the issue. The std-dev is also provided in Table A. RTN has a lower std-dev than the other quantization approaches. This is because RTN is deterministic while the other approaches introduce randomness as we shuffle the calibration sets for every time of quantization. The randomness in RTN (as well as Unquantized) only comes from the non-deterministic outputs (the LLM may produce different outputs for the same input to enhance creativity), however, such randomness hardly affects the evaluation.

Q2

Fig 5 uses different value ranges for the y-axes of 3-bit and 4-bit quantization, leading to misunderstanding. Table A shows 3-bit quantization is consistently worse than 4-bit quantization.

Q3

In Fig 6, GPTQ is the baseline that quantizes the model individually for each task, so the quantized model of GPTQ cannot be shared among different tasks. In contrast, MLGPTQ quantizes the model jointly for all tasks, ensuring the quantized model is shareable. Hence, it is reasonable that GPTQ is better on some metrics. We considered GPTQ as a reference since it fulfills the two key factors (discussed in lines 283-286 of Section 4.2), aiming to better anatomize the effectiveness.

Due to the space constraint, we only provided the results on 3 datasets in Fig 6. To address the comment, we have presented the results on the other 3 datasets of the translation task in Fig D of the one-page PDF (the other tasks are not considered since metrics like G_BLEU, S_BLEU, and NIST_MT do not apply). The results are consistent with Fig 6 --- both MLGPTQ and GPTQ, which fulfill the two factors, outperform the other approaches in almost all metrics. Moreover, since MLGPTQ produces a shareable quantized model while GPTQ cannot, the results verify that MLGPTQ is suitable for multi-task quantization.

Q4 (second half)

Although the model is quantized, the computation during inference is still executed in half-precision. Before the computation of each layer, the corresponding model weights are temporarily dequantized (e.g., MatMul4Bitin bitsandbytes: https://github.com/bitsandbytes-foundation/bitsandbytes/blob/0.43.0/bitsandbytes/autograd/_functions.py#L516). Thus, mmodel quantization does not decrease computational overhead but introduces a minor overhead of dequantization.

Besides, S-LoRA does not support deploying quantized models since existing quantization methods do not fit multi-task scenarios, as discussed in Section 3.1 of our manuscript, so we could only use half-precision for S-LoRA. (LoRA-Inlaid is the first system that supports deploying quantized models for multi-task serving.) Thus, we believe the system comparison in our work is fair.

W1 & Q5

Below we analyze the individual impact of each component.

Multi-task Quantization

As discussed in the response to Q4 (second half), moodel quantization does not decrease computational overhead. However, it reduces the memory consumption of storing model weights so that we can preserve more memory for KVCache to improve overall efficiency.

To assess the impact of multi-task quantization, we considered a variant of LoRA-Inlaid, which disables quantization (i.e., the served model is not quantized), denoted as "Ours (w/o quant)" in the right of Fig C of the one-page PDF. The results show that multi-task quantization brings 39% improvement ("Ours" vs. "Ours (w/o quant)") when serving the 7B model, and disabling it leads to OOM when serving the 13B model.

The time cost of quantization is discussed in the analysis of incremental quantization below. Note that we can quantize the model before serving, so it does not affect the performance of serving.

Incremental Quantization

The incremental quantization aims to support dynamic task addition without halting the serving. As introduced in our Global Responses, there are two steps in quantization, and we avoid redundant computation in the first step of incremental quantization. Meanwhile, a layer-by-layer mechanism is developed to reduce the memory consumption of incremental quantization.

To evaluate its impact, we conducted an experiment where there are 5 tasks in the ongoing service and another 5 tasks need to be added. We measured the time cost of three approaches:

• Full quantization with 10 tasks, which halts the serving.
• An offline variant of our incremental quantization with the 5 new tasks, which halts the serving. However, it does not need to perform the layer-by-layer quantization.
• Our incremental quantization with the 5 new tasks, which works concurrently with the ongoing service.

As shown below, by avoiding the redundant computation, the time cost of the first step can be reduced greatly, accelerating quantization. Moreover, although the layer-by-layer mechanism slows down the quantization by 1.26 times due to the extra IO, it reduces the memory greatly and does not halt the serving.

Output Length Prediction

As introduced in our Global Responses, the output length prediction is done on CPU and overlaps with the LLM inference on GPU. To measure its impact, we experimented with a variant of LoRA-Inlaid without the output length prediction (instead, the averaged output length of each task is used), denoted as "Ours (w/o prediction)" in the left of Fig C of the one-page PDF. The results show that it brings 18% and 27% improvement ("Ours" vs. "Ours (w/o prediction)") for the 7B and 13B models.

W3 & Q6

Please refer to the Global Responses.