Thanks for your valuable comments.

Q1. Although the router demonstrates strong classification ability for these domains, it would be worthwhile to explore its performance across more domains, particularly those with potential overlap. In such scenarios, MoE could leverage the combined knowledge of multiple experts, as it is fine-tuned based on a token-level router. A potential consequence is that, when adding a large number of experts to the MoE, fine-tuning may converge faster due to shared knowledge and parameters across domain experts. In contrast, a model-level router may struggle to distinguish between overlapping domain experts and always rely on a single expert, potentially causing overall performance degradation.

A1. We acknowledge that overlapping domains can present challenges for a model-level router. However, as mentioned in Q1 of the general response, our primary contribution lies in addressing the critical challenge of high storage demands when serving multiple LLMs. The model-level router is presented as a simple exploration on automating the switching process. Handling overlapping domains would require extending the current setup to a multi-label classification framework, which we consider a promising direction for future work.

Furthermore, as discussed in L295-306, our method offers a cost-efficient solution for handling diverse user queries by requiring training only the model-level router given a set of well-trained expert models. In contrast, MoE models necessitate training not just the token-level router but also all network parameters. For example, training an MoE using existing Math, Code, and Wikipedia experts requires over 900 GPU days (as reported by Sukhbaatar et al., 2024), whereas our approach only requires training at the hours level, completing within a single GPU day, making it significantly more efficient. This efficiency difference stems from the distinct characteristics of experts in MoE models versus existing well-trained experts. In MoE models, each expert often acts as a generalist across all domains due to the constraints imposed by the load balancing loss (Fedus et al., 2022; Jiang et al., 2024). In contrast, existing well-trained experts typically specialize in specific domains, which align more naturally with our approach.

Q2. This approach requires the construction of a domain classification dataset, which can be inconvenient. For instance, when adding a new domain expert, the router would need to be re-trained on an updated domain dataset.

A2. While our approach requires re-training when introducing a new domain, we believe the associated overhead is manageable. The trainable parameters in our model-level router are significantly fewer compared to those in MoE, as mentioned in Q1.

Q3. While the approach demonstrates effectiveness compared to other quantization methods in ablation studies, the experiment results do not include a comparison with an end-to-end token-level router approach, such as MoE baselines.

A3. As discussed in L295-306, our method fundamentally differs from MoE. MoE employs token-level routing, which dynamically assigns individual tokens in a sequence to different experts within the same model. In contrast, our ME-Switch utilizes model-level switching, selecting the most suitable model to handle an entire user request. This distinction also significantly affects training overhead. As mentioned in Q1, training an MoE with multiple experts demands hundreds of GPU days, making a direct comparison challenging due to the limited computational resources available.

Q4. The comparison only covers up to 4 models, after which the naive implementation encounters out-of-memory (OOM) issues. It is a little subtle to conclude that latency is lower than the naive inference as the number of models is greater than 4.

A4. Our method scales more efficiently than the naive inference method. In the naive method, each $\mathbf{x}\mathbf{W}$ is computed independently during the forward pass, requiring a distinct $\mathbf{W}$ for each user in the batch. As the number of models grows (>=4), this approach results in substantial I/O costs due to loading of large weight matrices. In contrast, our method leverages shared pre-trained model weights $\mathbf{W}$ along with a set of small deltas $\hat{\Delta}$, significantly reducing the inference I/O burden. This is demonstrated in Figure 8 of our initial submission. We have included the discussion in the revised manuscript.

Q5. The discussion of latency mentions that the naive implementation has a slight advantage when the number of models is fewer than 4, as shown in Figure 8, and the author attributes this to the quantization overhead. However, why does the quantization overhead stop being the dominant factor as the number of models increases? What is the trend in the profiling of quantization overhead as the number of models continues to increase? A deeper analysis of this would be valuable.

A5. Thank you for pointing this out. Our explanation in the main text was not entirely precise. Our method is slightly slower than the naive approach when the number of models is fewer than 4 due to the additional computational cost introduced by the second term $\mathbf{x} \hat{\Delta}$ in Eq. (4). This includes the cost of I/O, dequantization, and multiplication. To illustrate this, we profile their latency percentage in Table V. We observe that the dequantization cost is very small across different model numbers. Initially, latency is dominated by I/O operations because LLM decoding is a memory-bound process when the batch size is small (Lin et al., 2024; Liu et al., 2023). However, as the number of models grows, compute-related operations, such as matrix multiplications, begin to dominate the overall latency. We have included these results and discussions in the revised manuscript.

Table V. Decoding latency percentage (%) of different components for $\mathbf{x} \hat{\Delta}$. (Testing)

I appreciate the authors’ detailed response. The updated explanation regarding the cost of I/O, dequantization, and multiplication is helpful for readers to understand how latency changes as the number of models increases and the contribution of each component.

The author emphasis the novelty of the paper is the efficiency improvements compared to traditional MoE methods. Overall, this work has a good vision of the drawbacks of traditional methods' efficiency and their method does show efficiency improvements over traditional token-switch approaches, particularly in reducing storage requirements, training costs, and I/O latency for large language models. However, though the paper provides performance comparisons with MoE models across up to four distinct domains, further discussion would be valuable when comparing at larger scales in a more comprehensive level. I encourage the authors to keep expanding their performance analysis relative to previous methods on more domains and more models in following works.

In conclusion, I will keep my current rating.

Thanks for your valuable comments.

Q1. How does the saliency-based delta quantization technique in this paper differ from other non-magnitude-based saliency quantization techniques like Slim-LLM [E]?

A1. Slim-LLM measures weight salience based on the error induced by removing specific weights, using the same metric as SparseGPT [F] (see Eq. (3) in [F]), which is primarily designed for pruning. In contrast, our method uses reconstruction error (see Eq. (2)) to directly assess the impact of quantization on the model’s output, making it more suitable for quantization tasks. To highlight the advantages of our saliency metric, we replaced it with the metric used in Slim-LLM and presented the results in Table IV. From the results, our saliency metric consistently outperforms Slim-LLM’s metric across various datasets and bitwidths. These results are included in Figure 4 and Table C of the revised manuscript.

Table IV. Performance comparisons with different saliency metrics.

Q2. Model routing is a well-studied problem! Are there other existing approaches that study model routing in the same scenario where the appropriate model is not known? How is your solution different or better than them?

A2. Please refer to Q1 of the general response. We have moved the model-level router section to the appendix to maintain focus on our primary contribution in the main text.

Q3. Related work like GPT-Zip is not cited

A3. Thank you for pointing this out. We have included a citation and discussion of GPT-Zip in the related work section of the revised manuscript.

Q4. On L46, the paper claims that no single model can master all tasks simultaneously. While this is not currently the state of affairs is there formal proof or evidence that this is impossible?

A4. Our statement on L46 is based on practical challenges. For further details, please refer to Q2 of the general response.

Q5. Will the code for this be open source?

A5. Yes, we will release the code upon acceptance. We have also provided the core pseudo codes of our Triton kernel in Q13 of Reviewer 1fKA.

Q6. Are there any insights on why mixed precision quantization outperforms the full precision models in certain settings?

A6. The performance improvements over individual expert models are primarily attributed to our additional training through efficient distillation, as discussed in L393-395. This process improves the models’ task-specific performance by optimizing the quantization step size. Similar phenomena are also observed in many quantization literature [G][H][I].

Q7. Is there an estimate of how often in practice the appropriate model is not known in advance? Is this more common than the case where it is known?

A7. In public-facing applications or open-ended systems, user inputs may vary widely in content and intent, often lacking clear contextual information. This makes it particularly challenging to determine the appropriate model for queries in advance. Although it is difficult to precisely quantify how often this occurs, such scenarios are common in many practical applications.

Reference:

[E] SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models. arXiv 2024.

[F] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023.

[G] Learned step size quantization. In ICLR, 2020.

[H] Learnable Companding Quantization for Accurate Low-bit Neural Networks. CVPR 2021.

[I] Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation. In CVPR 2022.

Thanks for your constructive comments.

Q1. It would be great if both contributions can be evaluated more thoroughly against previous methods. There are a lot of previous works (especially for the router) tackling similar problems, it seems that the authors only compared with the most natural baseline, but not more advanced methods. It would be great if the authors can justify it or provide more results.

A1. For salient-aware delta compression, we have compared our method against various quantization techniques, as shown in Figure 4 and Table C of the initial submission. Additionally, during the rebuttal period, we conducted further evaluations using Slim-LLM’s saliency metric [E], with the results detailed in Q1 of Reviewer jYU9. These results consistently demonstrate that our method outperforms all compared approaches across different tasks and bitwidths.

For further discussion on prior work related to model-level routing, please refer to Q1 of the general response.

Q2. I am a little bit confused by how the router is related to the first contribution -- it would be great if the authors could elaborate more? These two parts look quite orthogonal to me.

A2. Please refer to Q1 of the general response.

Q3. The second contribution could be put better into context.

A3. Thank you for your constructive advice. To better focus the main contribution of our method, we have moved the contents w.r.t. model-level router to the appendix. For additional details, please refer to Q1 of the general response.

Reference:

[E] SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models. arXiv 2024.

Dear Reviewer qVf8

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We have carefully addressed your concerns and provided detailed responses, which we hope have resolved your queries. If you have any additional questions or further concerns, please do not hesitate to let us know.

Best regards,

Authors of #912

Dear Reviewer qVf8,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the discussion period is nearing its end, we wanted to kindly check if our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to clarify further.

Thank you again for your valuable feedback and time.

Best regards,

Authors of #912

def twobit_dequant_bmm_scale_kernel(
    # Pointers to matrices
    a_ptr,
    b_ptr,
    c_ptr,
    scales_ptr,
    # Matrix dimensions
    M,
    N,
    K,
    # The stride variables represent how much to increase the ptr by when moving by 1
    # element in a particular dimension. E.g. `stride_am` is how much to increase `a_ptr`
    # by to get the element one row down (A has M rows).
    stride_am,
    stride_ak,
    stride_bk,
    stride_bn,
    stride_cm,
    stride_cn,
    stride_scales,
    stride_batch_a,
    stride_batch_b,
    stride_batch_c,
    stride_batch_scale,
    # Meta-parameters
    BLOCK_SIZE_M: tl.constexpr,
    BLOCK_SIZE_N: tl.constexpr,
    BLOCK_SIZE_K: tl.constexpr,
    GROUP_SIZE_M: tl.constexpr,
    ACTIVATION: tl.constexpr,
):
    """Kernel for computing the matmul C = A x B.
    A has shape (B, M, K), float
    B has shape (B, K//n_bits, N), int, packed boolean
    C has shape (B, M, N),
    scales is of shape (N) float16
    """
    # -----------------------------------------------------------
    # Map program ids `pid` to the block of C it should compute.
    # This is done in a grouped ordering to promote L2 data reuse.
    # See above `L2 Cache Optimizations` section for details.
    pid = tl.program_id(axis=0)
    pid_batch = tl.program_id(axis=1)

    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
    num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
    num_pid_k = tl.cdiv(K, BLOCK_SIZE_K)

    num_pid_in_group = GROUP_SIZE_M * num_pid_n
    group_id = pid // num_pid_in_group
    first_pid_m = group_id * GROUP_SIZE_M
    group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)

    pid_m = first_pid_m + (pid % group_size_m)
    pid_n = (pid % num_pid_in_group) // group_size_m

    offs_m = (pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)) % M
    offs_n = (pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)) % N

    offs_am = tl.max_contiguous(tl.multiple_of(offs_m, BLOCK_SIZE_M), BLOCK_SIZE_M)
    offs_bn = tl.max_contiguous(tl.multiple_of(offs_n, BLOCK_SIZE_N), BLOCK_SIZE_N)
    offs_k = tl.arange(0, BLOCK_SIZE_K)

    a_ptrs = (
        a_ptr
        + (offs_am[:, None] * stride_am + offs_k[None, :] * stride_ak)
        + pid_batch * stride_batch_a
    )

    # Adapted from GPTQ-Triton (https://github.com/fpgaminer/GPTQ-triton)
    # b_ptrs is set up such that it repeats elements along the K axis n_bits times
    b_ptrs = (
        b_ptr
        + ((offs_k[:, None] // 16) * stride_bk + offs_bn[None, :] * stride_bn)
        + pid_batch * stride_batch_b
    )
    scales_ptrs = scales_ptr + offs_bn * stride_scales + pid_batch * stride_batch_scale

    # (BLOCK_SIZE_K, BLOCK_SIZE_N)
    # shifter is used to extract each bit of each element in the int matrix
    shifter = (offs_k % 16) * 2
    scales = tl.load(scales_ptrs)

    # -----------------------------------------------------------
    # Iterate to compute a block of the C matrix.
    # We accumulate into a `[BLOCK_SIZE_M, BLOCK_SIZE_N]` block
    # of bf32 values for higher accuracy.
    # `accumulator` will be converted back to bf16 after the loop.
    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
    for k in range(0, num_pid_k):
        # Load the next block of A and B, generate a mask by checking the K dimension.
        # If it is out of bounds, set it to 0.
        a = tl.load(a_ptrs)
        # b = tl.load(b_ptrs, mask=offs_k[:, None] < K - k * BLOCK_SIZE_K, other=0)
        b = tl.load(b_ptrs)  # (BLOCK_SIZE_N,)

        # Convert B from int to a.dtype
        # b: (BLOCK_SIZE_K, BLOCK_SIZE_N)
        b = (b >> shifter[:, None]) & 0x3
        b = (b - 2).to(a.dtype)
        b = b * scales[None, :]  # bf16
        # b = b.to(a.dtype)

        # We accumulate along the K dimension.
        accumulator += tl.dot(a, b)
        # Advance the ptrs to the next K block.
        a_ptrs += BLOCK_SIZE_K * stride_ak
        # b_ptrs += BLOCK_SIZE_K * stride_bk
        b_ptrs += (BLOCK_SIZE_K // 16) * stride_bk
    # You can fuse arbitrary activation functions here
    # while the accumulator is still in bf32!
    # if ACTIVATION == "leaky_relu":
    #     accumulator = leaky_relu(accumulator)
    c = accumulator.to(tl.float16)

    # -----------------------------------------------------------
    # Write back the block of the output matrix C with masks.
    offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
    offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
    c_ptrs = (
        c_ptr
        + stride_cm * offs_cm[:, None]
        + stride_cn * offs_cn[None, :]
        + pid_batch * stride_batch_c
    )
    c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
    tl.store(c_ptrs, c, mask=c_mask)

We have included the above pseudo-codes in Section C of the revised manuscript.

Q11. Why is fp16 used throughout the experiment setups rather than bf16, which is supported on A100s (used in the paper) and is far more common in inference settings?

A11. Thank you for pointing this out. We indeed use BF16 in our experiments. This has been clarified and included in the implementation details in Section 5 of the revised manuscript.

Q12. What is the effect of using different backbones for the same task with the proposed approach?

A12. We would like to clarify that we have conducted experiments on the instruction domain using the Mistral-7B, LLaMA-2-13B, and LLaMA-3-8B model families. The results, presented in Tables 1 and B of the initial submission, demonstrate that our method effectively adapts to different backbones while maintaining strong performance across tasks.

Q13. What additional details can be shared about the Triton kernel? What is the performance benefit of its inclusion? How is the implementation structured?

A13. The Triton kernel is designed to accelerate model inference. Without it, $\tilde{\Delta}$ must be loaded from high-bandwidth memory (HBM) into SRAM, dequantized into BF16 format, written back to HBM, and then reloaded for multiplication with $\mathbf{x}$. Our Triton kernel fuses dequantization and multiplication into a single step, reducing intermediate memory operations and eliminating unnecessary data transfers, resulting in faster inference. To provide a clearer understanding, we include the core pseudo-codes of the Triton kernel as follows:

Q4. Results are defined only on a very narrow set of task domains. This makes it quite unclear how the proposed approach generalizes. Domain-specific expert setups in a general setting might involve dozens of models in a large-scale system.

A4. To show how our salient-aware delta compression generalizes, we further include BioMistral-7B as an expert in the medical domain and Saul-7B-Base as an expert in the legal domain. Both models are fine-tuned from Mistral-7B-Instruct-v0.1. We evaluate their performance using subsets of MMLU corresponding to these domains. As shown in Table III, our salient-aware delta compression results in only a minor performance drop, maintaining nearly lossless performance. These results are included in Section 5.1 of the revised manuscript.

Table III. Results on the medical and legal domain for the Mistral-7B family.

Q5. In general, the paper’s writing is redundant, i.e. the core premises, contributions, and prior work are presented multiple times each, with similar levels of depth. The authors could make room to do more experiments by removing redundant components.

A5. Thank you for your feedback. We have simplified the writing of our manuscript, moved the router section to the appendix, and included additional experiments in the main text.

Q6. The approach with which the router is trained generalizes very poorly. A router must be trained based on a collection of domain-specific models for every single setup. A router trained on a specific domain-specific set of multiple choice questions is quite specific to the input tasks.

A6. Please refer to Q1 of the general response. Additionally, while it is true that the router needs to be trained for each new setup, the associated training cost is manageable due to the small number of trainable parameters, allowing it to complete within a single GPU day.

Q7. There is no presentation of an accuracy-memory trade-off across approaches.

A7.. We would like to clarify that Figure 4 in the initial submission presents the accuracy-memory trade-off of different methods, directly addressing this concern. The results demonstrate that our method consistently outperforms other approaches across various bitwidths and tasks.

Q8. Introduction — the authors might consider reframing motivations around MoE as well and listing memory pressure as a primary motivator for LLM task specialization.

A8. Thank you for the suggestion. We would like to highlight that our approach is focused on general-purpose LLMs rather than being tailored to MoE models, addressing the memory challenges associated with serving multiple specialized LLMs.

Q9. L78 the "information loss" that occurs with subsequent methods can be further specified. Is this something that affects downstream accuracy, or does this refer to something else?

A9. The “information loss” mentioned in L78 refers to the quantization error introduced by using per-tensor quantization (Liu et al., 2024a), as described in L76-80. This approach applies a single quantization step size ($s$ in Eq. (1)) for an entire layer, which limits its ability to capture fine-grained variations within the layer. Consequently, this quantization error can lead to a drop in final performance. To improve clarity, we have replaced the term “information loss” with “quantization error” in the revised manuscript.

Q10. Why doesn’t reconstruction loss merely end up choosing values which are closest to values which are fp16 quantized, i.e. representable with low approximation error? More analysis of reconstruction loss would be helpful to prove efficacy.

A10. As described in L208-215 and Eq. (2) of the manuscript, the reconstruction error is determined by both the input $\mathbf{x}\_i$ and the quantization noise $\Delta\_{ij} - \hat{\Delta}\_{ij}$ (i.e., approximation error). Even when the quantization noise is small, the reconstruction error can be significant if the input magnitude is large.

Thanks for your constructive comments.

Q1. The assumption that each LLM specializes in a distinct domain is a large one, and limits the general applicability of the authors' approach. Indeed, in many MoE-based settings, model diversity is sufficient to improve performance significantly, and task specialization is not even considered. The authors need to argue that this setting is representative and useful.

A1. We would like to emphasize that our approach is designed for general-purpose LLMs rather than being tailored to MoE models. As discussed in L295-300, MoE employs token-level routing that assigns individual tokens in a sequence to different experts within the same model, whereas our ME-Switch uses model-level switching to select the most suitable model for an entire user request. For a detailed discussion regarding the assumption that each LLM specializes in a distinct domain, please refer to Q2 of the general response.

Q2. There are missing baselines. The sensitivity and quality of the domain-based routing setup as compared to a baseline without adaptive quantization makes it difficult to disambiguate the effect of improved quantization alone.

A2. We present the results of ME-Switch without applying our salient-aware delta compression in Tables I and II. The results demonstrate that both salient-aware delta compression alone and model-level routing alone achieve performance comparable to their respective unquantized expert models across a variety of downstream tasks. We have included the results in Section D of the revised manuscript.

Table I. Performance comparison of different methods for the Mistral-7B family. "Baseline" refers to the unquantized experts, while "SADC" represents our salient-aware delta compression.

Table II. Performance comparisons of different methods for the LLaMA-2-13B family. "Baseline" refers to the unquantized experts, while "SADC" represents our salient-aware delta compression.

Q3. There is no baseline that foregoes supervised fine-tuning (SFT) or one that compares SFT without quantization. These components should be independently ablated.

A3. Please refer to Q1 in the general response.

Dear Reviewer 1fKA

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We have carefully addressed your concerns and provided detailed responses, which we hope have resolved your queries. If you have any additional questions or further concerns, please do not hesitate to let us know.

Best regards,

Authors of #912

Dear Reviewer 1fKA,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the discussion period is nearing its end, we wanted to kindly check if our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to clarify further.

Thank you again for your valuable feedback and time.

Best regards,

Authors of #912