DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment

Thank you for your expert evaluation and specific suggestions.

---

#1. The deployment of DP-LLM on larger LLMs needs evaluation.

We evaluate the performance, fine-tuning costs, and inference overheads of DP-LLM when applied to Llama-3-70B. We measure the perplexity and compare it against HAWQ-V2 when the target precision is 3.5.

On a single A100 80GB GPU, the fine-tuning of Llama-3-70B takes 3.5 hours. The cost of fine-tuning the average precision increases sublinearly with respect to the parameter size, considering that Llama-3-8B takes 0.5 hours to fine-tune on the same GPU (as reported in Appendix B.3).

We evaluate the inference latency overhead on Jetson Orin AGX 64GB, the only edge device with large enough memory to load a 70B model. The latency overhead is kept minimal even when the model size increases. When the effective bitwidth is 3.5, the latency overhead could not be observed when measuring tokens per second up to the second decimal place, which was measured as 4.56 tok/s. This is possible because the complexity of the relative error estimator increases linearly with the hidden dimensions ($\mathcal O(nk)$, where $k=64$ for our error estimator setup), while the linear layers have approximately quadratic complexities since most Transformer architectures scale both the hidden dimension and the intermediate dimension.

---

#2. Does the performance of DP-LLM depend on Any-Precision LLM? Also, low-bit evaluation scenarios are lacking.

DP-LLM is not inherently tied to Any-Precision LLM. Its core idea—dynamic, layer-wise precision assignment based on saliency estimation—is orthogonal to the underlying quantization method and can be used in conjunction with any backend that supports multiple bitwidths. We adopt Any-Precision LLM primarily for its memory efficiency, as it enables runtime model adaptation without requiring multiple copies of the model. However, if memory capacity is not a constraint, DP-LLM can also be used with other quantization approaches by separately maintaining multiple bitwidth variants in memory. Under tight memory constraints, Any-Precision LLM is a natural choice to support runtime adaptation; however, this reliance is not unique to DP-LLM and similarly applies to other approaches such as static mixed-precision methods.

Decoupled from Any-Precision LLM, we evaluate DP-LLM with a state-of-the-art quantization method to demonstrate its effectiveness in the low-bit regime (e.g., 2.x-bit). The table below reports the perplexity of DP-LLM on LLaMA-3-8B, where it dynamically selects among 2- to 4-bit weights quantized using GuidedQuant [1] in comparison to static mixed-precision baselines. In all cases, DP-LLM consistently outperforms the static approaches, demonstrating its robustness under aggressive quantization.

[1] Kim et al., “GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance”, ICML, 2025.

---

#3. Comparison with SliM-LLM.

We evaluate SliM-LLM when the effective bitwidth is 4, using the same perplexity measuring tools as DP-LLM to ensure consistency.

While DP-LLM archives comparable performance against SliM-LLM, it is worth noting the following limitations of SliM-LLM.

First, unlike DP-LLM, the precision reduction of SliM-LLM does not translate into speedup, which is crucial for runtime model adaptation. In particular, SliM-LLM reports to have 5.15% to 27.06% slowdown compared to GPTQ, often being slower than the FP16 baseline. DP-LLM, on the other hand, shows minimal latency overhead from 0.06% to 4.24% as shown in Table 4.

Second, SliM-LLM assigns a uniform target precision for every layer and only performs mixed precision allocation to groups within each layer. On the other hand, DP-LLM can assign different target average precisions to each layer, providing more flexibility in precision assignment.

---

#4. Is $B[i]$ equal to $b$ where $c_{i,b}=1$? Also, the use of layerwise average precision is puzzling.

Yes, $B[i]$ equals $b$ where $c_{i,b}=1$.

The runtime model adaptation calls for the distinction between layer-wise maximum precision and layer-wise average precision.

The layer-wise maximum precision is determined to fit the model into the memory capacity budget of the device. Since DP-LLM requires high precision weights to be available for dynamic precision selection, leaving high precision weights that are not important would be wasteful, or even cause out-of-memory situations under tight memory budgets.

The layer-wise average precision, on the other hand, is used for fitting the model for the current runtime budget (i.e., LLM inference slack of Figure 1). After the model is loaded into memory, to perform runtime model adaptation, the layer-wise average precision is used to match the current runtime budget.

---

#5. Comparison with sparse outlier protection methods is needed to address fair comparison under equal storage costs.

We apply the methodology of OWQ [2], a state-of-the-art quantization method that saves additional weight channels in FP16, to harvest outlier channels for each layer. We set the number of outliers to match the storage overhead of DP-LLM, which is 2.4% for Llama-3-8B as reported in Table 8. We then measure the perplexity after adding the outlier channels to the static baselines.

Although saving outliers shows notable performance improvements, DP-LLM still consistently outperforms the static baselines by a considerable margin.

It is also worth noting that outlier protection comes at the cost of latency overhead. We measure the slowdown of OWQ incurred by the outliers under the same configuration. Since the GitHub repository of OWQ reports that the kernels were optimized for A100 GPUs, we evaluate the kernel time on an A100 GPU using PyTorch Profiler. We find that outlier protection incurs an average of 18.42% kernel time slowdown. Applying a finer-grained outlier protection (e.g., element-wise) would lead to even greater slowdown.

[2] Lee et al., “OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models”, AAAI, 2024.

---

#6. Clarification on latency evaluation.

We present the time-per-output token (TPOT) values under several effective bitwidths. We also report the TPOT of the full precision (i.e., FP16) model for comparison.

By employing low-overhead error estimation techniques, DP-LLM shows latency improvements proportional to the precision decrements.

Thank you for your constructive feedback.

---

#1. A quantitative and deeper analysis on dynamically changing sensitivity of layers (Section 2.4).

We kindly ask to refer to our response of #1 to Reviewer QAUB.

---

#2. The paper is not well positioned in the wider context and history of quantization.

DP-LLM fundamentally differs from prior works on saliency metrics for model compression by addressing the dynamic nature of saliency (referred to as sensitivity in our manuscript). Previous approaches, such as Optimal Brain Surgeon [1], Deep Compression [2], and LLM Super Weights [3], primarily focus on static methods for estimating saliency at the layer or finer granularity. None of the existing methods capture dynamically changing sensitivity during inference, nor do they propose runtime techniques to handle such dynamics with minimal overhead. DP-LLM is, to the best of our knowledge, the first to address this issue. We will include this discussion on the positioning of DP-LLM within the broader context in the revised version.

[1] Hassibi et al., “Second order derivatives for network pruning: Optimal Brain Surgeon”, NIPS, 1992.

[2] Han et al., “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, ICLR, 2016.

[3] Yu et al., “The Super Weight in Large Language Models”, arXiv, 2024.

---

#3. Cheaper saliency estimators are not considered.

We evaluate a simple saliency estimator based on activation norm [4, 5] and compare it to our proposed estimator based on relative error norm. Specifically, we evaluate perplexity on WikiText2 and C4 using Llama-3-8B. To configure the norm-based saliency estimators, for each dataset, we collect the distribution of the desired norm, and for each layer and decoding step, select the top-k% to apply a 4-bit weight instead of a 3-bit weight. We experimented with k=20% and k=50%, which roughly correspond to constructing a 3.2-bit and 3.5-bit model, respectively.

It is clear that the relative error norm-based selection shows superior performance compared to the activation norm-based selection. This can be attributed to the fact that the relative error norm is a direct indicator of the quantization error, while the activation norm is an indirect measure of it.

[4] Lin et al., “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration”, MLSys, 2024.

[5] Zhang et al., “LQER: Low-Rank Quantization Error Reconstruction for LLMs”, ICML, 2024.

---

#4. Only two static mixed-precision baselines were evaluated.

As DP-LLM is designed for efficient runtime model adaptation with regard to fluctuating LLM inference slacks, the mixed precision scheme should have minimal latency overheads. More fine-grained mixed precision methods that employ channel-wise or element-wise mixed precision schemes cannot satisfy this constraint. For example, SliM-LLM [6] utilizes fine-grained block-wise mixed precision, but reports to have 5.15% to 27.06% slowdown compared to GPTQ, even often showing slower speeds than the FP16 baseline. The static baselines we evaluated, LLM-MQ and HAWQ-V2, employ a layer-wise mixed precision scheme, allowing for near-zero overhead, making them suitable baselines for evaluating the runtime adaptation ability of DP-LLM.

[6] Huang et al., “SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models”, ICML, 2025.

---

#5. Deploying a highly optimized model for the tightest margin could nullify the need for runtime model adaptation.

We consider the cases of applying PTQ and applying QAT for constructing a highly optimized model as the reviewer has mentioned.

• PTQ: While SoTA PTQ methods demonstrate competitive performance with the full precision model up to 4 bits, further reduction in precision leads to significant performance degradation. For example, GuidedQuant [7], a SoTA non-uniform quantization method, shows a large perplexity increase of 1.45 when quantizing Llama-3-8B to 3 bits compared to the full precision model. Thus, maintaining a statically optimized model that meets the tightest margin (i.e., low average precision) would incur a large loss in accuracy when a larger inference slack is available.

• QAT: QAT methods require a tremendous amount of GPU time, which could be impractical considering the rapid advancements of recent LLMs. Since model evolutions are accelerating, the needs for low-cost and fast-deployable quantization methods are rising.

[7] Kim et al., “GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance”, ICML, 2025.

---

#6. Do LLM inference slack really have such fluctuations?

In short, yes. As explained in Section 2.2, LLM inference slack is determined by the difference between the QoS budget and system utilization, both of which dynamically vary. We provide additional references to support such variance in QoS budget and system utilization.

• System utilization: Mendis et al. [8] show that memory bandwidth utilization varies both within a workload and across workloads in smartphones. Furthermore, it is common for multiple tasks to be concurrently running on edge devices, including both foreground applications launched by the user and background jobs. Prior work such as ApproxDet [9] specifically takes into account this fluctuation, proposing an adaptive solution that is aware of such resource contention.

• QoS budget: It is well known that different applications of LLMs, while sharing the same model, can have varying performance priorities and requirements [10, 11]. For instance, summarizing a long, complex document may prioritize accuracy over throughput, while live translation requires low time between tokens for better user experience.

[8] Mendis et al., “Impact of Memory Frequency Scaling on User-centric Smartphone Workloads”, SAC, 2018.

[9] Xu et al., “ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles”, SenSys, 2020.

[10] Agrawal et al., “On Evaluating Performance of LLM Inference Systems”, arXiv, 2025.

[11] Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI, 2024.

---

#7. Other fine-grained neural architecture modifying schemes are not mentioned.

While pruning-based methods, including Once-for-All [12] and EagleEye [13] mentioned by the reviewer, can derive multiple sub-networks from a full-sized model for model adaptation, such techniques primarily target non-LLM models, which have several orders of magnitude fewer parameters. Although some pruning methods have been developed specifically for LLMs [14, 15], they generally incur more performance degradation than quantization when applied at an equivalent model compression ratio.

[12] Cai et al., “Once-for-All: Train One Network and Specialize it for Efficient Deployment”, ICLR, 2020.

[13] Li et al., “EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning”, ECCV, 2020.

[14] Ma et al., “LLM-Pruner: On the Structural Pruning of Large Language Models”, NeurIPS, 2023.

[15] Ashkboos et al., “SliceGPT: Compress Large Language Models by Deleting Rows and Columns”, ICLR, 2024.

---

#8. The precision of activations is ambiguous.

As we employ a weight-only quantization scheme, the activations are kept at high precision (i.e., FP16).

---

#9. Is the error estimation accuracy a theoretical statement?

Limiting the relative error estimation difference within 8% with 95% confidence is a theoretical statement. From Equation (3), to get the vector norm, we take the square root of every term. To match $\sqrt{(1+\epsilon)}=1.08$, we set $\epsilon=0.1664$. For 95% confidence, we set $\delta=0.05$, and from $k=\mathcal O(\epsilon^{-2}log(\delta^{-1}))$, we can see that $k=64$ is a reasonable selection to match the confidence and error estimation difference statement.

---

#10. The use of geomean in section 6.3 is concerning.

In Section 6.3 (Line 335-336), we state that “Even at the 99th percentile, the geomean increase remains below 3%.” The geometric mean is calculated from the three values in the 99th percentile row of Table 6 (i.e., 3.02%, 2.25%, and 3.32%), which is 2.83%. We did not intend to disguise any outliers, but used the geometric mean to provide a condensed overview of Table 6.

Thank you for your thoughtful review.

---

#1. Detailed quantification of Section 2.4 is needed. Also, clarification for Figure 3 is needed.

As noted in Line 127–128 of the paper, Figure 3 was generated using the first sample from the C4 train dataset, decoded with the Llama-3-8B model.

To provide quantitative insights into the dynamic nature of sensitivity, we present statistics based on the experimental data in Figure 3 that captures how dynamically the set of sensitive layers changes across decoding steps. Specifically, we compute the overlap of sensitive layers (i.e., the top 20% most sensitive layers) across all pairs of decoding steps. The average overlap ratio is only 4.83%. For reference, if the sensitive layers were selected completely at random, the expected overlap would be 4% (i.e., 0.2 × 0.2). The closeness between the two values (4.83% vs. 4%) suggests that the set of sensitive layers varies almost as much as a purely random selection, highlighting the importance of addressing this dynamism to achieve better performance. To further analyze the dynamics of sensitive layer distribution, we present two additional analyses based on the same experimental data in Figure 3.

The first addresses the question: “Is there a consistent trend where the same tokens lead to similar sets of sensitive layers?” To investigate this, we compute the average overlap ratio of sensitive layers for repeated occurrences of the same token. We select the ten most frequent tokens in the sample for analysis (',', '.', 'to', 'will', 'BBQ', 'in', 'you', 'be', 'and', 'the'), and present the results in the following table. The maximum average overlap ratio observed is only 6.32%, which is not significantly different from the random baseline of 4% (i.e., 0.2 $\times$ 0.2). This indicates that even for identical tokens, the set of sensitive layers varies significantly depending on the context. Therefore, the answer to the question is inclined toward No. This motivates the need to account for such dynamic behavior, thereby justifying the design and value of our approach.

The second explores the question: “Do different layers tend to exhibit different levels of sensitivity?” Specifically, we examine sensitivity variations across layer types (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj). We compute the selection ratio—the proportion of decoding steps in which a given layer is selected as sensitive—for each layer, and then average these ratios by layer type. The results show that down_proj has the highest average selection ratio at 27.71%, while k_proj demonstrates the lowest at 16.97%. Notably, the down_proj of the final transformer block (block 31) achieves the highest selection ratio across all layers, reaching 45.27%. These findings reveal a discernible trend in sensitivity differences across layer types.

Such observations suggest that static mixed-precision methods may still offer some degree of effectiveness, as sensitivity shows partial correlation with layer type and can be partially anticipated. However, this effectiveness is limited at best—even the most frequently selected sensitive layer is chosen in only 45.27% of decoding steps. This means that statically fixing this layer as sensitive would result in selecting a suboptimal layer in more than half of the decoding steps, missing opportunities to allocate precision where it is most needed. These findings once again underscore the importance of dynamic approaches like DP-LLM.

---

#2. Real-world frameworks such as vLLM are not used for evaluation.

We believe the current latency evaluation setup is appropriate for measuring the overheads of DP-LLM. For this evaluation, we used gpt-fast, a highly optimized LLM inference implementation specifically designed for small batch sizes. This makes it well-suited for DP-LLM, which primarily targets single-batch on-device inference. In contrast, frameworks such as vLLM are optimized for datacenter serving scenarios with large batch inputs, and are therefore less relevant to the intended use case of DP-LLM.

---

#3. Statistical significance is unclear.

Two sources of variability for DP-LLM may exist: 1) the initialization point of each average precision when the fine-tuning process starts, and 2) the randomly sampled $A$ matrix when applying JL-Lemma. Once a configuration is ready for evaluation, all computations do not involve any randomness and result in a deterministic outcome.

The following table presents the average and standard deviation of DP-LLM perplexity using Llama-3-8B, evaluated across five different random seeds. The results are compared to those reported in the original paper. Only minimal variations are observed across runs.

---

#4. How does DP-LLM handle multi-batch scenarios?

DP-LLM is specifically designed for single-batch scenarios, which we believe to be the dominant use case in on-device LLM inference, where a single query comes from a single user [1]. Nevertheless, DP-LLM can be extended to support multi-batch scenarios, for example, by averaging the relative error across batched inputs. However, this may smooth out the dynamic sensitivity of individual queries, potentially diminishing performance gains. We believe that developing extensions of DP-LLM for multi-batch settings in data centers, while minimizing such degradation, represents a promising direction for future work.

[1] Lin et al., “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration”, MLSys, 2024.

---

#5. How is the trade-off between performance and latency manifested?

Latency is directly proportional to memory consumption and thus exhibits the same trade-off with performance. In general, slower models tend to yield better performance. The following table illustrates this trade-off by showing the time-per-output token (TPOT) for different effective bitwidths.

---

#6. How much does each part contribute to the final results?

Every component of our framework is integral to the performance and efficiency of the system.

The ablation study of the error estimator is available in Table 3. The ablation study for using asynchronous error estimation is in Table 5. An ablation study regarding thresholds is not possible, since dynamic adaptation per decoding step would become infeasible without them.

Thank you for your insightful comments.

---

#1. How does DP-LLM handle multi-batch scenarios?

DP-LLM is specifically designed for single-batch scenarios, which we believe to be the dominant use case in on-device LLM inference, where a single query comes from a single user [1]. Nevertheless, DP-LLM can be extended to support multi-batch scenarios, for example by averaging the relative error across batched inputs. However, this may smooth out the dynamic sensitivity of individual queries, potentially diminishing performance gains. We believe that developing extensions of DP-LLM for multi-batch settings in data centers, while minimizing such degradation, represents a promising direction for future work.

[1] Lin et al., “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration”, MLSys, 2024.

---

#2. Latency overheads are not discussed.

Table 4 reported in the paper shows the latency overhead against LLM-MQ. Latency overheads are measured while error estimation mechanisms are concurrently running. The overhead against HAWQ-V2 is omitted since static baselines have virtually the same latency under equal effective bitwidth. Additionally, Table 5 shows the effect on latency when utilizing different relative error estimation techniques.

---

#3. Clarification for Q in Equation (6) of Appendix A is lacking.

$W_Q$ should be $W_{i,b}$ in Equation (6). The underlying quantizer, which is Any-Precision LLM for the current implementation, will create every $W_{i,b}$ in advance. Given such quantized weights, the goal of Equation (6) is to decide $c_{i,b}$ to minimize the loss perturbation. We apologize for the ambiguity.

---

#4. Algorithm 1 is incoherent with the main text.

It is simply a typo. $s_{i,b}W_{i,b}x+t_{i,b}W_{i,b}x$ is supposed to be $s_{i,b}W_{i,b}x+t_{i,b}W_{i,b+1}x$. We apologize for the confusion.

---

#5. Is the per-query QoS deviation unique to DP-LLM?

Per-query QoS deviation is unique to DP-LLM. While static methods have equal average bitwidths over all decoding steps, DP-LLM aims to match the average bitwidth among multiple decoding steps to the target precision. This may result in small deviations in the actual average bitwidth used against the target precision.

---

#6. Is DP-LLM implemented using a Matryoshka quantization approach?

Yes, for evaluations, we build DP-LLM on top of Any-Precision LLM, which adopts the Matryoshka quantization approach. This eliminates the need to store separate models for each bitwidth; instead, it maintains only the high-bitwidth parameters, from which lower-bitwidth variants are derived via on-the-fly truncation.