Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling

We sincerely appreciate the reviewer’s thoughtful and detailed feedback, which has greatly enhanced the depth, clarity, and overall quality of our work.

W1: Strengthen the contribution with more detailed discussion or theoretical insights.

We appreciate the suggestion and agree that a deeper theoretical discussion can greatly enhance the contribution. In response, we would like to highlight Section C of our appendix, which provides a detailed theoretical analysis of cross-bit-width implicit knowledge transfer—a core aspect of our method. Specifically, we offer a formal perspective on how sub-networks with different quantization levels (e.g., 2-bit and 8-bit) can implicitly benefit from each other by sharing weights, even without overlapping data or explicit soft-label supervision. We show that the gradient updates from each sub-network lie within the span of their respective input data subspaces, and that the shared parameter is effectively updated over the union of these subspaces. This results in an implicit data-sharing effect that broadens the effective training distribution for each sub-network (Lines 143–155). Furthermore, we theoretically analyze the alignment between gradients from differently quantized sub-networks and argue that this serves as a form of indirect supervision—akin to soft distillation—without requiring additional targets or training overhead (Lines 156–165). This insight is, to the best of our knowledge, unique to our work and has not been previously formalized in prior studies on multi-bit quantization.

We believe this framework not only strengthens the novelty of our method, but also provides valuable insights into the design principles behind multi-bit quantized networks trained with shared weights. We will make sure to better highlight these points in the main text for clarity.

W2: More clearly highlight the efficiency gains of coreset sampling with a formal comparison to standard approaches.

We appreciate your helpful feedback. To more clearly highlight the efficiency gains of our proposed coreset sampling strategy, we have added a formal comparison to standard approches (i.e., Any-Precision) in the Table below. This breakdown presents the GPU hours required for training, adaption, calibration, and scoring steps under different frameworks using ResNet-20 on CIFAR-10 with pruning rate of 80%. We excluded the scoring time from GPU hours, as it is an offline process and can be reused across all experiments under the same or comparable experimental settings (e.g., dataset, model architecture).

[Table. Breakdown of GPU hours for ResNet-20 experiments with 80% pruning]

With these results, we would like to briefly clarify our framework. First, coreset sampling significantly reduces the training GPU hours for multi-bit quantization models. This efficiency gain grows in proportion to the pruning rate. However, coreset sampling on its own does not remove the cost of the calibration phase, which is required to support additional bit-widths. To address this, we apply Bias Correction and BN adaptation, effectively eliminating the need for calibration. Therefore, if we formally define the computational cost of each approach with the pruning rate $p \in [0,1)$, where $p$ is the fraction of samples removed from the full dataset, it can be expressed as:

Cost of the Standard Approach (Any-Precision):

Cost of Our Method (Bias Correction + Coreset Sampling):

Here, calibrated bit-widths refer to intermediate bit-widths that are not directly trained but require calibration, such as 3, 5, 6 and 7 bits. BN adaptation incurs only a negligible overhead of 0.004 GPU hours as shown in the table, so we omit it from the formula for clarity.

We hope this clarification, along with the updated comparison table, clearly demonstrates the efficiency benefits of our framework. By jointly applying coreset sampling, Bias Correction, and BN adaptation, our method achieves substantial reductions in computational cost while preserving accuracy across bit-widths.

We would like to extend our heartfelt appreciation to the reviewer for your careful and constructive feedback, which has profoundly enriched the quality and clarity of our work.

W1: Broader validation on other domains would strengthen applicability.

We fully agree that validating our approach on other domains such as generative LLMs would further strengthen its general applicability and impact. However, exploring multi-bit quantization in the context of LLMs is currently not technically feasible for us, primarily due to the substantial computational resources required to train such models from scratch. Moreover, this application domain remains relatively understudied, and additional foundational studies are needed to understand the behavior of multi-bit quantization in LLM settings. A further challenge is that existing sample-importance scoring methods are tailored to discriminative models, and adapting them for generative models remains an open problem. We therefore consider this an important direction for future work and will include a discussion of this in the revised manuscript.

W2: Provide the theoretical basis behind Eq. 2 and how it contributes to error reduction.

Thank you for pointing this out. We agree that Eq. 2 was introduced somewhat abruptly, and hope to clarify the theoretical groundings and provide a more detailed explanation of how it contributes to reducing activation errors below.

Our goal is to mitigate the mismatch in activation distributions across bit-widths by addressing the root cause: systematic bias introduced during weight quantization. Since we isolate the effect of quantized weights by fixing the input activation, we analyze the convolution output as follows. Let:

• $x \in \mathbb{R}^d$: fixed input activation (same across all bit-widths),
• $w, w_q \in \mathbb{R}^d$: full-precision and quantized weight vectors,
• $y = x^\top w$: full-precision output activation,
• $y_q = x^\top w_q$: quantized output activation.

Then the error induced by quantization is: $\epsilon = y_q - y = x^\top(w_q - w)$. This shows that any discrepancy in output activations is a direct result of the quantization error in the weight vector, scaled by the input activation. Since $x$ is fixed, the variation across bit-widths in $y_q$ comes solely from $w_q$. As shown in Fig. 3, quantized weights ($w_q$) tend to introduce systematic shifts in both mean and scale compared to the full-precision weights ($w$), leading to:

• $\mathbb{E}[w_q] \neq \mathbb{E}[w]$ (bias mismatch),
• $\mathbb{V}[w_q] \neq \mathbb{V}[w]$ (variance mismatch).

These mismatches result in distributional shifts in the output activation $y_q$, which become more severe under lower bit-widths. Our proposed correction aims to eliminate these mismatches at the source, i.e., the weight level. We seek a transformation of $w_q$ that makes it aligned with $w$ in terms of mean and variance. Let’s define a corrected quantized weight vector $w_q’$ such that: 1. $\mathbb{E}[w_q’] = \mathbb{E}[w]$ 2. $\mathbb{V}[w_q’] = \mathbb{V}[w]$

To align the quantized weights with the full-precision weights, we apply an affine transformation of the form $w_q’ = \alpha (w_q + \beta)$, where $\alpha \in \mathbb{R}$ controls scaling and $\beta \in \mathbb{R}^d$ controls shifting. To match the mean, we require $\mathbb{E}[w_q’] = \alpha (\mathbb{E}[w_q] + \beta) = \mathbb{E}[w]$, which gives $\beta = \mathbb{E}[w] - \mathbb{E}[w_q]$. To match the variance, we require $\mathbb{V}[w_q’] = \alpha^2 \mathbb{V}[w_q] = \mathbb{V}[w]$, which yields $\alpha = \sqrt{ \frac{\mathbb{V}[w]}{\mathbb{V}[w_q]} }$. Substitute $\alpha$ and $\beta$ into the affine form, and we get the Bias Correction equation:

Using this corrected $w_q’$, the new activation becomes:

The expression above can be interpreted as an affine transformation of $y_q$ (i.e., rescaled and shifted), controlled by the quantized weight distribution’s discrepancy from the full-precision distribution. Since the input $x$ is fixed, correcting the weight vector’s mean and variance directly adjusts the distribution of the output $y_q’$, reducing systematic activation shifts and scale mismatches across bit-widths. In doing so, the challenge of aligning activation distributions is effectively reduced to a simpler and more stable problem of aligning weight statistics. As a result, different quantized sub-networks can share BN layers more reliably, as the activations are already well-aligned at the source (i.e., before normalization), eliminating the need for separate BN parameters or additional activation calibration. We will incorporate this detailed theoretical grounding into the main manuscript and provide the full derivation and extended discussion in an appendix of the revised paper.

W3: Clarify whether BN adaptation has been applied to ViTs.

ViTs use Layer Normalization (LN) instead of Batch Normalization (BN), eliminating the need for BN-specific calibration, such as updating running mean and variance after training. Unlike BN, LN does not maintain such running statistics. As far as we are aware, there has been no prior investigation into how LN behaves in the context of multi-bit ViT networks. Our work is the first to experimentally demonstrate that LN parameters, once trained, remains relatively effective across all bit-widths without requiring additional calibration or adjustment. We will incorporate this discussion and the detailed results into the Appendix

W4: Implementation of bit-wise coreset sampling into existing pipelines seems complex, provide more guidance.

We understand that ease of implementation is crucial for broader adoption, and we truly appreciate the feedback. In practice, our scoring and sampling code is already modularized, making it easy to plug into various network architectures and base models. To further support integration, we will provide a detailed guide covering key components such as pruning rate $p$, sampling frequency, and the temperature variable—three of the few hyperparameters closely tied to our method. We also plan to automate the full pipeline with a single bash script and include clear instructions for each stage (scoring, sampling, training, and if applicable, BN adaptation). Finally, we will publicly release all code and scripts to facilitate easy adoption and reproducibility.

Thank you for your thoughtful feedback and insightful suggestions. In response, we have carefully followed your recommendations and provide additional results and clarifications below.

W1: Dynamic sample-wise importance

We appreciate the reviewer’s insightful question. You are absolutely right that accounting for the dynamic nature of sample importance is a key consideration of our method. Our decision to compute importance scores only once at the beginning—rather than periodically throughout training—was primarily driven by the high computational cost associated with full-dataset scoring. To address the potential rigidity of this one-time evaluation, we incorporate temperature-based probabilistic sampling, which introduces stochasticity into the coreset selection process. This allows our method to be aware of potential importance variations, even without explicitly updating the scores over time.

That said, we acknowledge the validity of the reviewer’s concern that high-probability samples may still dominate selection, especially when importance is fixed. To directly examine this, we conducted additional experiments where importance scores are dynamically re-evaluated every 10, 30, 50, or 100 epochs, and coresets are resampled accordingly. We evaluated how it impacts accuracy and GPU hours using ResNet-20 on CIFAR-10 and ResNet-18 on CIFAR-100 under both 80% and 90% data pruning.

The results below reveal a consistent pattern: while dynamic score re-evaluation leads to only marginal accuracy changes, it incurs a substantial increase in computational cost. In many settings, our one-time scoring strategy already matches or even outperforms more frequent re-evaluation in terms of final accuracy, while consuming significantly fewer GPU hours. This empirical finding validates our design choice, and shows that a single, well-computed importance estimate—when paired with stochastic sampling—offers an effective and efficient balance, capturing most of the benefits of dynamic importance tracking without incurring its heavy cost.

Looking ahead, with the observation that dynamic re-evaluation yields modest gains on the more challenging CIFAR-100, we believe dynamic sampling techniques could be the key to boosting performance on complex, high-variability datasets where importance scores drift more drastically throughout training. The main hurdle is the high cost of score re-evaluation during training, which currently limits the practicality of dynamic methods. In future work, we will explore lightweight techniques to reduce score-evaluation overhead while maintaining the quality of importance estimates. We will include a discussion of these future work directions in the revised manuscript.

[Table 1. Impact of score re-evaluation for ResNet-20 / CIFAR-10 & ResNet-18 / CIFAR-100 with 80% pruning]

[Table 2. Impact of score re-evaluation for ResNet-20 / CIFAR-10 & ResNet-18 / CIFAR-100 with 90% pruning]

W2: Effectiveness of BN sharing for 1-bit quantization

This is an important point, and we appreciate this insightful observation. In multi-bit settings, weight distributions tend to approximate a normal distribution, where our Bias Correction mechanism works effectively. However, in the 1-bit case, the weight distribution collapses into a near-uniform or binary form, causing a significant distribution shift that Bias Correction alone struggles to adequately address. For that reason, prior works on multi-bit networks often omit the 1-bit setting from their analysis [1,2,3]. In contrast, our method explicitly incorporates this case and demonstrates that assigning a separate BN layer for 1-bit quantization is both a simple and effective solution.

To further support this, we conduct ablation studies comparing two configurations: one where BN is shared across all bit-widths (including 1-bit), and another where the 1-bit case has its own BN, and the remaining bit-widths share a single BN. The results clearly show that including 1-bit in BN sharing significantly degrades 1-bit quantization performance. On the other hand, using a separate BN for 1-bit achieves strong performance with negligible additional overhead, that is the additional parameters for the separate BN layer, which accounts for less than 0.5% of the total number of parameters. This performance degradation stems from unstable BN statistics. The 1-bit weights produce activation distributions that are markedly different from those of higher bit-widths, leading to significant fluctuations in shared BN statistics. These fluctuations negatively impact the normalization of other bit-widths, ultimately harming overall model performance.

[Table 3. BN sharing with and without 1-bit quantization on ResNet-20 / CIFAR-10]

W3+Q3(a): Clarification on the training pipeline

Thank you for drawing our attention to this, as we fully agree that a more detailed explanation of the training pipeline is in need. For ImageNet-1K experiments, fine-tuning begins from the pretrained Any-Precision model before calibration. Specifically for our method, we use this non-calibrated model as the starting point and apply coreset sampling and Bias Correction during fine-tuning, followed by BN adaptation at the end. In contrast to the original Any-Precision [4], which requires an additional calibration stage after training to achieve comparable accuracy at calibrated bit-widths, our approach eliminates this step. Training concludes after fine-tuning and a lightweight BN adaptation phase.

W3+Q3(b): Provide GPU hours with more detailed granularity

We truly appreciate the comment and fully agree that providing a more detailed breakdown improves transparency and clarity in evaluating computational efficiency. In response, we report GPU hours for all ResNet experiments, broken down by training stage, including:

• Score evaluation
• Coreset training
• Calibration (if applicable)
• Adaptation

These results also help clarify the computational cost structure of our framework. Specifically, coreset sampling significantly reduces training GPU hours for multi-bit quantization models, and this efficiency gain increases with the pruning rate. However, coreset sampling alone does not eliminate the cost of the calibration phase, which is typically needed to support additional bit-widths. To address this, we apply Bias Correction and BN adaptation, which allow us to remove the calibration step entirely without sacrificing accuracy.

[Table 4. Breakdown of GPU hours for ResNet-20 experiments with 80% pruning]

[Table 5. Breakdown of GPU hours for ResNet-18 experiments with 80% pruning]

Reference [1] K.Xu et al., "Eq-net" ICCV 2023. [2] X.Sun et al., "CoQuant" ICCV 2024. [3] Y.Zhong et al., "MultiQuant" CoRR 2023. [4] H.Yu et al., "Any-precision DNN" AAAI 2021.

Thank you for addressing my concerns. Given that the key concerns regarding design choice and efficiency have been addressed, I have increased my score and support acceptance

We deeply thank the reviewer for their constructive and detailed feedback, and also for recognizing the insights and empirical results of our work. Below, we provide point-by-point responses to the concerns and questions raised.

W1: Alignment achieved by Bias Correction

Thank you for pointing this out. To demonstrate the effectiveness of our Bias Correction method, we conducted an analysis of the output activations, comparing how well activations from different bit-widths align with one another. In all experiments, the activation precision is fixed to 4 bits, and we measure the mean absolute error (MAE) between the activations of quantized and full-precision models, with and without applying Bias Correction. The results show that applying the correction consistently improves alignment, reducing the avg. MAE from 0.755 to 0.667. This improvement holds across all bit-widths, demonstrating our method’s effectiveness. We will include these results in the revised version to highlight the impact of Bias Correction on activation alignment.

[Table 1. MAE between different bit-widths and full precision models' activation with vs. without Bias Correction]

W2+Q1: Clarify the role and relative impact of Bias Correction vs. BN adaptation

Thank you for this point. As shown in Table 2, our ablation studies isolate each component to assess its individual contribution to performance. The results reveal that neither method alone is sufficient for optimal accuracy across all bit-widths. For example, using only Bias Correction leads to suboptimal performance at 2-bit, while using only BN adaptation underperforms at 1-bit. Using both techniques together consistently yields the best accuracy across all bit-widths.

While Bias Correction offers a low-cost way to compensate for systematic errors in the weight space, it does not directly influence the BN running statistics that govern activation distributions at inference time. BN adaptation, on the other hand, recalibrates these statistics through forward passes on the training data, directly aligning the activation distributions. In this sense, BN adaptation complements Bias Correction by targeting residual discrepancies that remain after correcting the weights. Together, the two methods correct both weight distribution shifts and activation misalignments, enabling consistent activation behavior and improved accuracy, particularly for intermediate calibrated bit-widths (e.g., 3,5,6,7 bits). We plan to include these ablation studies in the revised version to more clearly demonstrate the distinct and complementary roles of each technique.

[Table 2. The impact of Bias Correction vs. BN adaptation]

W3+Q2: Impact of resampling frequency

We agree that sampling frequency is an important factor to examine. In practice, the overhead of bit-wise coreset resampling is extremely small compared to the overall training cost. For example, even on an ImageNet-scale dataset, performing 100 resamplings takes only about 3.36 minutes. Given this negligible cost, resampling at every epoch is a practical and effective choice.

To quantitatively demonstrate this, we conducted experiments with different resampling intervals and measured both validation accuracy and total sampling time. The results show that resampling every epoch improves average accuracy by 1.33%p compared to resampling every 30 epochs, while adding just 53 seconds of overhead to a multi-hour training process. This demonstrates that frequent resampling can offer meaningful accuracy gains at virtually no additional cost.

[Table 3. Influence of sampling frequency on ResNet-20 / CIFAR-10]

Q2: Impact of coreset sampling

Thank you for raising this point. To examine the standalone contribution of coreset sampling to both accuracy and training speedup, we provide results on a CIFAR-10 baseline where only coreset sampling is applied, with all bit-widths except for 1-bit sharing BN layers.

In the table below, we denote setups with coreset sampling as A, and those with Bias Correction and BN adaptation as B. The configuration A only (B not applied) represents coreset sampling alone, while A+B combines coreset sampling with Bias Correction and BN adaptation. As shown in the results, while coreset sampling (A) contributes most to the speedup, it is not sufficient on its own to maintain strong accuracy at across every bit-width. In contrast, when Bias Correction and BN adaptation are applied alongside coreset sampling (A+B), we observe accuracy improvements everywhere, with pratically no additional GPU hours. The results indicate that although coreset sampling is the main driver of compute efficiency, Bias Correction and BN adaptation is essential for best accuracy.

[Table 4. Impact of only coreset sampling on ResNet-20 / CIFAR-10 (A: Coreset Sampling, B: Bias Correction & BN adaption); Bold the highest performance for each pruning rate.]

W4: Miscellaneous figure errors

Thank you for drawing our attention to this. For Figure 1, we will clarify that D and S denote the full dataset and the sampled coreset, respectively. For Figure 2, we will update the caption to explicitly define E[X] and V[X] as the running statistics of BN layers before adaptation, and E[X]’ and V[X]’ as those after the BN adaptation phase. These changes will be reflected in the revised version to improve clarity of our presentation.

Q3: Overhead of BN Adaptation

While BN adaptation does introduce additional computation, its overhead is negligible compared to the overall training cost since it involves only forward passes over a fixed number of samples and does not require backpropagation. For example, in the ResNet-20 with CIFAR-10 setting, BN adaptation takes only 0.004 GPU hours, accounting for less than 0.3% of the total training time in both the Bias Correction-only configuration (7.52 GPU hours) and the more efficient Bias Correction + Coreset Sampling configuration (1.52 GPU hours). Below, we include a detailed breakdown of GPU hours across all components, highlighting the negligeable overhead of BN adaptation step.

[Table 5. Breakdown of GPU hours for ResNet-20 / CIFAR-10 experiments with 80% pruning]

Q4: Pretraining settings for ImageNet-1K experiments

Thank you for raising this important question. In our experiments, we fine-tune from publicly available Any-Precision ResNet-50 checkpoints [1], rather than training from scratch. This follows prior work, as full Any-Precision training on ImageNet-1K is extremely time-consuming, typically requiring around 300 GPU hours for a single run—making it infeasible to reproduce during the rebuttal period. To our knowledge, no existing multi-bit quantization work has explicitly reported this pretraining cost. Accordingly, the GPU hours reported in Tables 3 and 4 include only the additional fine-tuning and calibration time. However, we agree that including full pretraining costs would improve clarity and will report those figures in the revised manuscript.

Q5: Bolded results in tables

The bolded values are meant to indicate results from our proposed method. We apologize that this was not clearly stated and will revise them accordingly. Additionally, we will consider using a different visual cue to distinguish our method from others more clearly.

Reference [1] H.Yu et al., "Any-precision DNN." AAAI 2021.