Response to Reviewer Comments

We thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of the clarity of our formulation, the novelty of our proposed quantization framework, and the strength of our experimental results. Below, we respond to each of the concerns raised.

(1) On the Scope of Experimental Evaluation

We agree that broader evaluation would strengthen our claims. We focused on ViTs due to their known sensitivity to quantization, where even small weight perturbations can cause large accuracy drops. Demonstrating strong performance in such settings highlights our method’s advantage.

To further demonstrate the broader applicability of BQQ, we evaluated our method on a compact large language model (LLM) optimized for edge deployment: Qwen2.5-0.5B. This model has recently gained attention for its strong performance despite its small size. We compared BQQ against a standard quantization baseline, GPTQ, under equal bit-width settings. As shown in the table below, BQQ achieves comparable results in both perplexity and downstream task accuracy, highlighting its effectiveness even in highly resource-constrained environments. Notably, in the extremely low-bit regime (2-bit quantization), BQQ demonstrates a remarkable advantage—achieving a 5.7% average improvement in downstream accuracy and reducing perplexity by a factor of 200 compared to GPTQ. This underscores BQQ’s robustness and superior performance in challenging low-bit scenarios. If accepted, we plan to include a broader range of experimental results to further showcase the versatility and effectiveness of BQQ across diverse model architectures and tasks.

Additionally, we emphasize that the current work proposes BQQ as a general binary quantization framework, rather than a dedicated quantization technique for neural networks. In our implementation, BQQ quantizes weights by minimizing reconstruction error in the weight space, without relying on output-based error signals that are commonly used in many neural network quantization methods. This implies that BQQ does not yet exploit task-specific loss functions or activation statistics. We therefore believe that adapting BQQ to incorporate such feedback—especially minimizing the downstream output error—could lead to significant further improvements in model performance, and this represents a highly promising avenue for future research.

(2) On the Effect of the Intermediate Dimension

Thank you for suggesting deeper analysis of the intermediate dimension $l$, which affects both accuracy and computation. In our main experiments, we used the heuristic $l = \mathrm{round}\left(\frac{mn}{m + n}\right)$ to match the original matrix size and adjust overall capacity via the number of binary stacks $p$.

To explore this further, we conducted additional experiments varying $l$ and $p$ while keeping total bit budget fixed. Results on the same benchmark matrices from Fig. 3 show that optimal $l$ depends on data characteristics. For instance, datasets with fast-decaying singular values (e.g., "Distance", "ImageNet") benefit from smaller $l$ in low-capacity regimes. In others, our default setting remained robust.

These results validate our heuristic while suggesting that tuning $l$ to specific data or architectures could yield further gains.

(3) On Latency and Computational Complexity

We appreciate the reviewer’s thoughtful comment regarding the lack of latency analysis. While our primary contribution lies in proposing the Binary Quadratic Quantization (BQQ) framework and demonstrating its advantages in terms of memory compression and approximation quality, we acknowledge that inference latency is a critical factor for practical deployment. Below, we address the three aspects the reviewer mentioned:

(a) Weight Loading Time: We fully agree that binary representations greatly reduce model size, which directly alleviates I/O bottlenecks during weight loading. This is particularly relevant in scenarios where memory bandwidth is the limiting factor. In such cases, BQQ's compressed binary format allows faster transfer from storage or DRAM to compute units, offering potential latency gains in loading time. Since I/O transfer time is approximately proportional to model size, reducing the model to 1/16 its original size—as in the case of 2-bit BQQ—can theoretically lead to up to 16× faster weight loading, assuming the I/O subsystem is the primary bottleneck.

(b) Low-Bit Computation Efficiency: Although binary and low-bit operations can be highly efficient on hardware that supports them (e.g., INT cores on modern GPUs), our current implementation involves casting 1-bit weights to floating-point values before matrix operations. This is due to the lack of native mixed-precision (e.g., INT-FP) kernel support on standard GPU backends. As a result, despite memory savings, the actual compute kernel is still performed in floating point, leading to no measurable improvements in raw latency under current software-hardware stacks.

However, recent work such as LUT Tensor Core (Zhiwen Mo et al., ISCA 2025) illustrates that with dedicated support for mixed-precision tensor operations (e.g., 1-bit × 16-bit), speedups over 7.5× are possible. We believe that BQQ is well positioned to benefit from such hardware innovations, and integrating with emerging low-bit inference libraries or hardware designs is a promising avenue for future research.

(c) Theoretical Computational Complexity: In terms of algorithmic complexity, the per-layer FLOPs of BQQ are on the order of $\mathcal{O}((m + n)ldp)$, where $m$ and $n$ are the dimensions of the original weight matrix, $l$ is the intermediate dimension, $p$ is the number of binary matrix stacks, and $d$ is the input dimension of the layer.

In comparison, standard scalar $p$-bit quantization requires $\mathcal{O}(mndp)$ FLOPs per layer. When $l \approx \frac{mn}{m + n}$, BQQ achieves the same asymptotic computational complexity as scalar quantization. While our current implementation does not fully leverage this due to software and hardware constraints, we emphasize that BQQ is structurally amenable to such optimizations.

We hope the reviewer recognizes that while we have not yet developed a latency-optimized implementation, our analysis highlights the conditions under which latency gains are likely to be realized. We thank the reviewer again for highlighting this important aspect and agree it is a crucial direction for future work.

(4) On the Relation to LoRA and Related Work

Thank you for pointing out the connection to LoRA. While LoRA is primarily designed for efficient adaptation of large models by introducing lightweight low-rank updates, our method targets compression of the original real-valued matrices themselves. Despite this difference in objective, there is a structural resemblance in the use of low-rank components. We agree that clarifying this distinction—and positioning BQQ within the broader context of related methods, including LoRA-based techniques and hybrid approaches such as QA-LoRA and LQ-LoRA—will help readers better understand the contributions and novelty of our work.

If accepted, we will mention the structural similarities between our method and LoRA-based approaches, such as the use of low-rank components, to help situate our method within the broader landscape of efficient model design.

Final Remarks

Once again, we thank the reviewer for their valuable insights. Your suggestions have significantly improved both our presentation and analysis.

Thank you very much for your thoughtful comments and for taking the time to carefully review our responses.

We sincerely apologize for the confusion regarding (2) in the rebuttal. While both Figure 3 in the paper and the table shown in the rebuttal are based on experiments using the same dataset, in the rebuttal table, we reported the MSE values multiplied by $10^3$ for better readability. We regret not clearly stating this, and we apologize for the oversight.

If the paper is accepted, we will make sure to incorporate the additional results into the main text or appendix, along with a clear explanation of the scaling and its relation to Figure 3.

If anything remains ambiguous or warrants further explanation, we would be more than willing to provide additional details.

Response to Reviewer Comments

We thank the reviewer for the insightful and constructive feedback. We are encouraged by the overall positive assessment and appreciate the valuable suggestions. Below, we address each comment in turn.

(1) On Mathematical Expressions

We appreciate the reviewer's careful reading of our algorithms and the helpful suggestions on improving notational consistency. We agree that the expression 0.5 in Algorithm 2, Line 3, should be more precisely written as a matrix, e.g., in boldface or an all-one matrix scaled by 0.5, as was done in Algorithm 1. If the paper is accepted, we will revise all such instances to improve clarity and consistency.

Regarding the use of clone, we understand the concern that it is Python-specific. If the paper is accepted, we will revise the pseudocode to either eliminate such implementation-specific terminology or annotate operations like deep copying consistently throughout, including Line 14 in Algorithm S.1.

(2) On Inference Computational Cost

The reviewer is absolutely right that the computational cost of DNN inference is an important consideration in quantization methods. While this point is currently discussed in Appendix A.5, we agree that it deserves to be presented directly in the main text. If the paper is accepted, we will incorporate a concise summary of the inference-time computational cost analysis into the main body to improve clarity and accessibility.

(3) On the Behavior of the Greedy Strategy

The reviewer correctly observes that in Algorithm 3, the scaling parameters $r_i$, $s_i$, and $t_i$ tend to have smaller magnitudes as the index $i$ increases. This is indeed the case. Since our method reduces the residual error in a greedy manner, later steps operate on smaller residuals, which naturally results in terms with smaller absolute values.

To further illustrate this behavior, we include an additional experimental result showing how the values of $r_i$, $s_i$, and $t_i$ decrease progressively during the greedy process. The table below summarizes this trend:

(4) On the Analogy with SVD and the Eckart–Young Theorem

We appreciate the reviewer's thoughtful question regarding the relationship between our greedy approach and the Eckart–Young theorem. As correctly observed, our greedy optimization procedure—where scaling coefficients are determined in a decreasing order of significance—bears a conceptual resemblance to the construction of low-rank approximations in SVD.

However, unlike SVD, our method does not guarantee optimality in a theoretical sense. Nevertheless, we empirically observe that it achieves a favorable trade-off between approximation error and bit-width compared to existing methods. In this sense, while heuristic, our method resonates with the spirit of the Eckart–Young theorem, suggesting that greedy, non-optimal procedures can still yield near-optimal approximations under certain conditions.

In addition, although this slightly departs from the question of greedy optimality, we have found that an upper bound on the approximation error for each subproblem in our greedy process can be derived using the Eckart–Young–Mirsky theorem and the triangle inequality:

where $\sigma_j$ and $\mathbf{W}\mathrm{svd}^{(i)}$ denote the singular values and the reconstructed matrix after applying an $l$-rank approximation to the singular value decomposition (SVD) of $\mathbf{W}^{(i)}$, respectively. Furthermore, $\mathbf{U}\mathrm{svd}^{(i)}$ and $\mathbf{V}\mathrm{svd}^{(i)}$ are the singular vectors obtained from the SVD of $\mathbf{W}^{(i)}$. Note that $\mathbf{U}\mathrm{svd}^{(i)}$ is represented as a single matrix that already incorporates the top $l$ singular values.

This bound highlights a theoretical connection between our method and classical SVD-based low-rank approximation, offering additional justification for the effectiveness of our greedy strategy.

If the paper is accepted, we plan to include this theoretical insight in the revised version, along with a more detailed discussion of its implications for total approximation quality.

Final Remarks

Once again, we sincerely thank the reviewer for the thoughtful questions, constructive suggestions, and positive evaluation. The feedback has helped us better articulate the contributions and implications of our work, and we believe it will significantly enhance the clarity and impact of the final version, should the paper be accepted.

Thank you for your detailed reading and response.

Regarding point (4), the expression $-\mathbf{W}\textrm{svd}^{(i)} + \mathbf{W}\textrm{svd}^{(i)}$ was written intentionally to make the subsequent application of the triangle inequality more transparent. However, we understand that this may have caused confusion instead, and we sincerely apologize for that.

To clarify, the derivation could be more explicitly written as follows:

$A := \sqrt{ \sum_{j = l+1}^{\min(m,n)} \sigma_j^2 }$, $\mathbf{B} := \mathrm{sgn}(\mathbf{U}\mathrm{svd}^{(i)}) \mathrm{sgn}(\mathbf{V}\mathrm{svd}^{(i)})$, $c := \frac{ \langle \mathbf{W}\mathrm{svd}^{(i)}, \mathbf{B} \rangle }{ \| \mathbf{B} \|^2 }$

If the paper is accepted, we will include a more detailed derivation and a complete proof in the camera-ready version to ensure clarity. If there are any remaining concerns or points of confusion, we would be happy to provide further clarification.

Response to Reviewer Comments

We thank the reviewer for the constructive and thoughtful comments. We are encouraged that the reviewer found our paper well-written and recognized the strength of our method in achieving state-of-the-art low-bit quantization results in both accuracy and compression.

Below, we address each of the concerns one by one:

(1) Lack of Real Speedup Measurements

We agree that measuring actual runtime performance is important for practical deployment. Our primary contribution, however, lies in proposing the BQQ framework and demonstrating its theoretical and empirical benefits in terms of memory efficiency and approximation quality. While we provide FLOPs and bit-level analyses as proxies for computational cost, real-world speedup on GPUs depends on hardware specifics and implementation details beyond FLOPs alone.

Current GPU architectures include optimized cores for integer-only (INT) operations, enabling efficient low-bitwidth computations when both operands are integers. However, because mixed-precision operations involving floating-point (FP) and integer (INT) data—such as those required when 1-bit weights are cast to floating-point formats before processing—lack dedicated optimized cores, 1-bit weights must be converted to FP for computation. As a result, the actual amount of computation performed on the GPU remains effectively the same as in unquantized models, despite reductions in memory access and data transfer. Consequently, despite reduced memory bandwidth and data transfer, our GPU implementation did not show latency improvements.

Recent research, for example LUT Tensor Core (Zhiwen Mo et al., ISCA 2025), shows that specialized hardware supporting optimized tensor operations between 1-bit and 16-bit data types can achieve over 7.5× speedups compared to standard GPU execution. Such innovations suggest that substantial acceleration is possible with appropriate hardware support for mixed-precision computations.

At present, we have not yet developed a hardware-optimized implementation tailored to BQQ and thus cannot provide actual runtime acceleration results. We sincerely wish the reviewer to value our theoretical contribution on memory and computation cost efficiencies. Implementing and evaluating practical runtime improvements on optimized hardware is an important direction for future work.

(2) Evaluation Limited to Vision Transformer

We thank the reviewer for pointing out that the experiments were conducted only on end-to-end image classification models (i.e., ImageNet) and for suggesting evaluation on large language models (LLMs) to strengthen the generality of our method. We agree that BQQ is broadly applicable to a wide range of architectures and tasks—including not only various neural networks but also non-neural applications such as Approximate Nearest Neighbor (ANN) search, where binary representations are highly beneficial.

In this work, we focused on Vision Transformers (ViTs), which are known to be particularly fragile to quantization—especially under low-bit settings where small perturbations in weights can lead to significant degradation. Demonstrating strong performance on ViTs thus highlights the robustness of BQQ in one of the most challenging quantization scenarios. While applying BQQ to LLMs is a promising direction, the sheer size of these models makes our current optimization pipeline computationally expensive, and full-scale evaluation remains an ongoing effort.

That said, to provide the reviewer with more concrete evidence of BQQ’s broader applicability, we additionally applied our method to compact LLMs suitable for edge devices. Specifically, we evaluate our method on Qwen2.5-0.5B, an efficient and high-performing LLM designed for edge deployment. Compared with GPTQ, which is a standard quantization method, BQQ demonstrates competitive performance in terms of perplexity and 0-shot downstream task accuracy under the same bit-width, as summarized in the table below.

However, we emphasize that the current work proposes BQQ as a general binary quantization framework, rather than a dedicated quantization technique for neural networks. In our implementation, BQQ quantizes weights by minimizing reconstruction error in the weight space, without relying on output-based error signals that are commonly used in many neural network quantization methods. This implies that BQQ does not yet exploit task-specific loss functions or activation statistics. We therefore believe that adapting BQQ to incorporate such feedback—especially minimizing the downstream output error—could lead to significant further improvements in model performance, and this represents a highly promising avenue for future research.

(3) Lack of Analysis on Intermediate Dimension and the Number of Stacks ($l$ and $p$ in Eq.(5))

We appreciate the reviewer’s insightful comment regarding the need for a more detailed analysis of BQQ’s internal parameters, such as the intermediate dimension $l$ and the number of binary matrix stacks $p$. In our main experiments, we fixed the intermediate dimension as: $l = \mathrm{round}\left(\frac{mn}{m + n}\right)$ to maintain the same number of elements as the original $m \times n$ matrix, and adjusted the model size by varying the number of stacked binary matrices $p$. However, we agree that varying $l$ under fixed model size constraints can provide a deeper understanding of the trade-off between compression and reconstruction accuracy.

To address this point, we conducted additional experiments where we fixed the total model size and varied the ratio between $l$ and $p$. We used the same set of matrix datasets as in Fig. 3. The results are summarized in the following table, where the “$l$-scale” denotes the relative value of $l$ normalized by: $\mathrm{round}\left(\frac{mn}{m + n}\right)$

Our findings show that the optimal value of $l$ can depend on the spectral characteristics of the data. For instance, in datasets with rapidly decaying singular values such as Distance and ImageNet (see Fig. S.1), a smaller $l$ (e.g., $l = 0.5$) yielded better performance at small model sizes. On the other hand, in other cases, using $l = 1$ produced consistently strong results. These results suggest that while our default choice of: $l = \mathrm{round}\left(\frac{mn}{m + n}\right)$ works well in general, adapting $l$ to the specific data characteristics could lead to improved trade-offs between reconstruction error and memory usage.

We thank the reviewer again for prompting this important analysis.

(4) Runtime and Optimization Iterations

Thank you for your insightful question. The reason why higher-bit quantization takes longer is due to our greedy optimization approach applied to each binary matrix component. In our main experiments, the intermediate dimension is fixed as: $l = \mathrm{round}\left(\frac{mn}{m+n}\right)$ and the model size is controlled by the number of stacked binary matrices $p$.

Importantly:

• When $p = 1$, only one binary matrix optimization subproblem is solved.
• When $p = 2$, two such subproblems are solved sequentially.
• Therefore, increasing $p$ directly increases the number of optimization steps, proportional to the number of binary matrices. As a result, higher-bit quantization leads to longer computation times.

Regarding the runtime results presented in the appendix:

• All experiments were conducted with a fixed 50,000 optimization iterations.
• Your understanding is correct that lower-bit quantization tends to be more sensitive, and generally benefits from a larger number of iterations.

As shown in the supplementary material:

• For 2-bit quantization, more iterations are required to reach satisfactory accuracy.
• For 3-bit and 4-bit quantizations, accuracy plateaus well before 50,000 iterations, indicating that fewer iterations suffice to maintain stable performance.

Final Remarks

We sincerely appreciate the reviewer’s thoughtful feedback and hope the final version of our work will address these concerns more fully.

Response to Reviewer Comments

We sincerely appreciate the positive feedback and constructive suggestions. Below, we address the main concerns raised in the review.

(1) Comparison with Other Quantization Methods

We thank the reviewer for suggesting additional comparisons with a broader range of quantization methods. While our original experiments focused on scalar quantization baselines, we agree that extending the evaluation to a more diverse set of techniques can provide valuable insights.

In response, we conducted additional experiments that compare our method against the following approaches:

• Vector Quantization using k-means (VQ)
• $E_8$ Lattice Vector Quantization (LVQ)
• Hadamard Transform (HT) + Uniform Quantization (UQ)
• Discrete Cosine Transform (DCT) + UQ
• JPEG (DCT + quantization + entropy coding)

To ensure fair comparisons, we clipped all benchmark matrices to the nearest lower power-of-two size. This process was necessary to enable the application of HT, which requires input dimensions to be powers of two. All methods were evaluated on these clipped matrices for consistency.

For VQ, we also evaluated a variant using quantized centroids (VQ+UQ) to account for codebook storage.

In LVQ, we used the $E_8$ lattice with 240 norm-$\sqrt{2}$ centroids, representing each 8-dimensional input as a linear combination of $n$ centroids, each stored in 8 bits, making it comparable to $n$-bit scalar quantization in memory.

For the HT and DCT methods, we experimented with two levels of energy preservation by retaining only a portion of the low-frequency components after transformation. Specifically, we evaluated settings that preserve approximately 50% and 100% of the transformed components, denoted as HT50, HT100, DCT50, and DCT100, respectively. Uniform quantization (UQ) was then applied to the retained components for compression.

We summarize the performance of different methods across 5 datasets. For readability, we show results as “Size [KB] / MSE $\times 10^{3}$” pairs.

The above results on the five benchmark matrices show that BQQ outperforms most other compared methods—including VQ, LVQ, and transformation-based schemes—on the majority of datasets. The only exception is JPEG on ImageNet, where BQQ performs slightly worse. These results demonstrate the strong empirical performance of BQQ and underscore its practical advantages over a diverse and competitive set of baselines.

Methods such as the Hadamard Transform (HT) and Discrete Cosine Transform (DCT), when used with quantization, serve as preprocessing steps to decorrelate or concentrate signal energy. Their compression efficiency and accuracy depend on the subsequent quantizer. From a methodological perspective, these hybrid schemes are not standalone quantizers and thus not directly comparable to BQQ, which is itself a quantizer. Nevertheless, BQQ's ability to outperform these popular combinations is notable and underscores its effectiveness. Moreover, combining such transforms with BQQ may further improve compression, which we consider a promising direction.

We will incorporate a formal version of this table into the main paper if the paper is accepted.

(2) Upper Bound of BQQ

We appreciate the reviewer's insightful comment regarding the lack of theoretical approximation bounds. In response, we analyzed a simplified version of Equation (5) by assuming $\beta_i = -0.5\alpha_i$, $\delta_i = 1$, and $\gamma_i = -0.5$. Although this setting does not yield a closed-form optimal solution, we found that an approximate solution can be constructed by aligning the binary components with the sign patterns of the singular vectors obtained from a truncated SVD, combined with appropriate scaling. This approximation can then be used to provide an upper bound on the error of BQQ. Specifically, we found an upper bound on the residual error in the greedy error minimization process of BQQ, denoted by $L_{\text{sub}}^{(i)}$, as given below. This bound is derived based on the Eckart-Young-Mirsky theorem and the triangle inequality.

where $\sigma_j$ and $\mathbf{W}\mathrm{svd}^{(i)}$ denote the singular values and the reconstructed matrix after applying an $l$-rank approximation to the singular value decomposition (SVD) of $\mathbf{W}^{(i)}$, respectively. Furthermore, $\mathbf{U}\mathrm{svd}^{(i)}$ and $\mathbf{V}\mathrm{svd}^{(i)}$ are the singular vectors obtained from the SVD of $\mathbf{W}^{(i)}$. Note that $\mathbf{U}\mathrm{svd}^{(i)}$ is represented as a single matrix that already incorporates the top $l$ singular values.

This bound is connected to the magnitude of the discarded singular values, which explains the empirical observation (e.g., Figure 3 and Figure S.1) that BQQ outperforms better on matrices with rapidly decaying singular spectra, and underperforms when the low-rank approximation leaves significant residual energy.

(3) Greedy Optimization and Its Computational Cost

We acknowledge the limitation of using a greedy strategy. However, solving polynomial unconstrained binary optimization (PUBO) problems to global optimality quickly becomes intractable as problem size grows. In BQQ, joint optimization would involve $(m \times l + l \times n) \times p$ binary variables for $m \times l$ and $l \times n$ matrices and $p$ BQQ terms, leading to significant memory and time costs even for moderate-sized matrices.

To mitigate this, we adopt a greedy residual minimization strategy inspired by Binary Coding Quantization (BCQ), which incrementally reduces the approximation error by adding one component at a time. In contrast to global optimization methods that require re-running the entire procedure from scratch when the number of stacks ($p$ in Eq.~(5)) is modified, the greedy approach preserves previously optimized components. This enables efficient and flexible adjustment of model capacity without discarding past computations. Nonetheless, as the reviewer rightly noted, applying this transformation to many matrices in neural networks incurs non-trivial computational costs. Reducing this overhead remains an important direction for future work.

While we acknowledge that our current optimization strategy is relatively simple and could certainly be improved with more sophisticated or learned methods, we would like to emphasize that the core contribution of this work lies in the formulation of the BQQ framework itself. The novelty of BQQ as a binary matrix approximation scheme is independent of the specific optimization algorithm used, and we hope the reviewer understands that our primary goal is to establish the expressiveness and potential of the framework, rather than to claim advances in optimization methodology. We will further clarify this point and add future directions regarding optimization improvements in the final manuscript.

Final Remarks

We hope these responses adequately address the reviewer's concerns. We thank the reviewer again for the valuable feedback, which has helped strengthen the paper both theoretically and experimentally.

Thank you very much for your reconsideration and for raising your score. We truly appreciate your thoughtful evaluation and are glad that the additional experimental and theoretical results helped clarify our contributions. If the paper is accepted, we will make sure to incorporate these results into the final version as promised.