Outlier-Aware Post-Training Quantization for Discrete Graph Diffusion Models

We very much appreciate your positive comments of our paper.

Q1: Some quantization methods [a,b,c,d], low-rank decomposition and importance-based weight (e.g. PB-LLM) dividing methods should be discussed.

A1: We sincerely appreciate your valuable suggestion. Due to the page limit of the rebuttal, we have included the comparison with these related works in this link. We will further expand our related work section to incorporate a detailed discussion of these studies.

Q2: The proposed method shows no advantage over the BF16 baseline in runtime memory.

A2: Thank you for your valuable comment. Our Bit-DGDM shows notable memory usage advantages compared to the BF16 baseline. In Table 1, on the QM9 dataset, Bit-DGDM only requires 2.3GB memory compared to BF16's 3.4GB usage, with a 32% reduction. The results are consistent in other datasets. Since DGDM is primarily a computation-intensive model rather than a large-parameter model, our main focus remains on improving inference speed while maintaining performance. We will incorporate memory analysis in our experimental results.

Q3: The idea of dividing weights and activations into difficult to quantize parts and easy to quantize parts is common.

A3: Thank you for your feedback. Though the idea of separating weights and activations has been explored before, our method introduces several key innovations that distinguish it from prior work. (i) For easy-to-quantize weights, we are the first to propose ill-conditioned low-rank weight decomposition, enabling stable decomposition into low-rank components even in the presence of significant outliers. (ii) For difficult-to-quantize weights, we propose the use of α-Sparsity, a structured sparsity property that facilitates subsequent inference acceleration. (iii) For activation outliers, our approach eliminates the need for calibration data to select thresholds, making it more practical.

Q4: Whether DGDM has practical and wide application is worth discussing.

A4: Thank you for your insightful question. Compared to Gaussian denoising diffusion models, DGDM possesses unique capabilities in modeling discrete data, enabling multiple real-world applications, particularly in biological and chemical tasks. In this work, we validate DGDM’s effectiveness through two critical applications: molecular generation and inverse protein folding. Furthermore, our proposed Bit-DGDM is specifically designed to facilitate the practical deployment of DGDM under real-world computational constraints.

Q5: What theoretical advancements the proposed ill-conditioned low-rank decomposition has compared to previous work.

A5: The key theoretical advancement of our ill-conditioned low-rank decomposition lies in its fundamental difference from conventional SVD-based methods when handling matrices with significant outliers. In standard SVD, the presence of outliers introduces several critical limitations [1,2]. They distort the singular value spectrum by amplifying small singular values associated with high-frequency noise components, consequently degrading the signal-to-noise ratio in the resulting low-rank approximation. This distortion manifests most prominently in the impaired rank selection capability, where the outlier-contaminated singular value spectrum exhibits slowed decay, making traditional truncation criteria unreliable. Built on prior ill-conditioned decomposition works [3,4], our method establishs rigorous recovery guarantees for the ill-conditioned low-rank composition of weights. The theoretical framework ensures stable outliers isolation properties even under significant outlier, while simultaneously maintaining better control over the dense matrix condition number throughout the decomposition process.

[1] Estimating the number of hidden neurons in a feedforward network using the singular value decomposition. 2006.

[2] Robust PCA via outlier pursuit. Neurips, 2010.

[3] Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. JMLR, 2021.

[4]Learned robust PCA: A scalable deep unfolding approach for high-dimensional outlier detection. Neurips, 2021.

Q6: What is the basis of rank selection for a low-rank components?

A6: Thank you for raising this important question. The selection of rank is based a systematic trade-off analysis between computational efficacy and precision. Through experiments on CATH dataset, we observed that low ranks (rank=8) led to significant degradation in graph generation quality, and higher ranks (rank=32) provided marginal quality improvements while incurring substantial latency overhead. We will include detailed studies on the rank selection in the revised manuscript.

We highly appreciate your positive reviews and constructive suggestions.

Q1: These methods [1,2,3] should be discussed in this context to provide a more comprehensive comparison.

A1: Thank you for your constuctive suggestion. We note that these methods [1,2,3] were designed for image diffusion models (IDMs). Specifically, [1,2] optimize quantized weights by minimizing the MSE between quantized and full-precision continuous outputs. [3] introduces time-step specific encoding for image diffusion models. However, these approaches cannot be directly applied to Discrete Graph Diffusion Models (DGDMs) due to fundamental incompatibilities. As shown in Remark 3.1, in DGDMs, the intermediate node attributes and graph structures are obtained through discrete sampling from categorical distributions. While IDMs relay on Gaussian noise and continuous denoising processes. Our method advances existing method in three innovations. (i) Recognizing the significant outliers in model weights, we first propose an ill-conditioned low-rank weight decomposition. This contrasts with basic SVD approach, allowing our method to achieve better numerical stability. (ii) For the residual component derived from the raw weight and low-rank decomposition, our method enforces α-sparsity, enabling efficient sparse matrix multiplication during inference. (iii) For activation outliers, our approach eliminates the need for calibration data to select thresholds, making it more practical.

We will add detailed discussion of these works in our related work. We sincerely appreciate your valuable feedback.

[1] Q-diffusion: Quantizing diffusion models. In CVPR, 2023.

[2] Post-training quantization on diffusion models. In CVPR, 2023.

[3] Temporal dynamic quantization for diffusion models. In Neurips, 2023.

Q2: In lines 275--277, the authors state that the matrix is ill-conditioned but provide limited explanation of its impact. It would be helpful to clarify how this ill-conditioning affects the convergence of SGD and whether it introduces optimization difficulties. Additional theoretical or empirical analysis would strengthen this claim.

A2: We sincerely appreciate the reviewer's insightful suggestion. To better illustrate how ill-conditioning affects the optimization of Eqn. (9) ($\|LR^\top + S_W - W\|_F^2$), we have conducted additional empirical analysis examining the relationship between the number of outliers and the reconstruction error (measured by Frobenius norm between the reconstructed matrix and the original matrix). These results are provided in this link. We observe that the reconstruction error (Frobenius norm) increases monotonically with the number of outliers. This correlation indicates that stronger ill-conditioning (caused by more outliers) leads to greater optimization difficulties in matrix reconstruction. Moreover, in this link, we present a comprehensive derivation of these two scale terms $(R^\top R)^{-1}$ and $(L^\top L)^{-1}$ in Eqn.(13).

We will include a more detailed description of this mechanism in the revised version to better clarify these points. Thank you for this valuable suggestion to improve the clarity of our work.

Q3: It would be beneficial for the authors to provide a more detailed analysis of the computational complexity and conduct experiments to evaluate the actual impact on efficiency.

A3: We sincerely thank the reviewer for raising this important point regarding the computational cost of our method. We agree that the additional multiplications introduced to compensate for quantization errors incur some computational overhead, but this overhead is marginal in practice.

To evaluate this, we conducted an ablation study (Figure 4 and Sec 5.4). The results show that the ablation variant (iii) Bit-DGDM-$S_XS_W$, which removes multiplications for sparse components, achieves only a modest speed improvement (from 2.5× to 2.6×) but suffers significant degradation in generation quality, as evidenced by the perplexity increase (4.5 → 4.7). This performance decline highlights the critical role of these operations in mitigating high-magnitude outliers. Furthermore, compared to threshold-based alternatives (variant (ii)), our method not only preserves competitive generation quality but also delivers better inference speed (2.5× vs. 2.2×). This efficiency gain stems from our use of ill-conditioned low-rank decomposition. Collectively, these experiments demonstrate that the additional computational overhead is negligible and does not substantially impact inference speed.

In the revised version, we will expand our empirical analysis to better demonstrate the impact of each proposed component in our work. Once again, we deeply appreciate this valuable feedback, which will undoubtedly help improve the clarity and rigor of our work.

We very much appreciate your constructive comments. For your concerns:

Q1: The font in the figures of the paper should be consistent to make it easier to read.

A1: Thank you for your helpful feedback. We have updated all figures to use Times New Roman font, ensuring formatting consistency. The updated figures are avaible here: https://anonymous.4open.science/r/ICML_Refined_Figures_9670

Q2: The time complexity analysis should be extensively introduced in the main paper.

A2: We appreciate your valueble suggestion. Due to space limitations in the main manuscript, we have included the detailed complexity analysis in Appendix D. In our future version, we will add a concise summary of the key complexity results in Sec 4.3 of the main paper.

Q3: The time cost for the quantization of DGDMs.

A3: Thank you for your insightful question. As detailed in Appendix F.3, the quantization process demonstrates practical efficiency. Specifically, for the DiGress model on the QM9 dataset, the ill-conditioned decomposition requires 13.6 minutes and memory usage remains manageable at 2.1 GB. We emphasize that this computational overhead is highly acceptable given the significant inference acceleration achieved through quantization.

We very much appreciate your positive comments and constructive suggestions.

Q1: What are the differences between the proposed method and SVDQuant?

A1: Thank you for your insightful question. Our method introduces two key innovations over SVDQuant. (i) Recognizing the presence of significant outliers in model weights, we propose an ill-conditioned low-rank weight decomposition method. This contrasts with SVDQuant's conventional SVD approach, allowing our method to achieve better numerical stability. (ii) For the residual component derived from the raw weight and low-rank decomposition, while SVDQuant maintains a dense structure, our method enforces $\alpha$-sparsity. This constraint ensures that no more than $\alpha$ proportion of elements in each row and column are non-zero, enabling efficient sparse matrix multiplication during inference. Consequently, our approach not only preserves precision but also achieves consistent acceleration, whereas SVDQuant’s dense residuals incur higher computational costs.

Q2: How significant is the tradeoff between the efficiency and effectiveness of the proposed method? This question is related to the design of the method as it contains high-precision multiplications in addition to the low-bit quantized operations.

A2: Thank you for your question. The trade-off between computational efficiency and model effectiveness is a central consideration in our model design. Our experimental results (Figure 4) demonstrate that while the high-precision multiplications do introduce some computational overhead, they are essential for maintaining the quality of generated graphs in precision-critical applications like molecular generation and protein inverse folding. Specifically, while removing the multiplication operations for sparse componenets(as in variant (iii) Bit-DGDM-$S_XS_W$) yields a slight speed improvement (2.5× → 2.6×), it comes at a significant cost in generation quality, as seen in the perplexity degradation (4.5 → 4.7). This confirms that the additional operations play a crucial role in handling high-magnitude outliers effectively. Moreover, compared to threshold-based variant (ii) removing weight outliers through thresholds, our method not only maintains competitive generation quality but also achieves better inference speed (2.5× vs. 2.2×). This gain stems from our use of ill-conditioned low-rank decomposition, which remains computationally efficient and model effective in significant outlier scenarios.

Q3: The reviewer suggests using Times font in the figures helps to make the paper more consistent in format.

A3: Thank you for your constructive suggestion. We have revised the font in all figures to Times font to ensure consistency with the paper’s format. The updated figures are available at the following link: https://anonymous.4open.science/r/ICML_Refined_Figures_9670. Please let us know if you have any additional feedback. We would be happy to make further improvements.