Response to Weakness 1: Classifier-free guidance scales

Thank you for pointing out this important concern. We set classifier-free guidance scales as 1.5 in all the experiments in our paper, following the origin DiT work [1].

[1] Scalable diffusion models with transformers. In CVPR, 2023.

Response to Weakness 2: W4A8 quantization

Thanks for the valuable feedback. Here, we would like to emphasize the challenging W4A8 quantization and the contribution of our PTQ4DiT.

a. Challenge of W4A8 quantization

Prior studies [1, 2] have revealed that W4A8 quantization presents non-trivial challenges due to the severe information loss in low-bit model weights and have attempted to mitigate the degradation of W4A8 U-Net-based diffusion models. However, we observed that these methods result in substantial performance loss when applied to W4A8 Diffusion Transformers (DiTs), indicating the significant difficulty of low-bit quantization for DiT architectures.

b. Contribution of PTQ4DiT

This is the first time a PTQ method has enabled high-quality generation at W4A8 bit-widths for DiTs, paving a promising path for future research in this field. Specifically, our work delves into DiT quantization and identifies two key challenges (Section 3). Then, we develop PTQ4DiT to address these challenges. Encouragingly, we find that PTQ4DiT consistently shows significant performance improvements over mainstream methods, as detailed in Tables 1 and 2 of our paper. Figure 5 further demonstrates that PTQ4DiT facilitates high-quality image generation despite the difficulty inherent in the lower bit-width of W4A8.

[1] Q-Diffusion: Quantizing Diffusion Models. In ICCV, 2023.

[2] PTQD: Accurate Post-Training Quantization for Diffusion Models. In NIPS, 2023.

Dear Reviewer a8C3,

We sincerely appreciate your prompt response and are grateful that you found our rebuttal beneficial. Thank you once more for your valuable feedback in enhancing our submission.

Thank you very much for your acknowledgement of our work and your constructive feedbacks and suggestions.

Response to Weakness 1: Statistical experiments for the important property

We validate the important complementarity property by additional statistical experiments on the well-established Jaccard similarity [1].

a. Jaccard similarity

To quantitatively demonstrate the complementarity property (large values do not coincide in the same channels of activation and weight), we measure the Jaccard similarity [1] of salient activation and weight channels:

where $S_A$ is the set of indices of salient activation channels, $S_W$ is the set of indices of salient weight channels, and $|S|$ calculates the number of elements in set S. In this way, a lower $Jaccard(S_A, S_W)$ reflects stronger complementarity.

b. Detecting salient channels

To accurately construct the sets $S_A$ and $S_W$, we utilize a robust statistical outlier detector, Interquartile Range (IQR) [2], to identify the salient channels among all channels. This method identifies data points that lie significantly outside the middle 50% of the distribution as outliers. We perform IQR detection on channels' maximal absolute values to identify the salient channels.

c. Statistical experiment

We randomly select 100 samples for ImageNet 256x256 generation with total timestep T = 250. We evaluate the average Jaccard similarities at t = $\frac{1}{4}T, \frac{1}{2}T,$ and $\frac{3}{4}T$. The results are averaged over 100 samples and linear layers in DiTs:

Our finding are:

(I) We obtain relatively low Jaccard similarities (note that $Jaccard \in [0, 1]$), suggesting a significant complementarity between salient activation and weight channels. The complementarity property motivates our Channel-wise Salience Balancing (CSB) to redistribute extreme values among non-overlapping salient channels.

(II) Different timesteps t exhibit various Jaccard similarities, aligning with the key observation in our paper: the temporal variation in salient channels. Such variability further indicates the necessity of our proposed Salience Calibration method (SSC) for various timesteps.

[1] Distance between sets. Nature 1971.

[2] Outlier detection: Methods, models, and classification. ACM Computing Surveys (CSUR) 2020.

Response to Weakness 2: Is the geometric mean well-designed?

a. The effectiveness of geometric mean

The geometric mean was selected due to its suitability for the multiplicative relationship between activations and weights (Eq. (3) in our paper) and its ability to balance and minimize the influence of extremely large values, which is critical in our quantization method.

b. The uniqueness of geometric mean

Mathematically, alternative means could include the arithmetic, quadratic, or harmonic mean. However, interestingly, we found that the geometric mean is uniquely capable of ensuring the mathematical equivalence of the balancing transformation (as expressed by Eqs. (12), (15), (16) in our paper). Here, we provide a brief demonstration:

This essential property offers the feasibility of reparameterization, thereby eliminating extra computation overhead of the balancing transformation during inference.

Response to Weakness 3: Generality of PTQ4DiT

Please refer to General Response 1: Generality of PTQ4DiT.

Response to Weakness 4: Work only with linear layers

a. Extending PTQ4DiT to other structures

Although our method is derived from the quantization of linear layers, the underlying idea of balancing transformation can be seamlessly extended to other structures, such as convolutional layers. This can be achieved by reformulating the convolution operation into a matrix multiplication between activations and weights [1], which is common in the practical implementation of convolution [2].

b. The reason for focusing on linear layers

The linear layers in Transformers incur significant computational and memory overhead due to the large matrix multiplications [3, 4, 5, 6], representing the primary efficiency bottleneck in Transformer models including Diffusion Transformers (DiTs).

[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.

[2] Optimizing hardware accelerated general matrix-matrix multiplication for cnns on fpgas. Transactions on Circuits and Systems 2020.

[3] PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization. ECCV 2022.

[4] RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers. ICCV 2023.

[5] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. ICLR 2024.

[6] QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models. ICLR 2024.

Response to Weakness 1: Effectiveness of SSC

We clarify the innovation of CSB and provide additional evidence of the efficacy of SSC.

a. Innovation of CSB

While CSB shares the general concept of distribution re-scaling and re-parameterization with Smoothquant, it has unique innovation. Our CSB is designed for quantizing DiTs, whereas existing methods, such as Smoothquant, focus on quantizing LLMs. The quantization of DiTs presents unique challenges due to the extreme values in both activations and weights. In contrast, Smoothquant identifies outliers only in activations within LLMs and addresses this by migrating outliers from activations to weights. Fortunately, the extreme values do not occur in the same channels of activation and weight in DiTs. Our CSB is then derived from this complementarity to balance the salience between activation and weight, alleviating quantization errors for both.

b. More SSC ablation

To validate the importance of SSC, we conduct additional experiment on the challenging ImageNet 512x512 with W4A8 using 50 timesteps:

The results demonstrate the significant impact of SSC, as it further enhances the benefits introduced by CSB.

c. Direct evidence of SSC's effect

Please see General Response 2: Experiment on direct evidence.

Response to Weakness 2: Overheads

We clarify that $B_\rho^X$ and $B_\rho^W$ are not time-varying. As defined in Eq. (10), $s_\rho(X^{(1:T)})$ aggregates activation saliences across all timesteps. Therefore, the resulting refined salience is a function of the number of timesteps $T$ and does not depend on any specific timestep $t$. We then formulate $B_\rho^X$ and $B_\rho^W$ based on $s_\rho(X^{(1:T)})$ and $s(W)$, which are not time-varying.

Thus, we only perform reparameterization once before quantization, without additional memory cost for model weights.

Response to Question 1: Recalibration

While SSC is orthogonal to solvers and models, it does depend on the number of timesteps ($T$ in Eq. (10)) due to its design of aggregating channel salience across T. Encouragingly, SSC is capable of generalizing across different T without recalibration. To verify its generalizability, we conduct two sets of experiments on ImageNet 256x256 with W8A8:

(I) Downsampling: We calibrate and quantize the model using PTQ4DiT with 250 timesteps and evaluate this model using 100 and 50 timesteps.

(II) Upsampling: We calibrate and quantize the model using PTQ4DiT with 50 timesteps and evaluate this model using 100 and 250 timesteps.

The results show that both downsampling and upsampling does not significantly effect the performance, demonstrating the generalizability of SSC. We conjecture that this is due to two factors:

(I) Inherent generalizability of DiTs. Previous studies [1,2,3] indicated that diffusion models possess strong ability to generalize across timesteps. Consequently, models trained with 250 timesteps can be directly applied to 100 and 50 timesteps with acceptable performance degradation. This interpolation ability is also evidenced by the performance of FP DiTs with 100 and 50 timesteps in Tables 1 and 2 of our paper (note that the original DiT work [4] only released the FP model for 250 timesteps).

(II) Effect of PTQ. PTQ is formulated as a numerical problem and does not re-train the original models, which will not significantly affect the inherent interpolation ability if the quantization error remains low enough.

[1] Diffusion Models Beat GANs on Image Synthesis. NIPS 2021.

[2] Improved denoising diffusion probabilistic models. ICML 2021.

[3] Post-training quantization on diffusion models. CVPR 2023.

[4] Scalable diffusion models with transformers. CVPR 2023.

Response to Question 2: Reparameterization

We address 3 types of linear layers exhibiting significant channel salience: FC1, Projection1, and Projection2. We design reparameterization strategies for these layers. Specifically, FC1 follows the 'Post-adaLN' scenario. Thus, the Balancing Matrices can be integrated to the adaLN before Pointwise Feedforward. We further absorb them into MLPs regressing $\gamma_2$ and $\beta_2$, a process detailed in Appendix D.

We illustrate the integration strategies in Figure 7 and discuss them in the caption. We apologize for causing any confusion.

Response to Question 3: Quantization details

a. Are all layers in the DiTs quantized?

We quantize all layers in the DiT models. The time embedding linear layers and attention matmuls are quantized.

b. Quantization optimization

Following Q-Diffusion, we involve gradient-based optimization for the quantization parameters, which has similar PTQ cost as that of Q-Diffusion (about 19 hours on a single NVIDIA A6000 GPU) since we do not introduce additional parameters.

Response to Question 4: Explaining the statement

Correlation salience refers to the degree of alignment between activation salience $s(X^{(t)})$ and weight salience $s(W)$. A low correlation indicates that channels with extreme values in activations are not the same as those in weights, and vice versa.

Leveraging this complementarity, we propose SSC to prioritize timesteps with lower correlation between activation and weight saliences. This approach helps distribute the quantization impact more evenly across channels and timesteps, improving overall quantization performance.

Dear Reviewer jrKh,

Thank you for your thorough review and constructive suggestion. We are grateful for your acknowledgment of our efforts to address the concerns. Your expertise has significantly contributed to the enhancement of our work.

Weakness 1: More experiments

a. W8A8 ImageNet 512x512

We evaluate PTQ4DiT on ImageNet 512x512 with W8A8, compared against strong Diffusion PTQ methods, including Q-Diffusion and PTQD:

PTQ4DiT consistently outperforms mainstream methods across various timesteps. Figure 8 provides the images generated by PTQ4DiT in this setting, which closely mirror the FP model.

b. New DiT model

Please refer to General Response 1: Generality of PTQ4DiT.

Weakness 2: Novelty discussion

a. The general concept of re-parameterization

The fundamental idea of re-parameterization in quantization is to absorb the extra affine transformation into adjacent layers, thereby enabling the construction of quantization-friendly distributions without incurring extra inference costs.

b. Existing works that use re-parameterization

Several quantization methods benefit from re-parameterization. For example, RepQ-ViT [1] uses scale re-parameterization to avoid the extra costs of channel-wise and $log\sqrt2$ quantizers for post-LayerNorm and post-Softmax activations in ViTs. OS+ [2] shifts and scales activations to address the asymmetry and concentration issues in LLMs and re-parameterizes them to avoid extra costs.

c. Unique innovation and contribution of PTQ4DiT

While PTQ4DiT shares the general concept of scale re-parameterization, its unique innovation lie in 3 aspects:

(I) We explore the quantization of DiTs, while existing works focus on ViTs or LLMs.

(II) We delve into the complementarity of activations and weights, while existing works [1,2] focus solely on activations. Specifically, we find that DiTs exhibit extreme values in both activations and weights, yet these extreme values do not simultaneously occur in the same activation and weight channels. We leverage this property to redistribute salience between activation and weight, alleviating quantization errors for both.

(III) Unlike the static re-parameterization in existing works [1,2], PTQ4DiT dynamically adapts to the temporal variability of channel salience, a special characteristic of DiTs.

[1] RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers. ICCV 2023.

[2] Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. EMNLP 2023.

Weakness 3&4: Direct evidence and Motivation

a. Direct evidence

Please refer to General Response 2: Experiment on direct evidence.

b. Motivation of equilibrium

Unlike ViTs and LLMs which exhibit outliers only in activations, extreme values are presented in both activations and weights in DiTs. Fortunately, they do not coincide in the same activation and weight channels. Such complementarity property inspired us to seek a balance between activations and weights, where their extreme values can be mitigated. This balance is the essence of equilibrium. To realize this, we adopt geometric mean as it is suitable for the multiplicative relationship between activations and weights.

c. Motivation of inverse ρ

The motivation of inverse Spearman's ρ stems from the need to balance salient weights with salient activations from different timesteps, which have various degrees of complementarity.

Spearman's ρ measures the rank correlation between two variables (the statistical dependence between their rankings). A high ρ suggests that two variables exhibit higher values simultaneously, which inversely reflects the complementarity. In our context, the high ρ implies that the same channel tends to have extreme values in both activations and weights, hindering the salience balancing.

To counteract this, our SSC inversely weights timesteps with high ρ between activation and weight salience, thereby prioritizing timesteps with more significant complementarity. This approach helps distribute the extreme values more evenly across channels and timesteps, improving overall quantization performance.

Question 2: Post-softmax layer

We quantize the post-softmax layer.

Question 3: Generation setting

a. Clarification on FID-10K

The origin DiT paper [1] achieves a 2.27 FID-50K on ImageNet 256x256.

In our work, the only difference is that we use FID-10K, following pioneering studies on diffusion such as [2,3,4]. Specifically, the work on IDDPM by OpenAI [3] indicates that while FID-10K may result in a slight increase, it significantly reduces computational resource demands.

Thus, we use FID-10K to facilitate more experiments for comprehensive evaluation. For fair comparisons, all baseline methods and our PTQ4DiT are evaluated using the same metrics.

b. Experiment on 50K samples

We perform an additional experiment generating 50K samples on ImageNet 256x256 as in [1], evaluating W8A8 quantization. The results indicate that PTQ4DiT effectively recovers the performance and closely matches the FP model:

In this setting, generating 50K samples takes around 167 hours on an NVIDIA RTX A6000, which is less feasible for extensive comparisons.

[1] Scalable diffusion models with transformers. CVPR 2023.

[2] Diffusion Models Beat GANs on Image Synthesis. NIPS 2021.

[3] Improved denoising diffusion probabilistic models. ICML 2021.

[4] Post-training Quantization on Diffusion Models. CVPR 2023.