BiDM: Pushing the Limit of Quantization for Diffusion Models

Thank you for your detailed review of our work. Here are our responses to your concerns:

Q1: The experimental ...

Thank you for your suggestion. We have added experiments and discussions on EfficientDM. You can check the Global Rebuttal (1) for more details.

Q2: Figure.4 ...

Thank you for your suggestion. We have already placed this exploration in the manuscript. Table.7 shows that $L_2$ represents the results using MSE Loss as the distillation loss. $L_{SPD}$, with an FID of 22.74, significantly outperforms MSE Loss, which has an FID of 26.07, demonstrating the suitability of SPD for DM.

Q3: The proposed ...

It should be noted that during inference, BiDM requires only a very small number of additional floating-point additions for the connections across timesteps compared to the classic binarization work XNOR-Net, and there are no differences in the majority of calculations, such as convolutions. XNOR-Net's original paper already stated that it achieves a 58× speedup in convolution operations in its abstract.

We understand your concerns about efficiency, but performing a floating-point convolution with a depth of 1 for scaling factors requires only a very small amount of computation, and the overhead for averaging matrix $A$ is also minimal. Statistical results show that this portion accounts for only 0.08% of the total OPs in binarized DM. Therefore, BiDM remains efficient during inference.

Q4: During inference ...

BiDM fully binarizes components such as convolutions and QKV, and the remaining components have a very minimal impact on computation. Specifically, the computational cost of this part accounts for only 0.38% of the total OPs in the binarized DM.

In particular, we fully binarized linear/conv1d layers with weights and MMs involving the input x in AttentionBlock, while MMs involving intermediate results (such as proj_out) were performed using floating-point operations. This decision was based on our observation that the latter does not involve weight storage, involves very little computation, and has a relatively noticeable impact on the final results.

Thus, we chose not to fully binarize the mentioned MMs, and this trade-off is necessary to achieve a more accurate fully binarized DM.

Q5: minor issues:

Q5-1: Line 93 ...

Sorry for the incorrect link; it should have pointed to reference [54] in the manuscript.

Q5-2: In Figure 4 ...

Here, we provide a clearer explanation to present the meaning of Fig.4 better:

• The first row labeled FP32 shows the visualization of the full-precision model output $\mathcal{F}^{fp}$.
• The second row labeled Diff represents the visualization of the difference between the full-precision model output and the binarized model output, $\mathcal{F}^{fp} - \mathcal{F}^{bi}$.
• The third row labeled SPD shows the visualization of the local attention for both the full-precision model and the binarized model calculated using the SPD method: $\frac{ A^{fp}}{ ||A^{fp}||_2 } - \frac{A^{bi}}{ ||A^{bi}||_2 }$.

From Figure 4, it can be observed that our SPD method allows the model to pay more attention to local information in each patch.

Q5-3: The term ...

Thank you for your feedback. Using "attention-guided" might be a more appropriate term. Our implementation of SPD is inspired by self-attention, involving matrix multiplication with its transpose locally to measure local self-attention. However, as you mentioned, self-attention is a widely accepted technical term, and using it to describe our SPD could lead to conceptual confusion.

We will update the explanations for these three issues in the revised version of the manuscript. Thank you very much for your detailed review of our work.

Q6: Are the Q ...

Sure, as we mentioned in our response to Q4 above, QKV components have all been binarized.

Q7: During inference ...

Yes, the calculations are dynamic. However, as explained in Q3, this part of the computation is identical to that in XNOR-Net and involves very minimal additional calculation, resulting in a very limited extra burden on the inference process.

Q8: In Line 224 ...

As a general quantization method, real-to-binary [1] suggests that using attention map-based loss during the distillation of a binary model from a full-precision model achieves better results. In contrast, BinaryDM, as the work most closely related to BiDM, directly points out that using L2 loss makes it difficult to align and optimize binary features with full-precision features. These studies indicate that the general L2 loss is inadequate for meeting the optimization needs of binary scenarios.

As a result, Table.7 of our manuscript shows that SPD indeed outperforms L2 loss with MSE.

We will include a discussion on this section in the revised version of the manuscript to make the motivation behind SPD more clear.

[1] Training binary neural networks with real-to-binary convolutions.

Q9: Figure 3 ...

(1) Why is Lt−1 ...

(2) Is this relationship ...

Sorry for the confusion. Here are our explanations for the two issues:

• (1) In reality, although the trainable $k$ results in changes to local elements, this does not imply that the entire vector is linearly scaled. Thus, ${L}^{t-1}$ is not a linear scaling up of ${B}^{t-1}$.
• (2) The relationship between these two is unrelated to Equation 12. ${L}^{t-1}$ and ${B}^{t-1}$ should be considered as differences in the output features of the model obtained after training under non-trainable/trainable settings of $k$ in Eq.10. This means that the learnable $k$ allows for more flexible and free feature representation.

You could refer to the Global Rebuttal (2) for a further explanation.

Dear Reviewer AtKg,

Thank you for your thorough review of our work, BiDM, during the review stage. We have carefully considered your concerns during the rebuttal stage and made revisions to the relevant sections of the manuscript.

We are looking forward you to reviewing our response and we are also willing to answer any further questions.

Best regards,

Authors

Thank you for your prompt response. We are glad to clarify the inference efficiency of BiDM.

Q10: Thank you for the authors' response. My concerns regarding Q1, Q2, Q4, Q5, Q6, Q8 have been satisfactorily addressed. Nonetheless, I remain uncertain about the deployment efficiency, particularly given that practical inference speeds often diverge from theoretical operations per second (OPs). The dynamic min-max operations, despite their minimal theoretical OPs, may introduce substantial latency in practical deployment, especially with smaller model sizes where matrix multiplications are less significant. I am keen to know if there are effective tools available to verify this.

In BiDM, there are no dynamic min-max operations; instead, our design involves only basic arithmetic operations such as convolution, matrix multiplication, addition and summation, and matrix dot multiplication. Additionally, as stated in Eq.(9) of our manuscript and in our response to Q3, the operators in BiDM behave the same as those in XNOR-Net[1] during inference, which corresponds to Eq.(11) in the XNOR-Net's original paper.

Following your suggestion, we expand upon the inference process described in Eq.(9) and provide a detailed explanation and testing. Since the divisor involved in calculating the mean of $A^{1,h,w}$ from $I^{c,h,w}$ (i.e., the channel dimension $c$) can be integrated into $k^{1,1,3,3}$ in advance, resulting in $k'^{1,1,3,3} = \frac{k^{1,1,3,3}}{c}$. Additionally, $\alpha^{n,1,1,1}$ derived from $W^{n,c,h,w}$ can also be computed ahead of inference. Therefore, the actual operations involved during inference are as follows:

[FP] Original full-precision convolution:

• (0) Perform convolution between full-precision $I_f^{c=448,w=32,h=32}$ and full-precision $W_f^{n=448,c=448,w=32,h=32}$ to obtain the full-precision output $O_f^{448,32,32}$.

[XNOR-Net/BiDM] The inference process for XNOR-Net/BiDM involves the following 6 steps:

• Sign operation:
  • (1) Compute $I_b^{448,32,32} = \text{sign}(I_f^{448,32,32})$.
• Binary operation:
  • (2) Perform convolution between the binary $I_b^{448,32,32}$ and the binary $W_b^{448,448,3,3}$ to obtain the full-precision output $O_f^{448,32,32}$.
• Full-precision operations:
  • (3) Sum the full-precision $I_f^{448,32,32}$ across channels to obtain $A^{1,32,32}$.
  • (4) Perform convolution between full-precision $A^{1,32,32}$ and $k'^{1,1,3,3}$ to obtain $O_1^{1,32,32}$.
  • (5) Pointwise multiply $O_f^{448,32,32}$ by $O_1^{1,32,32}$ to obtain the full-precision output $O_2^{448,32,32}$
  • (6) Pointwise multiply $O_2^{448,32,32}$ by $\alpha^{448,1,1}$ to obtain the final full-precision output $O^{448,32,32}$

We utilized the general deployment library Larq[2] on a Qualcomm Snapdragon 855 Plus to test the actual runtime efficiency of the aforementioned single convolution. The runtime results for a single inference are summarized in the table below. Due to limitations of the deployment library and hardware, Baseline achieved a 9.97x speedup, while XNOR-Net/BiDM achieved an 8.07x speedup. Besides, the improvement in generation performance brought by BiDM is even more significant, and we believe that it could achieve better acceleration results in a more optimized environment.

[1] Rastegari et al. Xnor-net: Imagenet classification using binary convolutional neural networks. ECCV 2016

[2] "LarQ". https://github.com/larq/larq

Thanks for the authors' response. My concerns regarding Q1, Q2, Q4, Q5, Q6, Q8 have been satisfactorily addressed. Nonetheless, I remain uncertain about the deployment efficiency, particularly given that practical inference speeds often diverge from theoretical operations per second (OPs). The dynamic min-max operations, despite their minimal theoretical OPs, may introduce substantial latency in practical deployment, especially with smaller model sizes where matrix multiplications are less significant. I am keen to know if there are effective tools available to verify this.

Regarding Q3, could you elaborate on the computations in Eq.(9)? Specifically, how is the convolution between $A$ and $k\alpha$ executed efficiently?

Thank you very much for your high recognition of our work and the valuable suggestions you provided. Our response is as follows:

Q1: The description of the cross-time step connection during the training phase is somewhat vague. During the inference phase, the impact of this connection is iteratively attenuated. For example, when α is 0.3, the influence of step T on step T-2 is 0.3×0.3=0.09, rather than 0. However, upon reviewing the source code in the supplementary materials, I found that the authors only consider steps T-1 and T-2 during training. I would like to know if this affects the final accuracy.

From the optimization principles of DDPM[1], this does not affect the accuracy of the results. Essentially, our training process is no different from that of DDPM, as both use efficient training to optimize random terms of the usual variational bound on negative log likelihood with stochastic gradient descent. Due to the cross-time step connection in TBS, which requires considering the values from the previous inference step, we included T-2, T-1, and T in our considerations. This approach still adheres to the DDPM training methodology.

[1] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. NeurIPS, 2020.

Q2: The scenario assumed in Figure 3 is overly idealized. When only L^t changes, for instance, when L^t is at the lower right of L^(t-1), the weighted average might result in T^(t-1) being further from F^(t-1). How is this situation explained?

You can refer to the Global Rebuttal (2) for the explanation. Additionally, in practical applications, the learnability of $\alpha$ ensures the effective collaboration between $L^{t-1}$ and $L^{t}$. Moreover, the ablation study results in Table.6 also demonstrate the effectiveness of connecting $L^{t-1}$ and $L^{t}$ across time steps.

Q3: Although BiDM shows significant improvements over other binarization methods, there remains a substantial performance gap compared to full-precision models. This could still hinder its practical application.

BiDM is the first fully binarized DM method capable of generating viewable images, significantly surpassing advanced binarization methods like ReSTE in quantitative metrics (on LSUN-Bedrooms, BiDM achieves an FID of 22.74, notably better than ReSTE's FID of 59.44). This demonstrates the feasibility of fully binarized DMs, marking a step towards practical applications with great potential.

Besides, as you highlighted in the Strengths section, the improvements to techniques like XNOR will have a positive impact on the entire binarization field. We will further explore this area in future work to achieve broader practical applications.

Q4: The manuscript only addresses diffusion models using a U-Net backbone primarily based on convolution. It appears to be inapplicable to diffusion models like DiT, which are primarily based on transformers and not related to U-Net.

The connections across timesteps in TBS and SPD should be directly applicable to DiT, with SPD likely being even more compatible with DiT's inherently space-patched input. While the convolutional design in TBS isn't suitable for DiT's linear-based architecture, we have observed that finer quantization granularity (such as per-group quantization) is gradually being proposed for architectures like transformers. Using a similar approach of dynamically calculating statistics for scaling factors followed by convolution or linear transformation could also be applicable. We plan to explore these aspects further in our future work.

Dear Reviewer cNoX,

Thank you for your thorough review of our work, BiDM, during the review stage. We have carefully considered your concerns during the rebuttal stage and made revisions to the relevant sections of the manuscript.

We are looking forward you to reviewing our response and we are also willing to answer any further questions.

Best regards,

Authors

Thank you for reviewing our manuscript and providing valuable suggestions. Here are our responses to some of the concerns you raised:

Q1: The main weakness of the paper is that the proposed method, BiDM, increases the training time of DMS compared to the original process. This issue should be more precise in the paper. The authors should compare the convergence rates of the proposed method and its competitors in terms of wall-clock time and identify which components contribute to the increased training time by breaking down the complexity of all main components of the proposed model.

Thank you for your suggestions. BiDM consists of two techniques: TBS and SPD. The time efficiency analysis during training is as follows:

• TBS includes the learnable convolution of scaling factors (Eq.10) and the cross-time step connection (Eq.12):

  • The increase in training time due to the convolution of trainable scaling factors is minimal, as the depth of the convolution for scaling factors is only 1, and the size of the trainable convolution kernel is only $3\times3$.
  • The cross-time step connection is the primary factor for the increase in training time. Since it requires training $\alpha$, we introduce this structure during training, so each training sample requires not only noise estimation for $T^{t-1}$ but also for $T^{t}$, directly doubling the sampling steps.

• SPD may lead to a slight increase in training time (an additional 0.18 times), but since we only apply supervision to the larger upsampling/middle/downsampling blocks, the increase is limited.

We have supplemented the actual training time on an NVIDIA A100 40GB GPU, and the results in Global Rebuttal (3) align well with the theoretical analysis mentioned above. Due to the actual software and hardware frameworks, the actual training time per iter for BiDM did not completely double compared to the baseline.

Following your suggestion, we compared the training loss under the same training iterations or training time. BiDM achieved significantly better generative results than baseline methods under the same training cost, demonstrating that it not only has a higher upper limit of generative capability, but is also relatively efficient when considering generative performance. You can refer to Global Rebuttal (3) for a more detailed explanation.

We also tested the FID after uniformly training for 0.5 days, and the results in Global Rebuttal (3) show:

• BiDM has the best convergence, even in a short training time.
• No.3 significantly outperforms No.5 because connections across timesteps greatly increase training time, making No.3 converge faster in the early training stages.
• No.5 slightly outperforms No.7 because $\mathcal{L}_{SPD}$ causes a slight increase in training time.

We emphasize that the biggest challenge in fully binarizing DM lies in the drop in accuracy. Although BiDM requires a longer training time for the same number of iters, it significantly enhances the quality of generated images, as no other method has been able to produce effective images.

Q2: The chosen baselines are quite weak as all of them were originally designed for discriminative tasks such as image classification. The authors should consider stronger baselines dedicated to generative tasks such as [1] and the baselines therein.

Thank you for your suggestion. We have supplemented the results and analysis for BBCU, and we have also included experimental results for other quantization methods suited for generative models, such as EfficientDM. You can refer to the Global Rebuttal (1) for more comprehensive information.

We will include the above discussion in the revised version of the manuscript. These discussions will further clarify the motivation and necessity behind each component of BiDM.

Q3: In Table 3, it is not clear why adding SPD alone makes the results worse, whereas combining it with TBS leads to improved performance. Do the authors have any explanation for this phenomenon? This point should be addressed to clarify the underlying reasons for the observed behavior and to provide a deeper understanding of the interaction between SPD and TBS in the proposed method.

Sorry for the confusion, but there might be a misunderstanding. In fact, SPD alone improves the results compared to Vanilla. Specifically, the ablation results in Table 3 can be more clearly restated in the table below. In all metrics, the binarized DM with only SPD outperforms the Vanilla method. For example, FID decreases from 106.62 to 40.62, demonstrating its superiority in optimizing binarized DMs.

When SPD and TBS are used together, SPD brings even better generative performance on top of TBS, further reducing the FID from 35.23 to 22.74.

Dear Reviewer 4HDt,

Thank you for your thorough review of our work, BiDM, during the review stage. We have carefully considered your concerns during the rebuttal stage and made revisions to the relevant sections of the manuscript.

We are looking forward you to reviewing our response and we are also willing to answer any further questions.

Best regards,

Authors