DM-Tune: Quantizing Diffusion Models with Mixture-of-Gaussian Guided Noise Tuning

Major Questions:

See weaknesses.

Additional Questions:

1. There are visible artifacts in your FP-MP samples in Figure 1 (e.g. the robot is less cohesive). While it clearly performs better than I-MP, it is not immediately clear that the evaluation metrics are true representations of the generated images. Could you address this discrepancy?
2. Your theoretical noise modeling applies directly to $X_t$ rather than the internal layers. The two are only equivalent if the model is linear, which is not the case. Could you address this or explain why the same approximation applies to the hidden activations?
3. Why did you keep the unconditional path at full precision instead of the conditional path? It seems like the inverse may perform better, and empirical evidence supporting either choice would strengthen your argument.
4. Does splitting the quantization per path negatively impact computational performance? It will reduce parallelism for the quantized layers.
5. Could you provide an explanation as to why you use the standard deviation to measure the sensitivity of precision? It’s not clear to me that a wider distribution indicates increased sensitivity when fixed layer-wise scaling could be applied. Other formats such as NF4 (Dettmers et al., 2023) take advantage of this fact for non-uniform spacing.
6. You show that DiT performs exceptionally well with quantization; however, is the same true for text-conditioned DiTs (e.g., Pixart-alpha, SD3, etc.)? If it is true, then it would be valuable to add this claim, otherwise if the behavior is not identical, consider adding a text conditioned DiT along with the other 3 models.
7. In Figure 6, your results suggest equivalent performance with NT-1 and NT-3, although NT-1 requires more epochs to converge. Is there any real advantage to NT-3 if it necessarily has a higher computational overhead during inference? The number of tuning epochs seems largely irrelevant at this scale, as it is presumably done once per model.
8. Were the evaluation results in Tables 1 and 2 generated with CFG? If not, please do so. Model behavior can vary drastically with and without CFG. Including both with and without would be preferable, where simplifying the columns could save space while presenting the full table in the appendix. Furthermore, please state the sampler used since that too can have a significant impact.

Comments for Clarity:

• The motivation sentence in Section 3.1 discussing diffusion models as a stochastic process is unclear. Consider rewriting.
• In Section 4.1.1, you mention "quantization is applied to all linear layers", please clarify if this includes convolutions for the conv-nets.
• In Section 4.5, please explain what is meant by “O3: we use vector instructions”, since warps always execute at the vector level (SIMT). Consider rephrasing.
• Mentioning batch size for evaluation in Section 5.1 is irrelevant, where you should instead directly state the batch sizes used for tuning.
• The $5.2\times$ speedup in Section 5.5 should be clarified by directly stating the reference point. Additionally, consider using an arrow indication in Figure 7 for easy visual clarity.

Comments for Presentation:

• In Figure 1a, which method is yours? This should be clearly marked and included if not done so.
• Non-cherry-picked samples should be added to the appendix to better compare the different versions (I-MP can be omitted in this case as your focus is on FP-MP).
• Figure 2 is difficult to read as the curves often overlap, applying a relative scaling may improve legibility.
• The critical point in Figure 3a is not legible due to the axis scale, consider using log scale or adding a sub-frame-region around the origin.
• Figure 7 should use a pattern fills along with color coding to improve readability. Also consider adding the geomean, which is typical to aggregate overall performance.
• Ensure consistency in the ordering of figures. Every time you show DiT, SD, and DDIM, they should appear in the same order from left to right.
• The plot titles in Figure 3 (“FID vs…”) should be removed. Additionally, both axis and label font sizes are too small in most figures.

According to the weakness section, I have the following questions:

1. How does the proposed DM-Tune framework perform on larger diffusion models, such as SDXL, or tasks requiring higher resolution? Can the authors provide a detailed analysis of the runtime and memory trade-offs when scaling to these more complex scenarios to validate the scalability of the approach?

2. Why is BF16 omitted as a baseline together with FP32 baseline when evaluating the effectiveness of FP8 configurations?

3. why does the proposed sensitive-layer selection strategy show minimal improvement over randomized selection for the Stable Diffusion model? What insights do the authors have regarding this outcome?

4. Some of the attached figures are really tough to follow. For Fig.1 & Fig.2, even though I zoomed in for the max extend, I still cannot get a clear visualization of the generated images. For Fig.2, the lines are too close to be distinguished.

5. The pseudocode in Alogrithm.1 depicts the fact that overflow detection is the first criteria when it comes to sensitive layer selection. But how STD actually affects the sensitivity of the certain layer is remain unclear in the paper.

6. The paper could be more demonstrating if real world testing prompts could be attached for visualization comparison between the FP8(proposed low-precision) and baselines.

1. What is the total number and memory size of the trainable noise-tuning parameters that are assigned in a layerwise and timestep-wise manner?
2. What are the overflow ratios in Algorithm 1, and how is "overflow" defined? Does it mean the overflow that occurs when converting high-precision FP to FP8?

Q1: In Table 7, it is not clear why Q-Diffusion and PTQD achieve worst performance than FP32. I am assuming that FP32 denotes the original non-quantised model.