Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization

Thanks for recognizing the novelty of our paper. We believe there are some misunderstandings about our implementation, and we will address your questions with the following experiments.

Methods And Evaluation Criteria: Q1

MoDiff focuses on quantization rather than the latest models or datasets. To clarify its generality, we evaluate MoDiff on Stable Diffusion with MS-COCO and DiT on ImageNet using public quantized checkpoints [1,2]. Results show that MoDiff generalizes well across architectures and datasets.

Table 1: Stable Diffusion

Table 2: DiT

[1] Q-Diffusion. ICCV 2023.

[2] PTQ4DiT. NeurIPS 2024.

Theoretical Claims: Q1

The corollary validates our assumptions and guides design; it's not a strict per-layer rule nor the tightest bound. We aim to extend this to mixed-precision in future work.

Theoretical Claims: Q2

We warm up by running the first timestep in full precision and then using low-bit activations. This adds negligible overhead. We apply warm-up to baselines for fairness, showing the performance gain is not from warm-up.

Experimental Designs Or Analyses: Q1

First, MoDiff is quantization-method agnostic. We follow [1]'s channel-wise dynamic quantization (hardware-unfriendly, but illustrative), and also test Q-Diffusion and tensor-wise quantization (Table 1, 11), showing consistent FID improvements.

Second, low-bit activation cases are rare due to the large quantization error they introduce. MoDiff breaks the limitations of existing methods by effectively reducing activation quantization error. Similar configurations are also used in prior works [2,3].

[1] DGQ. ICLR 2025.

[2] PikeLPN. CVPR 2024.

[3] WRPN. ICLR 2018.

Experimental Designs Or Analyses: Q2

The baseline does not necessarily outperform MoDiff since MoDiff (1) has smaller quantization errors and (2) has better performance on sFID. Specifically, MoDiff closely matches the full-precision model across all metrics compared to the baseline, meaning its output is nearly the same as the ground truth. Moreover, MoDiff outperforms the baseline in sFID. It is inaccurate to conclude that the baseline is better considering all metrics.

Experimental Designs Or Analyses: Q3

To address your concern, we show how error compensation improves sFID for CIFAR-10 with DDIM.

Experimental Designs Or Analyses: Q4

Our comparison with Q-Diffusion is fair. We only make minimal changes and necessary adaptations to integrate our method. Specifically, we (1) apply modulated computation and remove bias for correctness, (2) use temporal differences instead of raw activations for calibration since they serve as inputs during sampling. The last implementation is (3) omitting block-wise quantization for training stability. To demonstrate that (3) does not account for MoDiff’s performance gains, we applied it to Q-Diffusion with W8A8 on CIFAR-10, yielding FID 4.76 (worse than baseline). Our implementation also reproduces Q-Diffusion results without modulation, confirming consistency.

Relation To Broader Scientific Literature: Q1

We exclude MixDQ due to a lack of public code. EfficientDM uses quantization-aware training, which differs from our post-training quantization focus.

Other Strengths And Weaknesses: Q1

Lines 237–260 explain how errors accumulate in modulated computation and how our compensation method addresses this. Specifically, $\hat a\_t$ represents the activation computed in the previous timestep, while $a\_t$ is the desired (unquantized) activation. Modulated computation reuses $A(\hat a\_t)$ instead of $A(a\_t)$, so computing temporal difference based on $a_t$ in the next step will introduce errors. Our compensation technique corrects this by computing temporal difference based on $\hat a_t$.

Lines 324–329 emphasize that compensation is more critical in practice than theory suggests. In practice, diffusion models suffer from significant error accumulation as quantization errors build up layer by layer. The severity of error accumulation highlights the necessity of applying error compensation.

Other Comments Or Suggestions: Q1

Thanks for pointing out the missing right parenthesis. We will correct the typo in the revised version.

Questions For Authors: Q1

We run tensor-wise quantization on CIFAR-10 with 20-step DDIM. MoDiff consistently improves low-bit quality.

Thank you for recognizing the novelty, effectiveness, and clarity of our paper. We are glad to address your questions.

Experimental Designs Or Analyses: Q1. All experiments were conducted on small-scale datasets such as CIFAR-10, LSUN-Churches, and LSUN-Bedroom, whereas it is standard practice to validate a research idea on larger datasets like ImageNet—especially since the proposed methods are training-free, making scalability feasible.

To address your concern, we conduct experiments following [1], using DiT on ImageNet 256×256 with tensor-wise dynamic quantization. The results demonstrate that our method consistently improves generation quality at low activation bit widths:

[1] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.

Experimental Designs Or Analyses: Q2. Should extend its evaluation to include more recent samplers, such as flow models, EDM, and DPM-solver.

To address your concern, we perform tensor-wise quantization using the DPM-Solver-2 on CIFAR-10 with 20 sampling steps. Additionally, we conduct experiments with the PLMS solver using 50 steps on Stable Diffusion with MS-COCO-2014. In both cases, MoDiff consistently improves FID scores across different solvers, even with reduced sampling steps.

Table1: DPM on CIFAR-10

Table 2: PLMS on MS-COCO

Experimental Designs Or Analyses: Q3. The ablation study on error compensation would also benefit from employing direct image quality metrics like FID, IS, or P&R instead of the L2 metric, which is less convincing for assessing image quality.

To address your concern, we conduct experiments to show the sFID measurement with or without error compensation on CIFAR10 with DDIM. The results show that error compensation reduces the error accumulation in low bit activation cases.

Experimental Designs Or Analyses: Q4. Table 1 contains an error in the bolding.

Thank you for pointing out the incorrect bold formatting. We will revise it in a new version.

Essential References Not Discussed: Q1. There are some caching methods missing discussion.

Thank you for mentioning the relevant papers. We will include them in the related works section of the revised version.

Thank you for recognizing the novelty, effectiveness, and clarity of our paper. We believe there are some misunderstandings about our implementation, and we are glad to address your questions.

Essential References Not Discussed: Q1. Compare with PTQD.

Compared to PTQD, MoDiff is (1) more general and flexible, (2) free from strong assumptions about error distribution, and (3) significantly more effective in low-precision scenarios.

(1) PTQD requires solver-specific adaptations to address variance and bias, while MoDiff can be applied across solvers without modification. Moreover, PTQD is restricted to standard diffusion models, whereas MoDiff also supports cached diffusion models by compensating for reuse errors in cached components.

(2) PTQD relies on strong assumptions about error distribution, specifically that quantization errors follow a Gaussian distribution after input rescaling. This assumption can introduce inaccuracies in error estimation. In contrast, MoDiff leverages the widely observed similarity between timesteps, which is well-supported by prior works [1].

(3) MoDiff performs well in low-precision activation settings, whereas PTQD fails entirely. To demonstrate this, we evaluate both methods on CIFAR-10 with W8A4 quantization. PTQD yields an FID of 397.12 and fails to produce meaningful images, while MoDiff achieves a much lower FID of 13.41.

We will include these comparisons in the revision.

[1] Deepcache: Accelerating diffusion models for free. CVPR 2024.

Other Strengths And Weaknesses: Q1. Lack experiments on large models (Stable Diffusion).

To address your concern, we conduct tensor-wise quantization on Stable Diffusion v1.4 using the 50-step PLMS solver on MS-COCO-2014. The resulting FID scores demonstrate that MoDiff consistently performs well on large-scale diffusion models.

Other Strengths And Weaknesses: Q2. MoDiff is restrictive, which does not support norm layer folding and depends on dynamic quantization.

There are a few misunderstandings about our paper. We clarify that MoDiff (1) supports norm layer folding and (2) does not depend on dynamic quantization.

(1) MoDiff supports norm layer folding and is practical in use. It applies to any linear operation, not just linear layers. By folding norm layers with other linear components such as convolution layers, MoDiff can be applied to the resulting block due to preserved linearity. In our implementation, we did not perform such merging, following standard practices in the diffusion quantization community [1].

(2) MoDiff is agnostic to the quantization method and not limited to dynamic quantization. As shown in Table 1 of the main paper, it consistently improves performance in low-precision settings with Q-Diffusion, which is static quantization. Additionally, dynamic quantization is well-studied in the literature [2] and supported by certain hardware platforms [3].

[1] https://github.com/Xiuyu-Li/q-diffusion.

[2] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.

[3] NVIDIA TensorRT Documentation. https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/work-quantized-types.html

Other Strengths And Weaknesses: Q3. Suboptimal results in some settings.

First, after carefully reviewing all log files, we identified some errors in writing the results for the $W8A8$ and $W4A8$ cases. The revised results are now provided. As shown in the updated table, MoDiff consistently matches both the baseline and full-precision models in terms of FID and sFID.

Second, our MoDiff is agnostic to quantization methods, where channel-wise dynamic quantization (we follow [1] and agree that it is hardware-unfriendly) is used to show the feasibility and potential of MoDiff. In addition to channel-wise dynamic quantization, we report the results on Q-Diffusion and tensor-wise quantization in Table 1 and 11 with consistent improvement over FID on low-bit activation cases.

[1] DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models. ICLR 2025.

Other Comments Or Suggestions: Q1. Missing right parenthesis in Eq (18).

Thank you for pointing it out. We will correct the typo in the revised version.

Questions For Authors: Q1. If MoDiff needs more training time.

MoDiff is even more efficient than Q-Diffusion in scaling factor calibration. As noted in our implementation, we skip block-wise quantization for training stability while maintaining the same number of iterations to learn scaling factors. This results in reduced time for post-training calibration.

Experimental Designs Or Analyses: Q1

Although the goal of this paper is to validate our method, not to benchmark it across all diffusion architectures, we conduct experiments following [1] to address your concern, using DiT on ImageNet. The results demonstrate that our method consistently improves generation quality:

[1] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.

Experimental Designs Or Analyses: Q2

We will include the mentioned literature in the revised version. Regarding the papers you cited, MoDiff (1) offers orthogonal contributions and can complement quantization methods; (2) generalizes caching approaches; and (3) reduces quantization error better compared to PTQD.

(1) ViDiT-Q is a quantization method that does not exploit time step similarities or compensate for quantization errors. MoDiff addresses these limitations to reduce quantization error and prevent error accumulation, which is orthogonal to ViDiT-Q. Moreover, MoDiff is a general framework compatible with ViDiT-Q's techniques, including group-wise quantization, channel balance, and mixed-precision computation.

(2) $\Delta$-ViT and dual feature caching rely on empirical, heuristic designs to cache different components without compensating for quantization error. MoDiff is a general framework that encompasses these methods as special cases. Specifically, MoDiff reduces to them when the cached component is quantized to 0 bits while the remaining components use full precision.

(3) PTQD reduces quantization error by post-processing quantized models based on assumed error distributions. However, it ignores timestep similarities and fails under low-precision activations. To verify this, we evaluate PTQD with W8A4 on CIFAR-10, where it yields an FID of 397.12 and fails to generate meaningful images, while MoDiff achieves a significantly better FID of 13.41.

Other Strengths And Weaknesses: Q1

We believe there are some misunderstandings about the novelty of MoDiff. Specifically, (1) $\Delta$-DiT only approximates temporal differences without modulating the cache, and (2) it lacks any form of error compensation.

First, $\Delta$-DiT does not approximate temporal differences but instead estimates differences between transformer blocks. Moreover, it directly reuses cached values, which introduces errors, whereas MoDiff applies lightweight modulation to reduce these errors.

Second, MoDiff explicitly traces quantization errors from the previous timestep and compensates for them in the next with theoretical guarantees. In contrast, caching methods, including $\Delta$-DiT, simply reuse cached components without addressing accumulated errors.

Questions For Authors: Q1

We want to emphasize that (1) MoDiff is hardware-friendly and benefits from hardware in practice, though the hardware implementation is out of the scope of our study, and (2) MoDiff focuses on improving computational efficiency rather than memory efficiency. In addition, exploring lower-bit activation, even down to 1 bit, is valuable and aligns with ongoing research in quantization [2, 3].

First, GPU architectures support 4-bit activation computation [1], but performance degrades significantly at such low precision. MoDiff addresses this limitation by overcoming the performance bottleneck of low-precision activations.

Second, low-bit activations offer much more speedup over 8-bit activations by reducing FLOPs [1], making them a valuable target for acceleration.

Finally, MoDiff represents a step toward enabling low-bit activations, not the endpoint. This direction could eventually push activation precision down to 1 bit, potentially driving future hardware designs to better support ultra-low-bit computation, such as logic-based operations.

[1] Int4 Precision for AI Inference. https://developer.nvidia.com/blog/int4-for-ai-inference.

[2] BinaryDM: Accurate Weight Binarization for Efficient Diffusion Models. ICLR 2025.

[3] BiDM: Pushing the Limit of Quantization for Diffusion Models. NeurIPS 2024.

Questions For Authors: Q2

We want to emphasize that MoDiff is agnostic to quantization methods. Channel-wise dynamic quantization is used to show the feasibility and potential of MoDiff following the baseline [1]. In addition to channel-wise dynamic quantization, we have also reported the results on Q-Diffusion and tensor-wise quantization (more friendly to hardware) in Table 1 and Table 11 with consistent improvement over measurements.

[1] DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models. ICLR 2025.

Questions For Authors: Q3

To address your concern, we perform tensor-wise quantization using the DPM-Solver-2 on CIFAR-10 with 20 sampling steps.

Thank you for recognizing the novelty, effectiveness, and clarity of our paper. We are glad to address your questions.

1. In Table 1, FID column, some values are incorrectly bolded, e.g., 4.21

Thank you for pointing out the incorrect bold formatting. We will revise it in a new version.

2. In the proof of Theorem 4.2, is there a factor of two missing in equation (56)?

Thanks for pointing out the missing constant scalar. Here is the revised proof, which does not affect the conclusions:

$$\tilde{\mathbf{e}}\_t^2 = \\|\mathbf{o}\_t - \tilde{\mathbf{o}}\_t\\|\_2^2$$ $$= \\|\mathbf{o}\_t - \mathcal{A}(Q(\mathbf{a}\_t - \mathbf{a}\_{t+1})) - \tilde{\mathbf{o}}\_{t+1}\\|\_2^2$$ $$= \\|\mathbf{o}\_t - \mathbf{o}\_{t+1} - \mathcal{A}(Q(\mathbf{a}\_t - \mathbf{a}\_{t+1})) + (\mathbf{o}\_{t+1} - \tilde{\mathbf{o}}\_{t+1})\\|\_2^2$$ $$= \\|\mathcal{A}(\mathbf{a}\_t - \mathbf{a}\_{t+1}) - \mathcal{A}(Q(\mathbf{a}\_t - \mathbf{a}\_{t+1})) + (\mathbf{o}\_{t+1} - \tilde{\mathbf{o}}\_{t+1})\\|\_2^2$$ $$= \\|\mathcal{A}(\mathbf{a}\_t - \mathbf{a}\_{t+1} - Q(\mathbf{a}\_t - \mathbf{a}\_{t+1})) + (\mathbf{o}\_{t+1} - \tilde{\mathbf{o}}\_{t+1})\\|\_2^2$$ $$\leq 2\\|\mathcal{A}(\mathbf{a}\_t - \mathbf{a}\_{t+1} - Q(\mathbf{a}\_t - \mathbf{a}\_{t+1}))\\|\_2^2 + 2\\|\mathbf{o}\_{t+1} - \tilde{\mathbf{o}}\_{t+1}\\|\_2^2$$