DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space

We sincerely appreciate the insightful suggestions and comments provided by the reviewer.

Q1: The benchmarks used for the evaluation are sufficient in size, authors even include some high-resolution datasets. Nevertheless the method is only compared with the baseline model with the same architecture run in the pixel space

In addition to the comparison with pixel-based diffusion models, we also compared DCTdiff with the latent diffusion model UViT which utilizes the VAE of StableDiffusion for image compression. Please refer to Table 3 of our paper for the details.

Q2: Theorem 5.1 might be a bit relevant to the proposed approach, but it does not provide theoretical grounds for the proposed method.

Theorem 5.1 provides a theoretical foundation for understanding how noise affects different frequency components in the diffusion process. It formalizes the observation that high-frequency signals are more susceptible to noise perturbations due to the energy concentration properties of the DCT. This insight is crucial for justifying why DCTdiff exhibits a faster noise accumulation process than pixel diffusion, ultimately motivating the introduction of the SNR scaling method. We will improve the writing of Section 5.3 to highlight the connection.

Q3: The evaluation is a comparison to the baseline UViT on a number of different datasets including some high-resolution ones. In those experiments DCTdiff consistently outperforms pixel-space UViT. However, the observed results are far from state-of-the-art (e.g. FID on FFHQ256 2.19 for StyleGAN from 2022 vs 5.08 from this work).

Thanks for the interesting observation. GANs have been a strong class of models that generate high fidelity images (often measured by FID) (more evaluations can be found in https://paperswithcode.com/sota/image-generation-on-ffhq-256-x-256) yet do not produce diverse images. The main advantages of diffusion models are that they are stable to train and have higher diversity (often measured by recall) than GANs (see papers [1][2]). Our DCTdiff actually achieves better FID than many other diffusion-based models, e.g [3] and [4]).

[1] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." NIPS. 2021.

[2] Boutin, Victor, et al. "Diffusion models as artists: are we closing the gap between humans and machines?." ICML 2023.

[3] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." CVPR. 2022.

[4] Kim, Dongjun, et al. "Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation." ICML. 2022.

Q4 I think the one important reference missing is [1] where authors combine a hierarchical VAE with a diffusion model by using a Discrete Cosine Transform (DCT) to create low-dimensional pseudoinputs, which are then modeled via a diffusion-based prior to efficiently approximate the aggregated posterior and enhance latent space utilization

Except for explicit signal compression, it is inspiring to see that DCT can be treated as a low-dimensional latent representation and applied to VampPrior framework to obtain the flexible prior distribution modeled by diffusion. We will add the discussion of [1] to our literature review.

[1] Kuzina, Anna, and Jakub M. Tomczak. "Hierarchical VAE with a Diffusion-based VampPrior." Transactions on Machine Learning Research.

Q5: Have you considered using different terminal distribution for the diffusion model instead of gaussian? Maybe something like Pareto distribution would be more suitable for the unusual distribution defined by DCT? Such an approach might also help for the SNR Scaling

We sincerely appreciate the reviewer providing this insightful suggestion. In our DCTdiff work, we follow the Gaussian prior of diffusion due to the fair comparison. But we do think that Pareto distribution which exhibits the power-law (same as image spectral distribution) is very likely to be a great solution for DCT distribution modeling. Moreover, Laplace distribution is also worth exploring for DCT modeling. We will investigate this topic in our next work.

Thank you very much for the reviews and constructive suggestions.

Q1: the main paper lacks qualitative figures, and the appendix only includes Figs. 9-11, missing key results for FFHQ 512

We will add more qualitative samples in the appendix, including the randomly drawn ones, and compare them with the baseline as well. Surely, the qualitative results on FFHQ 512x512 will be added to the paper, too.

Q2: Theorems 5.1 and 5.2 appear correct but do not significantly contribute to the main claims. Theorem 5.1: States that diffusion removes high-frequency components first and low-frequency components later. This is well known (Patashnik et al., ICCV 2023) and not specific to DCTdiff. Theorem 5.2: Reformulates existing DCT-based upsampling methods in a mathematical form.

Theorem 5.1 provides a theoretical foundation for understanding how noise affects different frequency components in the diffusion process. It formalizes the observation that high-frequency signals are more susceptible to noise perturbations due to the energy concentration properties of the DCT. This insight is crucial for justifying why DCTdiff exhibits a faster noise accumulation process than pixel diffusion, ultimately motivating the introduction of the SNR scaling method. Regarding Theorem 5.2, we present an alternative derivation that differs from the existing approach in the literature. We believe that DCT-based upsampling holds significant potential in deep learning applications. We consider moving Theorem 5.2 to the appendix while emphasizing its potential in the main paper. In summary, we will refine the writing in these two paragraphs by adding proper citations and explicitly highlighting the message we want to deliver in this paper.

Q3: The metrics and baselines are limited (only FID and UViT), which may not ensure a fair comparison. A broader evaluation with models trained on the same datasets (even if #Parameters, GFLOPs, and training steps differ) would improve the study. Additional metrics, such as Inception Score, could be included.

In addition to UViT, we have also provided the results on the baseline DiT (please refer to our paper). The reason for choosing these two base modes is that they are well-known models and widely used in the community. Evaluating DCTdiff on more base models is valuable as long as the base model is easy to implement. For example, we have preliminarily implemented DCTdiff using the UNet architecture, the results are promising (Table 11). We believe that DCTdiff is a general method to be applied in different diffusion networks.

Table 11. FID of UNet-based DCTdiff on CIFAR-10 (NFE=100).

Regarding extra metrics, we have added IS, precision, recall, CMMD (as suggested by reviewer VaLT) to evaluate the models. The results are shown in Tables 3, 4, 5.

Table 3. Comparison between UViT and DCTdiff on CIFAR-10 using DDIM sampler (NFE=100)

Table 4. Comparison between UViT and DCTdiff on FFHQ 128 using DPM-solver (NFE=100)

Table 5. Comparison between UViT and DCTdiff on AFHQ 512 using DPM-solver (NFE=100)

Q4. Latent diffusion models using VAEs typically produce high-quality images but introduce issues with encoder-decoder invertibility. Hong et al., propose a correction algorithm for this, suggesting that DCTdiff could bypass such issues entirely

Thank for suggesting the case of encoder-decoder invertibility. We do agree that DCT is a flexible and invertible method of image representation, which differs from the neural network-based image tokenizers. We will include this discussion with paper [1] in the final version of our paper.

[1] Hong, Seongmin, et al. "Gradient-free Decoder Inversion in Latent Diffusion Models." NIPS (2024)

Thank you very much for the detailed and helpful reviews.

Q1: The paper only evaluates some generative tasks, while the capability of DCT space on other generative and discriminative tasks is still unknown

We will first rephrase this sentence to avoid any misunderstanding. Our work primarily aims to explore the feasibility of performing diffusion in the DCT space for image generation, with a particular focus on generative capabilities. While we acknowledge the potential discriminative properties of DCT-based modeling, we believe these properties are primarily derived from the generative pre-training mechanism rather than the information format itself. This could also explain why both pixel and latent diffusion models exhibit similar characteristics. We leave the investigation of the discriminative, editing, inpainting and restoration ability of DCT-based diffusion models as future work.

Q2: The comparison is restricted to latent diffusion using SD-VAE, an outdated compression model, Can the DCT space outperform these modern tokenizer?

First, we will rewrite this claim by clarifying that the latent diffusion is SD-VAE based, making the scope of our claim clearer. Second, Our goal is not to beat these emerging tokenizers with dedicated designs, we explore how far the image diffusion modeling can go without a pretrained tokenizer. Moreover, our work does not conflict with image tokenizers, e.g. we totally agree that exploring DCT tokenizer is promising, and the technique of these new tokenizers can also be used to develop DCT tokenizer.

Q3: Can you provide some results on ImageNet-256?

Given the limited GPUs, we have prioritized our experiments on the other tasks: scaling up, extra evaluations, and explorations of UNet-based DCTdiff.

Q4: The baseline on CIFAR-10 appears under-optimized

For fair comparison, our implementation of UViT and DCTdiff on CIFAR-10 used patch size (ps=4) (different from ps=2 in original UViT paper). If we apply ps=2 on DCTdiff, block size will be 1, yielding only the DC frequency and preventing frequency reweighting. To address your concern, we have implemented another baseline of UViT using ps=2 (256 tokens). Correspondingly, we increase the model size of DCTdiff to ensure the same computational complexity for a single network forward pass. Results on Tables 9 and 10 show that DCTdiff is significantly better than UViT in terms of FID. We acknowledge that the FID 5.05 on the UViT is still higher than the 4.5 you tested (might be due to the warmup steps, machine and software difference etc.), but our code and all trained models will be released for reproducibility.

Table 9. Extra CIFAR-10 benchmark, FID-50k using DDIM sampler

Table 10. Extra CIFAR-10 benchmark, FID-50k using DPM-solver

Q5: Can the proposed DCT-based diffusion be adapted to UNet backbones and also outperform pixel-space counterparts?

It is an interesting question. Although we think that Transformer has many advantages than UNet regarding the image DCT implementation: ease of (1) Cb Cr 2x subsampling, (2) elimination of high frequencies, and (3) frequency loss reweighting, we have investigated the possibility of UNet-based DCTdiff using the ADM codebase. Concretely, we remove the Cb Cr 2x subsampling and loss reweighting in the preliminary experiment, and just convert the RGB channel to YCbCr followed by DCT transform, the resulting frequencies yield the input tensor with shape (32, 32, 3) on CIFAR-10. We trained the UNet-based DCTdiff for 400k steps, the FID is shown in Table 11 and the results are promising. We believe further exploration of UNet-based DCTdiff will lead to better generation quality.

Table 11. FID of UNet-based DCTdiff on CIFAR-10 (NFE=100).

Q6: ViT-based architecture is more preferable when scaling to larger network capacities

We provide the scaling experiments below.

Table 6. FID on CIFAR-10 using DDIM sampler, ps=4

Table 7. FID on FFHQ 128 using DPM sampler

We thank the reviewer for the valuable comments which help improve the paper.

Q1: The paper claims that "DCT Upsampling Outperforms Pixel Upsampling". However, it is only compared against interpolation methods

The upsampling we mean in the paper is indeed interpolation. We will make this statement clearer.

Q2: What is the inference time comparison

Pixel diffusion and DCTdiff share the same GFLOPs and inference time (Table 1)

Table 1. wall-clock time (NFE=100, 10k samples, one A100)

Comparing latent diffusion and DCTdiff, the inference time is different because

• GFLOPs of UViT (latent) = 34*NFE (diffusion) + 1240 (Decoder)
• GFLOPs of DCTdiff = 133*NFE (diffusion)

The decoder of the VAE is expensive, but DCTdiff has a larger complexity in diffusion as DCTdiff has 1024 tokens and UVIT has 256 tokens. Table 2 shows that DCTdiff is faster in low NFE but is slower in the high NFE condition.

Table 2. Wall-clock inference time on AFHQ 512 (10k samples, A100 GPU). GFLOPs is appended in the brackets.

Q3: The performance gap diminishes as NFE increases

In most cases, our DCTdiff has significantly lower FID (20%~30%) than the base model. We believe the small improvement on ImageNet 64 is due to the small model capacity since we have limited GPU to scale on ImageNet. However, we do provide the scaling experiments on other datasets (Table 6,7)

Q4: How about evaluation using IS, Precision/Recall and CMMD

Table 3. Comparison between UViT and DCTdiff on CIFAR-10 using DDIM sampler (NFE=100)

Table 4. Comparison between UViT and DCTdiff on FFHQ 128 using DPM-solver (NFE=100)

Table 5. Comparison between UViT and DCTdiff on AFHQ 512 using DPM-solver (NFE=100)

Q5 Missing experiments of model scalability

We performed scaling experiments on CIFAR-10 and FFHQ 128 during the short rebuttal period.

Table 6. FID-50k on CIFAR-10 using DDIM sampler, patch_sz=4

Table 7. FID-50k on FFHQ 128 using DPM sampler

Q6 Compare DCTransformer with DCTdiff

DCTdiff differs from DCTransfomer in several key aspects: probabilistic modeling, image representation, network and image tokenization. We summarize their differences in Table 8. Overall, DCTdiff exhibits a straightforward way for image frequency generative modeling. Performance-wise, DCTdiff achieves FID 7.28 on FFHQ while DCTransformer has FID 13.06.

The only overlap between DCTransformer and DCTdiff is using the DCT transform and YCbCr color transform. But these well-known techniques of JPEG compression cannot be credited to DCTransformer.

Table 8. Differences between DCTransformer and DCTdiff

Q7: What is the limitation of this paper?

The limitation is that we did not explore other generative applications (e.g. image inpainting) and discriminative tasks. Also, frequency-oriented Transformer architecture and super-resolution image generation were not covered in this paper. We think these are promising directions for future work.