The GAN is dead; long live the GAN! A Modern GAN Baseline

Thank you for your feedback.

“... still need to be scaled to large-scale text-to-image setting.”

For scaling, we are currently running ImageNet-64 experiments to include in our paper; please see our discussion in response to Reviewer 2Hzz. In general, we hope our work is a meaningful first step and that future works can better assess its scaling.

“Many GPUs are used, even for CIFAR-10; this is also in line with diffusion models, but it seems a lot, are all these GPUs needed for CIFAR-10?”

Using 8 GPUs to train CIFAR-10 is not required; our models can be trained with fewer GPUs at the cost of slower training speed. But, it is becoming a common practice, e.g., for diffusion models; Karras et al. used 8 GPUs to train their CIFAR-10 models in EDM.

“Any intuition to why worse performance with GELU and swish activations since these are very successful with transformers? Is this true for both G and D?”

In experiments, we tried to apply GELU/Swish/SMU to G and D and found that doing so deteriorates FID considerably. We did not try on G xor D. We posit two independent factors:

• ConvNeXt in general does not benefit much from GELU (and possibly similar activations). Table 10 and Table 11 in https://arxiv.org/abs/2201.03545: replacing ReLU with GELU gives little performance gain to ConvNeXt-T and virtually no performance gain to ConvNeXt-B.
• In the context of GANs, GELU and Swish have the same problem as ReLU: that they have little gradient in the negative interval. Since G is updated from the gradient of D, having these activation functions in D could sparsify the gradient of D and as a result G will not receive as much useful information from D compared to using leaky ReLU.

This does not explain the strange case of SMU (https://arxiv.org/abs/2111.04682): SMU is a smooth approximation of leaky ReLU and does not have the sparse gradient problem. It is unclear why it also underperformed and future work awaits. We are happy to add this discussion to supplemental.

“Have you considered a transformer architecture?”

We have not conducted any experiments, but are also interested to see how they perform. In particular, whether adding attention blocks to a ConvNet (similar to BigGAN and diffusion UNet) or using a pure transformer architecture (similar to DiT) will result in stronger performance. Given the impressive results of EDM2 (which uses UNet), it seems the argument has not yet settled for generative modeling.

“You mention that normalization can harm GANs, but groupnorm is used in diffusion models and layernorm is used in language models, could this be useful for your approach? Has it been tried before? I saw the references you mentioned to say that normalization is bad for GANs.”

We tried adding groupnorm to G and D and it did not improve FID or training stability. We would like to clarify that we do not intend to claim that all forms of normalizations are harmful. Our claim only extends to normalization layers that explicitly standardizes the mean and stddev of the activation maps. This has been verified by prior studies:

• Instance norm harms GANs: https://arxiv.org/abs/1912.04958
• Group norm harms diffusion models: https://arxiv.org/abs/2312.02696
• Group norm harms VAEs: https://arxiv.org/abs/2405.14477, pg. 5

The harm of normalization layers extends to the adjacent field of image restoration (https://arxiv.org/abs/1707.02921, sec 3.1; https://arxiv.org/abs/1809.00219, supplemental sec 1). To the best of our knowledge, EDM2 is currently the strongest diffusion UNet and it does not use normalization layers. However, it does apply normalization to the trainable weights and this improves performance considerably. We expect that applying the normalization techniques in EDM2 would improve our model's performance.

“You mention that PresGAN and DDGANs obtain good performances, do you think that some of their ideas could be useful?”

Yes, it is possible. We excluded these ideas as they are not an indispensable part of a minimal GAN framework. DDGAN in particular combines GANs and diffusion models - a promising research direction. Prior work has explored this in various flavors: DDGAN formulates GAN as a fast diffusion solver; Diffusion GAN (https://arxiv.org/abs/2206.02262) applies diffusion to GANs as a non-leaky augmentation; Diffusion2GAN (https://arxiv.org/abs/2405.05967) distills a diffusion model into a GAN; PaGoDA (https://arxiv.org/abs/2405.14822) inverts a pretrained diffusion model to obtain the posterior p(z | x) and facilitates the combination of GANs and MLE.

“What is EMA half-life? Is the EMA .999?”

We follow the notation of Karras et al. in StyleGAN and EDM. EMA beta can be obtained using the formula ema_beta = 0.5 ** (batch_size / ema_half_life).

“Why do you schedule Adam B2? This seems very atypical.”

It is atypical. The gradient magnitude of G and D varies drastically early in training. This effect is further amplified when we use a large initial learning rate, leading to large loss oscillations early on. Using a constant Adam beta2 such as 0.99 produces a loss that always stabilizes in a few hundred steps, and the initial oscillations do not seem to have any long-term negative impact. However, by annealing Adam beta2, we can reduce the initial loss oscillations: now, we no longer need to wait a few hundred steps for the loss to settle. In particular, a lower initial Adam beta2 allows the optimizer to adapt to large gradient magnitude changes much quicker and so stabilize training earlier. Similar reasoning for tweaking Adam beta2 is given in EDM2 (https://arxiv.org/abs/2312.02696, pg. 18). The benefit of scheduling Adam beta2 is marginal, but since we already schedule multiple hyperparameters, we might as well schedule Adam beta2 too.

Hyperparameter scheduling (question truncated for space).

We agree that your suggestion will ease the hyperparameter tuning, and we plan to introduce something similar for the automatic configuration of our model.

Thank you for your feedback.

“The novelty of the method is somewhat limited, as both relativistic pairing GAN (RpGAN) and zero-centered gradient penalties (0-GPs) are previously proposed and validated approaches in the field of GANs.”

While these components have been proposed separately, none matches our contributions:

• New theoretical and empirical contribution: 0-GPs have been proposed in the context of classic GANs, but nobody has studied how they interact with the training dynamics of RpGANs. We provide both theoretical and empirical evidence on how they address the non-convergence of RpGANs, which ultimately allows us to derive a well-behaved GAN loss that is resilient to GAN optimization problems.
• New practical contribution: We establish a clean and cohesive GAN framework with just the must-have components. With the new loss, we show that a simple non-StyleGAN based architecture can beat StyleGAN2 by a large margin.

“How scalable is the proposed model? It is suggested to validate the effectiveness of the method on higher-resolution datasets and more complex tasks. This would align with the motivation to serve as a modern baseline, especially in the current context where both model parameter size and training data scale are increasing.”

We present early evidence: To complement our ImageNet-32 result, we are currently running ImageNet-64 experiments by stacking another stage onto our ImageNet-32 architecture. Note that, to evaluate EDM, Karras et al. (2022) used ImageNet-64 as their biggest dataset. Without hyperparameter tuning, we achieved FID 2.56, which is in between ADM (FID: 2.91, 250 DDIM steps) and EDM (FID: 2.23, 79 steps with EDM sampler). We achieve this without techniques like adaGN and attention in ADM/EDM. Our result uses ~90M parameters while EDM and ADM use ~300M. This is evidence that our model is more efficient at its current scale and will hopefully scale well.

For more complex tasks, we are unsure what the reviewer would like to see - please suggest something. To the best of our knowledge, ImageNet is the one of the largest and best studied closed-world datasets. If the reviewer meant extending the setting to open-world text-to-image generation, then this requires significantly more compute (and monetary cost!). It also may require additional techniques like text encoders, self/cross-attention, and conditional modulation.

More generally, addressing scalability is secondary to our top priority of addressing the GAN “myths” that led to the shrinking (as YF9j puts it) of GAN research: GANs are difficult to train, GANs do not work without many tricks, GANs are fragile and do not work with powerful modern vision backbones. Scalability, while also highly important to GAN applications, is dependent upon proving the feasibility of a clean GAN framework, via a minimal model that beats prior work. Of course, we agree with the reviewer on the practical importance of scalability. We are not OpenAI or Meta with thousands of GPUs, but our added experiment will provide evidence for the scalability of our model given our compute constraints.

Finally, we will open source our code and pretrained models and we welcome the community to test the scalability of our model.

"Small-scale experiments to validate the significant improvements in stability and diversity provided by the used loss are insufficient; more complex datasets should be used to verify its effectiveness."

We acknowledge that stability and diversity are only directly validated on StackedMNIST (Figure 1 / Table 1). We have added diversity via the recall metric for all other experiments below. Concerning stability, we propose to add convergence plots for all datasets and for loss-ablated models, similar to Figure 1, in supplemental material, with discussion in the main paper.

That said, StackedMNIST itself should not be dismissed so easily: it is a challenging case for stability because of the limited data and that data having an all-black background. This makes D's job to reject fake samples extremely easy early on, causing instability. FFHQ is generally more stable and the original StyleGAN loss is likely to work in terms of stability, but we show that our approach gives better results in terms of FID.

[Following on; concerning the impact of stability on performance] “As shown in Table 2, the improvement from config C [well-behaved loss] compared to config B [no well-behaved loss] is not significant.”

Improving performance may need both a well-behaved loss and a more powerful backbone, as is the case here. Both Config-B and C have a weak DCGAN-style backbone. Assuming convergence is stable, G and D must still be sufficiently powerful for a given dataset, otherwise performance will saturate regardless of the loss used. With a weak backbone, it is not surprising that replacing the loss does not result in a drastic improvement. With a stronger backbone in D and E, performance increases. Adding convergence plots for this more complex FFHQ data will contextualize this result with respect to stability.

“... and diversity can be measured by metrics such as recall …”

Thanks - our oversight - we report recall numbers below.

On ImageNet-32, we obtain a recall of 0.63, comparable to ADM (recall 0.63).

On FFHQ-256, after gamma hyperparameter tuning, we achieved a lower FID of 2.77 and a recall of 0.50, outperforming StyleGAN2 (FID: 3.78, recall: 0.43).

On CIFAR-10, we achieved a slightly better FID of 1.97 and recall 0.57 after gamma hyperparameter tuning. By comparison, StyleGAN-XL has a slightly lower FID of 1.85 but a much worse recall of 0.47. StyleGAN2-Ada has a considerably worse FID of 2.42 but a slightly higher recall of 0.59. We attribute the recall gap to hyperparameter tuning, as we tune hyperparameters for best FID performance.

For ablations, Config-C, using our new loss, achieved recall 0.24, and Config-B with the original StyleGAN loss achieved a worse recall of 0.22.

Thank you for your feedback.

“there seems to be a lack of information about the training setup, hyperparameters used, number of inference steps, etc pertaining to the diffusion based models in Tables 4, 5 and 6.”

The diffusion model numbers in these tables are directly taken from reports in existing papers. Below, we have added the commonly-reported number of function evaluations (NFEs) for each. For the camera ready, we will add more detailed training setup descriptions and a hyperparameter table.

Table 4:

• LDM: from https://arxiv.org/abs/2112.10752, Table 18, NFE: 200
• ADM (DDIM & DPM-Solver) & Diffusion Autoencoder: from https://arxiv.org/abs/2312.06285, Table 1, NFE: 500

Table 5:

• DDPM: from https://arxiv.org/abs/2006.11239, Table 1, NFE: 1000
• DDIM: from https://arxiv.org/abs/2010.02502, Table 1, NFE: 50
• VE & VP: from https://arxiv.org/abs/2206.00364, Table 2, NFE: 35

Table 6:

• ADM -> https://arxiv.org/pdf/2301.11706 Table 3, NFE: 1000
• DDPM-IP> https://arxiv.org/pdf/2301.11706 Table 3, NFE 1000
• VDM -> https://proceedings.neurips.cc/paper_files/paper/2021/file/b578f2a52a0229873fefc2a4b0 -> NFE 1000
• DDPM++ -> https://openreview.net/forum?id=PxTIG12RRHS -> NFE 1000

“To clarify, on line 132 on page 4, “857” corresponds to StyleGAN backbone trained on WGAN-GP loss and “881” corresponds to the mini-batch standard deviation trick added to the same? Would be better to clarify the exact configuration…”

Not quite. Although the mini-batch standard deviation trick was popularized by StyleGAN, it was introduced in Progressive GAN (https://arxiv.org/abs/1710.10196). Thus, we took these numbers from Table 4 of the Progressive GAN paper. “857” corresponds to a low-capacity version of the progressive GAN trained with WGAN-GP loss and “881” adds the minibatch stddev trick. We will clarify the reference in the camera-ready paper.

“... and include the two cases in table 1 and figure 1.”

While in principle this is a good idea to contextualize the gain, in practice it is a little tricky: ProGAN/StyleGAN are quite different models, and Table/Figure 1 aim to show without complication the effect of different losses on our small simple model. We cannot ‘strip’ ProGAN/StyleGAN meaningfully, and so their performance may be better than what our simple model achieves. We do not want to confuse the reader by adding new lines to the plots that are incomparable, obfuscating the point we are trying to make about stability. If the reviewer has other ideas for how to make this comparison, we welcome them.