Towards Well-distributed Generative Networks Using Adversarial Autoencoders

1.$\$There are several relevant works that are not being mentioned or fully discussed, which maintain high similarity to the proposed method. [1] has already provided a more precise definition and analysis of mode collapse and proposes the mode completeness condition that requires consistent probability measures regarding samples. Moreover, in methodology, matching the real data's inverses with the latent prior is somehow a general design as similar mechanisms are adopted in [1,2,3,4]. There are also recent (in two years) efforts on solving mode collapse that need to be discussed and compared, including [1,5,6].

2.$\$The claims about the existing metrics like "we found that these metrics correlate poorly with the reduced chi-square, which shows that they are not suitable for exact measurements" in Sec. 5.1 are confusing. Previous metrics like FID evaluate the distance between the feature distributions of the generated data and real data, which surely incorporate diversity evaluation as well as image quality and are meaningful. It is meaningless to evaluate diversity without a certain quality guarantee.

3.$\$The empirical results are limited and confusing. 1) The comparison merely involves StyleGAN2 variants, leaving alone other recent efforts on solving mode collapse. 2) The results of precision and recall failed. It would be better to consider other metrics like density and coverage [7]. 3) FID results in Table 1 indicate that the proposed method is worse in generation quality even with an advantage in diversity, which makes the results somehow meaningless.

[1] IID-GAN: an IID Sampling Perspective for Regularizing Mode Collapse. IJCAI 2023.

[2] Vee-gan: Reducing mode collapse in gans using implicit variational learning. NIPS 2017.

[3] Mggan: Solving mode collapse using manifold-guided training. ICCVW 2021.

[4] Dist-gan: An improved gan using distance constraints. ECCV 2018.

[5] Improving Generative Adversarial Networks via Adversarial Learning in Latent Space. NeurIPS 2022.

[6] UniGAN: Reducing Mode Collapse in GANs using a Uniform Generator. NeurIPS 2022.

[7] Reliable fidelity and diversity metrics for generative model. ICML 2020.

• The presentation and clarity of the paper could be improved.
• The statements without support should be substantiated with appropriate references or experimental results.
• The suggested reduced chi-square metric does not correlate with widely adopted evaluation metrics, such as FID, precision, and recall (Table 1). The proposed method achieves better results only in the reduced chi-square metric. This paper should provide a justification for the superiority of the reduced chi-square metric to claim the superiority of the proposed model.

• The claim is not well justified. The authors mentioned in the abstract that their proposed method ensures that the density of the encoder’s aggregate output distribution closely matches the prior latent distribution. However, there is neither theoretical justification nor sufficient empirical verification for this. I understand that the authors plotted in Fig. 2 of the distribution of bins, but still the curve produced by the proposed method does not match the theoretical curve. The experiment is only done on the 3DShapes dataset, and the latents have a uniform distribution. Experiments are not sufficient. The authors need to provide quantitative results to measure the overall performance of the proposed method. The method should be compared to StyleGAN in terms of generated images using FID and Inception Scores on large datasets such as LSUN, FFHQ and ImageNet datasets.

• Even for the 3DShapes dataset, the experimental results are not good. In table 1, while the method is slightly better than StyleGAN-XL in $\mathcal{X}_{\nu}^2$, but it is much worse than StyleGAN-XL in FID. Ultimately, we hope the generated images to be as realistic as possible.

• The organization of this paper is not good. The authors introduced the dataset and evaluation method in Sec. 3. I think the proposed method is specifically designed to address the challenges for the 3DShapes dataset. However, the 3DShapes dataset can be created automatically. A generative model is not necessary for generating these images. Why do we need to spend effort design specific methods for this task?

• Many parts of this paper require clarification:

  • Why "being able to generate every image in the data distribution does not imply reproducing the correct distribution"?
  • What data are used to produce Fig. 1? Are they two-dimensional toy data? Is the visualization in Fig. 1 in the data space or latent space? Looks like the discriminator is a linear classifier. Since the data points forms a circle distribution, I don't think a linear discriminator has sufficient capacity of differentiate between real data and generated data points. This experiment can not prove that oscillation problem is caused by adversarial training, but could be caused by discriminator's capacity.
  • "a recognition network that computes the six generating factors from an image". How accurate is that network? I think the performance of this network could impact the evaluation of different methods.
  • The authors mentioned: "Note that $x − y$ will always have a mean of 0 and thus cannot oscillate". What does oscillation mean for random variables? Why having a mean of 0 prevents oscillation?
  • In the experiments, the authors need to demonstrate how large batches improves the performance.
  • The well-structured latent space of $G$ is approximately a Gaussian distribution, since in Eq. 4 the authors regard the Gaussian distribution as real data. The well-trained encoder $E$ will encode the real data to a Gaussian distribution in the latent space. Why the authors call it well-structured?
  • While the discriminator in Eq. 7 accepts a pair of images is interesting, the authors need to justify the effectiveness of concatenating a pair of images along the channel dimension as input for the discriminator in Eq. 7.

The paper's motivation seems unclear and divided between solving multiple problems that do not necessarily fall in line. The solution proposed also seems incremental with inadequate quantitative and qualitative evidence to show in favour. Also, there are numerous typographical/grammatical errors present in the article.