Unseen Image Synthesis with Diffusion Models

This article leans more towards a technical discussion. Its ability to generate OOD samples lacks competitiveness compared to models trained in relevant domains. At the same time, many fundamental viewpoints in the article are based on other works, making the overall article lack originality.

Although the paper provides some original ideas and analysis, I was not convinced by the main claims.

• $\underline{\text{Generalization}}$. To claim generalization, it is required to demonstrate that the "new" generated images of OOD domain are significantly different from the $N$ real images used to estimate the gaussian distribution. At the moment, the paper does not convincingly demonstrate that this is indeed the case. While there is a discussion in the context of "mode collapse", it only suggests that generated images are different from each other (and not from $N$ provided images). In general, my intuition is that the "new" generated images look good if their latent representation is close to one of $N$ computed real OOD representations, but can be significantly deteriorated in quality between them. Another aspect that strengthens this concern is the FID of 43+ compared to the upper bounds of 6-8, which demonstrates that the general quality of produced "new" images is not great (which is not a sign good generalization).

  • To extend the analysis the generalization, I think it would be interesting to see a batch of uncurated OOD-generated images put aside their nearest neighbours based on some measure like LPIPS. Another interesting vizualization would be a trajectory of generated images from latent point of one real image to another (e.g., similar to latent space interpolations in GANs).

• $\underline{\text{Claims in bold}}$. I also have questions regarding the insights offered in the paper.

  • $*"DDPM ... has sufficient representation ability to reconstruct arbitrary unseen images"*$. This is hardly an observation, since this ability is constructed directly through the deterministic inversion via DDIM. It is not yet a sign of generalization, since the model can still just reproduce the provided $N$ images or their slight variations.
  • $*"... an interesting fact that contradicts conventional wisdom"*$. This claim is supported only by different values $B$ that were picked manually by hand, based on my understanding. Apart from the fact that manual handpicking is subject to biases, it is not clear how statistically significant is the difference between 0.2 and 0.3 in Table 2. Lastly, the $B$ values themselves are not yet indicators of generalization, as they only show how easy it is to reconstruct a provided image, not how to generate new OOD examples that differ.

Lastly, I would suggest the authors to move the Algorithm 3 to the main paper, since it is in essence the main technical implementation step contains the proposed method.

The proposed method, while innovative, employs existing techniques to identify latent vectors for OOD image generation, which may not be seen as groundbreaking. Along with this, the process outlined in Algorithm 2 (Geometric Optimization) for rejecting and re-sampling latent from empirical mean and covariance resembles the approach of fitting a basic machine learning model to produce these latent vectors, which might not be perceived as a novel solution.

The paper lacks a detailed discussion on the practical implications and applications of the proposed method, particularly in real-world scenarios like text-to-image generation, which could potentially limit its broader relevance and utility. The paper could benefit from a more thorough exploration of the practical use cases or potential domains where the proposed method could be effectively employed.

The process of generating OOD images, as described, follows a logical sequence of steps from inverting images to identifying the conducive region for OOD image generation, which may not evoke surprise among readers familiar with such techniques.

1. The organization. Algorithm 1,2, 3 or something similar need to be put in the main paper. Section 3 jumps to the experiments suddenly.

2. The experiments and observations. Although the observations are quite interesting, it needs more experiments to verify this point. The paper focuses on the tuple <dog, face, bedroom, church>. Given this paper is based on empirical observation, more evidences need to be shown that the observations hold.

3. Some other metrics need to be included, such as diversity and dissimilarity to the original target images. Becasuse a trivial solution can be using the inverted noise to generate the images, which is just the reconstruction of the target domain. So, the FID can be very close to 0 even the proposed algorithm fails totally.

4. The rejection rate of the algorithm need to be discussed. Authors have three rules to reject the sampled latents from a high-dimensional Gaussian prior. The computation time and rejection rates need to be discussed.