Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks

We thank the reviewer YDjR for the positive feedback as well as the relevant comments. We especially appreciated that the reviewer highlighted the interdisciplinary approach, which was at the heart of the article. Unfortunately, interdisciplinarity has its own limitations, especially when it comes to comparing humans and machines. This is the main reason why we have limited ourselves to rather simple datasets. We detail our answer below:

• About the generalization to other and more complex datasets: We want to clarify that our experiments on the Omniglot datasets are actually not 'limited'. We placed the Omniglot results in the supplementary information section to keep a concise main-text article, but the Omniglot analysis is consistent with that conducted on the QuickDraw Dataset. In this article, our focus has been primarily on the one-shot drawing task because it allows fair comparison between humans and machines; more complex tasks like natural image generation by contrast are beyond human capability. Nevertheless, the latent diffusion models and the regularizers we have used in the article are known to scale well on more complex datasets (e.g. [1]). We anticipate similar performance improvements on natural image datasets as observed with the QuickDraw data, especially since regularizers like Barlow, SimCLR, and prototypical have shown strong performance in one-shot classification tasks of natural images. However such natural image generation models won’t be comparable to human performance as humans can hardly synthesize such images.

• About the limitations: We agree that the limitation should be in the main text and not in the appendix. We have therefore added a paragraph summarizing our limitation in the discussion section (line 351)

Overall, we think our response has addressed the primary concern of the reviewer, clarifying that our deliberate choice of a relatively simple one-shot drawing setting allows us to draw a fair comparison between humans and machines to effectively answer our scientific question. We hope we have convinced the reviewer to increase their rating. Should there be any remaining issues, we are more than willing to engage in further discussion.

[1] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Should there be any remaining issues, we are more than willing to engage in further discussion.

We thank the reviewer 1Pi8 for the meaningful comments. Please find below a point-by-point response that addresses the reviewer’s concerns:

• About the incremental work compared to [30] : We agree with the reviewer that our article builds on the comparison framework and some ideas introduced in [30], but we respectfully disagree that our article does not bring significant novelties compared to [30]. Here are the 2 main novelties:

  • While [30] focuses on comparing various types of generative models (GANs, VAEs, and diffusion models), we focus on comparing the effect of inductive biases (through regularization) in the latent space of latent-diffusion models. This difference might seem minor to the reviewer, but this question of effective inductive biases is prevalent in cognitive psychology (and is still an open question, see [1, 2] below) and has never been systematically studied in latent-diffusion models from a machine learning perspective. Our results confirm how crucial such inductive biases are as it has a strong impact on the generalization performance of the one-shot drawing setting.
  • The method used in [30] to generate feature importance maps leads to maps in which the background is dominant (see Fig 5 in [30]), preventing the authors from making any meaningful and quantifiable comparison with human feature importance maps. In our article, we introduce a novel method to derive importance maps. In contrast to [30] our method leads to importance maps that are directly comparable with those of humans. We think this is an important difference with [30] as our results are backed by two independent methods (the recognizability from originality methods, and the importance maps comparison), strengthening our claim.

  To clarify and emphasize the differences with [30] we have added a few extra sentences to summarize those 2 major differences in the related work section (lines 129-130).

• About the claim that “the gap between humans and machines is almost close” that might be a bit overstated: We agree with the reviewer that this claim may be somewhat overstated, considering that our comparison between humans and machines is based on a very specific task, which does not cover the full spectrum of abilities in both humans and machines. We have therefore downplayed this claim and we have replaced it in the narrower context of our article (in the abstract, and in the discussion section).

• About the comparison with the DDPM: We have actually included the DDPM from [30] in our comparison, this is what we call the pixel baseline (see Fig 2). Note that our baseline is a guided version of the DDPM because previous articles have shown that guided DDPM performs better than their non-guided counterpart in the one-shot generation task. We call it ‘pixel-baseline’ , because the DDPM is directly applied in the pixel space in contrast to other latent diffusion models that are applied in a regularized latent space. But we agree with the reviewer that this designation is rather ambiguous. We have therefore changed it to ‘pixel-space DDPM’.

• About the sketch codebook of the VQ-VAE: The codebook in the VQ-VAE could be viewed as a dictionary of vectors, with each vector being learned so that it minimizes the L2 distance with the latent coordinate. Note that the way we learn the codebook is similar to the standard procedure described in the original VQ-VAE article [3]. To be more concrete, let’s consider the case where the latent space (before discretization) is of size (4, 128) (here we ignore the batch dimension for the sake of concision). Let’s also consider we have a codebook with 512 elements (this is the codebook size we use in the article). In this case, all 512 vectors of the codebook will be of size 4, so that they match the number of channels of the latent space. During learning, the codebook vectors are learned so that they minimize the L2 distance between 128 vectors (of size 4) of the latent space (see section A.2.1. for a pseudo-code). During inference, the discretization process associates to each of the 128 vectors of the latent space the address (i.e. an integer) of the closest vector in the notebook. This allows the transformation of the continuous-valued latent space into a discretized one. We agree with the reviewer that a clear explanation of codebook learning is missing in the article. We have included a scheme to explain this process in section A.2.1 to clarify this point.

Overall, we hope that our detailed response has addressed the reviewer's concerns, and convinced them to increase their rating. If some concerns remain, we will be pleased to engage into more discussion.

[1] Goyal, Anirudh, and Yoshua Bengio. "Inductive biases for deep learning of higher-level cognition." Proceedings of the Royal Society A 478.2266 (2022): 20210068.
[2] Marjieh, Raja, et al. "Using Contrastive Learning with Generative Similarity to Learn Spaces that Capture Human Inductive Biases." arXiv preprint arXiv:2405.19420 (2024).
[3] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." Advances in neural information processing systems 30 (2017).
[30] Boutin, Victor, et al. "Diffusion models as artists: are we closing the gap between humans and machines?." Proceedings of the 40th International Conference on Machine Learning. 2023.

Should there be any remaining issues, we are more than willing to engage in further discussion.

We thank the reviewer ZReU for the detailed comments. Here is our point-by-point answer:

• About the lack of analysis of the different regularizer components: We agree with the reviewer that we did not discuss enough the reasons for such differences. As this issue was shared with other reviewers, we have answered this concern in the bullet point 2), in the general rebuttal form.

• About considering multimodal and more complex datasets: We acknowledge that the latent diffusion models we use do not face scalability issues and can handle multiple modalities. As our scientific question involves tight comparisons between humans and machines, we focus on tasks accessible for both humans and machines (which is not the case for natural image synthesis.Therefore, the focus on the one-shot drawing task is deliberate and aligned with the scientific question we explore in this article.

• About the methods used to compare machines and humans and about stroke analysis: In this article, we used two independent methods to compare humans and machines —the recognizability vs. originality framework and feature importance maps. Both methods lead to the same conclusion: certain regularizers help narrow the gap with humans in one-shot drawing tasks. But we agree with the reviewer that stroke analysis could be another interesting approach to analyze human drawings. We plan to include such analysis in further work, so that we could compare all 3 different comparison methods.

• About the low originality value on impact when evaluating human samples: There might be a misunderstanding on this specific point. The originality plays a major role when comparing various types of generative models (e.g. VAEs, GANs, …) to humans as shown by previous authors (see Fig 3 of [1], or in Fig 2 of [2] ). But in this article, we focus on one particular type of generative model that are the latent diffusion models. Because such models tend to fall close to humans in terms of originality, we have rescaled the originality vs. diversity framework to zoom in and better highlight the differences between humans and machines (note the originality scale starting at 0.5). Such a rescaling might be the reason why the reviewer thinks that originality plays a minor role in the evaluation process. It is also important to note that there is an inherent tradeoff between originality and recognizability: while recognizability assesses how likely the data point falls in the classifier decision boundary, originality measures how ‘diffuse’ the sample distribution is (the Fig 1 in [1] well illustrate this trade-off)... Therefore, a very ‘original’ agent (producing highly diverse samples) will tend to have low recognizability as the samples are likely to fall outside of the classifier decision boundary. In Fig 2, we observe that models need to trade their originality for more recognizability to better approximate human performance. We have included two sentences in the main text (line 245) to clear up this misunderstanding.

• About the possible biases of the classifier to evaluate the recognizability: We fully agree with the reviewer that the classifier used to evaluate the recognizability might be biased. Hoewer, we think these biases have a low impact on our analysis for two main reasons. First, we use a one-shot classifier to evaluate the recognizability (similar to the original paper that introduces the originality vs recognizability metrics [1]). Such classifiers are less prone to overfitting by construction (as they learn a metric space, see [3] for more explanation), mitigating the impact of potential biases in the training distribution. Second and most importantly, all samples (either human-drawn or machine-generated) are evaluated with the same classifier. Therefore, any potential biases in the recognizability metric will equally affect both human and machine performance, ensuring that the comparison remains meaningful. We propose to include such an explanation in the main text (line 242) to clarify this point.

• About other approaches to improve performance on the one-shot drawing task (i.e. more combinations of regularizer): As this specific point was also raised by reviewer vmMb we have run more experiments to systematically explore more combinations of regularizers. Those experiments lead to one more figure. More details are in the bullet point 3 of the general rebuttal form.

• About the pixel baseline and the rasterization: Our pixel baseline is indeed a diffusion model (without any latent projection) to learn to generate the distribution of pixel value. It indeed leverages rasterized images, as the original quickdraw images are vectorized.

• About the alignment of the metrics with human judgment: We fully agree with the reviewer that having evaluation metrics (such as originality and recognizability) that align with human judgment would make our analysis more impactful. We are currently working on finding more aligned metrics (using harmonization techniques combined with psychophysics experiments). We have added 2 sentences in the discussion section (line 381) to discuss this interesting point. We thank the reviewer for the valuable comment.

We believe that our response, along with the additional curves and paragraph included in the article, addresses most of the reviewer's concerns and should encourage them to raise their rating.

[1] Boutin, Victor, et al. "Diversity vs. Recognizability: Human-like generalization in one-shot generative models." Advances in Neural Information Processing Systems 35 (2022): 20933-20946.
[2] Boutin, Victor, et al. "Diffusion models as artists: are we closing the gap between humans and machines?." Proceedings of the 40th International Conference on Machine Learning. 2023.
[3] Snell, Jake, Kevin Swersky, and Richard Zemel. "Prototypical networks for few-shot learning." Advances in neural information processing systems 30 (2017).

Should there be any remaining issues, we are more than willing to engage in further discussion.

We thank the reviewer vmMb for the relevant and valuable comments. Please find below a point-by-point to answer that answer the main concerns of the reviewer :

• About the lack of analysis on the impact of the different regularizers on the performance: In Fig 2, the comparison between the regularized latent diffusion models (shown in color) with the no-regularization baseline (hexagon marker) shows how the different regularizers contribute to the global performance (as measured by the originality vs. recognizability framework). However, we acknowledge the reviewer's concern that we did not discuss enough why the regularizers produce such effects. To mitigate this issue we have added an entire paragraph (at the end of the results section) that better explains these differences. Because it was also requested by another reviewer, we have included this paragraph in the general response (see bullet point 2). We thank the reviewer for raising this issue, as we think it improves the quality of the article.

• About the human evaluation of the samples: Human evaluation of the generated samples has already been proposed by previous studies. For example, [1] did a Turing test where humans had to distinguish between human-drawn and machine-generated samples. While insightful, such experiments give few insights into how the samples differ from machine to human. On the contrary, we think originality vs. recognizability offers a finer comparison between the distribution of images drawn by humans and those generated by machines.

• About systematic investigations of the different regularizers combinations: To address the reviewer’s concern, we have run additional experiments to systematically explore various combinations of regularizations. In particular, we have systematically explored the following combinations of regularization Barlow + Prototype: KL + prototype, SimCLR+Prototype, VQ + Prototype. Among all combinations of regularizers, it is the Barlow + Prototype and the KL + Prototype combinations that perform the best. Interestingly, the good performance of the combination KL + Prototype was not expected, because the KL regularizer (when used separately) shows a low recognizability (see Fig2). On the other hand, the VQ + Prototype combination shows no improvement compared to the Prototype regularizer. Overall, these additional experiments confirm the potential of combining an unsupervised and a supervised regularizer to match human performance. These experiments lead to one additional figure (included in the one-page PDF allowed in the general response). Note that we have also explored all combinations between unsupervised and the classifier regularizer, but we did not observe any significant improvements (we have included those experiments in the supplementary information of the revised version).

• About generalization to more complex databases: Note that the diffusion models as well as the RAEs we use in these articles are all known to scale well to larger datasets [2]. So in theory nothing prevents us from applying the proposed regularization in more complex settings. However, the point of our article is to compare humans with machines. To do so, one needs to leverage tasks that are accessible by both humans and machines, which is not the case with natural image synthesis: humans can hardly produce images that resemble natural images. We have chosen the one-shot drawing task because it offers a leveled playfield to compare humans and machines. To make this point clearer, we have included a sentence in line 133.

Overall, we hope that our detailed response has addressed the reviewer's concerns, and convinced them to increase their rating. Should there be any remaining issues, we are more than willing to engage in further discussion.

[1]: Lake, Brenden M., Ruslan Salakhutdinov, and Joshua B. Tenenbaum. "Human-level concept learning through probabilistic program induction." Science 350.6266 (2015): 1332-1338.
[2]: Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Should there be any remaining issues, we are more than willing to engage in further discussion.