Modelling complex vector drawings with stroke-clouds

Thank you for the detailed feedback and appreciation of our use of the set transformer. We address each question one by one:

(R2-1) The visual quality of the sketches We focus on complex vector sketch generation as a new genre where sketches can have a large number of constituent strokes. We make no assumption about the exact nature of sketches in our theory as long as it fits into the said criteria. However, one needs to take into account the fact that there are no large-scale vector datasets of complex sketches available. The limitation on the visual quality of sketches primarily comes from the training data and the quality of the outputs of the available vectorization methods.

(R2-2) Bezier curves complexity Firstly, we would like to mention that, even though we used quadratic Bezier curves as a design choice, we never made any hard assumptions about the exact stroke representation in our theoretical setup. To further clarify any doubts, we have now added Figure 17 to the appendix, generating sketches from a model trained on the QuickDraw dataset. In this case, we used Bezier curves with 15 control points (able to represent significantly complex strokes). It can be seen that our model copes well with more complex strokes when the appropriate training data is available.

(R2-3) Additional stroke parameters Thanks, great suggestion. While we leave extensive evaluation of adding additional properties to strokes to future work, as a proof of concept, we experimented with adding line thickness to a set of control point positions, describing each stroke. A newly added Figure 16 in the supplemental shows that our model manages to leverage line thickness as well and represents eyelashes with thicker lines, showcasing flexibility in our set transformer approach.

(R2-3) Quantitate results

Thank you for the suggestion. We have added a comparison of the FID scores of the generated drawings under different sampling conditions in section 4.2 Table 1.

(R2-4) Interpolation results in Figure 8 We agree that the interpolated results in Figure 8 (now Figure 10.) are of lower quality. We however argue that this might not be a limitation of our SRM module but is a result of the lack of training data for the latent generative model. This however should not weaken our main contribution which is complex vector sketch generation. We will now move this Figure to the Appendix and provide a careful discussion around it.

(R2-5) The advantage of our approach of direct vector sketch generation over raster sketch generation followed by a vectorization step.

This is exactly where our method shines! Our approach is very time-efficient, something we believe is critically important for complex vector sketch generation — it generates a 1000-stroke drawing in just ~0.05 seconds. For comparison, it takes 10-15 seconds to vectorize a raster sketch with the approach we used to prepare the training set. Moreover, being able to generate vector sketches directly has the potential to provide more control over the generation process and editability of the output.

Thank you for your detailed comments!

(R3-1) Introduction
Thanks, we have now revised the introduction to remove subjective claims. Our main point is that the vector format is not only a scalable alternative for a raster format, but it also allows us to model additional information lost in the raster representation: that of each drawing and sketch being a collection of individual strokes. Such representation enables more fine-grained editing taking the artwork creation process into account. Please do inform us if this is now more accurate than previously stated — it certainly was not our intention to overclaim!

(R3-2) Additional references

We focus our discussion on recent papers as the focus of our paper is on deep generative approaches for vector graphics. We are happy to provide a more extensive discussion of other related topics if reviewers think that this will benefit the paper's exposition.

(R3-3) The use of quadratic curves to represent strokes (strokes complexity)

First, we would like to stress that, theoretically, SRM has no restriction on the exact stroke model to use. The reason why we used quadratic Bezier curves is that the method used to create vector sketches produces fairly simple strokes, and quadratic Bezier curves are more than enough to model strokes in the training set.

To show that our model supports Bezier curves of a high degree (can represent more complex strokes), we train on the sketches from the QuickDraw dataset with Bezier curves with 15 control points. We added Figure 17 to the Appendix which shows that our model can flawlessly support more complex curves.

(R3-4) Additional stroke parameters (Same as R2-3) Thanks, great suggestion. While we leave extensive evaluation of adding additional properties to strokes to future work, as a proof of concept, we experimented with adding line thickness to a set of control point positions, describing each stroke. A newly added Figure 16 in the supplemental shows that our model manages to leverage line thickness as well and represents eyelashes with thicker lines, showcasing flexibility in our approach.

(R3-5) Results view angles and classes

The short answer is the results simply reflect the training data. We train only on the dataset of faces. But of course, our method is agnostic to the exact nature of data, as long as they can be categorized as complex sketches.

Additional results for a different dataset (QuickDraw) are shown in the newly added Figure 17 in the Appendix.

(R3-6) Generating a sketch of a particular noun class

Enabling more classes is possible in a naive setting by sourcing a similar dataset where we can readily obtain vectorised training sketches, and training on millions of these sketches. This was however not possible within the rebuttal period, especially recognising our limited computation resources. Another option could be conditioning our SRM module on feature spaces of large pretrained models and focusing on the transfer learning problem, this is however out of the scope of our work.

(R3-7) Numerical evaluation

Thanks. We have added a comparison of the FID scores of the generated drawings under different sampling conditions in Section 4.2 Table 1.

Summary

Given the answers 3-6, we expect our model to generalize well to new classes and styles, with stroke design being an area of study moving forward.

Thank you for a very detailed review, particularly your appreciation of the challenging and forward-looking nature of the problem we solve, and our idea of using a set representation approach to model vector graphics.

Below we answer each question one by one.

Strokes sampling during inference

SRM (Set Reconstruction Module) is trained on sketches with a variable number of strokes, thanks to the arbitrary sum (i.e. $\sum_{\mathbf{s}\in \mathcal{S}}$) present in the true likelihood in Eq(3). Our SRM is conditioned on a latent vector, and at inference time, we can indeed generate any number of strokes. However, this is a parameter that has to be set manually. In Section 4.2 we discuss what happens when we vary the number of strokes during generation, and Figure 4 shows the results with varying numbers of strokes. For the results in the paper (unless specified otherwise), we generated 1000 strokes per sketch as we found it to produce good results. Since we assume that strokes are IID, we need to sample more strokes than the average cardinality of sketches in the training set (in Anime-Vec10k, there are 305 strokes per sketch on average). We discuss in detail in Section 4.2 oversampling and under-sampling.

Assumption of IID strokes:

We do not claim that strokes are independent — the strokes are indeed correlated in their true sense. However, we use the fact that the strokes are conditionally independent given a set representation (i.e. a compact latent code), powered by De-Finetti’s Theorem. The implication of this is that conditioned on a latent set representation, our SRM model will not result in a degenerate solution.

How is the sequence $\mathbf{s}$ generated at training time:

Thanks, but there might be a misunderstanding here. We provide below clarifications on our interpretation of the question — “In the same way, how is the sequence generated at training time? Do you let the randomness of the random strokes take care of matching the proper final stroke? Why is there no Hungarian matching? Assuming normalized coordinates in [-1,1]x[1,1], how does it make sense for the denoising network to take a random stroke at (-1, -1) and try to denoise it to (1,1), instead of just recognizing (with Hungarian matching) or simple Euclidean distance that it is much easier to denoise the stroke closer to the final location?”. Please do get back if this is incorrect — we will endeavor to respond promptly.

First and foremost, we do not deal with sequence representations. Each stroke is a 6-dimensional vector, as described in the beginning of Section 3. Also, we do not match strokes explicitly during training. During training, for each noise level, we pass to our MLP the noisy strokes (noisy control points, essentially) with added noise corresponding to the given noise level, and the latent code representing the set. Our MLP is simply trained to return a noise estimation per stroke, conditioned on the set representation. The simplicity of the SRM formulation is that it can simply be seen as training any other conditional diffusion model where the set latent is the condition and the strokes of one sketch as IID data points.

Numerical comparisons

Thanks. We have added a comparison of the FID scores of the generated drawings under different sampling conditions in Section 4.2 Table 1.

Additional references

Thanks for the additional references! We added a discussion of additional works to a revised introduction.

Additional qualitative results

Thanks for the suggestions! We have added additional qualitative results in Figure 7 and Figure 12 as well as examples of the training data in Figure 13.

Figure 5: Why does the DDPM sampler seem to have more strokes than the DDIM sampler? or are the strokes just placed closer together in the DDIM sample?

All samples generated in Figure 5 have the same number of generated strokes. Since DDPM sampling is stochastic, there is more diversity in stroke positioning. In DDIM sampling (which is deterministic), strokes are positioned so closely that it is hard to see that there are multiple strokes on top of each other. In Figure 15 in the Appendix, we reduce stroke width to better show the alignment between strokes for the DDIM sampling method.

Do you plan to release the dataset?

Yes, most definitely — this is a dataset that (we think) is much needed to progress the sketch community in terms of studying complex sketches. We will also release the code as well.