Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision

Thank you for the feedback on our work.

W1. Regarding the technical novelty, please note that Grimore’s contribution is threefold: 1. A primitive-agnostic model that allows generation of many SVG attributes using raster supervision, 2. The novel use of a discrete auto-encoder architecture for vector graphic generation that simplifies the problem by deconstructing it into the learning of fundamental vector primitives that can be effectively composed to create complex graphics. 3. The introduction of the geometry loss as a new training objective function, which efficiently prevents artifacts from control points as discussed in Appendix A.8.

W2. Thanks for the great question. We have implemented your suggestion regarding the development of a segmentation strategy to create patches, and tested this approach on more complex data.

We have added results using an additional layer-extraction method based on segmentation masks generated from the Segment Anything (SAM) model [1] and a new compositing method that allows us to reconstruct all the layers similarly to real SVGs. SAM provides very meaningful masks, which require just minor refinement to ensure – for instance – one single connected component per layer. Overall, the VSQ outputs look good.

We have updated the manuscript to provide some results for these experiments, and added an extensive explanation of the segmentation-based patch extraction method (Appendix A.1). We would be happy to incorporate these results into the main paper for the camera-ready version.

W3. We compare with Im2Vec because all other methods require vector data for training or do not support training at all (see next point). Although we agree that for some datasets it would be possible (e.g., FIGR-8 comes with an SVG version), this comparison becomes challenging on a raster-only dataset such as MNIST or emojis.

W4. Could the reviewer clarify which loss comparison this point is referring to?

W5. In Appendix A.2 we now report a comparison of inference times with different SDS-based methods showing that Grimoire is more than two orders of magnitude faster and those methods have impractical inference time (VectorFusion takes more than five minutes for one generation! Our ART module takes around two seconds by average). We report the table below for your convenience.

We also show how when we want to train a neural network, the zero-shot generations of SDS methods, despite looking more artistic, are incoherent with the dataset and do not fit for a reasonable comparison.

W6. Thanks for pointing this out. We have updated the revised version of the paper accordingly.

In response to your questions:

Q1. We have further trained our VSQ model on more complex data (emojis) based on patches extracted using the segmentation dataset. Despite training only on 120 emojis, the results on validation data look promising even on emojis with many details. Despite these results being already of higher quality than what is reported by Im2Vec, we also wish to mention that the authors of Im2Vec had conveniently hardcoded the colors in their experiments on emojis (e.g., one yellow path, two black paths for the eyes, etc.). Without this hack, their model would not converge at all.

Q2. In general, this number is known during the training of the VSQ, which has to reconstruct each patch, and is learned dynamically by the ART module using the <EOS> code token.

Specifically for the MNIST dataset, this number is fixed to the number of tiles during the VSQ training.

Q3 Since a better position of the control point translates into minor artifacts, we perform a sweep for multiple values of the weighting factor on the validation set of FIGR-8 and choose the one that produced the best reconstruction. We have reported qualitative examples of reconstruction with different weights in Appendix A.8.

[1] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., ... & Girshick, R. (2023). Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4015-4026).

[1] We used the ViT-H default backbone.

Thanks for your valuable input and questions, especially about more complex results and a comparison with SDS-based methods.

W1. Thanks for your suggestion. We extended Grimoire to predict closed shapes as layers, using a segmentation neural network to automatically extract the regions to model as layers. This allowed us to report results on more complex imaging such as emojis as suggested. The final image is obtained by compositing all layers together similarly to standard SVGs. We have added an extensive explanation of the approach in Appendix A.1.

Moreover, we have included a qualitative comparison of generations with SDS-based methods in Appendix A.2 on the revised paper. Our experiments show how even if those models generate more artistic results, they offer no control on the target data. Compared to Grimoire, they fall short with respect to a target dataset and make finetuning extremely hard.
Also, unlike Grimoire, those methods are extremely slow and impractical, requiring up to five minutes for a single generation. We have included this comparison in Table 4 of Section A.2 in our revised version of the paper. We report these values below for your convenience.

W2. Regarding the technical novelty, please note that Grimore’s contribution is threefold: 1. A primitive-agnostic model that allows the generation of many SVG attributes using raster supervision, 2. The novel use of a discrete auto-encoder architecture for vector graphic generation that simplifies the problem by deconstructing it into the learning of fundamental vector primitives that can be effectively composed to create complex graphics. 3. The introduction of the geometry loss as a new training objective function, which efficiently prevents artifacts from control points as discussed in Appendix A.8.

Regarding the questions on the method:

Q1.

"each of the xi codes is projected to q dimensions", does this mean the results are d x q and there are xi codes for each patch?

No, $d$ is the dimension being projected. Each of the $\xi$ codes is projected from $d$ to $q$ ($q$ being the number of dimensions of our codebook), resulting in $\xi \times q$ values per shape in total.
We updated this part in the paper to make it more explicit that the dimension d is the one being projected.

$\hat{z}_i$ is the quantized code value for one of the $i$-th of $\xi$ codes.

We chose this notation instead of reporting the dimensions of all codes together because it comes handy for Equation (1). We have added a new image in Appendix A.3 that clarifies each dimension of the embeddings.

How does the "transformation ensures that each unique combination of quantized values is mapped to a unique code"?

Each code $v_i$ is a single-dimensional integer value obtained according to Equation (1).

Equation (1) ensures a “unique combination of quantized values” because the quantized value $\hat{z}_i$ for a given dimension $j$ of the hypercube is first multiplied by the basis value $b_ij$, which accounts for all the values of all the dimensions below in the hypercube. This ensures that the contribution of the code of each dimension is a unique value, before all contributions are summed together. We refer to the original FSQ paper for more information, but we are also available for further clarifications.

Q2.

“How to make sure that strokes from different patches are connected to each other?”,

Our post-processing algorithms reported in Appendix A.2 play a relevant role in making all strokes connected. However, we found the grid size that quantizes the stroke coordinates (described in Line 286) to have major importance in the smoothness of the connected strokes. Small grid sizes yield a disconnected appearance regardless of the post-processing, while higher grid sizes mitigate the need for post-processing because the points are already closer.

Thank you for your valuable feedback.

W1.

• We fixed the typo in the notation at Line 265

• We have created an Experimental Setup paragraph as you recommended inside the dataset section and streamlined the method section.

• We have included a new diagram in A.3 to clarify the training of the VSQ, as you suggested. We plan to further polish the figure for the camera ready.

• We can shrink the related work for the camera ready and instead incorporate into the main paper the qualitative results on emojis currently in Appendix A.1 or the new diagram.

W2. Vector-supervised methods cannot be tested on the raster datasets. FIGR-8 offers an SVG version, which the authors from IconShop have used to train their model. This model has slightly better performance than Grimoire with a CLIPScore of 31.18 (vs 29.00) and FID score of 0.40 (vs 0.64).

However, this comparison should be interpreted with caution, as the nature of the training signal differs significantly. Unlike raster strokes, IconShop requires vector supervision and learns directly from precise position tokens.

We are working to provide you with results on DeepSVG, depending on the complexity of their tokenization pipeline we will share some updates by the end of the extended discussion period.

Concerning your question:

Q1. All the tables (Table 1, 2) and figures (Figure 7, 8) about reconstruction use the test sets. All Tables for generations (Table 3) use the test set for the FID score. Generated images (Figure 4, 5, 6) are not linked to any split. We have now included this information – when relevant – in the captions of the revised manuscript.

Great! This analysis, including comparison along different axes, helps better position the proposed approach and adds a significant value to the evaluations. With that, all my concerns have been addressed. As before, I am leaning positive.

Dear Reviewer ymmo,

We have extended our analysis by training DeepSVG on the same FIGR-8 split used for Grimoire.

Unlike Grimoire, DeepSVG supports conditioning only on class identifiers rather than natural language prompts. Therefore, we assigned a unique identifier to each class in FIGR-8.

The evaluation of the generative capabilities highlights that DeepSVG struggles to handle the complexity and diversity of FIGR-8, yielding the lowest CLIPScore and FID performance among all models.

Below, we present a table with all the scores, highlighting the key features of each approach. We are willing to incorporate those additional results into our manuscript.

We hope that our enriched analysis helps to better position our method and ultimately addresses all your concerns about our submission.

Thank you for your valuable and thoughtful feedback.

W1. We would like to reemphasize the positioning of our method:

Grimoire is a scalable, effective and primitive-agnostic generative framework for the SVG standard that uses only raster supervision for training.

All the works mentioned in your review use vector supervision directly and require vector data, and CAD is even a different standard. Grimoire learns from raster images, which is a significant difference.

Our model offers different opportunities compared to vector-supervised methods. For instance, how would you condition IconShop to your own input shape designed with a specific color or thickness? This must be created in a vector format (aligned with the tokenizer's expected input) and constrained to the features enabled by the original vector training data. In contrast, raster drawings offer greater flexibility for this task.

However, we acknowledge your concerns about the complexity of the raster data used in this work. To address this, we have included results on an additional patch-extraction pipeline, based on the output of the Segment Anything Model (SAM)[1]. In this variant, each mask generated by SAM is used to extract a single raster layer. We applied this pipeline to emojis in PNG format, training Grimoire to reconstruct each shape as an independent vector layer. Finally, we composite all layers for visualization, mimicking the structure of real SVGs. How could this data be handled with a vector-supervised model like IconShop?

W2. Vectorizing raster data for training a vector-supervised model is counter-intuitive and can be extremely inefficient, especially in contexts where new training data is constantly being produced and makes vectorization a bottleneck. In general, vectorizers are slow (The paper you mentioned takes at least 10 seconds per image, LIVE takes minutes per image), and their processing time must be put on top of the training time of the vector-supervised model.

W3. After working on this project for more than a year, we can confidently claim that although the wholistic representation might at first glance look better, this is actually extremely sub-optimal when using raster supervision; at least in the early stages of the training.

As we already mentioned to other reviewers, the experiments on emojis from Im2Vec had the ground truth colors hard-coded to facilitate convergence. Without this hack, their model does not work. We overcame Im2Vec limitations by breaking down the complexity of SVGs into single, simpler components to composite together. The two stage architecture also plays a crucial role to achieve valid generations: we let the model first focus on the visual primitives, and learn to place those onto the canvas in a second stage.

We have included the additional experiments on more complex images in the Appendix A.1.

[1] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., ... & Girshick, R. (2023). Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4015-4026).

[1] We used the ViT-h default backbone.

Thank you for the further clarification on your concerns.

Indeed, the goal of Grimoire is not to reconstruct vector data, but rather learning to generate new images from raster data.

Regarding the practical uses of the generation capabilities, we would like to draw your attention to the results in Appendix A.1, which involve more complex data and realistic SVG representations. To complement this, we wish to share with you some samples of these new reconstructions. We have uploaded those SVGs into the “ Showcase” folder in our repository.

Here is a link for your convenience: LINK

We hope that upon inspection, you will acknowledge that although there is room for improvement in quality in future work (as the goal of this study is not to produce a finalized product but to advance research on generative models under raster supervision), Grimoire demonstrates its potential to effectively generate vector graphics that closely resemble authentic SVG creations.

We believe this method can clearly be beneficial for practical use cases such as logo design.

In the Showcase folder, we have also included a font that uses letters and numbers generated by Grimoire. We encourage you to install the .TTF file and try out the generations.