Highly Compressed Tokenizer Can Generate Without Training

Thank you for your review! We hope to address some of your concerns in this response.

• Image Editing vs. Generation

We decided to choose the "image generation" nomenclature in line with previous work on so-called retrieval-augmented generative models, such as RDM $1$ .

Furthermore, as requested by Reviewer mQYF (see "Initializing with Random Tokens" for more details and quantitative results), we find that it is indeed possible to generate images "from scratch", by starting with random tokens.

Link to uncurated generations starting from random tokens: https://i.ibb.co/20L8xk1q/rand-tok.png

• Additional Failure Cases

The "from scratch" generations referenced above are uncurated, and we plan to include additional visualizations in the appendix of our revised submission.

You may also be interested in our response to Reviewer apSr on additional out-of-domain examples.

Finally, you can find a discussion of some limitations of the token optimization-based editing approach in our response to Reviewer N1ti's question regarding limited control over the editing process. In particular, there are no guarantees that text-guided editing will not result in unintended modifications to the input images, although we believe that this issue could be alleviated with further engineering effort (e.g., a more advanced objective function combining reconstruction and CLIP components).

• Determinism and Duplicated Seed/Prompt Inputs

All experiments in Section 5 do in fact include some small amount of randomness in the loss due to nondeterminism in some of the CUDA kernels used in our implementation $2$ .

Interestingly, even the small amount of nondeterminism caused by implementation details of the CUDA kernels used in PyTorch is sufficient to lead to diverse generations by the test-time optimization procedure.

We have experimentally verified that using deterministic implementations of these algorithms leads to deterministic results, in turn leading to degraded FID in the case of smaller number of seed images or deterministic seed selection (e.g. 500 seed images or 1000 seed images with top-1 selection). In the case of at least 1000 seed images with top-1% selection, there is enough diversity in the seed-to-prompt association to yield about 4000 unique inputs, such that the FID-5k results would be relatively unaffected by the use of a fully deterministic implementation.

Thank you for pointing out this issue. We will address this point explicitly in the main text of the revised submission.

References

$1$ Blattmann, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B. Retrieval-augmented diffusion models. NeurIPS 2022.

$2$ https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html

We thank you for your thorough and detailed comments and suggestions, as well as the thoughtful questions. We hope the following answers can address your concerns.

• mec.q.1. Optimizer Iterations

The assessment that the number of optimizer iterations will result in the input image deviating further and further from the input matches our results. In fact, the FID and IS scores are relatively sensitive to the number of iterations, and the choice of iterations can provide control over the FID/IS score tradeoff.

We have therefore computed FID and IS for different number of optimizer iterations (between 50 and 500 in 50 iteration increments). We report FID and IS at the number of iterations corresponding to the best FID, and separately, to the best IS. Note that these differ from the results in Table 2 because Table 2 reports results using the same number of optimizer iterations for all models.

For best results, the number of iterations should be picked on an example-by-example basis: "smaller" edits may require less iterations than more significant ones. One could also design adaptive stopping criteria based on monitoring the objective function's value and change across iterations.

• mec.q.2. Optimizing without VQ

We find that the vector quantization step serves as regularization that prevents the test-time optimization procedure from behaving akin to an adversarial attack on the CLIP objective. In particular, the CLIP objective, when provided with the output of the 1D tokenizer with VQ, appears to be very adversarially robust (see also [1]). We observe weaker robustness in the case without VQ.

The best FID-5k achieved by the no-VQ optimization is 15.8 (at 100 optimization iterations, with an IS of 118) and the best IS is around 130 (with 250 optimization iterations, at an FID of 17.1). Even with L2 regularization applied on the tokens, we are not able to improve FID and see a very small improvement in IS. In all cases, the optimization with VQ significantly outperforms the no-VQ one with a top FID of 15.1 and IS of >160.

We have generated examples of how this manifests itself qualitatively, which we will include in the revised submission. We observe that the no-VQ optimization sometimes leads to more artifacts and more exaggerated and larger areas of repetitive texture.

Link to visualization: https://i.ibb.co/3yYd9vF6/no-vq-vis.png

• eda.q.1. FID 8.6 Experiment

We report this metric (FID-50k of 8.6) in Table 4, alongside the FID-5k and IS, CLIP and SigLIP scores. As you point out, IS, CLIP and SigLIP over 50k samples are not reported. We find that the metrics other than FID are very similar between the 5k and 50k sample evaluations, which is why we omit them.

• eda.q.2. Decoder Size vs. Number of Tokens

We agree that this is a shortcoming in our evaluation, so we have run additional experiments comparing the VQ-BL-64 and VQ-BL-128 tokenizers, which share the same model and codebook sizes, differing only in the number of 1D tokens. The results are included in the tables from our response to question mec.q.1, and show that the VQ-BL-64 tokenizer always outperforms the VQ-BL-128 tokenizer.

• eda.q.3a. Role of 2D Tokens

Our main reason for including the 2D VQGAN in this table was to demonstrate the poor performance of simple test-time optimization with widely used “traditional” tokenizers. Since we are not aware of any 2D tokenizers with VQ that achieve a similarly high compression ratio as TiTok, the conclusion from this experiment may be better reworded as the high degree of compression enabled by 1D tokenization is key to generative performance, rather than 1D tokenization itself being key.

• eda.q.3b. 2D Tokens Qualitative Results

Please see our response to Reviewer mQYF’s Question 1 for some qualitative examples of 2D tokenizer generations.

• sm.q.1. Code

We will make the code publicly available.

• Preserving Input Image Features

There is indeed no guarantee on preservation of aspects of the input image.

This could be mitigated with further engineering effort, e.g., the difference in tokens compared to the tokenized version of the input image could be slightly penalized. One could also combine the text-guided objective with the inpainting one to explicitly preserve user-defined regions.

Certain differences wrt. the input are introduced by the tokenizer itself, due to imperfect reconstruction (VQ-LL-32 example: https://i.ibb.co/qMnp23fq/recons.png).

References

[1] S. Santurkar et al. Image synthesis with a single (robust) classifier. NeurIPS 2019. https://arxiv.org/abs/1906.09453

We thank you for your review! As you point out, the idea of latent space optimization is indeed not new. However, we do believe that its application in the case of highly compressed latent spaces is noteworthy for a few reasons:

1. Previous attempts to use test time latent space optimization for image editing, i.e. VQGAN-CLIP, have not demonstrated high quality generation of photorealistic scenes, and rely on "tricks", such as the usage of a large number of different augmentations (various crops of the image being optimized, color-space corruptions, flips, etc.), in order to be successful. While these additional tricks can also be used to improve image generation in the case of highly compressed latent spaces, we demonstrate that they are not necessary. Remarkably, even an extremely straightforward application of test time optimization can produce reasonable results when operating in the highly compressed latent space (baseline in Table 4).

2. Since our baseline test time optimization algorithm is very simple (7 lines of code, see Algorithm A1 in the appendix), we do not claim any significant engineering contribution. Instead, we view our findings as strong motivation to view tokenizers with increasingly high compression ratios as generative models. In particular, we find it surprising that tokenizers trained with a standard VQGAN-like objective can be used to perform a variety of generative tasks such as text-guided image editing or inpainting. We would also like to point to our response to Reviewer mQYF, in which we provide experimental evidence that the VQ-LL-32 tokenizer can even generate “from scratch”, starting from pure noise.

---

• Comments on Out-of-Domain Generalization

Since TiTok tokenizers are trained only on ImageNet, we expect limited ability to generate complex out-of-distribution scenes, such as those involving classes that are not part of the ImageNet-1k dataset or requiring composition of multiple subjects (since ImageNet images often feature a single prominent subject). Unfortunately, no highly compressed 1D tokenizer trained on larger scale datasets was available at the time of submission, so our experiments are confined to ImageNet.

However, as tokenizers with even higher compression ratios or trained on larger datasets become available, we expect application to more diverse domains to become possible. Further, we hope that the view presented in our paper – that the lossy autoencoding task performed by tokenizers with very high compression ratios can be thought of as a generative modeling problem – can provide an insightful perspective in scaling such tokenizers to these larger datasets.

• Additional Out-of-Domain Examples

As this was also requested by Reviewer mQYF, we generated additional qualitative results in the out-of-domain setting of text-guided style transfer. In particular, we use CLIP prompts like "a watercolor/pixel art/abstract/cartoon painting of a <subject>", and find that the model can produce qualitatively very compelling looking results for text-guided style transfer, even for styles which we expect to be mostly absent from the ImageNet-1k dataset. We will include the generations in the revised submission.

Link to visualization: https://i.ibb.co/Bd1HkmX/style.png

We are happy to hear you enjoyed our paper, and would like to thank you for the great questions, which we hope to answer below.

• Q1. 2D Tokenizer Results

We have produced visualizations of the optimization process using MaskGIT’s VQGAN, alongside the 32-token TiTok tokenizer, which we will include in the revised submission.

Examples from Figure A5: https://i.ibb.co/YB72k9Fh/2d-tok-vis-A5.png

Additional example: https://i.ibb.co/WZnHmLx/2d-tok-vis.png

In this visualization, we observe that the highly compressed tokenizer allows the optimization procedure to perform globally consistent edits to the image. On the other hand, the VQGAN optimization is easily able to change local texture and color to align with the prompt, but fails to perform "global" edits (for example, when changing the species of an animal, the shape of the head and position of the ears are relatively unchanged in the VQGAN case, but attributes such as color can be adapted more successfully).

Regarding VQGAN-CLIP: To the best of our knowledge, there is no existing evaluation of VQGAN-CLIP for ImageNet generation. We hope that our comparison of MaskGIT's VQGAN which is already included in Table 3 can provide such a comparison, as it should be very similar to VQGAN-CLIP while being fair in the sense that the same seed image selection and cost function are used as for the 1D tokenizer experiments.

• Q2. Token Regularization

VAE Tokens. The experiment in the paper did not include regularization on the VAE tokens. We agree this is crucial, and have repeated this experiment. Results are provided in the table below.

*weight chosen for best results from sweep including {0.02, 0.1, 0.2, 0.5}

While the results for VAE tokens with regularization are improved over the numbers reported in the original submission, FID and IS are still poor compared to the VQ model, so we do not change our conclusion regarding the importance of VQ.

2D Tokens. For MaskGIT’s VQGAN, we do not use regularization in the experiment from Table 3. We do not find that it makes a substantial difference.

• Q3. Additional OOD Examples

We have run an additional out-of-domain text guided style-transfer task, which we describe in our response to Reviewer apSr.

• Initializing with Random Tokens

We decided to run some additional experiments and find that it is indeed possible to generate images "from scratch", starting from randomly sampled tokens!

While results are worse than in the case of CLIP-based prompt-to-seed association, FID and IS scores are still reasonable. Qualitatively, we find that it is still possible to generate relatively high quality samples, especially with more detailed CLIP prompts and longer optimization times (400-500 iterations). We plan to include uncurated generations starting from randomly sampled tokens (with no seed image) in the revised submission.

Link to uncurated generations starting from random tokens: https://i.ibb.co/20L8xk1q/rand-tok.png

Quantitative results (using the same settings as the line marked with (*) from Table 4 in the paper) can be found in the table below:

With additional tweaks from Table 4 (token L2 regularization, extra token noise in the case of starting from a seed image, extra 100 iterations in the case of starting from random tokens), the gap in performance between the seed-image-initialized and random-token-initialized optimizations diminishes further:

*For both tables, tokens are randomly initialized from normal distribution with std=0.3, chosen as the best performing value from a sweep including {0.05, 0.3, 1.0}.

Note: Starting from random tokens leads to better performance than starting from random seed images (i.e., without CLIP-based seed-to-prompt association) – cf. Table 1 row 3 (line 284).

• Initializing from Single Best Image

In Table 1 (L290), we show the results of an experiment where only the best seed image from the seed image pool is picked for each prompt. This allows achieving high IS at the expense of degraded FID.

If you are referring to picking only one overall best image, and using that same image for every prompt, we suspect performance may not be great, since our previous experiment initializing with random tokens can achieve better results than initializing with random images.

Thank you for your review and helpful comments!

• Per-Token Attributes as Emergent Properties

The direct token editing examples do indeed rely on emergent properties that are not controllable using the standard autoencoder-style/VQGAN training scheme used by TiTok.

As such, we agree that practical applications of token copy-paste for image editing is restricted to a few tasks (such as brightness change, background blur, etc.), and these tasks have to be manually discovered using methods such as those described in Sections 3.1 and 3.2.

However, we find the fact that such editing is possible at all very surprising. In particular, perturbation of individual tokens does not lead to semantically meaningful and globally coherent edits in the case of 2D tokenizers due to the spatial relationship of tokens and regions of the input image (see, for example, Figure 1 in $1$ ). We therefore intended to present this finding to draw attention to underexplored emergent properties of highly compressed latent spaces, as well as to motivate our more scalable gradient-based image editing approach from Section 4, which can be directly applied for practical applications such as text-guided editing or generation, or inpainting.

• Notation: Top-1/Top-1%

Regarding L290 (top-1 vs top-1%): Thank you for pointing out the confusing notation. "Top-1" is used to denote a seed image association procedure by which only the single most aligned seed image is chosen. In contrast, top-1% associates a random seed image from the subset of the top 1% most similar seed images (for example, with 1000 seed images, top-1% would correspond to random association to one of the top-10 images from the seed image pool). We will try to clarify this in the final version.

• Stability of Optimization Process

We do observe that the optimization process is sensitive to noise, such that even small perturbations to the objective and its gradients can lead to quite different generations. However, this does not mean that the generations are generally low quality – instead, we observe that this leads to generations that are diverse (even when adding only small amounts of noise), while still being reasonably high quality (as evidenced by our FID and IS scores).

References

$1$ Cao, S., Yin, Y., Huang, L., Liu, Y., Zhao, X., Zhao, D., and Huang, K. Efficient-VQGAN: Towards high-resolution image generation with efficient vision transformers. ICCV 2023, https://arxiv.org/abs/2310.05400