FlexTok: Resampling Images into 1D Token Sequences of Flexible Length

We thank reviewer jx6L for the thoughtful and constructive feedback. Below we address the main points raised:

1. Relation between image size, complexity, and token count

This is a good point, and we haven't explored this explicitly yet. We performed all our experiments at 256x256 resolution specifically to enable direct comparison with standard methods (e.g., VQ-GAN, LlamaGen). While FlexTok could support variable-resolution inputs (e.g. using architectures like NaViT), that's a separate research direction we leave for future work. We also agree the interplay between image complexity, resolution, and token counts is an interesting open question.

2. “Lossless” compression

We appreciate this observation about reconstruction quality at low token counts. It's important to clarify that all image tokenizers are inherently lossy, just to different degrees. Some tokenizers (e.g., SEED (Ge et al. 2023)) focus purely on semantic features without pixel accuracy, while others (e.g., VQ-GAN) are more pixel-aligned. FlexTok sits in between, explicitly transitioning from semantic-level reconstructions at low token counts (<16 tokens) toward highly pixel-aligned reconstructions as token count increases (e.g. at 256 tokens). We explicitly measure pixel-level reconstruction fidelity using Mean Absolute Error (MAE), which steadily improves with more tokens (Fig. 4). Regarding the rFID metric, we agree that additional reconstruction metrics would strengthen our analysis; we'll include PSNR and SSIM results compared directly to baseline tokenizers like VQ-GAN (see the comparison in response to reviewer ekCv).

3. Why a diffusion-type decoder rather than VQGAN?

We chose a diffusion-type decoder (rectified flow) exactly because it models conditional uncertainty. When using few tokens, the degree of compression is high and reconstructions naturally have uncertainty; the diffusion decoder handles this gracefully, producing plausible, semantically coherent outputs rather than blurry averages. As the number of tokens increases, the uncertainty naturally reduces, making reconstructions progressively more accurate and deterministic (see image reconstructions in Appendix J.1). A VQGAN-style decoder wouldn't offer this flexible control over reconstruction uncertainty.

4. AR sampling vs. diffusion sampling impact

We found both AR sampling (top-k, top-p, temperature) and diffusion sampling parameters influence image quality and diversity. AR sampling tends to be stable across reasonable settings (see Appendix F), while diffusion decoding, particularly adaptive projected guidance (Sadat et al., 2024), had a significant impact on final image quality. In short: both matter, but diffusion guidance is particularly critical. We'll clarify this explicitly.

We thank reviewer ekCv for the thoughtful and constructive feedback. Below we address the main points raised:

1. Variable-length generation limitation (fixed token length in generator)

To clarify: in our current setup, we train a single autoregressive (AR) model capable of generating a full 256-token sequence. During inference, shorter outputs are obtained simply by early stopping this sequence at the desired length. While effective, you're right that it could be even more flexible if the AR model itself determined when to halt generation, and this is an interesting future research direction. We expect that we can create such halting conditions by augmenting the training data with per-token-subsequence reconstruction metrics (e.g., MSE, DreamSim), and when training the stage 2 AR model, simply truncating the training sequences if the score reaches a certain pre-defined threshold.

2. Weakness at longer token sequences (limited improvement beyond ~32 tokens)

The reviewer raises an insightful point about long-sequence performance. Indeed, our "Pow2" nested dropout schedule intentionally emphasizes shorter sequences during training. This design optimizes FlexTok’s ability to reconstruct images effectively at extreme compression (few tokens), a core contribution of our work. However, it is correct that this strategy results in diminishing improvements for longer sequences (>32 tokens). Adjusting dropout sampling strategies to more evenly balance short and long sequences could potentially mitigate this trade-off, and we appreciate the suggestion here.

3. Semantic-level vs. pixel-level reconstruction (PSNR/SSIM comparisons)

We agree, rFID is not a good measure for pixel alignment. We measure rFID to demonstrate that the FlexTok decoder is capable of producing outputs that could "plausibly" come from the image distribution, no matter the number of tokens given. To measure pixel-level reconstruction alignment, we show MAE and DreamSim in Fig 4, and observe a roughly log-linear relationship between the scores number of tokens. We additionally show PSNR and SSIM in the table below for various number of tokens, and find that at 256 tokens used, FlexTok reaches comparable compression performance to common 2D-grid tokenizers that use 16x16 discrete tokens.

FlexTok d18-d28 reconstruction metrics on IN1K validation set, resolution 256x256:

Comparison with common discrete tokenizer baselines (numbers from Cosmos paper):

4. Impact of REPA vs. longer training schedules

This is a good question, however, the difference in convergence speed is so significant that we found it computationally too expensive to ablate this. We expect that the ~17x convergence speedup (in terms of FID) demonstrated in the original REPA paper may roughly translate to our setting too. Unfortunately we are unable to add qualitative examples to this text-only reply, but we find the non-REPA reconstructions (after training for the same number of steps) to be significantly worse in terms of fidelity, and overall less semantic. We will add a discussion of these points as well as qualitatives to the camera-ready.

We thank reviewer 6zZ5 for the thoughtful and constructive feedback. Below we address the main points raised:

1. Higher computational cost of rectified flow decoder and token-count limitations

Please see our response to reviewer MMdz. Our preliminary experiments suggested that higher token counts can significantly reduce reconstruction errors, and enable decoding with fewer steps, however, for simplicity of training subsequent autoregressive models on the token sequences, we decided to choose 256 tokens as the upper-bound for this submission.

2. System-level comparison for text-to-image generation

We fully understand the motivation for requesting comparisons against external text-to-image baselines. However, unlike the somewhat standardized class-conditional ImageNet setting, proper comparison on text-to-image generation with external baselines is usually extremely difficult and nuanced due to differences in compute and dataset (size, caption quality, diversity, aesthetics, similarity to COCO, etc.) having significant impact on downstream evaluations (e.g. see "On the Scalability of Diffusion-based Text-to-Image Generation", Li et al. 2024). For that reason we decided to perform a controlled experiment in which we train a 2D grid tokenizer with the same data, compute, and rectified flow decoder objective, and perform autoregressive generation on its tokens. The results in Fig. 6 suggest that FlexTok performs comparably to classical 2D grid tokenizers at 256 tokens, but offers more flexibility overall.

3. Additional standard metrics for class-conditional generation

We appreciate the suggestion and provide the requested metrics (Inception Score, Precision, Recall, and gFID) in the table below. Our results are comparable to state-of-the-art methods such as VAR-d30 (323.1 IS, 0.82 precision, 0.59 recall, 1.92 gFID) and TiTok (gFID between 1.97 and 2.77 depending on tokenizer choice) across a broad range of token counts. We will incorporate these results into the camera-ready version of the paper.

4. Additional suggestion: summary of appendix content

Thanks for the helpful suggestion. Adding a concise appendix summary at the start is indeed beneficial. We will add this to the camera-ready version.

We thank reviewer MMdz for the thoughtful and constructive feedback. Below we address the main points raised:

1. Single-token generation claim

Image tokenization is commonly performed with lossy autoencoders that abstract away imperceptible information, meaning all tokenizer decoders (whether trained with diffusion/flow methods or GANs) are inherently generative to some extent. The degree to which they must be "generative" directly corresponds to the amount of compression. At the coarsest level (single FlexTok token = 2 bytes), our representation necessarily operates at a highly semantic level, conceptually similar to semantic tokenizers like SEED (Ge et al. 2023). At the finest level (256 tokens = 512 bytes), we achieve compression comparable to classical 2D grid tokenizers like VQ-GAN (also, see response to reviewer ekCv). FlexTok's unique strength is providing a single model that learns these different hierarchies, effectively offering an alternative way to describe images in a coarse-to-fine manner. The single-token scenario is indeed best interpreted as semantic conditioning rather than pixel-level compression.

2. Computational overhead (25-step flow matching decoder) We acknowledge that FlexTok's rectified flow decoder adds computational complexity during inference (though encoding remains efficient). We explicitly chose this architecture because it consistently maintains high reconstruction fidelity across a wide range of token counts (Fig. 4). Importantly, decoding happens after the autoregressive (AR) generation step, which is already computationally intensive, so the flow decoding step adds a constant overhead rather than introducing an entirely new computational bottleneck. We also anticipate that common distillation methods (e.g., consistency decoders, Reflow) can substantially lower inference complexity.

3. Token-level faithfulness (single-image reconstruction quality) Regarding reconstruction quality at 256 tokens (512 bytes), it's important to recognize that this still represents extremely high compression: a full-color image compressed to just 512 bytes will naturally show some loss of detail. This token count was explicitly chosen to match standard tokenizers (e.g., VQ-GAN, LlamaGen), allowing direct and fair comparisons. At this standard token count, FlexTok achieves reconstruction fidelity comparable to these established methods (see the comparison in response to reviewer ekCv). The visible imperfections at this compression rate are expected, and our results show a clear log-linear trend (Fig. 4), strongly suggesting that higher token counts (e.g., 1024 tokens or more) would yield increasingly faithful reconstructions. This trade-off between compression and fidelity is fundamental to all image tokenization approaches, and exploring higher-token-count scenarios is a natural next step.

4. Technical novelty compared to TiTok, ALIT, ElasticTok, and DALL-E 3 Regarding FlexTok's novelty beyond causal attention masking: while ElasticTok and ALIT indeed share similar concepts, these methods were developed concurrently and independently. FlexTok directly builds on TiTok's 1D tokenizer framework and diffusion/flow-matching ideas, but introduces several critical new technical components: specifically, causal attention masking combined explicitly with nested token dropout and rectified flow decoding. Crucially, these innovations collectively enable hierarchical tokenization, smoothly transitioning from coarse, semantic-level representations to detailed pixel-level reconstructions within one tokenizer. This hierarchical tokenization was not shown by TiTok or other prior methods. Additionally, we provide a detailed analysis of this hierarchical behavior in generative settings, clearly showing its practical benefits and trade-offs.