Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations

Dear Reviewer 7n9v,

Thank you for your detailed comments!

Q1: LLM training strategy. It is unclear whether the backbone LLM is trained from scratch or fine-tuned.

Ans: In Sec.3.4, we follow previous works on multimodal LLMs [2,31,35] and Unified MLLMs [6,55], which typically divide the training pipeline into two stages: Pretraining and Supervised Finetuning.

In this context, Pretraining refers to training the multimodal LLM, initialized from a pretrained LLM backbone. Specifically, we initialize our model using Qwen2.5-Instruct, a pretrained LLM, and then conduct MLLM Pretraining followed by Supervised Finetuning. We will clarify the difference between LLM and MLLM pretraining in the revision.

Q2: More explanation of Scale Adaptive Pooling and Decoding

Ans: (a) Mechanism: As described in Figure 3 (caption), Lines 122–129, 130–140 (especially Lines 131–132), and Lines 226–227, Scale-Adaptive Pooling (SAP) is implemented using 2D adaptive pooling with a predefined scale set {1, 2, 3}. During training, a scale is randomly sampled; at inference, a fixed scale is used. For a 384×384 input image, these scales produce 729 (27×27), 169 (13×13), and 81 (9×9) tokens, respectively. To support variable token lengths, we introduce Scale-Adaptive Decoding (SAD): a lightweight ViT decoder (identical to SigLIP2’s ViT, see Lines 127–129) that can process variable-length sequences by dynamically resizing its 2D positional embeddings.

(b) Role: SAP and SAD allow TA-Tok to extract multi-granularity features, as noted in Lines 122–125. This flexibility enables the MLLM to better adapt to different tasks:

• Coarse-grained features (e.g., 81 tokens) are suitable for text-to-image generation, especially under limited training data.
• Fine-grained features (e.g., 729 tokens) are crucial for detailed image understanding tasks. This design is empirically validated in Table 5: visual understanding tasks consistently benefit from more tokens, while generation tasks prefer fewer tokens in low-data regimes, but improve with more tokens when training data is scaled up.

Q3: Comparisons to other visual representations.

(1) Whether all methods are trained on the same data?

Ans: Yes. Line 250-252: “For training MLLMs with these representations, we sample a subset of our training data for controlled experiments: 10M T2I data, 10M I2T data and 5M text-only data. All models are trained with the same configuration and tested on visual understanding [15, 20, 22, 26] and generation tasks [19].”

(2) How the authors isolate the impact of visual tokenization from the influence of different decoding strategies.

In Section 4.3, we compare TA-Tok with other discrete visual tokenization methods, including VQ-VAE, Janus (which also uses VQ-VAE for image generation), and a hybrid tokenizer (i.e., UniTok [36]). All these methods adopt an encoder–decoder architecture, where the encoder (tokenizer) converts input images into discrete tokens, and the decoder (de-tokenizer) reconstructs the image from those tokens. We understand that the reviewer may question whether our decoding setup is fair compared to the baselines. However, we would like to emphasize that the proposed de-tokenizer (AR-DTok and Dif-DTok) is one of the key contributions of our paper. They are specifically designed to decode fully semantic discrete tokens (produced by TA-Tok) back to pixel-level images, using autoregressive and diffusion-based models, respectively. Therefore, while decoding strategies do differ across methods, these differences are an integral part of the overall design and contribution of our approach, not confounding factors in the comparison.

Q4: Pre-training strategies

(1) More detailed explanation of the proposed I2I and TI2I tasks.

Ans: As described in Sec. 3.4 and Appendix Sec. D.2, the proposed I2I and TI2I tasks encourage the MLLM to first understand the input image and then generate another image, which means the model needs to improve its understanding and generation task in one training sample. We will add more details for the two tasks.

(2) "A dog running on the grass" becomes: "A dog running on <image>" and "the grass". Explain this example.

Ans: As described in Line 206-208 and Appendix Sec. D.2, we use FLUX to generate two images based on the prompts “A dog running on the grass” and “the grass”. The image with prompt “the grass” is used as an input image. The task is to generate a final image based on a multimodal prompt: “A dog running on <image>”.

Q5: The two decoding strategies in Figure 4

(1) How the two-de-tokenizers relate to the autoregressive LLM at inference time?

Ans: As described in Figure 2 and Sec. 3.3 Inference (Line 186-188), the LLM first generates a sequence of visual tokens during inference. These tokens are then passed to either AR-DTok or Dif-DTok to reconstruct the final image. In other words, the de-tokenizers operate after the LLM and are responsible for converting the predicted tokens back to the pixel space.

(2) Why does the autoregressive model only appear in one of the two decoding pathways?

Ans: The “Autoregressive Model” shown in Figure 4(a) refers specifically to the image generation model used within AR-DTok, which is a separate autoregressive model trained to reconstruct images from visual tokens. It is not the same as the autoregressive LLM depicted in Figure 2. To clarify: (a) Both AR-Dtok and Dif-DTok receive the LLM-generated tokens at inference time. (b) During training, they both consume tokens from TA-Tok.

Q6: Self Reflect requires more details in the main text

Ans: Due to the space limit, we mainly introduce the Self Reflect strategy in Appendix Sec. F and its ablation experiments. We will add more detail of Self Reflect in the main text and give a link to the appendix for more details.

Q7: Since the projection matrix W in Text-Aligned Codebook is trainable, is it possible that this alignment is compromised?

Ans: As shown in Figure 3, we initialize the codebook using frozen LLM embeddings from a text-pretrained LLM. This provides a strong semantic prior: each token in the codebook corresponds to a position in the LLM’s text space. So even though the projection is learned, it must learn to align with a fixed semantic space. This constraint naturally preserves semantic consistency during training.

Q8: What is the criterion used to select the top-k most representative embeddings when constructing the visual codebook?

Ans: As described in Line 120-121 and the following code, we select the top-k most representative embeddings based on their average distance to others.

# embeddings.shape=[150K, D]
norms = torch.norm(embeddings, dim=1, keepdim=True)
cosine_sim = torch.matmul(embeddings, embeddings.T) / (norms * norms.T)
average_similarity = torch.mean(cosine_sim, dim=1)
topk=65536
representative_tokens = torch.argsort(average_similarity)[:topk]

Q9: In Table 2, individual best-performing scores are not highlighted.

Ans: We will highlight them in the revision.

Dear Reviewer 7n9v,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. In response to your comments, we have provided detailed clarifications to address your specific concerns.

As the discussion period will conclude in a few days, we would greatly appreciate it if you could kindly take a moment to review our rebuttal at your convenience. If there are any remaining questions or suggestions, we are more than happy to address them promptly.

Thank you once again for your thoughtful feedback and valuable contribution to our work.

Best regards,

The Authors

Dear Reviewer 7n9v,

Thank you again for your thoughtful and constructive feedback.

As the discussion period is coming to a close, we were wondering if you have had a chance to consider updating your score. Your active participation and perspective are incredibly valuable to the final assessment of our work, and we would be grateful for your continued input.

Best regards,

The Authors

Dear Reviewer Gja8,

Thank you very much for your insightful comment!

Q1: Add intuition for the imbalance gain of understanding and generation tasks in Tab.6.

Ans: Thanks for your valuable suggestion. We will clarify this point in the revision. We believe the observed imbalance arises from two main factors:

(a) Task and supervision mismatch. Generation tasks directly benefit from joint training because they require the model to understand the concept (prompt)-whether text or image-before generating coherent images. In this sense, improvements in visual understanding directly enhance generation quality.

In contrast, understanding tasks are often discriminative in nature (e.g., classification or multiple choice), which do not directly benefit from generation supervision. For example, predicting visual token sequences may not significantly help with answering a Yes/No question.

(b) Asymmetry in task difficulty and data requirements: The model’s understanding ability is already relatively strong, often requiring less data to achieve good performance. For example, models like LLaVA reach strong image understanding with only 0.6M training samples. In contrast, image generation is a more complex task that requires substantially more data (often 10M+ samples) and benefits more from improved multimodal representations. As a result, joint training helps generation more noticeably—since it benefits from enhanced visual understanding and alignment. Meanwhile, understanding tasks gain less from generation supervision, as their performance tends to saturate earlier, and the benefits from joint training diminish once the understanding capability reaches a certain level.

Q2: The relation between the proposed TI2I task to traditional TI2I task.

Ans: Traditional TI2I tasks, such as image editing, subject-driven generation, or next-frame prediction, primarily emphasize pixel-level consistency. For example, image editing requires unedited regions unchanged, while subject-driven generation ensures the identity or appearance of a subject is preserved between input and output images.

In contrast, our proposed TI2I task aims to bridge the gap between visual understanding and generation, rather than enforce pixel-level similarity. Given a sample triplet (T, $I_1$, $I_2$), the model is trained to first understand both the text T and the input image $I_1$, and then generate a new image $I_2$ conditioned on both modalities. Pixel-level consistency is not required-only semantic alignment.

For example, if $I_1$ is “a picture of grass,” $I_2$ is “a dog running on grass,” and the text is “A dog running on <image>,” the model is expected to understand that “<image>” refers to “grass” and generate an appropriate scene. The grass in $I_1$ and $I_2$ may differ visually, as long as they share the same semantic concept. This formulation allows the model to learn multimodal alignment and reasoning within a single training sample, and gradually unifies image understanding and generation in a semantically coherent way.

We also give some prompt format design and application in Appendix Figure 1, where the model trained on the TI2I task can be adapted to subject-driven generation and zero-shot style transfer.

Q3: Several images in Figure 7 are not well aligned with the prompt.

Ans: Thanks for pointing this out. In Figure 7, we used a set of prompts collected from the web and prior works, and generated the images without cherry-picking. While many results are reasonable, some generations are not well aligned with the prompts. This is primarily due to limited training data and compute budget. Our current model is not strong enough to handle some unseen prompts, and may produce incorrect results. We believe it can be mitigated by scaling up training data and incorporating more diverse scenarios in the future work.

Q4: Why AR-DTok shows better capability in generating complete and correct text within images in Figure 6?

Ans: We believe this phenomenon is due to two key factors: (a) Training data. As shown in Appendix Table 1, AR-DTok is trained with more data than Dif-DTok (50+23M vs 23M). This provides AR-DTok with better alignment to TA-Tok tokens, particularly for fine-grained content such as text within images.

(b) Architecture difference. As shown in Figure 4, the two de-tokenizers differ in how they integrate TA-Tok tokens:

• In AR-DTok, TA-Tok tokens z and VQVAE tokens y are concatenated as [y, z] and processed jointly via self-attention in an autoregressive manner.
• In Dif-DTok, TA-Tok tokens are injected through cross-attention with the noise input during denoising.

Because TA-Tok tokens carry explicit 2D structure and visual details, the self-attention fusion in AR-DTok is more effective in learning fine-grained token correspondences between TA-Tok and VQVAE. This contributes to AR-DTok’s superior ability in generating coherent and correct text within images.

Q5: Evaluation of SAP design for I2I and TI2I tasks.

Ans: As discussed in Q2, the proposed I2I and TI2I tasks are designed as auxiliary tasks to mitigate the gap between visual understanding and generation, and are different from traditional TI2I tasks. Therefore, we evaluate the design of SAP on image editing, a traditional TI2I task. Specifically, we simply finetune Tar 1.5B on OmniEdit [1] and evaluate it on Emu-Edit [2]. The results are shown below. We find that different SAP settings perform similarly on CLIP-I and CLIP-T, which measures the semantic similarity between the input image, prompt and edited image. However, for the L1 metric, which reflects pixel-level consistency, using more tokens (e.g., 729) provides better reconstructions. We will include the evaluation of TI2I tasks in the revision.

[1] Wei, Cong, et al. Omniedit: Building image editing generalist models through specialist supervision. ICLR.

[2] Sheynin, Shelly, et al. Emu edit: Precise image editing via recognition and generation tasks. CVPR.

Dear Reviewer xYB2,

Thank you very much for your helpful comment!

Q1: Are representative LLM embeddings selected through clustering?

Ans: We do not use clustering algorithms (e.g., K-Means) for selection. Instead, we adopt a simple and efficient distance-based ranking strategy. Specifically, we compute the average cosine similarity between each token embedding and all others, and select the tokens with the lowest average similarity (i.e., most representative and diverse). The selection process is shown below and will be included in the revision:

# embeddings.shape=[150K, D]
norms = torch.norm(embeddings, dim=1, keepdim=True)
cosine_sim = torch.matmul(embeddings, embeddings.T) / (norms * norms.T)
average_similarity = torch.mean(cosine_sim, dim=1)
topk=65536
representative_tokens = torch.argsort(average_similarity)[:topk]

This method allows us to efficiently obtain a diverse and informative subset of the LLM's embedding space, forming the initial visual vocabulary.

Q2: The 65K vocabulary is still large.

Ans: We agree that 65K may still seem large. However, in vector quantization, a larger vocabulary is generally beneficial, which helps preserve more visual detail and reduce quantization error. Ideally, we would use the full 150K vocabulary from the LLM.

In practice, though, using all 150K high-dimensional embeddings (≥1536) results in excessive GPU memory usage during TA-Tok training. To address this, we reduce the vocabulary size to 65K, so that a 40GB A100 GPU can hold it and keep training stable.

Q3: Where is the result of randomly initialized codebook expanded to 65536 tokens?

Ans: Traditional VQ codebooks typically use smaller vocabularies (8K–64K) with low-dimensional embeddings (e.g., 4–32). In contrast, to preserve the semantic richness of SigLIP features (1152-D), we increase the codebook dimension to 1536. However, this combination of large vocabulary size and high dimension is extremely difficult to train when initialized randomly. As shown in Appendix Table 3, even with 32,768 tokens and 1536-D (setting (b)), the model struggles to converge due to codebook collapse. Expanding the random-initialized codebook to 65,536 tokens only worsens this issue, making training infeasible in our experiments. This challenge further motivates our use of LLM-initialized embeddings, which offer much more stable and meaningful training dynamics.

Q4: Performance from random initialization and LLM-based initialization appears similar.

Ans: First, the improvement of LLM-based initialization compared to random initialization is still significant after sufficient training, about 2.7%, 13% improvement on the challenging GQA and MME respectively, which are non-trivial for these tasks.

More importantly, under data-limited settings, the benefit becomes more significant. As shown in the table below, when we reduce the training data from 25M to 12.5M and 6.25M samples, LLM-initialized codebook consistently outperforms random initialization, especially on tasks that rely more heavily on semantic alignment, such as MME and GQA. This highlights that LLM-based initialization helps align TA-Tok with the LLM’s embedding space more efficiently, particularly in low-resources scenarios.

Q5: Why Janus performs worse in generation quality in Figure 5?

Ans: As noted in Line 246-247, the ‘Janus’ baseline used here is different from the original Janus Pro [6], it denotes a baseline method which adopts SigLIP for image understanding and VQVAE for image generation.

As illustrated in Line 256-257, we believe the poor generation performance stems from a conflict between the two visual representations: SigLIP (semantic) and VQVAE (pixel-level). Since these representations are not aligned, the training signals from image understanding and generation tasks do not reinforce each other.

This hypothesis is further supported in Table 6, where joint training fails to improve Janus on either task, while it brings significant gains for both VQVAE baseline and our proposed method. This highlights the benefit of using a unified visual representation, as enabled by our TA-Tok design.

Q6: Why is the visual codebook smaller than the text codebook?

Ans: As mentioned in Q2, the visual codebook size reflects a trade-off between encoding quality and computational cost. While a large codebook can better capture visual details, it significantly increases the computational burden during both tokenizer and MLLM training. Moreover, increasing the codebook size will also encounter the low codebook usage issue.

Q7: Is it appropriate to assume a shared visual-textual feature space?

Ans: Thanks for your thoughtful question! We agree that visual and textual modalities differ in semantic granularity. Ideally, a unified model should capture both pixel-level detail and semantic-level features.

However, in practice, integrating pixel-level information directly into fully semantic LLMs is extremely resource-intensive and technically challenging. As shown in Figure 5, methods like VQVAE, which focus on pixel-level reconstruction, fall behind Hybrid approaches and our method in both understanding and generation tasks.

To achieve better cross-modal alignment, we use a shared semantic feature space, where visual and textual tokens are compatible. We then use a lightweight de-tokenizer to project these semantic visual tokens back to the pixel space for image generation. This setup allows us to maintain semantic consistency while still supporting high-quality image synthesis.

Q8: Tar’s generation results are weak in identity consistency?

Ans: Thank you for the observation. As discussed in our Limitations (Appendix Sec. I), autoregressive and diffusion models are not original designed for image reconstruction, but in our framework, we repurpose them as de-tokenizers for this task. Since they are not explicitly optimized for preserving fine-grained identity features, some loss in identity consistency may occur.

To address this, future work can explore adding local consistency constraints, such as token-to-token or patch-to-path supervision,which may better preserve identity than the current image-level objectives.

Dear Reviewer fcrb,

Thank you very much for your thoughtful and constructive review.

Q1: Comparisons to continuous visual representations (MetaMorph and MetaQuery)

Ans: We appreciate your suggestion to clarify the motivation behind using discrete representations, and we will expand in the revision to include a more detailed comparison with MetaMorph and MetaQuery.

(1) MetaMorph builds a visual representation by regressing SigLIP latents through an additional vision head, followed by training a diffusion decoder with an image autoencoding objective. Due to this regression and reconstruction pipeline, MetaMorph is not able to sample diverse images from a prompt. It lacks a natural generative process like that in autoregressive LLMs. In contrast, our discrete visual tokens allow direct integration with autoregressive sampling strategies, enabling diverse and controllable generation in a unified token space.

(2) MetaQuery leverages a frozen MLLM for multimodal condition extraction. However, the image understanding and generation modules remain decoupled. The former is handled by the frozen MLLM, the latter by a diffusion decoder. This is conceptually closer to traditional diffusion-based methods (e.g., DiT) than to a unified model with shared representations. In contrast, our method explicitly aims to unify understanding and generation through a single discrete semantic space (i.e., TA-Tok).

(3) Why do we refer to Dif-DTok as a “De-Tokenizer” instead of a diffusion decoder (generator)? Our Dif-DTok is trained to reconstruct the input image from discrete visual tokens produced by TA-Tok, rather than to generate new images conditioned on TA-Tok’s tokens. Since TA-Tok tokens already preserve 2D structure and rich visual details (e.g., 27*27=729 tokens 2D grid), Dif-DTok’s role is to faithfully transform these structured tokens into the pixel space. We believe “de-tokenizer” more accurately captures its function as part of a modular pipeline, rather than a full generative decoder.

(4) The generated visual tokens are not merely conditioning signals, but 2D visual tokens that encode the full information necessary for image reconstruction—including structure and detail. This is demonstrated in Appendix Table 5, where images encoded by TA-Tok can be faithfully reconstructed by a De-Tokenizer. We believe it is a key difference from MetaQuery. Our Tar model can directly compose and organize visual content in discrete token space, similar to how LLMs generate coherent text sequences. Importantly, our system does not rely on a strong diffusion model to handle both multimodal prompt understanding and image generation like MetaQuery. Instead, a simple De-Tokenizer that reliably converts TA-Tok’s tokens into VQ-VAE or VAE tokens is sufficient.

Therefore, we consider our method to be in line with prior work such as VILA-U and UniTok, but with a more expressive and LLM-aligned tokenizer design. While we acknowledge that discrete tokens currently underperform continuous features in some understanding tasks. However, as noted in Appendix I, we see this as a technical rather than conceptual limitation. We expect the gap to narrow with longer token sequences, higher image resolutions, and improved vector quantization methods.

Q2: Motivation to use discrete tokens?

Ans: The community recognizes the value of developing a unified model for both visual understanding (VLM) and generation. Among potential architectures, LLMs provide the most natural and scalable foundation for unifying these tasks-offering a flexible framework centered on next-token prediction.

The central challenge is how to represent visual information within the LLM paradigm. While approaches like Transfusion adopt separate diffusion objectives for generation, they break the unified modeling pipeline for visual understanding and generation. In contrast, discretizing images into tokens enables the use of a single autoregressive modeling strategy, next-token prediction, for both text and image modalities.

However, existing discrete visual tokenizers (e.g., VQVAE) tend to operate at the pixel level, which limits their semantic alignment with LLMs. Unified tokenizers like VILA-U and UniTok attempt to achieve two fundamentally conflicting objectives simultaneously: pixel-level reconstruction and semantic-level alignment, making it difficult to strike a good balance between the two. To address this, we propose TA-Tok, a tokenizer designed to produce fully semantic, discrete visual tokens, initialized from LLM embedding. This alignment allows visual tokens to seamlessly interact with language tokens during autoregressive modeling.

While TA-Tok does not directly decode images, it preserves rich 2D spatial and structural information. This makes the output tokens more than just abstract conditions: they encode image layout and visual detail in a structured form. Then a lightweight de-tokenizer suffices to reconstruct the image, unlike previous works that require powerful diffusion decoders to synthesize images from scratch.

In summary, discrete tokens are essential for unifying visual understanding and generation within an LLM framework. Our design of TA-Tok bridges the semantic gap between vision and language, enabling a more integrated and efficient modeling pipeline.

Q3: Whether both modalities are learned within a single latent space?

Ans: In Line 32, we emphasize the advantages of a unified visual representation for both image understanding and generation, when compared to approaches that rely on two types of visual tokens (e.g., Janus). Our approach aims to unify modalities at the token level, where both vision and language operate in a compatible discrete semantic space.

As clarified in our response to Q1 and Q2, the de-tokenizer in our pipeline does not serve as a standard generative model, but rather transforms the semantic tokens from TA-Tok into VQVAE/VAE latent space for image reconstruction. The planning of image structure and visual details-traditionally done by the image generator-is instead handled by the LLM through discrete token generation.

Methods like VILA-U and UniTok also claim to unify modalities with simple CNN decoders. In contrast, we introduce autoregressive and diffusion-based de-tokenizers, which are more expressive and better suited to decoding semantic visual token from TA-Tok. Thus, despite architectural differences, our method belongs to the same line of unified modeling works like VILA-U and UniTok, but pushes further in terms of semantic alignment and generation fidelity.

Q4: The Understanding Only Models in Tab.1 are outdated.

Ans: We followed prior works [6, 55] in selecting these baselines and will include more recent models in the revision. However, Table 1 mainly aims to compare unified models, with understanding-only models serving as reference points. Direct comparison is also challenging, as recent understanding models often use more data and complex training pipelines. Despite this, our Tar model shows state-of-the-art performance among unified approaches.

Q5: Empirical results of training and inference latency of two De-Tokenizers.

Ans: Thank you for pointing this out. We have conducted a comparison between AR-DTok and Dif-DTok in both training and inference phases, as summarized in the table below. While the two models differ in architecture, parameters, and codebases, we carefully controlled for factors such as batch size, token count, and hardware setup to ensure a fair comparison. Some key observations:

• Training speed is comparable across both models.
• Inference: AR-DTok uses less GPU memory and runs faster at 256 tokens, but becomes slower at 1024 tokens due to its autoregressive nature.
• Dif-DTok scales better at high token counts but consumes more memory.

We will include this analysis and clarify the differences in the revision.

b/s: batch per second. s/b: second per batch. The batch size is 6 for training and 4 for inference.

Thank you to the authors for their detailed rebuttal. They shared additional insights on their design which I am greatly appreciated. Maybe the "De-Tokenizer" occupies a middle ground, functioning in between as a generation model and a reconstruction model. While I’m not fully convinced, I think it is a meaningful exploration backed by solid experimental support. As a result, I have decided to raise my rating to borderline accept.