Towards Semantic Equivalence of Tokenization in Multimodal LLM

[-] Complex images. SeTok seems to be effective in handling simple images, but when faced with complex images, the quality of clustering seems difficult to guarantee (for example, there are dense people, objects, or significant repetition between people and objects in the image). As a result, clustering errors are injected into visual tokens, which corrupt the final output results.

[-] Source of gains. For MLLMs, there are usually two types of visual bottlenecks: A. maintaining detail resolution to form hierarchical information [1]; B. one reasonable tokenizer. When demonstrating the value of the tokenizer, this paper still uses the detokenizer (in Fig. 2) to reconstruct pixel level images, i.e., keeping visual details. Can you decouple these two to demonstrate the effectiveness of each component separately? For example, report the performance with only \mathcal{L}_{citc}, or make more comparisons with [1].

[1] Eagle: Exploring the design space for multimodal llms with mixture of encoders

I have some concerns regarding the comparisons in Table 1. The authors claim that SeTok is superior to Q-former or cross-attention framework. However, SeTokim and the other compared MLLMs do not share the same training scheduler, base ViT, LLM, or training datasets. A fair comparison is to evaluate various kinds of visual tokenizers in an environment with tightly controlled variables.

1. The performance of MLLMs is greatly influenced by SFT data and LLMs. Therefore, when comparing image understanding capabilities, it is important to make sure under the same settings, such as using the same settings as LLaVA, to better reflect the effectiveness of the proposed methods. However, there is lack of this comparison in the experiments.
2. In the comparison of image understanding capabilities, there is a lack of evaluation on more fine-grained benchmarks such as TextVQA, OCRBench, SEED-Bench-2-Plus, as well as more general benchmarks like MMBench and SeedBench. A potential concern is that SeTok's dynamic clustering strategy might lead to the loss of fine-grained information, making it challenging to complete more difficult tasks.

• Unfair comparison. 1) The model is finetuned on a lot of multimodal instruction datasets, whereas the compared works, like the ones in Tab. 1, use smaller data sizes for finetuning. Performing comparison on different training data hardly demonstrates the priority. 2) The visual encoder introduces more parameters for clustering (totally 20 transformer layers). 3) A more advancing pretrained encoder SigLIP-384 is used. In conclusion, many settings, including the data, the model parameters, and the pretrained encoder, exist unfair comparison. These settings conceal the true effectiveness of the proposed tokenization method. I understand that it is hard for an MLLM to perfectly align with previous settings, and it is ok. But my intension is to make sure the authors can explicitly present these differences in the tables, which helps prevent misleading and ensure fair comparisons in future research.

• Potential data leak. The benchmarks used in Tab. 1 all have released answers, which may be potentially seen during training. It is suggested to perform additional evaluation on some addition benchmarks that do not have released answers, like MMB.

• Potentially poor performance on fine-grained visual perception. The clustering-based visual tokenization may be unfriendly to some fine-grained, like OCR, because clustering always fuses information for coarse representation.

1.Table 2 compares the performance of SETOKIM method and some similar methods under the task of zero-shot text-to-image generation and editing under several data sets. The results show that the indicators of some data sets are not optimal, and there is still some gap with SOTA.