Subobject-level Image Tokenization

Thank you for your constructive feedback! We realize that there are some concerns that arise from the interplay between token segmentation and token embedding. Therefore, before addressing your comments point-by-point, we first clarify how and why we disentangle these two components in our paper:

In this study, we explicitly separate the image tokenization into two stages: 1. token segmentation and 2. token embedding. Our paper emphasizes the segmentation step, which has been largely overlooked in existing research focusing primarily on improving embeddings. Our claim is that: adaptive segmentation facilitates better image understanding, and this is consistent across different embedding methods.

We adopted this separation due to several considerations:

• Segmentation and embedding have fundamentally different roles and characteristics. They are agnostic to each other and can be flexibly combined.

• Disentangling these two components allows controlled experiments and clearer attribution of improvements.

• This separation aligns well with previous works investigating token segmentation methodologies, both in language modeling and image understanding literature.

Below, we provide detailed responses addressing each of your specific comments.

---

1. Token Embedding for Subobject Segmentation (Weakness 1)

TL;DR Rather than overlooking token embeddings, our work demonstrates that adaptive segmentation consistently enhances image understanding across various embedding methods.

Existing studies typically focus on improving token embedding quality while defaulting to simple, uniform patch-based segmentation. In contrast, our paper explicitly addresses the overlooked aspect of adaptive token segmentation. This segmentation method is designed to remain agnostic of embedding choices, facilitating a controlled evaluation of segmentation effectiveness across diverse embedding schemes, as in Fig.6 (left).

Although popular ViT-based encoders are convenient and commonly used, they are by no means mandatory for our method. Indeed, as in Fig. 5(e), our EPOC achieves stronger advantages when coupled with convolutional VAE encoders or using raw RGB pixels.

---

2. Extrinsic Evaluation Settings (Weakness 2)

TL;DR We indeed largely adopt the LLaVA framework, but with some minimal sufficient modifications. Without these changes, the original LLaVA framework cannot provide meaningful evaluation for adaptive token segmentation methods.

The goal of this extrinsic evaluation is to probe the effectiveness of adaptive segmentation, not to propose any new VLM framework. The VLM evaluation setup in our paper closely follows common practices already established in the literature. The commonalities includes:

• We adopt the standard "Visual Encoder → MLP Connector → LLM" architecture from LLaVA.

• We adopt visual encoders (CLIP, DINOv2, VAE), MLP settings (2-layers), and LLM (autoregressive decoder-only Transformer) that is well aligned with existing studies.

• We adopted ShareGPT4V and Pixmo-cap, which are both widely used for pretraining VLMs.

However, minimal changes were necessary:

• Standard LLaVA assumes fixed-size and raster-ordered patches, thus omitting positional embeddings. We introduced positional embeddings specifically to handle adaptive tokenization, which involves irregular segment shapes and arrangements.

• Standard VLM benchmarks commonly evaluate visual reasoning through tasks like VQA, reflecting both vision and LLM capabilities. Since our goal is exclusively to measure improvements in image understanding capability, we only measure the caption quality—thereby minimizing confounding from LLM knowledge and reasoning factors.

---

3. Throughput Comparison (Questions For Authors)

According to the disentanglement between token segmentation and token embedding, we interpret “patch-wise tokenizer” in the question as combining patch segmentation with CLIP or DINOv2 embeddings. A fair throughput comparison thus requires combining the same embedding backbones with subobejct segmentation. Since embedding methods do not impact segmentation latency, the throughput conclusions from Table 1 directly apply – although patch segmentation is negligible in latency, the overhead introduced by our EPOC is minimal and practically insignificant relative to overall VLM training.

---

4. Structure and Clarity (Other Comments Or Suggestions)

Thank you for this constructive suggestion. While our original intention was to clearly separate the token segmentation from embedding strategies for conceptual clarity, we acknowledge that they are methodologically relevant. We will reorganize our paper in the final version to move "Token Embedding for Adaptive Segmentation" (Section 5.1) into the "Method" section (Section 3) as suggested.

---

Once again, we sincerely appreciate your detailed review and valuable suggestions!

Thank you for your thorough and insightful review. Below, we provide detailed responses addressing your specific questions and suggestions:

---

1. Formulating Token Monosemanticity Score (Question 1)

Yes. Below is the explicit formal definition, which will be added to the final version:

Definition 1 (Token Monosemanticity Score): Given an image $ \mathbf{X}\in\mathbb{R}^{H\times W\times 3} $, a predicted token segmentation $ \mathbf{M}\in\{0,\dots,N-1\}^{H\times W} $, and a ground-truth segmentation $ \mathbf{M}^{\ast}\in\{0,\dots,K-1\}^{H\times W} $, define the $ i $-th predicted token $ \mathbf{t}_i $ as the set of pixel coordinates assigned token index $ i $, i.e., $ \mathbf{t}_i=\{(h,w)\mid \mathbf{M}(h,w)=i\} $.

We say a token $ \mathbf{t}_i $ is monosemantic if it lies entirely within exactly one ground-truth semantic region. Formally, the indicator function of token monosemanticity is defined as follows:

$

\mathbb{I}_{\text{mono}}(\mathbf{t}_i)= \begin{cases} 1, & \text{if } \exists k \text{ such that } \forall (h,w)\in \mathbf{t}_i, \mathbf{M}^{\ast}(h,w)=k \\ 0, & \text{otherwise.} \end{cases}
$

Then, the Token Monosemanticity Score is defined as the fraction of all predicted tokens that are monosemantic:

$

\text{monosemanticity}(\mathbf{M}, \mathbf{M}^{\ast})=\frac{1}{N}\sum_{i=0}^{N-1}\mathbb{I}_{\text{mono}}(\mathbf{t}_i).
$

Intuitively, this metric quantifies how effectively the predicted segmentation avoids polysemantic tokens, which span multiple semantic regions.

---

2. Boundary Detector Backbone (Question 2 & 4)

We chose SegFormer primarily due to its popularity, simplicity, and computational efficiency. However, EPOC is definitely not confined to SegFormer. Boundary detection itself is a historically well-studied and relatively straightforward task, effectively handled by various convolutional or Transformer backbones [1-4].

[1] DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection. CVPR’15

[2] Richer convolutional features for edge detection. CVPR’17

[3] EDTER: Edge detection with transformer. CVPR’22

[4] DiffusionEdge: Diffusion Probabilistic Model for Crisp Edge Detection. AAAI’24

To directly address your questions regarding scalability and alternative backbones, we conducted an additional scaling experiment comparing the original SegFormer-b0 to a substantially larger SegFormer-b5. Both models were trained on SA-1B dataset for 2 epochs, and the followings are performance evaluation (boundary recall) results:

These results clearly show that while EPOC can indeed scale up with larger models, the performance gains are marginal relative to the substantial increase in computational overhead. This further justifies our original choice of the efficient SegFormer-b0.

---

3. Ablation on Position and Content Embeddings (Question 3)

We agree that an ablation study on position and content embeddings provides valuable insights into how VLMs utilizing adaptive token segmentation interpret images. We conducted an experiment with our EPOC-based VLM trained on CLEVR-cap, where we shuffled positional or content embeddings across visual tokens, and then evaluated the impact on performance.

Since CLEVR-cap is synthetic and the captions follow a structured template—e.g., “Total 5 objects: a small gray metal cube, a small red rubber sphere, …”—it allows us to parse the generation and calculate average accuracy (%) for individual attributes. The results are summarized below:

These results show that both positional and content embeddings contribute to accurately capturing semantic attributes and correctly associating them with their corresponding objects, confirming their complementary roles in adaptive token segmentation-based image understanding.

---

4. Comments and Suggestions on Section 5.2 and Fig. 5

Thank you very much for these practical suggestions. Indeed, Section 5.2 is densely packed due to the strict page limits of the ICML submission format. The final version is allowed an extra page, enabling us to reorganize the presentation. Additionally, we will ensure Figure 5 includes consistent grid lines across all subplots.

---

Finally, thank you again for your constructive and detailed feedback, which significantly improves the quality and clarity of our paper.

Thank you for your encouraging review and thoughtful questions. Your recognition of the strengths of our method is greatly appreciated. Below are our clarifications on your valuable questions:

---

1.Extrinsic evaluations on caption data rather than general VQA (in “Methods And Evaluation Criteria”)

Our main goal was to isolate image understanding performance clearly from complex reasoning tasks. Standard VQA datasets tend to fuse perception with significant knowledge and reasoning capabilities of LLMs. For example, questions like “Where is he looking?”, or “What time is it on the clock?” which commonly occurs on VQA data requires the VLM to understand concepts like “third-person pronoun”, “sense of direction”, and “time” which can potentially confound the assessment of the effectiveness of adaptive token segmentation. Thus, we focused on detailed captioning datasets to precisely evaluate token-level perception capability.

---

2. Practical value regarding the comparison between DINO and CLIP (in “Methods And Evaluation Criteria”)

The observed advantage of DINO over CLIP in our experiments primarily results from the higher resolution and dense supervision available in DINO embeddings. CLIP, especially the original OpenAI version, lacks resolution flexibility with a 7x7 feature map, limiting fine-grained perception that is essential for subobject tokenization. We agree CLIP still holds significant practical value, especially when high-resolution features are integrated, which can further enhance subobject tokenization.

---

3. Future direction towards video tokenization (in “Questions For Authors”)

Indeed, as we stated in Appendix E, subobject-level tokenization's greatest potential may lie in video understanding, where computational efficiency and token management are even more critical due to the temporal dimension. Our work provides foundational insights that naturally extend to video data, and we consider this a key future direction.

Thank you once again for your valuable input and for clearly identifying the broader applications of our method.

Thank you very much for your detailed and comprehensive review. We sincerely appreciate the considerable effort and depth of analysis you provided! Below, let us address each of your concerns point-by-point and outline concrete steps we will take to improve the final manuscript:

---

1. VLM Architecture in Extrinsic Evaluation (in “Methods And Evaluation Criteria”)

TL;DR Our approach is precisely a minimal adjustment of the LLaVA architecture. The modification involves adding positional embeddings to support adaptive token segmentation.

We appreciate your comment and realize that our original description might have caused confusion. To clarify, our VLM architecture directly builds upon the well-established LLaVA framework (visual encoder → MLP connector → LLM). We do not introduce any new paradigm, such as "separate visual encoder and a VLM on top" as suggested. In the final version, we will explicitly emphasize this fact in Sec 5.

The sole reason for the modification is to handle adaptive token segmentation, which inherently involves variations in token size, position, and arrangement—an aspect not supported by existing architectures. Apart from this, our training settings and datasets (e.g., using ShareGPT-4V, Pixmo-cap) align well with common practices in VLM pretraining.

---

2. Perplexity-based Metric in Extrinsic Evaluation (in “Methods And Evaluation Criteria”)

We appreciate this important suggestion. To clarify, we used validation perplexity mainly to maintain consistency and clarity across different datasets. We fully agree that additional metrics are valuable. For all data points in Fig. 5, we calculated their accuracy or BLEU as suggested, and measured their correlation with perplexity. As below, two metrics strongly correlate with each other. We will include these results in the appendix of the final manuscript:

$^*$The correlation on CLEVR-cap seems weaker compared to others. This is because they exhibit a log-linear relationship according to our visualization. The correlation between perplexity and log(accuracy) is -0.95.

---

3. Boundary-based Metric in Intrinsic Evaluation (in “Experimental Designs Or Analyses”)

TL;DR Mask-based metrics are fundamentally unsuitable for evaluating image tokenizers. Boundary-based metrics accurately assess tokenization quality in class-agnostic and multi-granularity settings and have been widely adopted in CV and NLP.

Mask-based metrics, such as mIoU, depend heavily on clearly defined semantic classes, which makes them inappropriate for evaluating our tokenizer. They also struggle when segmentation granularities differ between predictions and ground truth. Accurately subdividing an annotated object into several meaningful subparts may produce misleadingly low IoU scores.

Boundary-based metrics naturally overcome these limitations. These metrics have also long been standard in boundary detection literature and have been used by SAM-related studies. They are also applied in NLP tokenizer evaluation, assessing the alignment to linguistic morphological boundaries.

Additionally, the Token Monosemanticity Score is essentially a mask-based metric (see response 1 to reviewer Ev7G).

---

4. Reporting Training Loss in Fig.6. (in “Questions For Authors”)

The current Fig.6 is presenting average training loss, which is an indicator of convergence speed. To align with Fig.5, we will update it to report validation perplexity following your suggestion. In fact, the two metrics closely correlate with each other, as it can be seen in Fig. 15. With validation performance in Fig.6, the relative performance relationship stays exactly the same and the conclusions in Sec. 5.3 remains unchanged.

---

5. Relation to Recent Works (in “Questions For Authors”)

Thank you for highlighting these strongly related recent works. The submission already included [B] (Aasan et al., 2024), and we will cite and discuss [A] in our final version. Both studies explore adaptive token segmentation, closely aligning with our research, yet with important differences:

• Compared to EPOC, the superpixel approach in [B] performs bottom-up pixel grouping without semantic understanding, and SAM-based segmentation in [A] cannot ensure efficient and panoptic segmentation. They are already involved as baseline methods in our paper.

• These works focus on ViTs and image classification, while our approach extends to VLMs and evaluates performance on the more challenging task of detailed image captioning.

---

Once again, thank you very much for your comprehensive and insightful comments!