ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation

We thank the reviewer for insightful and constructive feedback on our work.

W1.1: Provide a quantitative evaluation

A: We clarify that the effectiveness of the proposed adaptive-length tokenizer (ALTo) is quantitatively evaluated on our MultiSA1B dataset, which contains complex shapes and multi-object masks, as shown in Figure 8. The evaluation includes multiple settings of the length penalty coefficient. With a coefficient of 0.01—which is the value selected for our final model—we achieve an average output length of 21.51 (standard deviation: 6.38) and a high average mask IoU of 0.938, demonstrating the strong effectiveness of our approach in balancing token efficiency and segmentation accuracy.

Beyond the tokenizer itself, the advantages of our method are further supported by the state-of-the-art performance and significantly reduced token usage of ALToLLM-8B in Table 2.

W1.2: How does ALTo handle extremely complex objects that might require more than 32 tokens

A: Theoretical Basis. The maximum token length of 32 in ALTo is a deliberate design choice, inspired by the finding in TiTok [11] that "an image is worth 32 tokens," which demonstrates that a compact 1D sequence of 32 tokens can effectively represent complex visual information. Empirical Validation. (1) Our extensive experiments, including those on the highly complex Multi-Target-SA1B dataset, have shown that 32 tokens are sufficient to achieve high-quality segmentation. (2) As shown in Figure 8, our selected penalty coefficient (0.01) naturally leads to token lengths below 32 for most samples, with fewer than 0.5% of cases reaching the maximum. (3) As evidence in the response to W3, even for complex multi-object scenes, such as an image containing six distinct objects, our model could still mask all objects within 32 tokens. This design ensures a favorable balance between representational capacity and computational efficiency.

W2: Provide a quantitative analysis of the impact of GRPO

A: In fact, Figure 7 illustrates how GRPO influences mask quality and token efficiency. The x-axis shows IoU, while the y-axis represents the average sequence length. Blue points correspond to ALToLLM variants trained with different GRPO penalty coefficients, whereas red points indicate the fixed-length baseline. When comparing models with similar IoU (i.e., at the same x-coordinate), ALToLLM achieves comparable segmentation performance while reducing token usage by more than 5 tokens per sample on average—demonstrating superior mask quality and token efficiency.

Furthermore, as the length penalty coefficient in GRPO training decreases, mask quality improves at the expense of longer output sequences, confirming a controllable trade-off between accuracy and efficiency. This enables users to flexibly tailor the model’s behavior based on whether mask quality or token efficiency is prioritized in a specific application. As shown in Figure 6, this adaptation is achieved within just a few hundred training steps, supporting our claim in L170 that GRPO provides flexible and efficient control over the model’s output characteristics.

W3: Provide more complex scenarios

A: We sincerely appreciate the reviewer's insightful feedback regarding the demonstration of ALTo's adaptive capabilities. While our initial Figure 1 illustrated the progressive token allocation for objects with increasing edge complexity (from 4 tokens for a simple moon to 17 tokens for complex land boundaries), we fully agree that more sophisticated multi-object scenarios would better showcase our method's advantages. To comprehensively address this point, we analyze a complex scene with sequentially added objects. The results demonstrate ALTo's dynamic scaling of token allocation: beginning with 7 tokens for a single woman, the count appropriately increases to 30 tokens when incorporating fine details like flowers on a table. This systematic evaluation confirms our method's ability to automatically adjust token usage based on both object quantity and geometric complexity. Due to the NeurIPS rebuttal constraints, we present these findings in tabular form below and will include the complete visual analysis in the supplementary material for the final version.

Limitations

A: We sincerely apologize for this oversight in the supplementary material. The "Limitations and Future Work" section was inadvertently omitted during the final formatting stage. We will immediately add this section to the supplementary material.

Limitations and Future Work

While ALTo effectively handles most segmentation tasks with adaptive token lengths (1-32 tokens), two key directions merit further exploration. First, our approach requires multiple training stages, which increases the engineering complexity. A simpler training pipeline is desirable for future applications. Second, though our pipeline is designed to be modality-agnostic (treating mask tokenization as a special case of image tokenization), our experiments currently focus on mask validation. Future work will extend ALTo to RGB image tokenization and validate its effectiveness across more general vision tasks.

Thanks for the constructive comments on the intuitive, interesting and effective method.

W1: Most of the parts are borrowed from the HimTok work, and only token adaptation is the main contribution of this work

A: We would like to clarify that the proposed adaptive-length mask tokenizer and its integration into LMM -- resulting in ALToLLM -- are novel and non-trivial contributions, even though they are experimentally built upon the HiMTok architecture. The method introduces crucial innovations:

• The First Autonomous Adaptive-Length Mask Tokenizer. As shown in Table 1, ALTo represents the first framework enabling models to dynamically determine visual mask token lengths (2-32) based on shape complexities.
• Adaptive-Length Token Generation in MLLM. ALToLLM enables adaptive mask token generation for (referring) image segmentation tasks through SFT and GRPO, with elaborately-designed learning objective and RL reward. We can balance the quality and efficiency conviniently.

This has the potential to be applied to a broader range of visual scenarios in the future.

Q1: Extended to other tasks

A: We appreciate the reviewer's suggestion to evaluate broader capabilities. Our experiments demonstrate ALToLLM's versatility across multiple vision-language tasks.

(1) Reasoning Segmentation
We evaluate our model on the ReasonSeg benchmark, which requires complex reasoning for segmentation. Our method achieves superior performance compared to HiMTok, indicating stronger reasoning capabilities. These improvements are consistent with findings on other segmentation benchmarks such as gRefCOCO, further validating the robustness of our approach.

(2) Region Understanding
Following the setup of GLaMM, we assess Region-Level Captioning on the RefCOCOg dataset. Our method outperforms GLaMM in both METEOR and CIDEr metrics, demonstrating enhanced ability to generate semantically accurate and descriptive captions for localized image regions.

(3) Image Understanding
We further evaluate image captioning performance on NoCaps and Flickr30K, following the protocol of GLaMM. Two settings are considered: (i) the original setting, and (ii) a fine-tuned setting (denoted with †) using a small number of COCO training samples provided by GLaMM.

• In the original setting, our model tends to generate more detailed and descriptive captions—often exceeding the length and specificity of ground-truth annotations—leading to higher SPICE scores but lower CIDEr values due to style mismatch. This reflects our model’s strong scene understanding and descriptive richness.
• In the fine-tuned setting (†), our model successfully adapts to the concise captioning style typical of these benchmarks. On NoCaps, we achieve a CIDEr score of 113.6, surpassing GLaMM, while maintaining competitive SPICE performance. This adaptability underscores the flexibility and broad applicability of ALToLLM across different captioning styles and domains.

Q2: The difference in terms of problem setting between different types of datasets

A: Thank you for raising this important point. We clarify the key differences in the problem settings across the benchmark datasets:

• RefCOCO [48] focuses on basic referring expressions that rely on spatial cues (e.g., "left woman").
• RefCOCO+ [48] prohibits the use of absolute spatial terms, requiring models to understand attribute-based descriptions (e.g., "woman in red").
• RefCOCOg [49] emphasizes unambiguous expressions to distinguish between visually similar objects (e.g., "woman who is touching her head").
• gRefCOCO [50] includes generalized referring expressions, covering single-object, multi-object, and empty references (e.g., "all women").
• RefCOCOm [53] introduces part-level segmentation, requiring fine-grained understanding of object parts (e.g., "woman's hat").

This progression from simple to complex referring expressions systematically evaluates a model's ability to handle various linguistic and visual challenges. Among them, gRefCOCO, with its focus on multi-object references, is particularly relevant for assessing the effectiveness of our adaptive tokenization mechanism in complex, real-world scenes.

Q3 & 7: Ablation studies contributing to the final results

A: Thank you for your interest in our ablation studies. Building upon the results on the gRefCOCO val set (Table 2), we further incorporate the GRPO results with a penalty coefficient of $1 \times 10^{-3}$ (as shown in Figure 7). We perform more ablation experiments on the following components:

• Multi-Target-SA1B Pretraining: Utilizing our curated dataset containing complex and multi-object masks during Stage 1 pretraining.
• Pixel Encoder: Incorporating a pixel-level encoder to provide fine-grained visual features to the mask de-tokenizer.

The results are summarized below:

Q3: RL phase compared with the results of the SFT phase

A: As shown in Figure 7, the performance gain from the GRPO phase compared to the SFT phase (Table 2) is relatively modest in terms of absolute metrics (e.g., +0.1 cIoU). However, as noted in L170, the main advantage of GRPO lies in its flexibility—it allows users to control the trade-off between mask quality and token efficiency.
By adjusting the length penalty coefficient during GRPO training, we observe that reducing the penalty leads to higher mask quality at the cost of increased token usage, while increasing the penalty yields more compact outputs with slightly lower mask fidelity. This level of controllability enables the model to be adapted to different application scenarios, depending on whether users prioritize high-quality masks or efficient token generation.

Q7: Main factors contributing to the performance gains over HimTok in Tables 2 and 4

A: The table above shows that the improvements are primarily attributed to two key components:

• (1) Multi-Target-SA1B Pretraining, which significantly enhances the model’s ability to generalize to complex scenes with multiple objects.
• (2) Pixel Encoder, which provides fine-grained visual representations that enable more accurate reconstruction of detailed mask boundaries.

Importantly, the performance improvement on gRefCOCO (Table 2) is notably larger than that observed on RefCOCO, RefCOCO+, and RefCOCOg (Table 4). This indicates that our model excels particularly well in multi-object settings (see Q2), which aligns with the design goals of the Multi-Target-SA1B pretraining and the fine-grained visual modeling enabled by the pixel encoder.

Q4 & 5: Problems on writing (L153&156)

A: Thank you for your careful comments on the writing issues. We will thoroughly review the manuscript.

• In L153, $\hat{P}=P.detach()$, which is detached from the computation graph, so that $\hat{T}$ is differentiable with respect to $P$.
• In L156, thanks for pointing out this typo — it is indeed $\mathcal{L}_{\text{Length}} = \lambda \hat{L}$.

Q6: The method list is not similar for Table 2 and Table 3

A: Different dimensions are compared using different method list in Tables 2 and 3, respectively.

• It is natural to compare performance metrics (e.g., cIoU and gIoU) with many related works (method list in Tab. 2), but the average token length is only meaningful for methods following HiMTok paradigm (i.e., discrete mask token generation). Thus we only compare ALToLLM-8B (AL) with ALToLLM-8B (FL) and HiMTok on average token length in Table 2.
• To better illustrate the efficiency gains brought by the adaptive token mechanism, we also report the corresponding average inference time (Table 3). Since the LMM part of ALToLLM-8B (FL) follows the architecture design in HiMTok, it should have the same inference time.

Q7: What is the main factor that makes the difference between Fix-length ALTo with Fix-length tokenization compared with HimTok in Table 2 and 4?

A: Please kindly refer to Q3 & 7 above.

Thanks to the authors for the response.

After reading the rebuttal, I still have concerns about the application scope of the adaptive length token mechanism:

1. In the task that does not require a segmentation mask (image or region understanding), how can we apply the adaptive token strategy, since no ground truth mask is available in this task, while in the paper, we need a segmentation mask to train the model?
2. Why are the results not better than Glamm in the Flickr 30k dataset in terms of CIDEr metrics, even with the fine-tuned models? And in this setting, what is the advantage of the proposed methods and Glamm baselines in terms of the number of tokens?

We sincerely appreciate the reviewer’s thoughtful and constructive feedback on our work. Below, we address each point with detailed clarifications.

W1.1: The paper evaluates it solely on mask tokenization tasks. Why the method is especially suitable for masks?

A: Although the color space of masks is simple, it is non-trivial and essential to generate masks that not only ground object positions but also capture fine-grained shape details, as pointed in HiMTok [16]. Our method is well-suited for mask generation tasks, given the recent advances in the field and characteristics of the task.

(1) Recent studies like FlexTok [14], HiMTok [16], have shown that representing masks or images of varying complexity may require different numbers of tokens. For masks, simple shapes can be captured with just a few tokens, while complex ones demand more. Although HiMTok adopts a hierarchical (coarse-to-fine) token sequence, it lacks the ability to adaptively determine how many tokens are needed based on the shape complexity. Building on this insight, ALTo introduces a method that enables the model to autonomously decide the token sequence length in an adaptive manner. (2) The quality of generated masks can be quantitatively evaluated in a straightforward manner (e.g., IoU), and segmentation instructions typically correspond to deterministic target masks—unlike the ambiguity and diversity inherent in tasks like image generation. As the first to propose and focus on a model that autonomously adapts token sequence length, we chose to validate it on mask generation rigorously to minimize the influence of confounding factors during both training and inference, before extending to more complex domains.

W1.2: Whether it can be generalized beyond this domain?

A: As highlighted in our paper (Introduction and references [8, 9]), both image genration and segmentation masks can be represented within a unified tokenization framework. As demonstrated by prior works like HiMTok [16] (for masks) and FlexTok [14] (for RGB images), the coarse-to-fine token generation paradigm effectively handles both domains. Our current validation on segmentation establishes the method's effectiveness for mask generation. We fully acknowledge the potential for broader applications and plan to explore image tokenization in future work.

W2: Lack of baseline with fixed optimal length

A: In fact, Figure 7 directly compares our adaptive-length model (ALToLLM after GRPO) against the fixed-length baseline (FL) set to the TLP-determined optimal lengths. The x-axis and y-axis show the IoU and average sequence length for each experimental setting, respectively. The blue points represent the adaptive-length models obtained with different GRPO penalty coefficients, while the red points represent the fixed-length baseline. When we compare models with the same average token length (i.e., at the same y-coordinate), the adaptive model consistently achieves higher IoU than its fixed-length counterpart, demonstrating that the performance gain stems from adaptivity itself, not merely from better length calibration. Furthermore, when comparing models with similar IoU (i.e., at the same x-coordinate), the results show that ALToLLM maintains comparable IoU while reducing token usage by more than 5 tokens per sample on average. This efficiency advantage holds consistently across all tested datasets, including zero-shot scenarios, confirming that dynamic length adaptation is a key factor underlying the superiority of our method.

Q1: The tail tokens' dropping approach

A: The procedure is carefully designed across different training stages. Stage 1: We randomly drop tail tokens (keeping 1-32 tokens) during training to learn hierarchical mask representation. Stage 1.5: We introduce the differentiable Token Length Predictor (TLP) which learns to optimally truncate tokens through the cumulative stopping probability. This allows the model to: (a) predict appropriate lengths based on content complexity, and (b) maintain differentiability for end-to-end training via our straight-through estimator. Stage 2: During training, we use the pretrained TLP to determine optimal sequence lengths and perform deterministic truncation. For inference, the model dynamically generates mask tokens until predicting the end token, eliminating the need for explicit token dropping.

Q2: Rewritten of fomulation

A: Thanks for this valuable and careful suggestion. We will revise the formulation to emphasize the contribution of earlier tokens to the stopping decision. We will carefully check all equations throughout the paper to ensure mathematical rigor and clarity.

We sincerely thank the reviewers for their thoughtful feedback and for recognizing the merits of ALToLLM, particularly its adaptive token length mechanism, improved accuracy-efficiency trade-off, and insightful analysis of length regularization.

W1: Lack of ablation studies.

A: Good suggestions. To more comprehensively illustrate the key factors behind ALToLLM's performance gains over HiMTok, we conducted additional ablation studies on the gRefCOCO val set, although Table 2 and Figure 7 have already presented part of the relevant results.

We analyze the following design choices:

• Pixel Encoder: Introducing a pixel-level encoder to supply fine-grained visual features to the mask de-tokenizer.
• Tuning Mask De-tokenizer (MD): Whether the mask de-tokenizer is updated or kept frozen during Stage 2 training.
• Token Length Predictor (TLP): Predicting the number of mask tokens adaptively, enabling variable-length sequences.
• GRPO Training: Applying GRPO in Stage 3 with a length penalty coefficient of $1 \times 10^{-3}$ for sequence regularization.
• Multi-Target-SA1B used in Pretraining: Leveraging our dataset containing complex and multi-object masks during Stage 1 pretraining.

The results are summarized below:

W1.1: Where the gains over HimTok come from.

A: The ablation study reveals that the performance improvements of ALToLLM over HimTok primarily stem from two key components:
(1) Multi-Target-SA1B Pretraining boosts cIoU from 70.4 to 72.8, significantly enhancing the model’s ability to handle scenes with multiple and complex objects.
(2) The Pixel Encoder further increases cIoU by 2.0 points (to 74.8), thanks to the fine-grained visual feature that enables more accurate mask boundary reconstruction.

Additionally, the TLP improves token efficiency—reducing average output length by around 50%—while still increasing cIoU to 75.4, indicating that adaptive token generation reduces redundancy without sacrificing quality.

W1.2: Effect of GRPO

A: We acknowledge that GRPO contributes only marginally to absolute performance metrics (e.g., +0.1 cIoU). However, as discussed in L170, its primary value lies in controllability: GRPO offers a flexible mechanism to balance mask quality and token efficiency. As shown in Figure 7, increasing the length penalty coefficient in GRPO training improves token efficiency at varying degrees of cost to mask quality. This tunability allows us to adapt the model to different application needs—prioritizing accuracy in high-fidelity scenarios or efficiency in bandwidth-constrained environments. Moreover, as demonstrated in Figure 6, this adaptation can be achieved within just a few hundred training steps, highlighting the efficiency and practicality of GRPO training in dynamically shaping model behavior.

W1.3: Difference from HimTok in Stage 2 training

A: In Stage 2, unlike HimTok—which jointly fine-tunes both the mask de-tokenizer (MD) and the multimodal large language model (MLLM)—we only train the MLLM while keeping the MD frozen. Our ablation shows this design achieves comparable performance, as MD has been already well-learned during Stage 1 and capable of accurately generating diverse masks. Notably, we observed that the public implementation of HimTok omits the original LayerNorm layer when it is integrated into LMM, which is inconsistent and necessitate the tuning of MD in Stage 2. In contrast, our architecture is consistent before and after integrating ALTo into LMM, which eliminates the need to update the MD in Stage 2.

Q1: Mask token codebook size clarification

A: The mask token codebook size is 1024, consistent with HimTok for fair comparison.

Q2: Rewards about incorrectly formatted sample

A: In our mask de-tokenizer step, any responses that do not conform to the correct format will raise an exception, and its corresponding IoU reward is set to 0. As a result, incorrectly formatted samples always receive lower composite rewards compared to correctly formatted ones. We have empirically observed the desired behavior with this design: during GRPO training, the average format reward consistently converges to 1 within approximately 20 steps, and thereafter the model rarely produces samples with formatting errors.

Q3: Provide a plot of the distribution of mask token sequence length

A: (1) In Figure 8, we present the distribution of mask token sequence lengths in Multi-Target-SA1B under different penalty coefficients during Stage 1.5. When the coefficient is set to 0.01, the token lengths are primarily concentrated between 20 and 30, which may be attributed to the more complicated masks in Multi-Target-SA1B. (2) To see the actual distribution on real benchmarks, we plotted the token length distribution of ALToLLM (SFT) on the gRefCOCO val set, where most samples fall within the 10–20 range. Due to rebuttal constraints, we present the distribution for lengths 10–20 in tabular form instead of images.