Seg-VAR:Image Segmentation with Visual Autoregressive Modeling

Thank you for your thoughtful and constructive feedback on our work. We appreciate the recognition of Seg-VAR’s conceptual novelty, the effectiveness of seglats, and the strength of our experimental results. Below, we address each concern, clarify ambiguities, and provide additional details to strengthen the work. We will add this refinement to the final version of the paper.

1. Component-level ablation

Sorry for the confusion. The ablation in Table 5 is the ablation about each module design: stage 1 represents seglat encoder/decoder, stage 2 refers to latent encoder and decoder, and stage 3 represents the image encoder learning. That is, we discuss the impacts of different module designs. Since latent learning is the core idea of Seg-VAR, we keep the latent encoder and decoder in ablation studies.

To explicitly validate the role of seglats, we conducted additional experiments using a "vanilla VAR" baseline (removing latent encoder and decoder, and using plain VAR). As shown in the table, simply using vanilla VAR without seglat modules, the result is 8.4 lower than our Seg-VAR.

Table 1. Vanilla VAR v.s. Seg-VAR

2. Pipeline complexity and resources

We acknowledge the training pipeline (pretraining seglat encoder/decoder, latent modules, final alignment) is more complex than standard discriminative setups. However, our module designs prove their effectiveness in the ablation study, and that staged approach is necessary to effectively learn the structured latent space and ensure stable convergence of the autoregressive process. End-to-end training from scratch proved unstable in ablation studies (4.6 decrease in mIoU). What's more, under comparable parameters (Table 3), our Seg-VAR still outperforms Mask2Former in R50 backbone. These results suggest the efficiency and effectiveness of Seg-VAR.

Table 2. Staged training vs. End-to-end.

Table 3. Comparable parameters

3. Terminology and figure

We apologize for ambiguities in terminology and figures, which we clarify below:

• for the latent encoder, we aim to indicate that this structure encodes the segmentation mask and output segment latents (seglat). Worth mentioning, this module is a combination of the transformation of segmentation masks and color encoding, rather than merely a "spatial embedding mechanism". As for the figure, we thank you for your careful suggestions, and we will revise it in the final version! (Our intent is to highlight the contents of the image, because binary will lose details.)

4. Sensitivity of grid size and colormap We evaluate the robustness of grid number and Palette on ADE20K. As shown in the table, as the grid number changes greatly, the performance decreases as the number decreases, because the granularity can help modeling better distinguish instances. As for the palette size of the colormap, we discover that 6 is the optimal number because we have to balance between a large color space and loss of generality. (Ideally, the size should be larger than the category number). Overall, we find that grid numbers are more sensitive than palette size, but they still contribute to the performance gain. These results confirm Seg-VAR is robust to reasonable variations in grid/color parameters.

Table 4. Grid & Colormap

5. Seglat and latent decoder

The difference in structure naturally prohibits the combination of two decoders, since the seglat decoder is part of VQ-VAE, while latent decoder is a transformer-based module. What's more, the seglat decoder and latent decoder serve distinct roles:

• Seglat decoder: Maps discrete latent tokens (from the image encoder) to seglats, preserving instance-level spatial structure encoded in color mappings.
• Latent decoder: Refines seglats into pixel-accurate masks, correcting residual errors in color-to-mask conversion (e.g., edge blurring)

End-to-end training of a single decoder failed to balance these two granularities (as proved in Table 2), confirming the need for two stages.

6. [CLS] and [TYP]

These two tokens are required to distinguish between different segmentation tasks. [CLS] provides semantic context for the generated image (categories), and the [TYP] token is used to select the type of segmentation tasks. Both tokens are vital to Seg-VAR and can not be ablated. We will include these details in the revised version.

Conclusion

We agree that architectural complexity is a key consideration. Our ablations (summarized above) demonstrate that each component (image encoder, seglat encoder/decoder, latent encoder/decoder) contributes meaningfully to performance. Removing any module resulted in significant drops.

We plan to explore model simplification in future work, such as sharing weights between encoders or using distillation, while maintaining Seg-VAR’s unified capabilities. We thank the reviewer for highlighting opportunities to strengthen the work. The additional ablations, figure revisions, and clarifications outlined above will enhance the clarity and rigor of our submission. We believe these changes address the concerns raised while reinforcing the novelty and effectiveness of Seg-VAR, which could potentially lead to more explorations in unifying perception and generation.

Dear Reviewer BXbx,

Thank you for the time and effort you have dedicated to reviewing our paper. We would like to know if our response has satisfactorily addressed your concerns and if you have the opportunity to provide further feedback on our rebuttal. We are open to any additional discussion.

Best regards,

Authors of paper 6113

Dear Reviewer BXbx,

This is the last moment to account for the author feedback and engage in a minimal discussion. Without any discussion your review will weight less than the others.

Best,

Your AC

Thank you for your insightful evaluation and high appreciation for our work. We greatly appreciate your recognition of the paper’s clear narrative structure, superior performance, and innovative autoregressive paradigm for segmentation. Your constructive feedback on areas for improvement is invaluable, and we address each concern below:

1. Mathematical Formulation of Seglat

Seglat is designed as a structured latent representation that encodes both instance identity and spatial geometry of segmentation masks, bridging low-level mask pixels and high-level semantic tokens. Mathematically, we define it as follows: Let $M∈\{0,1\}^{H×W×N}$ denote a segmentation mask, where H,W are spatial dimensions, and N is the number of instances. Then, through the spatial convolution and transformer layer, the encoder transforms the original masks into $\mathcal{S}\in\mathbb{R}^{H\times W \times 3}$, which can be viewed as a kind of RGB image. To add more explicit identity and positional control, we implement colormap encoding to generate an additional colormap $M_{c}∈\{0,1\}^{H×W×3}$. These are added and fed into the seglat encoder as the final seglat $\mathcal{S}$.

Let $k∈{1,2,...,K}$ denote a scale in the hierarchical transformer structure (where S is the total number of scales). At scale k, image tokens and seglat tokens are tokenized by a shared tokenizer $\Phi$: Image features are $X_{k}∈[V]^{h_k×w_k}$, and Seglats are $S_{k}∈[V]^{h_k*w_k}$, where V indicates the vocabulary size. A flattening operation is adopted to convert the sequence of 2D features into 1D. Full attention is enabled for both control and image tokens belonging to the same scale, which allows the model to maintain spatial locality and to exploit the global context between control and image. $X_k',S_k'=Attention(X_k,S_k,S_k)$. As for the algorithms for tokenizers, we follow the design of VAR, which consists of interpolation, queue push, and lookup operations.

2. Open Vocabulary Segmentation Capability

We evaluate the open-vocabulary capability by evaluating on LVIS. By adopting CLIP to the model for classification, our model achieves a preliminary result of 24.3 mAP on novel classes (vs. 17.1 mAP for Mask2Former). This is because the seglat design inherently supports open vocabulary adaptation: Seglat tokens are semantically agnostic—they encode geometry (boundaries, centroids) rather than fixed class labels.

3. Parameter Efficiency

We adjust the parameters of Mask2Former by extending its transformer layer number so that the parameters can be comparable. As shown in the table, our Seg-VAR still outperforms Mask2Former in COCO panoptic dataset.

Thank you for your detailed feedback and thoughtful evaluation of our work. We appreciate the recognition of our novel direction in bridging generative and discriminative paradigms for image segmentation, as well as the constructive criticism that will help us improve the clarity and rigor of the paper. We will reorganize the paper with better structures for readers to comprehend. Below, we address each concern and question raised:

1. Generative Model Architecture and Training Details

   We apologize for the potential ambiguity in describing the generative components of Seg-VAR. Seg-VAR is built on visual autoregressive (VAR) modeling and employs a hybrid design that combines hierarchical autoregressive decoding with next-scale prediction principles. As for the sampling strategy, it refers to an approach of sampling both pixel- and token-level controls for image generation with teacher-forcing guidance, which is an extension of classifier-free guidance by adding controls to the Bayesian-based conditional distribution, as mentioned in ControlVAR.

   As for the training objectives, the training objective is based on the Evidence Lower Bound (ELBO), optimizing three components: (1) mask reconstruction loss via seglat decoder, (2) KL divergence to align image-derived latents with seglat distributions, and (3) cross-entropy loss for token prediction.

   Overall, this design inherits VAR’s next-scale prediction strengths while adapting it to segmentation via conditional mask generation.

2. Tokenization and Latent Representation

   The seglat encoder is tightly linked to traditional VAR tokenization but extends it for segmentation-specific needs:

   1. Like classic VAR, Seg-VAR uses a multi-scale, residual tokenization process: the image encoder generates latent tokens at multiple scales via a multi-scale fusion module (transformer layers + projection), and the seglat encoder (VAR-like structure) maps masks to discrete tokens across these scales.
   2. Unlike standard VAR, seglat tokens incorporate spatial information via a location-sensitive color mapping. This mapping assigns unique RGB values to instances based on their centroids (gridded into a*a regions), enabling the transformer to distinguish overlapping objects through positional cues.

   Latent encoder for segmentation is implemented to transform segmentation masks (H*W*K) into corresponding seglats (H*W*3) to be then fed into the Seglat Encoder & Decoder (which refers to the VAR-like structure), itself is a transformer-based structure with colormap encoding.

3. Connection to Image Autoencoders

   Seg-VAR’s segmentation autoencoder (seglat encoder/decoder) shares similarities with VQ-VAE but is specialized for segmentation (add extra image controls into modeling): Like VQ-VAE, it uses discrete latent tokens and a reconstruction loss.

   As for the latent tokens, they refer to the output from the image encoder, which is (H/d*W/d), which is optimized in the image encoder learning to minimize the differences with the latent codes by the seglat encoder (VQ-VAE-like structure). The segmentation masks use the latent encoder to generate seglat tokens with a transformer-based structure and colormap encoding, which incorporates spatial and instance-aware encoding.

Thank you for your insightful comments and valuable questions. We appreciate the opportunity to address these points and clarify our work.

1. Compute Cost and Latency

We acknowledge that even though Seg-VAR has consistent improvement across standard benchmarks, it exhibits higher computational overhead (parameters, FLOPs) and slower inference speed compared to discriminative models like Mask2Former. This is related to our two key innovations: (1) the autoregressive decoder, which sequentially predicts seglat tokens to ensure spatial coherence, and (2) the multi-module architecture (image encoder, seglat encoder/decoder, latent aligners) required for latent space learning.

While full parallelization of sequential decoding is infeasible, targeted optimizations could reduce latency without sacrificing core autoregressive modeling:

• Hierarchical Parallelism Across Scales

Seg-VAR uses hierarchical decoding, where coarse-scale tokens (establishing global context) are generated first, followed by fine-scale tokens (resolving local details). Coarse-scale tokens could be generated in a compact sequence, after which fine-scale tokens within the same coarse region might be parallelized. For instance, once a global object boundary is defined by coarse tokens, local patches within that boundary could be processed in parallel, as their dependencies are constrained to the local context.

• Cached Attention Mechanisms

The transformer decoder’s attention over previously generated tokens can be optimized using cached key/value pairs. Instead of recomputing attention scores for all prior tokens at each step, cached values from previous steps can be reused, reducing redundant computations

2. Multi-stage Training

We acknowledge the training pipeline (pretraining seglat encoder/decoder, latent modules, final alignment) is more complex than standard discriminative setups. However, this staged approach is necessary to effectively learn the structured latent space and ensure stable convergence of the autoregressive process. End-to-end training from scratch proved unstable in ablation studies.

Table 1. Staged training vs. End-to-end.

3. Statistical Significance

We apologize for the oversight in not explicitly reporting error bars or confidence intervals. While our experiments were run 3-5 times (as noted in the checklist) , we will add in the supplementary material: (1) standard deviations of key metrics (PQ, AP, mIoU) across 5 independent runs on COCO and Cityscapes, and (2) 95% confidence intervals for ablation studies (e.g., Table 5) . This will better support the statistical robustness of our results.

4. Grid Size and Colormap

We evaluate the robustness of the grid number and Palette on ADE20K. As shown in the table, as the grid number changes greatly, the performance decreases as the number decreases, because the granularity can help the model better distinguish instances. As for the palette size of the colormap, we discover that 6 is the optimal number because we have to balance between a large color space and loss of generality. (Ideally, the size should be larger than the category number). Overall, we find that grid numbers are more sensitive than palette size, but they still contribute to the performance gain.

Table 2. Grid & Colormap

5. Practical Advantages of Autoregressive Formulation Beyond segmentation quality, the autoregressive design offers unique benefits:

• Video extension: The sequential token prediction naturally models temporal dependencies, making Seg-VAR adaptable to video segmentation (e.g., tracking instance motion via time-step token conditioning).
• Unifying generation and perception: The autoregressive formulation in Seg-VAR inherently paves the way for unifying generation and perception, as its design bridges the latent spaces of image content and segmentation masks through sequential modeling. (Which is also recognized by Reviewer avcA)