UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning

Thank you for your careful review and for the insightful feedback! We now provide detailed responses to address your concerns.

Q1: VisionLLM v2 [1] can also perform both referring and segmentation tasks.

Thanks for highlighting this important baseline! We clarify that our statement of "Unified" refers to the unification of (1) referring & segmentation and (2) processing images & videos. Although VisionLLM v2 [1] can also jointly perform object referring and segmentation, it is an image-only model and thus cannot achieve pixel-level video understanding. We will carefully revise the relevant statements in our paper to ensure no overclaim. Below, we also provide performance comparisons between VisionLLM v2 and UniPixel.

Comparison on Referring Expression Segmentation (RES) and Reasoning Segmentation tasks:

Comparison on Referring Expression Comprehension (REC) task:

Results in both tables demonstrate that UniPixel significantly outperforms VisionLLM v2 in RES, ReasonSeg, and REC tasks. We will add the above comparisons into Table 3 and Appendix Table 4, and include more detailed discussions in Sec. 2 (Related Work).

Q2: The two-stage inference strategy increases the computational complexity.

Thanks for mentioning this important point! We agree that the object memory bank design inevitably introduces additional latency. However, we clarify that:

• This design affects efficiency only when the inputs contain points or boxes. Scenarios involving masks as visual prompts (e.g., region captioning) or segmentation-only tasks (e.g., referring video object segmentation) remain unaffected.
• Predicting masks for point and box prompts before providing responses can help better interpret the model’s underlying reasoning process.
• This is a reasonable trade-off considering the improved region-level understanding performance. Moreover, the extra computational overhead can be effectively minimized using proper inference techniques.

Below we compare the efficiency and performance of different strategies on PixelQA$_{mixed}$. The inference speed was tested on a single RTX 6000 Ada GPU with Flash Attention 2.

• A vs. B: Adopting the object memory bank can effectively enhance both segmentation and regional understanding capabilities. This is due to the disentanglement of object localization and regional understanding, allowing both to benefit from training on related tasks (e.g., referring object segmentation and region-based QA).
• B vs. C: Given that the two inference stages share the same visual inputs, the vision encoder outputs and the KV cache for these visual tokens can be cached and reused across both stages. This technique improves the inference speed by 56%, bringing it very close to that of single-pass inference.

In addition, how to train the model with this mechanism?

We adopt specialized training samples and design a rule-based pipeline to achieve this mechanism:

• During training, as shown in Appendix Table 1, we repurposed video QA samples from VideoRefer-700K [2] and Inst-IT [3] as memory pre-filling samples, in which the model is expected to predict masks for a question with visual prompts. An example is shown below.

 Input: What are [1] <REF> and [2] <REF> doing in the video?
Output: The relevant regions in the question are [1] <SEG> [2] <SEG>.

• During inference, when <REF> tokens are detected in the input prompt, we manually construct the corresponding model response using the output template above, ensuring it contains the same number of <SEG> tokens. We then apply teacher forcing to guide the model in predicting masks for these visual prompts. The predicted masks are subsequently used to construct <MEM> tokens to enable regional understanding.

We will carefully revise Sec. 3.2 to provide a clearer explanation of how the object memory bank works.

Q3: It is important to point out the base model used by the method.

Good suggestion! We will revise the Size column to Base Model in all the tables to clearly indicate the underlying LLMs/MLLMs of different methods.

Q4: What is the performance of the proposed model on the general VQA tasks, such as VQAv2, MME?

We compare UniPixel with a strong counterpart Sa2VA and its baseline Qwen2-VL on these benchmarks below. The evaluations were conducted using lmms-eval.

The comparison shows that UniPixel outperforms Sa2VA while is comparable to Qwen2-VL on general image QA tasks, demonstrating the effectiveness of the proposed fine-grained understanding designs.

We hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!

References:

[1] VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks, NeurIPS 2024.

[2] VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM, CVPR 2025.

[3] Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning, arXiv 2024.

Thank you for your quick feedback! We are glad to hear that your concerns have been solved. We will revise the paper accordingly and include more in-depth discussions in the final version. Thanks again for spending time reviewing our paper!

Thank you for recognizing the value of our work and for providing the constructive feedback! Below we provide detailed responses to your concerns and suggestions.

Q1: Whether the proposed model can consistently outperform larger-scale LMMs?

Thanks for the insight! We agree that comparisons with larger models enable a more comprehensive evaluation of the proposed method. In our work, we have already included models larger than 7B such as LISA-13B [1], VISA-13B [2], M2SA-13B [3], and InternVL-26B [4]. Below, we supplement our analysis with additional larger models.

Compared with larger models such as GPT-4o, Qwen2-VL-72B, and Sa2VA-26B, the segmentation and referring capabilities of UniPixel-7B are still competitive, as demonstrated by the SOTA performance on ReVOS, MeViS, VideoRefer$^Q$, and PixelQA datasets.

Q2: The authors could consider including the GPU requirements for training the proposed model.

Thanks for the suggestion! We trained our model with 8 RTX 6000 Ada (48GB) GPUs. Training the 2B and 7B variants took around 2.5 days and 4 days, respectively. This information will be included in our revision.

Q3: The authors have not released the code or the pretrained weights for community use.

We clarify that all the code, model checkpoints, training logs, and online demos for this project will be fully open-sourced upon acceptance of the paper. We also welcome and will support the community to develop stronger unified referring and segmentation models based on our codebase.

We hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!

References:

[1] LISA: Reasoning Segmentation via Large Language Model, CVPR 2024.

[2] VISA: Reasoning Video Object Segmentation via Large Language Models, ECCV 2024.

[3] MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation, ICLR 2025.

[4] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks, CVPR 2024.

[5] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, arXiv 2024.

[6] Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos, arXiv 2025.

Dear Reviewer NW7Z,

Thank you again for your insightful review!

Since the rebuttal period is halfway through, we would greatly appreciate it if you could kindly review our response to ensure it adequately addresses your concerns.

Thank you for your time and consideration.

Sincerely,

Authors of Paper 394

I appreciate the responses from the authors. Most of my concerns have been addressed. However, I still have one question regarding the authors’ statement that all code will be released after the paper is officially published. Shouldn’t we release the code as soon as the work is completed to help accelerate progress in this field? Isn’t that, after all, one of the main purposes of arXiv? Why must we wait until official publication? The field of large language models evolves extremely rapidly; by the time the code is released, the research may no longer be state-of-the-art and thus may provide limited value to the community.

Many thanks for your careful review and insightful comments! We are encouraged by your recognition of our technical novelty, experiments, and writing. Below we provide responses to your concerns in detail.

Q1: More Ablation Studies on the Base Model

Thanks for your suggestion! We provide the following ablation study to clarify the attribution of UniPixel's performance gains. The evaluation metrics are J&F (RVOS datasets) and Acc (VideoRefer$^Q$).

Several conclusions can be drawn from the above results:

• A vs. B: When using the same base model InternVL2.5-4B and mask head SAM2 (Large), UniPixel clearly outperforms Sa2VA on MeViS (+9.4%) and Ref-DAVIS17 (+1.2%), suggesting that the performance gains of UniPixel largely come from architectural innovations and learning strategies rather than simple backbone/module upgrades.
• C vs. D: Following your suggestion (whether comparable results could be achieved using only Qwen2-LLM), we trained an MLLM with SigLIP-SO-400M + Qwen2-1.5B with the same training recipe as VideoGPT+ [2], and adopted it as the base model. The results show that slightly lower but comparable results could still be achieved by a weaker base model. Notably, this variant remains competitive with Sa2VA (InternVL2.5-4B) on MeViS (+2.9%) and Ref-DAVIS17 (-0.3%).
• D vs. E: We also tried to upgrade the base model from Qwen2-VL-2B to Qwen2.5-VL-3B. It improves the performance on ReVOS and VideoRefer$^Q$, but slightly downgrades the results on Ref-DAVIS17.

More detailed discussions on the contributions of base models will be included in our revision.

Q2: Inference Efficiency when Using Object Memory Bank

Thanks for bringing up this important point! We agree that the object memory bank design inevitably introduces additional latency. However, we clarify that:

• This design affects efficiency only when the inputs contain points or boxes. Scenarios involving masks as visual prompts (e.g., region captioning) or segmentation-only tasks (e.g., referring video object segmentation) remain unaffected.
• Predicting masks for point and box prompts before providing responses can help better interpret the model’s underlying reasoning process.
• This is a reasonable trade-off considering the improved region-level understanding performance. Moreover, the extra computational overhead can be effectively minimized using proper inference techniques.

Below, we compare the efficiency and performance of different strategies on PixelQA$_{mixed}$. The inference speed was tested on a single RTX 6000 Ada GPU with Flash Attention 2.

What are the advantages of this design compared to simply plugging in an external segmentation model?

Adopting the object memory bank forces the model to predict masks (and leverage them to crop visual features) before actual reasoning. This enables the disentanglement of object localization and regional understanding, allowing both to benefit from training on related tasks (e.g., referring object segmentation and region-based QA). The comparison between A and B in the above table demonstrates that this design can effectively enhance both segmentation and regional understanding capabilities.

Given that the two-stage approach inevitably increases computational overhead and inference latency, how did you balance this trade-off?

The extra computational overhead can be minimized using proper inference techniques. For example, given that the two inference stages share the same visual inputs, the vision encoder outputs and the KV cache for these visual tokens can be cached and reused across both stages. As shown in the above table (B vs. C), this technique improves the inference speed by 56%, bringing it very close to that of single-pass inference.

Were there attempts to achieve effective single-pass inference?

Yes. In fact, A in the above table represents our simplified implementation of segmentation + QA. This is achieved by manually appending a <SEG> token after each visual prompt token. During inference, we decode these <SEG> tokens using the mask head to obtain object masks corresponding to the visual prompts, while treating the LLM outputs as textual responses. However, this paradigm does not allow the model to "re-watch" key objects by mask-pooling important regions, leading to sub-optimal performance in both segmentation and QA.

Q3: Key Architectural Innovations or Distinctions Compared to Draw-and-Understand [1]

Thanks for highlighting this highly related work! We acknowledge its contribution to visual prompting understanding, and will include more discussions in Sec. 2 (Related Work). However, UniPixel is significantly different from their proposed VP-MLLM [1] in several aspects.

• UniPixel supports both image and video inputs, whereas VP-MLLM can only perceive images.
• Through joint positional and temporal encoding, UniPixel can accept visual prompt inputs at any frames, while VP-MLLM is limited to prompting on only one frame.
• UniPixel integrates referring and segmentation into a unified framework, while VP-MLLM is a referring-only model and cannot perform segmentation tasks.

Conceptually, UniPixel can be regarded as a superset of VP-MLLM, as it supports unified object referring and segmentation in both images and videos.

Q4: The Mutual Improvement of Referring and Segmentation Tasks

Good question! We also thought that these are fundamentally different tasks, but the experimental results in Table 7(a) demonstrate that they can be mutually improved through multi-task co-training. We hypothesize that this is because training on both tasks helps the model focus more on fine-grained semantics rather than solely on global information. Further investigation into this phenomenon is left for our future work.

Is the observed mutual improvement during training solely attributable to UniPixel's decoupled design?

No. It is also affected by the ratio of the two dataset types. During model training, the proportions of samples for regional understanding and object segmentation were 32.3% and 33.9%, respectively, which is approximately a 1:1 ratio. We observe that the mutual improvement effect disappears when the data distribution is imbalanced.

Would this conclusion hold with a single one-turn inference model?

Yes. According to Table 7(a) (the 3rd line), training on both tasks without the object memory bank can also jointly improve segmentation and regional QA performance.

We hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!

References:

[1] Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want, ICLR 2025.

[2] VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding, arXiv 2024.

Dear Reviewer D9pB,

Thank you again for your valuable comments!

Since the rebuttal period is halfway through, we would greatly appreciate it if you could kindly review our response to ensure it adequately addresses your concerns.

Thank you for your time and consideration.

Sincerely,

Authors of Paper 394

We sincerely thank you for the detailed and constructive comments. Below we provide point-by-point responses to address your concerns.

Q1: Attribution of Performance Gains

We highlight the differences between Sa2VA and UniPixel in terms of architectural design, supported input/output formats, and frame sampling strategy below.

• Architectural Design: Sa2VA uses SAM2-Large (224M) while UniPixel utilizes SAM2.1-Base+ (81M). These mask heads exhibit similar segmentation capability. Ablation studies on both the base model and mask head will be presented below. Compared with Sa2VA, our key architectual innovations are the Viusal Prompt Encoder w. Joint Positional & Temporal Encoding (Sec. 3.1) and the Object Memory Bank (Sec. 3.2), both effectively enable unified object referring and segmentation.
• Supported Input/Output Formats: Our method supports flexible visual prompt inputs (i.e, points, boxes, and masks) at any frame(s). This is achieved by our visual prompt encoder that jointly encodes spatial position and temporal frame index of each visual prompt. Although Sa2VA's preprint also claims support for point and box prompts, their actual code, data, and experiments do not reflect this (Sa2VA GitHub Issue #19). Only mask-pooling-style region caption experiments are presented in their Table 15 and Figure 7.
• Frame Sampling Strategy: Sa2VA samples the first 5 video frames as keyframe inputs to the MLLM, after which SAM2 is employed to track the object in the remaining frames. This is sub-optimal as the MLLM cannot perceive the entire video. Instead, UniPixel leverages more flexible uniform or fixed FPS sampling strategies, allowing the LLM to access more temporal context and thereby enhancing overall performance.

We also provide ablation studies to clarify the attribution of performance gains. The metrics are J&F (RVOS datasets) and Acc (VideoRefer$^Q$).

• A, B, C: Sa2VA's comparison presents the ranking of base MLLMs for RVOS tasks: InternVL2.5-4B > InternVL2-4B > Qwen2-VL-2B. This is aligned with our ablation study (E vs. F).
• D, E: Changing from SAM2.1 (Base+) to SAM2 (Large) has minimal effect on model performance.
• B, F: When using the same base model InternVL2.5-4B and mask head SAM2 (Large), UniPixel clearly outperforms Sa2VA on MeViS (+9.4%) and Ref-DAVIS17 (+1.2%).
• D, G: Upgrading from Qwen2-VL-2B to Qwen2.5-VL-3B improves the performance on ReVOS and VideoRefer$^Q$, while slightly downgrades the results on Ref-DAVIS17.

The above ablation study clarifies that the performance gains of UniPixel largely come from architectural innovations and learning strategies, rather than simple backbone/module upgrades. We will include more detailed discussions and the table above in our revision. Qwen2.5-VL based UniPixel will also be open-sourced.

Q2: In Table 3, add C3VG [1] as a competitive non-LLM-based baseline.

Thank you for highlighting this strong and important baseline! We have incorporated its evaluation results on RES task in both Table 3 and Appendix Table 4, and included detailed discussions in Sec. 2 (Related Work). Comparisons between C3VG and UniPixel-7B are presented below. Note that the cIoU (cumulative IoU) metric used in our work is equivalent to the oIoU in C3VG's paper. We compare results under both mIoU and oIoU metrics for comprehensive evaluations.

Using mIoU as metric:

Using oIoU as metric:

Here, FT denotes fine-tuning on RefCOCO/+/g datasets after multi-task co-training. Without such fine-tuning, our model outperforms all LLM-based methods but slightly lags behind the non-LLM-based SOTA (C3VG [1]). Such a gap can be effectively closed by fine-tuning on these downstream datasets, as can be seen from the competitive post-FT results. The comparisons suggest that UniPixel can achieve better or comparable results to non-LLM-based SOTAs like C3VG on RES.

Q3: In Appendix Table 5, add OneRef [2] and SimVG [3] as strong non-LLM SOTA methods.

Thanks for this valuable insight! We have added the REC performance of OneRef-L (BEiT3-L) [2] and SimVG-TB (ViT-L/32) [3] in Appendix Table 5. The comparison between these methods and UniPixel is shown below.

Our 7B model slightly outperforms SimVG, but falls short of surpassing OneRef. We attribute it to the sub-optimal bounding box prediction, where the boxes are derived from masks rather than regressed by task-specific heads. This could potentially be improved by introducing dedicated bounding box heads following [1, 3], suggesting a promising direction for our future work. Nevertheless, to our best knowledge, UniPixel is currently the best-performing LLM-based method (7B-level) on REC.

Q4: Details about the Object Memory Bank.

It seems that OpenReview failed to render some parts of your questions. Based on the available content, we have attempted to reconstruct them as follows. Please kindly correct us if our interpretation is inaccurate.

In Lines 152–153, for objects [1]–[4], are the <MEM> tokens constructed using a single <SEG> token decoded by SAM2 to obtain masks across all frames?

Yes. For each object, one <SEG> token is decoded by SAM2 and be used to construct multiple <MEM> tokens. In this case, there are four objects, thus four <SEG> tokens are generated during memory pre-filling.

How is the multi-frame memory for each object built from a single <SEG> token?

During memory pre-filling, the model predicts one <SEG> token for each object. This token is passed to SAM2 and serves as a "hidden prompt" on the first frame. SAM2 then predicts the object mask for the first frame and propagates it across the remaining frames. The output from SAM2 is a multi-frame mask (a binary mask with shape [$T$, $H$, $W$]), which would be saved into the object memory bank.

In the memory injection stage, the saved multi-frame mask is split into $T$ single-frame masks (each with shape [$H$, $W$]). We then construct the memory-enhanced prompt according to these masks. For example, for a video with 4 frames, if the mask for object [1] at frame <2> is all-zero (i.e., the object is not visible in that frame), the prompt would be [1]: <1> <MEM> <3> <MEM> <4> <MEM> (<2> is omitted since the mask is empty). In this case, only the masks at frames <1>, <3>, and <4> are valid and are used to crop the corresponding visual features. The cropped 2D features are then average-pooled into 1D vectors, projected, and used to replace the <MEM> tokens in the prompt.

How does the model determine that <4> <MEM> does not exist in the last frame?

This is determined by the mask head. If the predicted mask in a frame is all-zero, it means that the object does not exist in that frame.

We will carefully revise Sec. 3.2 to provide a clearer explanation of how the object memory bank works.

We hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!

References:

[1] Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints, AAAI 2025.

[2] OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling, NeurIPS 2024.

[3] SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion, NeurIPS 2024.

Dear Reviewer HVUw,

Thank you again for your constructive feedback!

As the rebuttal period is halfway through, we would greatly appreciate it if you could kindly review our response to ensure it adequately addresses your concerns.

Thank you for your time and consideration.

Sincerely,

Authors of Paper 394