One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

We thank the reviewer for reviewing our paper.

Q1: This work utilizes two vision encoders, which will influence the speed of the model.

Response:

• We thank the reviewer for pointing out the valuable problem. In fact, the two vision encoders in the model are both necessary as they have their own unique role respectively. The CLIP-based vision encoder (termed as Visual Tokenizer in the paper), being pre-trained with image-text pairs, is responsible for semantic feature extraction, which is suitable for multimodal understanding. On the other hand, the SAM-based vision encoder, providing low-level vision features, is specially designed for producing segmentation masks, which is also adopted in a wide range of existing works, e.g., LISA.
• At the time we implement VideoLISA, as far as we know, the best model for these two types of features was still the expert model on each task. I.e., CLIP for semantic features and SAM for low-level features. In the future, if there are models that unify these features, we will consider replacing the two encoders with the new one to improve speed. We will discuss this problem in the revised paper.

Q2: It seems like the model has difficulty segmenting multiple objects at the same time.

Response:

• Firstly, we would like to emphasize that single object segmentation is the standard and most popular setting in the general field of language-guided object segmentation. In referring image/video segmentation [1,2,3] tasks and the reasoning image segmentation task [4], typically a text query describes a specific object in a straightforward or implicit way. The model is tasked to segment the object based on the given text description.
• Our work follows the widely adopted setting and democratizes reasoning segmentation to videos. In our paper, we did not make any claim regarding multi-object capability. We thank the reviewers for providing the valuable suggestion. We regard this as a meaningful future work and will discuss the possibility in the revised paper.

Q3: Curious about the results when replacing the SAM with other kernel-based segmenting and tracking models.

Response: We thank the reviewer for raising this interesting question. In fact, our One-Token-Seg-All approach exactly follows the philosophy of kernel-based tracking, as discussed in line 185-188 of our paper. The one [TRK] token is trained to serve as the semantic kernel while the visual features are the context to be contrasted.

Following the reviewer's suggestion, we explore other kernel-based segmentation and tracking models and report the experiment result in Table 4 of the rebuttal PDF. Specifically, we adopt XMem [5], which is a popular tracking model. It is equipped with a system of memory. The standard XMem model accepts the segmentation mask of the first frame, builds the memory, and inferences the masks of subsequent frames. The first frame memory here can be regarded as a semantic kernel for reference purpose.

In our experiment, we adapt this model into LISA [4] and VideoLISA. The results are reported in Table 4 of the rebuttal PDF.

• Our VideoLISA is compatible with XMem and shows remarkable performance in public benchmarks.
• When comparing LISA and VideoLISA under the same XMem setting, VideoLISA outperforms LISA by a noticeable margin.
• When comparing XMem and our One-Token-Seg-All, our method still shows superior performance, validating the effectiveness of the proposed method.

Q4: What are the advantages of the Sparse Dense Sampling strategy compared with QFormer and QFormer-like architectures (e.g. LLaMA-VID [6]) ?

Response: Q-Former and QFormer-like architectures (e.g. LLaMA-VID [6]) extract highly abstract semantic visual features through cross-attention, which significantly reduces computational overhead, especially for video data. However, although such highly abstract features might be okay for general video understanding tasks, such as VQA, they are not sufficient for the segmentation task, which is validated by our experiments. In contrast, our Sparse Dense Sampling achieves a delicate balance between preserving visual details and temporal context, making it favorable for video object segmentation tasks.

We conduct a comparison experiment between LLaMA-VID [6] and VideoLISA as reported in Table 3 of the rebuttal PDF. The result shows that our proposed strategy achieves remarkable performance in video segmentation tasks, surpassing the QFormer-style architecture.

We sincerely hope the rebuttal answers your questions and addresses your concerns.

[1] Kazemzadeh S, Ordonez V, Matten M, et al. Referitgame: Referring to objects in photographs of natural scenes[C]//Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014: 787-798.

[2] Seo S, Lee J Y, Han B. Urvos: Unified referring video object segmentation network with a large-scale benchmark[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. Springer International Publishing, 2020: 208-223.

[3] Ding H, Liu C, He S, et al. MeViS: A large-scale benchmark for video segmentation with motion expressions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 2694-2703.

[4] Lai X, Tian Z, Chen Y, et al. Lisa: Reasoning segmentation via large language model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9579-9589.

[5] Cheng H K, Schwing A G. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 640-658.

[6] Li Y, Wang C, Jia J. Llama-vid: An image is worth 2 tokens in large language models[J]. arXiv preprint arXiv:2311.17043, 2023.

We thank the reviewer for reviewing our paper.

Q1: It is not clear whether the model can segment multiple objects

Response:

• Firstly, we would like to emphasize that single object segmentation is the standard and most popular setting in the general field of language-guided object segmentation. In referring image/video segmentation tasks and the reasoning image segmentation task, typically a text query describes a specific object in a straightforward or implicit way. The model is tasked to segment the object based on the given text description.
• Our work follows the widely adopted setting and democratizes reasoning segmentation to videos. In our paper, we did not make any claim regarding multi-object capability. We thank the reviewers for providing the valuable suggestion. We regard this as a meaningful future work and will discuss the possibility in the revised paper.

Q2: The [TRK] token design seems to be simple.

Response:

• We would like to highlight that we do not pursue complexity in the model design. Instead, we aim to design a framework that is effective and suited to solve the problem. In fact, the approach of using a single [TRK] to segment all video frames is intentionally designed to be simple to avoid unnecessary complexity.
• The underlying rationale of the design comes from in-depth analysis and extensive experiments. Our motivation is that a single unified representation has the potential to segment video frames, as revealed by related works. Thus, a single [TRK] token, serving as an information bottleneck, is tasked to learn a compact object representation. However, this is non-trivial to achieve, as shown by our investigation on existing models, including SAM and LISA. Through experiments and analysis, we identify two key factors: incomplete input information and inappropriate training objectives. We thus enhance the cross-frame association capability of the [TRK] token from the two perspectives. We argue that the One-Token-Seg-All approach is simple yet effective by design. The analysis, experimental exploration, and the training objective design make the approach effective and non-trivial.

Q3: It is better to compare with another related work, PixelLM.

Response: We thank the reviewer for providing the valuable suggestion. We discuss the PixelLM model and compare it with our model. We will include the discussion into Related Work and the comparison into Experiment in the revised paper.

• Model design discussion. PixelLM is a large multimodal model for pixel-level image reasoning and understanding. Compared to LISA, PixelLM exhibits unique advantages in handling multiple objects. PixelLM excels in image-based tasks while it is inherently incapable to handle videos. Our VideoLISA aims to democratize the reasoning segmentation capability into videos. When comparing VideoLISA with PixelLM, VideoLISA highlights video segmentation while PixelLM emphasizes multi-object segmentation.
• Experimental comparison. We compare LISA, PixelLM, and VideoLISA in public benchmarks, as shown in Table 1 of the rebuttal PDF. Since LISA and PixelLM are not designed for videos, we adopt the One-Token-Seg-All approach similar to VideoLISA. Specifically, the prompt embedding from the first frame is used to segment the subsequent frames. The results show that VideoLISA outperforms LISA and PixelLM by a large margin. This performance gap comes from VideoLISA's dedicated designs that handle video temporal dimension, making it able to understand video content and output temporal-consistent masks.

Q4: It is better to show how does the model perform on text-generation tasks.

Response: We thank the reviewer for pointing this out.

• In VideoLISA, we intentionally re-purpose the large multimodal model into an expert model on reasoning video object segmentation. Our main focus is to democratize the reasoning segmentation capability into videos. The proposed designs are specially tailored for video temporal dimension. Thus, the text-generation capability is not specially considered.
• To assess this capability, we evaluate VideoLISA on popular multimodal understanding benchmarks, including MME, Science-QA, GQA, and TextVQA in Table 2 of the rebuttal PDF. We found that compared to the original large multimodal model, i.e., LLaVA, VideoLISA shows significant performance degradation in text-generation tasks. This is not surprising as the training datasets of VideoLISA are mainly composed of segmentation data. The model has been optimized for reasoning and localization.
• We notice that this is a common limitation among reasoning segmentation models. As shown in Table 2 of the rebuttal PDF, LISA model even shows a much worse performance. Developing segmentation capability while preserving the chat capability is not trivial as it involves various aspects, such as data curation, training strategy, etc. We will discuss this in the revised paper to inspire future work.

Q5: I am curious about the performance of adding a segmentation head to models like LLaMA-VID.

Response: LLaMA-VID compresses each video frame into two tokens, reducing computational cost. We adopt LLaMA-VID architecture and add a segmentation head similar to VideoLISA. We report the experiment results in Table 3 of the rebuttal PDF. To answer the reviewer's question: 1) LLaMA-VID, equipped with a segmentation head, is capable of doing segmentation and achieves decent performance across benchmarks; 2) when being compared with VideoLISA, LLaMA-VID shows worse performance across all evaluated benchmarks. The performance gap comes from that LLaMA-VID compressing the visual tokens into extremely low resolution, i.e., two tokens only. This compression inevitably lost visual spatial details, which are essential for segmentation.

We sincerely hope the rebuttal answers your questions and addresses your concerns.

We thank the reviewer for reviewing our paper.

Q1: The two proposed modules seem relatively simple.

Response:

• We would like to first emphasize that we do not pursue complexity in the model design. Instead, we aim to design a framework that is effective and suited to solve the problem. The underlying rationale of the model design comes from in-depth analysis of the task and extensive experiments. In our VideoLISA, to address the unique challenges in marrying LLM with video object segmentation, we propose two key innovations.
  • Firstly, a Sparse Dense Sampling strategy is designed to enable LLM to capture and understand temporal dynamics in videos. By leveraging the inherent temporal redundancy property of videos, this strategy achieves a delicate balance between preserving visual details and temporal context, making it favorable for video object segmentation tasks. In the evaluation of this approach, we also unveil two important properties of the ReasonVOS task: 1) Sparse temporal tokens can boost the performance by introducing more temporal cues; 2) Preserving dense spatial tokens is essential for segmentation tasks. In contrast, pooling on the spatial dimension will cause performance degradation. Both of the two conclusions are important lessons to the community and can support the rationality of our approach.
  • Secondly, we propose a One-Token-Seg-All approach to achieve temporally consistent segmentation masks in the promptable mask decoding paradigm. Our motivation is that a single unified representation has the potential to segment video frames, as revealed by related works. Thus, a single [TRK] token, serving as an information bottleneck, is tasked to learn a compact object representation. However, this is non-trivial to achieve, as shown by our investigation on existing models, including SAM and LISA. Through our in-depth analysis and experiments, we identify the two key factors: incomplete input information and inappropriate training objectives. We thus enhance the cross-frame association capability of the single token from these two perspectives. We argue that the resultant One-Token-Seg-All approach is simple yet effective by design. The analysis, experimental exploration, and the training objective design make the approach effective and non-trivial.

Q2: The settings of a few ablation studies were not clearly stated.

Response: We thank the reviewer for pointing out the writing flaw. Due to page limit, we present the ablation studies in a concise and compact way in the main paper. In the Appendix, we report more detailed settings and results of the ablation studies. Here we address the two points mentioned by the reviewer.

• N-frame experiment setting. This setting is stated in line 616-622 of the Appendix. In Table 5, the n-frame setting directly concatenates the visual features from n sampled frames as input to the large language model. In our implementation, the value of n is set to the same as T_dense for comparison. This implementation is mainly based on computational cost, as T_sparse can be a relatively large number (e.g., 32 in our implementation).
• How to combine XMem with LISA in Table 6. This setting is stated in line 646-653 of the Appendix. “During inference, LISA outputs the segmentation mask of the first frame based on language instruction. The tracker then tracks the segmented object through the video, yielding segmentation masks for the subsequent frames. Specifically, we adopt the popular XMem model as the tracker, as shown in the second row of the table.”

We sincerely hope the rebuttal answers your questions and addresses your concerns.