Text4Seg: Reimagining Image Segmentation as Text Generation

• W1: Decoder-free framework.
  As discussed in the general response, our TextSeg does not necessarily rely on the SAM refiner during inference, even though using it may yield better performance. Without SAM decoder, simpler alternatives like Conditional Random Fields (CRF) or $32 \times 32$ semantic descriptors still achieve competitive or superior results compared to existing works. Therefore, we maintain that TextSeg remains a decoder-free framework.

• W2: Results without SAM.
  We reported results with SAM in the main table because, by default, we use the off-the-shelf SAM with ViT-H as our mask refiner. Following your suggestion, we attempted to include the results without SAM across all the main tables. However, doing so required excessive space. To address this, we have instead included these results—covering various MLLMs, semantic descriptor lengths, and tasks—in the appendix (Tables 12~14) for comprehensive reference.

• W3: Improvement points to make paper clearer
  We greatly appreciate the reviewer’s valuable feedback and have addressed the points as follows:

  • The row length in R-RLE.
    By default, the row length in R-RLE is 16, but we used a row length of 6 in Fig. 3 for simplicity. We have clarified this in the section Definition of semantic descriptors in our revision.

  • Add backbone column to main results table.
    As recommended, we have added an LMM column for backbone to the main results tables (Tables 1, 2, and 3).

  • Add references of datasets/benchmarks in tables.
    We have included references for each dataset in the captions of Tables 1, 2, and 3 as suggested.

  • Explain what 'mix' under 'Training Data' of LISA means.
    The term "Mix" under LISA’s training data denotes a blend of different datasets proposed in LISA, including semantic segmentation datasets, referring segmentation datasets, and visual question answering datasets. For more details, please see the "Training Datasets of Continued Pre-Training (CPT)" table or refer to the LISA paper. This clarification has been added to the caption of Table 4 in the revised manuscript.

  • References from the end of page 6 to halfway of page 8 seem to be missing
    Thanks for your suggestion. The omitted references in the main text are already cited in Table 3. To maintain readability, we have avoided duplicating these references.

  • Training scheme was used for the ablation study.
    For the ablation studies on resolution of semantic descriptors and mask refiner with SAM variants, TextSeg was trained using the same combined RES datasets as in the main experiments. For the ablation study of I-RLE v.s. R-RLE, TextSeg was trained on the train split of the refCOCO and refCOCO+ datasets. The refCOCOg dataset was excluded in this case due to its longer referring sentences, which lead to excessively long semantic descriptors when compression techniques are not applied.

  • Typos and incorrect numbers.
    Thank you for pointing out these careless mistakes. We have corrected them in our revision as below:

    Notably, TextSeg with SAM-L improves the average performance on RES tasks from 73.5 to 79.1 cIoU compared to TextSeg without a mask refiner, with only a minimal increase in inference time.

    We have also updated the performance in Table 6 from 79.3 to 79.2 cIoU.

  • Mark best performances in bold in Table 5.
    We have bolded the best performances in Table 5 in the latest revision.

• Q1: Explanation about worse segmentation results of higher resolution.
  There are two primary factors that can affect the segmentation performance.

  • The first is the visual grounding ability, which refers to the model's capacity to follow instructions and to localize the position of objects.
  • The second factor is the shape boundary accuracy of the predicted mask with the groundtruth mask.

  While TextSeg, with higher-resolution semantic descriptors, can produce more fine-grained masks, increasing descriptor length also poses challenges for model training. This may negatively impact the model's visual grounding ability, particularly when compared to a model using a $16\times16$ resolution.

  Since the SAM refiner mainly helps to increase the shape boundary accuracy, it may not be that effective when the visual grounding performance is worse. Therefore, the results of $32\times32$ may be sometimes worse than those of $16\times16$ due to the decrease of visual grounding ability. We anticipate that this limitation could be addressed through further scaling of the model and dataset, enabling improvements in both grounding ability and boundary accuracy.

Thank you for your valuable and constructive feedback that really help improve the quality of our paper. We agree that including results without SAM helps to make our paper clearer. We therefore have implemented the following revisions:

1. Inclusion of Results Without SAM in Main Tables
   To address your suggestion, we have included results without the SAM (using CRF or higher resolution of semantic descriptors) for RES and GRES in Tables 1&2, results without any mask refiner for REC in Table 3. Kindly refer to the latest revision for details.
2. Additional Discussion in Section "Result of Single Object" (Table 1, RES)
   We have added further discussion to highlight the competitiveness of our method without SAM refiner:

   Even without using the SAM refiner, our method remains competitive. For instance, TextSeg_{\textup{InternVL2-8B}}^\dagger, refined with CRF, and TextSeg_{\textup{InternVL2-8B}}$$^\ddagger, based on $32\times32$ semantic descriptors, achieve results that rival or exceed existing methods.

3. Additional Discussion in Section "REC: Results" (Table 3, REC)
   We have also included additional insights regarding REC results:

   It is worth noting that TextSeg_{\textup{LLaVA-1.5-7B}}* and TextSeg_{\textup{LLaVA-1.5-13B}}$$^*, without a mask refiner, outperform their respective baseline counterparts. These results emphasize the superiority of TextSeg in following instructions, leading to enhanced visual grounding ability.

We hope these revisions address your concerns and provide greater clarity to our analysis.

Technical Contributions

Our main contribution lies in introducing the text-as-mask paradigm, driven by novel semantic descriptors, which redefines image segmentation as a text generation problem. The objective of our work is to equip existing MLLMs with segmentation capabilities without necessitating architectural modifications. As such, we intentionally avoid making technical modification to existing MLLMs, as our focus is on harnessing their inherent text generation capabilities to achieve segmentation in a streamlined and efficient manner.

Q1: Training Datasets

Most prior methods follow a two-stage training paradigm: Continued Pre-Training (CPT) using large datasets, followed by Supervised Fine-Tuning (SFT) for specific tasks. The datasets used in these approaches are summarized in the following tables:

• Table R1-1: Datasets for Continued Pre-Training (CPT)
• Table R1-2: Datasets for Supervised Fine-Tuning (SFT) in Referring Expression Segmentation (RES)
• Table R1-3: Datasets for Supervised Fine-Tuning (SFT) in Generalized Referring Expression Segmentation (GRES)

We can note that:

1. For CPT, previous methods rely heavily on large and diverse datasets, whereas our approach, TextSeg, eliminates this requirement, demonstrating superior efficiency and effectiveness.
2. For SFT, we ensure a fair comparison by following previous works and trained on:
   • The train split of refCOCO series for RES and REC tasks.
   • The train split of grefCOCO for the GRES task.

We have also included this discussion in the Appendix, under the section Comparison of training datasets, in our latest revision.

Table R1-1: Training datasets of Continued Pre-Training (CPT).

Table R1-2: Referring Expression Segmentation Datasets of Supervised Fine Tuning (SFT). $^*$Other methods have already incorporated refCLEF dataset in their CPT training datasets.

Table R1-3: Generalized Referring Expression Segmentation Datasets of Supervised Fine Tuning (SFT).

Q2: Use of SAM Mask Refiner

Please refer to the general response.

Q3: Inference Speed

We report the comparison of inference time (second per sample) as well as the image size in Table R1-4. TextSeg achieves an inference time of 5.92 seconds per sample, which is competitive compared to PixelLM and NEXT-Chat. Although slower than GSVA (0.42 seconds), this is due to GSVA processing smaller image sizes ($224^2$).

Table R1-4: Inference time comparison.

Q4: Missing Recent Specialized Segmentation Models

Thank you for your suggestion. We have included these models in Table 1 and Table 3 in our latest revision to ensure a more comprehensive comparison, as well as in Table R1-5 and Table R1-6 below. Notably, our TextSeg still demonstrates competitive performance compared to these specialized segmentation models.

Table R1-5: Referring Expression Segmentation results (cIoU) on RefCOCO (+/g) datasets.

Table R1-6: Referring Expression Comprehension results (Acc@0.5) on RefCOCO (+/g) datasets.

Thank you for your detailed feedback and for sharing your concerns. We deeply appreciate the opportunity to address them in further detail below.

Q2: We can clearly see that without any mask refiner (including SAM or CRF), the performance dropped significantly and just on the par or in some cases, lower than previous methods.

Without any mask refiner, as a lower bound, our TextSeg ($32^2$) achieves 71.4 cIoU (Table 1) on average on the RES tasks, outperforming most existing methods, including NEXT-Chat (68.9), LISA (69.9), PixelLM (69.2), LaSagnA (71.1), and equaling GSVA (71.4). Additionally, it achieves an average score of 67.6 (Table 2) on the GRES tasks, higher than all comparisons, including LISA (62.9) and GSVA (65.6). Lastly, on REC tasks, it achieves an average score of 85.0 (Table 3) , better than most comparisons, including Shikra (82.9), Ferret (83.9), LISA (79.8), GSVA (80.3), NEXT-Chat (80.4), which indicates a better visual grounding ability of TextSeg and further explains its superiority in RES and GRES (require strong visual grounding and shape boundary accuracy). Not to mention, it is entirely fair to compare our TextSeg with a mask refiner to enhance shape boundary accuracy against these existing methods.

Q3: if only considering the methods with the same image size, the proposed method is a bit better than NEXT-Chat, but notably slower than LISA, and I believe that the same thing will happen to PixelLM if we consider the same image size.

We have not claimed in our manuscript that TextSeg offers faster inference than other methods. In fact, it achieves comparable efficiency to several existing approaches. Additionally, PixelLM’s design mandates an image size of 448; reducing the size will degrades its performance.

Q4: If we consider full version of the proposed method, they outperform the new suggested methods and the mentioned methods in the initial manuscript. However, if we consider the non mask refiner variant of the proposed method (for a fair comparison as other methods don't have post-processing steps), the proposed method is just on par or lower than previous methods.

All the specialized segmentation models and generalized segmentation models in comparison inherently include a decoder in their architectures, which will be optimized during training. This is why additional post-processing is neither necessary nor effective for them. We question the rationale for restricting our method from utilizing a simple off-the-shelf mask refiner, merely because previous methods do not require it. If we were to exclude the SAM decoder and other post-processing steps from our method, then for a fair comparison, should the decoders in all other methods also be removed during evaluation?

We find it difficult to understand why it is necessary for the lower bound (non-mask refiner variant) of our method to outperform the optimal performance (with decoder) of all other methods. We believe this was not claimed in our manuscript.

• Rely on SAM to get the final pixel level prediction.
  Please refer to the general response.

• Compare with other VLLM approach also with SAM refinement.
  Please refer to the general response.

• Extend to instance level segmentation.
  Referring expression segmentation models inherently perform segmentation at the instance level, as they are capable of isolating a specific instance from multiple objects. For example, as shown in Figure 9 of our paper, a prompt like tallest young giraffe identifies and segments the tallest giraffe among four giraffes. Similarly, a prompt like green boy not kicking focuses on the individual in a green t-shirt who is not kicking, out of six people in the image. Furthermore, our method can be easily adapted for traditional instance segmentation tasks. For example, a user query might be: Could you segment all the people in the image and label each individual separately? The corresponding response could be structured as: <seg> others, ..., person 1, ..., person 2, ..., person 3, ..., others </seg>. In this manner, our method can be extended to traditional instance segmentation. Semantic descriptors can be constructed directly from existing instance segmentation datasets, making our approach versatile and easily adaptable to various instance-level segmentation tasks.

• The name of TextSeg.
  We have revised the name of our method from TextSeg to Text4Seg in our latest revision. However, for the purpose of this rebuttal, we continue to use the name TextSeg to maintain clarity and avoid confusion for the reviewers.

Dear Reviewer SUs4,

Thank you for your valuable feedback on our submission. We have addressed the concerns raised in your initial review, and we would like to confirm if there are any additional questions or points that require further clarification from our side.

If our responses have satisfactorily addressed your concerns, we kindly request your reconsideration of the overall assessment. We greatly appreciate your time and effort in reviewing our work and your contributions to improving its quality.