SLiMe: Segment Like Me

Clarification on Tables 5 and 9

We apologize for the confusion. We clarify that WAS-attention can be employed either solely during inference or during both optimization and inference. When used solely during inference, it has no impact on the optimization process.

The reference made by the reviewer to Table 5 refers to the scenario where WAS-attention is applied exclusively during inference. On the other hand, Table 9 presents results when WAS-attention is entirely excluded from both optimization and inference steps in the car parts segmentation task. Notably, this exclusion leads to a notable decrease in IoU, dropping from 68.3% to 62.7%.

We ran another experiment with completely removing WAS-attention from both optimization and inference steps on the horse parts segmentation task, where we also see a significant IoU drop (63.3% to 52.4%).

This table provides a clear breakdown of the values for each body part, comparing the results without and with the Weighted Average Score (WAS).

==================

Clarification on prompt/tokens

Please refer to our answer to Reviewer kAQy

Hedlin et al.’s work

We acknowledge that the high level idea of our work might have similarities to Hedlin et al.’s work, but their objectives are completely different from ours: we perform image segmentation while their task is image correspondence. Furthermore, in terms of architectural similarities, the optimization of cross-attention maps has been previously explored for various downstream tasks, as demonstrated in the work by Hedlin et al. However, our current task differs, and adopting their approach for our segmentation task did not yield effective results. Hedlin et al.’s exploration is confined to the optimization of a single text embedding, while our approach allows optimizing multiple text embeddings, enabling the simultaneous segmentation of various parts or objects.

We attribute the significantly enhanced performance of our method to the utilization of WAS-attention which we propose in this work, as demonstrated in Table 9 and also the following results on horse part segmentation which we ran to address reviewer Fjuo’s comment.

===================

Related works [a] and [b]

Thank you for bringing this to our attention. We've noticed that neither of the two works are peer-reviewed. While we appreciate their contributions, it's essential to highlight the distinctions that make them not directly comparable to SLiMe. In summary, Peekaboo requires an object description for each segment, a requirement absent in SLiMe. Additionally, Peekaboo lacks the capability to perform segmentation at the object part level, which is a fundamental feature of SLiMe.

Regarding the second work by Junjiao et al., as we understand it, the method primarily segments images into different parts based on feature similarity, akin to the approach in the Segment Anything paper. This method lacks control over segmentation granularity and cannot perform targeted object/part segmentation. For example, it cannot be directed to segment a specific part of an object. These distinctions emphasize the unique capabilities and advantages of SLiMe in comparison.

=================

The number of methods that are compared against are limited

We have conducted thorough comparisons with three most relevant papers to SLiMe. Notably, these papers also provide comparisons with other existing methods, offering readers valuable insights into our performance relative to those cited works (although we didn't include them in our study as their setups differed from ours). In response to a reviewer's suggestion (please refer to our responses to reviewer Fjuo for details), we further conducted experiments on two additional datasets, providing readers with a broader context for evaluating the performance of our work. We are open to suggestions on specific papers that you believe warrant comparison, and we genuinely welcome any recommendations for meaningful comparisons to be incorporated into the final version of the paper. Thank you

=================

Not including SegDDPM in Table 1

We conducted a comparative analysis of SLiMe and SegDDPM on two segmentation tasks: face and horse part segmentation (Tables 2 and 3). It's worth noting that the SegDDPM paper did not report results for the car part segmentation task (Table 1). Our attempts to run their method on this task revealed a challenge: for SegDDPM to segment cars into their parts, it requires a DDPM trained to generate car images. Unfortunately, the SegDDPM authors have not provided a DDPM checkpoint for car image generation, preventing us from assessing SegDDPM's performance in car part segmentation.

WAS-attention's effect

We would like to clarify that WAS-attention can be employed either solely during inference or during both optimization and inference. When used solely during inference, it has no impact on the optimization process.

The reference made by the reviewer to Table 5 refers to the scenario where WAS-attention is applied exclusively during inference. On the other hand, Table 9 presents results when WAS-attention is entirely excluded from both optimization and inference steps in the car parts segmentation task. Notably, this exclusion leads to a notable decrease in IoU, dropping from 68.3% to 62.7%.

We ran another experiment with completely removing WAS-attention from both optimization and inference steps on the horse parts segmentation task, where we also see a significant IoU drop (63.3% to 52.4%).

This table provides a clear breakdown of the values for each body part, comparing the results without and with the Weighted Average Score (WAS).

=====================

Post-processing e.g., GrabCut

When it comes to post-processing techniques like traditional GrabCut, it's important to note that not only do they introduce additional computations during inference, but they may also necessitate user intervention for refining segmentations in complex cases (refer to Figure 5 in the GrabCut paper). These methods often struggle to generalize in scenarios where there is no distinct color difference between object and background, or in cases with subtle color/texture transitions between different parts of an object. For instance, in face part segmentation, distinguishing noise from the actual face becomes challenging due to their almost identical texture and color.

=====================

Results on ADE-Bedroom-30 and FSS-1000

We would like to emphasize that while the SegGPT model was tested on unseen samples/datasets, its foundational base model is trained on extensive supervised segmentation data—comprising images and corresponding segmentation masks. This means that SegGPT's underlying supervised model may have encountered categories or samples resembling the test data. However, it's important to note that we do not utilize a base supervised segmentation model. Nevertheless, when applied to the FSS-1000 dataset, our method yielded a reasonable IoU:

SLiMe (ours): 73.53 % Painter: 61.7% SegGPT: 85.6%

Painter is cited in the SegGPT paper as a similar technique which also leverages a supervised segmentation base model (as SegGPT does). Notably, our SLiMe outperforms Painter.

For the ADE-Bedroom-30 dataset, we obtained an average mIoU of 32.3 using the same settings as SegDDPM, which is comparable to the 34.6 achieved by segDDPM. We identified a slight underperformance in SLiMe, attributed to its imperfect handling of small and featureless objects, such as man-made items like lamps in the ADE-Bedroom dataset. This limitation arises from the fact that these objects become tiny or even vanish in the cross-attention maps we employ, sized at 16x16 and 32x32, while the input images for this dataset are at a resolution of 256x256. Addressing this issue could be a valuable improvement for SLiMe in future work.

=====================

Interpretability of learned embeddings

We calculated cosine distance between some of the learned text embeddings with our SLiMe and the words/embeddings in the Stable Diffusion dictionary. The closest text tokens in the Stable Diffusion’s dictionary to the learned ones using SLiMe do not necessarily match. For instance, the 10 closest words in the Stable Diffusion dictionary to the optimized embedding of 'bear' segmentation are:

{brasil, nightout, mccall, boudo, confirmation, oldschool, stockholm, regardless, beacon, tik}

which do not correspond to the concept of 'bear.' However, a future work could involve a deeper exploration of the initial and the evolution of the word embeddings during optimization, which may lead to improved segmentation results or more interpretable tokens.

Clarification on prompt, tokens, and embedding

We appreciate the reviewer's insight regarding the potentially confusing tokenization step. To address this concern, we plan to include a new figure similar to this: https://imgur.com/a/0dutj8r that provides a clearer illustration of how each part relates to tokens for granular level segmentation. As illustrated in the figure, each part (p) in the image corresponds to a token (t) in the embedding, which is initially set as a null text (i.e., "") to be later optimized. Consequently, we do not utilize any prompt as input. The inputs to SLiMe include empty token placeholder(s), an image, and a corresponding segmentation mask.

We utilize Stable Diffusion's text tokenizer to specify text tokens (i.e., breaking the sentence into tokens). Since Stable Diffusion is configured to operate with 77 tokens, if the length of the tokenized text is less than 77, the tokenizer pads it with a special token (e.g., end of string: EOS). If its length exceeds 77, the tokenizer crops it to a length of 77. We initialize the tokens with null text (i.e., ""). So they are like placeholders in the embedding.

We hope this clarification resolves any confusion. Please feel free to let us know if any ambiguity persists.