Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation

Q1: "Exact model to generate segmentations"

R1: We use the official implementation and pretrained checkpoint from ReLA. The final segmentation mask is produced by the pixel decoder whose outputs are weighted by a language-supervised object activation map.

Q2: "In plot, is IoU calculated against the ground-truth?"

R2: Yes.

Q3: "Each dot means?"

R3: Yes, each dot corresponds to a single image paired with multiple descriptions. The mean and standard deviation are calculated along the caption dimension.

Q4: "Use different colour"

R4: We will revise the color coding in the figures.

Q5: "IoU inside brackets?"

R5: Yes it is IoU against GT.

Q6: "how to generate mutated expressions?"

R6: We generate the mutated expressions by querying ChatGPT4o, with a template "Genrate a list of n synonyms of the noun phrases in the following [sentence] and output the list separated by '&'", where n is in randint(2, 5) and [sentence] is set to text referring expression.

Q7:"Figure 3 plot add your model?"

R7: Please find link3 and link4 in our reply R10 to reviewer 4Tbz.

Q8: "Table with average std"

R8: We provide the table with average std below:

Q9: "the average std should drop for your method?"

R9: That's correct. We focus on test-time prompt embedding alignment. We show that out method is simple but very effective to reduce the sensitivity of input variance.

Q10: "Slow inference speed"

R10: Thank you for your feedback. Please refer our reply to reviewer jmV5.

Q11: "inference time comparison"

R11: We present a Table comparing the inference time with related baselines, including CLIPasRNN, Peekaboo.

All implementations were evaluated on a single NVIDIA A100 40GB GPU. As previously discussed, test-time optimization methods inherently involve a trade-off between inference speed and mask quality. Nevertheless, we demonstrate that our method can be significantly accelerated—reducing inference time from 470s to 28s—by fine-tuning the text encoder on a small number of target-domain samples and decreasing the number of inference steps from 1100 to just 50, with minimal performance degradation.

Q12: "Training-free overclaim?"

R12: Please refer our reply R1, R2 and R3 to reviwer 4Tbz.

Q13: "Concerns on expression parsing"

R13: Please refer our reply R4 and R5 to reviewer ixNi

Q14: "baseline + additional info"

R14: We observe that certain baseline methods, such as GLamm, LISA, and OMG-LLaVA, already incorporate similar linguistic information into their input pipelines. For instance, by integrating complete noun-phrase during feature fusion, or translating a special [SEG] token with context into segmentation masks. These models achieve impressive results but require significant training efforts.

Other baseline methods focused on test-time optimization, such as CLIPasRNN, OVDiff, and Peekaboo, could theoretically leverage this linguistic information as auxiliary input; however, empirical observations suggest that their performance remains suboptimal under these conditions.

In contrast, our proposed method explicitly formalizes this linguistic knowledge as prompt regularization. This strategy enables robust mining of noise-tolerant mask prompts and consequently yields refined and higher-quality segmentation masks, even without extensive training or configuration overhead.

Q15: "Discussing related works."

R15: We thank the reviewer for highlighting these important works, which also aim to leverage intermediate features from large vision-language models (VLMs) for downstream segmentation tasks.

Both CLIPpy and PACL assign labels by contrasting image patch embeddings with object text label embeddings. However, these methods still require training configurations.

Peekaboo is closely related to ours, as both aim to learn visual concepts using off-the-shelf diffusion models. However, Peekaboo heavily relies on alpha map initialization and is sensitive to its quality.

We will add this discussion to our manuscript.

We sincerely thank the reviewer for their detailed feedback. We are glad that the reviewer found our paper to present clear claims with evidence, supported critiques. And our method is computationally lightweight.

Q1: "training-free warrants scrutiny"

R1: The term “training-free” refers to the fact that our method does not require access to or processing of the training dataset. In contrast to prior approaches that demand significant computational resources to train or fine-tune on curated datasets, our method avoids such data-dependent training procedures.

Instead, we perform at test-time, which is more suitable for real-world, open-set scenarios—where test samples often involve novel concepts that has not been seen during training. In such cases, updating prompt embeddings during inference is necessary to align them with the target objects effectively.

To avoid confusion, we will revise our terminology throughout the manuscript, replacing “training-free” with “test-time optimization.”

Q2: "fine-tune the BERT encoder increase model complexity."

R2: We demonstrate that it is possible to reduce the number of diffusion steps during test time textual embedding update from 1100 to as few as 50—by adapting the text encoder using a small number of samples, without accessing the full training dataset. This lightweight adaptation, achieved via LoRA, is intended purely for accelerating test-time optimization, not for conventional training or fine-tuning on a full dataset.

Importantly, this step does not involve large-scale training or supervision and serves as a practical means to align the text encoder with the target domain efficiently. We emphasize that this adaptation remains consistent with our goal of avoiding full model training and supports fast, sample-efficient test-time optimization.

We will add this discussion to our manuscript.

Q3: "avoid overstating claim."

R3: We are sorry for the confusion terms and we will replace with "test-time optimization" for clarification.

Q4: "Proposed method's impact on performance."

R4: We thank the reviwer for raising this issue. We presnet additional experiments of how different prompt influence the downstream mask generation. We compare our Segment Anyword's mask prompt against pure text input, using official implementation of LanguageSAM. We present both quantitive and qualitative results link1 on GranDf validation set. Results show that our method is very effective on improving mask prompt quality.

We will add this additional experiment to our manuscript.

Q5: "Discuss with CoOp and CoCoOp"

R5: We thank the reviewer for bringing up CoOp and CoCoOp, two important works in visual prompt learning. We agree with reviewer's comments and will include the discussion in our manuscript.

Q6: "difference from MCPL"

R6: MCPL serves primarily as the inversion backbone in our method. However, our approach is not limited to MCPL and can be seamlessly integrated with other inversion or concept-discovery methods as well. We present a qualitative results with different cross-attention source, showing that our method is composible with various diffusion models and inversion algorithms link2. We will add this discussion to our manuscript.

Q7: "green, blue, and red represent?"

R7: These colors illustrates easy/medium/hard samples for achieve accurate and stable segmentation, categorized by its IoU mean and std.

Q8: "PL, R1, and R2 represent?"

R8: PL stands for prompt learning. R1 and R2 are two linguistic guided prompt regularizations.

We will revise the figure and table captions to enhance clarity and avoid ambiguity.

Q9: Typos

R9: We will revise our typos throughout the paper.

Q10: "novelty of this paper"

R10: As demonstrated through comprehensive experiments and additional results provided in link3 and link4, our method effectively improves both accuracy and stability by leveraging an off-the-shelf diffusion model to extract mask prompts via an inversion process. This approach is modular and plugable, offering several advantages—including the ability to handle novel visual and linguistic concepts—without requiring additional supervision or complex training procedures.

We thank the reviewer for their thoughtful questions and detailed feedback, which have significantly helped improve the quality of our manuscript. Below, we address the reviewer’s concerns point by point.

Q1: "Definition on "Word-Grounding" and "Novel-Concept""

R1: We refer "word-grounding" to the model’s ability to associate or align words from a sentence prompt with specific object regions in an image. We refer "novel-concept" to that the model can recognize or handle new concepts it has not encountered during training.

We will make amendements on Table caption.

Q2: "Additional motivation study."

R2: We thank the reviewer for raising this question. We present additional motivational study with another state-of-the-art model ETRIS (ICCV23) (link1), which focus on aligning representation from pre-trained visual and language encoder by adding intermediate fine-tuned adapters. The result also validates our previous findings that without test-time alignment, current open-set segmentation models suffer from input variance, resulting in unstable segmentation results.

We will extend this discussion to motivation section in our manuscript.

Q3: "why baseline methods failed with certain conditions."

R3: Both GSVA and SAM4MLLM generate multiple object masks within a single binary segmentation map, but these masks are not explicitly associated with specific word indices. Additionally, neither model is capable of handling novel concepts, as both require training or fine-tuning on the target dataset. This limits their ability to generalize to unseen concepts during test time.

We will add this extended discussion to our manuscript.

Q4: "Text parsing relies on external model"

R4: The primary focus of the proposed Segment Anyword is to improve the quality of automatic mask prompts without relying on complex training configurations (Page 2 Contribution 2). The external language model is merely used to parse and index object-related words for cross-attention map retrieval. Importantly, the use of a fine-tuned LLM (e.g., Vicuna) is not required—our method is compatible with state-of-the-art language models such as GPT-4o, which can provide strong reasoning capabilities out of the box.

Additionally, standard NLP libraries such as NLTK and SpaCy can be used for text pre-processing as well. These tools are widely adopted in prior work, known to be fast and reliable.

We will add this discussion to our manuscript.

Q5: "Text could be noisy"

R5: We acknowledge that noisy inputs, including sentences with incorrect grammar, may occur in the test dataset. However, addressing noisy text parsing is not the focus of our work. Instead, our goal is to generate mask prompts that are robust to such noise, without relying on supervised training. We show an example that our method can still handle typos such as "catstatue" and "kittytoi" in link2

In extreme cases, each word’s cross-attention mask can be matched directly against the ground-truth segmentation masks to determine the best ranking and pairing, thereby compensating for parsing inaccuracies.

We will add these additional results.

Q6: "Fairness on GranDf comparison"

R6: We thank the reviewer for raising this issue. To address it, we conducted an additional experiment using GLamm-generated captions as the textual input for our method, we report both quantitative and qualitative results link3

In general, since GLamm-generated text is not always accurate, it can affect the alignment of prompt embeddings in our method. Nevertheless, our approach remains competitive, as it is capable of refining and adapting the prompt embeddings—even from noisy or imprecise text—to better match the target object during test-time optimization.

We will update the experiment and result.

Q7: "In the test-time cross-attention collection step, which layer(s) of the denoising UNet are considered when collecting the cross-attention values?"

R7: We use the cross-attention maps at the 16×16 resolution, averaged across all denoising time steps to obtain the final cross-attention. This setup is inherited from prior works such as MCPL and Prompt2Prompt to ensure a fair and consistent implementation for textual embedding update. We will add this to our implementation section.

We sincerely thank the reviewer for their valuable questions and insights. We are pleased that they find our proposed method novel, resource-efficient, and highly generalizable.

We have made several amendments and addressed the reviewer's specific queries as detailed below:

Q1: "Inference time cost / Slow speed.""

We appreciate the reviewer highlighting the concern regarding test-time computational costs. We acknowledge there is indeed a trade-off in test-time optimization speed and mask prompt quality. Specifically:

1. Previous methods typically require extensive resources and effort to train or fine-tune large vision-language models on training datasets. This approach is resource-intensive with limited generalization capabilities.
2. In contrast, our proposed test-time prompt optimization method is more efficient and better suited to real-world open-set applications, where test samples may contain novel linguistic and visual concepts. In such scenarios, textual embeddings need to be dynamically updated and aligned with the target object during testing. Consequently, this necessary trade-off between inference speed and embedding alignment cannot be entirely avoided.
3. However, we have demonstrated that it is possible to substantially reduce the number of test-time steps from 1100 to 50 by adapting the text encoder to the target dataset domain fine-tuned by only a samll number of samples. Furthermore, future work could focus on additional engineering optimizations, such as replacing the current backbone with HyperDiffusion, to further enhance inference speed.