RESAnything: Attribute Prompting for Arbitrary Referring Segmentation

Dear reviewer tcpr, thank you for your encouraging feedback and recognition of our training-free approach's substantial zero-shot performance gains, strong generalization capabilities, and comprehensive experimental validation. We particularly appreciate your thorough review of our supplementary materials. Please find our responses to the identified weaknesses below.

Robustness and usage of foundation models

We acknowledge these limitations regarding foundation model dependency, prompt sensitivity, and runtime overhead. Our ablation studies demonstrate robust performance across diverse MLLMs (Pixtral 12B, Claude 3.5, and Qwen 2) using consistent prompts, and we provide comprehensive prompt analysis in the supplementary materials to verify the effectiveness of attribute-based prompting. More importantly, our focus is on solving referring expression challenges that previous methods fundamentally cannot handle. While current powerful MLLMs have extensively learned concepts from standard benchmarks like RefCOCO series through their training data, the critical research challenge lies in handling specialized, domain-specific and explicit, part-level queries in ReasonSeg and proposed ABO-ARES benchmarks. We understand that future foundational MLLMs may eventually be able to handle these queries through scaling, our work offers a viable pathway to obtain such annotated data for training and explores the possibility of learning rich properties, attributes, and other sparse information about target regions of queries in images without requiring extensive fine-tuning.

Our current method has not been optimized for inference speed. With parallelization on candidate mask generation from SAM, reference text generation, candidate text generation and similarity comparison between the target and candidates, the RESAnything time can be reduced to 3s (on multiple 4090 GPUs), as shown below: max(SAM, reference text generation) ($\approx$2s) + candidate text generation ($\approx$0.5s) + max(MLLM similarity computation, CLIP similarity computation) ($\approx$0.5s)

This optimization leverages the fact that candidate mask generation and reference text generation can be executed in parallel, and similarity computations using MLLM and CLIP can also run concurrently. Furthermore, inference speed could be improved even more by replacing SAM with FastSAM for mask generation.

Future works

Thank you for these insightful suggestions for future research directions. We agree that investigating proposal generation mechanisms when current SAM proposals are insufficient represents a valuable extension, as this directly addresses a key bottleneck in our approach. Our grouping and selection algorithm handles targets consisting of multiple proposals, though we acknowledge this mechanism is not always reliable. As shown in Section 10 of our supplementary materials, we included failure cases that are primarily attributed to poor mask proposals from SAM. This analysis confirms that proposal quality represents the primary bottleneck of our method, making your suggested research direction particularly relevant for advancing training-free referring expression segmentation.

Dear reviewer R2q5, thank you for your positive feedback and recognition of our ingenious integration of different tools to achieve training-free performance comparable to task-specific models. We appreciate your acknowledgment of our ABO-Image-ARES benchmark and novel attribute prompting paradigm that extends MLLMs' applicability to broader grounding tasks. We address your concerns regarding the identified weaknesses below.

REC capabilities of current MLLM and RESAnything

We appreciate the reviewer's thoughtful observation about the relationship between REC and RES capabilities. While we acknowledge that some MLLMs like Qwen2.5-VL demonstrate strong performance on RefCOCO datasets, we respectfully note several important considerations. First, RefCOCO series, being published 9 years ago, represents a relatively saturated benchmark where both supervised methods and foundation models have been extensively optimized, making strong performance on this dataset less indicative of genuine REC capabilities for novel, complex expressions. As demonstrated in Section 6.1 and Figure 9 of our supplementary materials, when MLLMs are directly asked to provide comprehensive descriptions of target queries in images, they generally fail to give accurate responses, suggesting that they may not have robust REC understanding. More importantly, our method addresses precisely that gap that existing approaches, including well-performing MLLs, cannot handle complex referring expressions involving reasoning and explicit attributes (See the quantitative result comparisons on ReasonSeg and ABO-ARES). These scenarios are absent from traditional benchmarks like RefCOCO series. While the interplay between REC and RES capabilities is indeed worthy of investigation, our focus is on extending both capabilities to challenging scenarios that represent genuine research frontiers rather than optimizing performance on well-established benchmarks. We believe this distinction is crucial for advancing the field toward handling real-world complexity in referring expression tasks.

Ablation on MLLMs and attributes

We have indeed conducted comprehensive ablation studies across MLLMs with varying capabilities and scales, as shown in Table 5 of our paper. Our evaluation includes Claude 3.5 Sonnet (a closed-source commercial model with strong reasoning capabilities), Pixtral 12B (a relatively large open-source model), and Qwen2-VL 7B (a general-scale model), representing a diverse spectrum of MLLM understanding and grounding performance levels.

To validate the effectiveness of attribute prompting, we conducted ablation studies showing significant advantages over conventional prompting on ReasonSeg:

We show details of standard prompt, attribute prompt, and corresponding generated descriptions in supplementary materials, Section 6.1 and Figure 9. Both quantitative and qualitative results highlight the contribution of our attribute prompting mechanism.

Usage of CoT

We appreciate the clarification regarding Chain-of-Thought implementation. To better contextualize our approach, RESAnything employs a step-by-step decomposition strategy that breaks down complex referring expression segmentation into manageable stages: (1) attribute decomposition of the referring expression, (2) mask proposal generation, (3) multi-modal similarity assessment, and (4) optimal selection and grouping. This systematic approach transforms the complex task into interpretable intermediate steps, enabling our method to handle sophisticated referring expressions that challenge end-to-end approaches. Our attribute prompting guides the MLLM through structured analysis at each stage, facilitating robust understanding of spatial and semantic relationships before reaching the final segmentation decision. We are happy to reword this if the reviewer find that this is confusing.

Others

We will update the quantitative comparison with the latest model in the camera ready version, and correct any formatting errors.

Dear reviewer DVej, thank you for your detailed review and insightful questions! We appreciate your recognition that the proposed zero-shot method is broadly applicable across different domains with support for a wide spectrum of expressions. We are also encouraged by your positive assessment of our proposed benchmark and the outstanding performance demonstrated by our method. Regarding the weaknesses and questions, please find our responses below.

Novelty

While our method integrates existing foundation models, the key contribution lies in the novel attribute prompting framework that enables systematic reasoning about part-level attributes and implicit queries—capabilities that current methods cannot achieve. Our novelty is not prompt engineering, but the systematic multi-stage reasoning framework that decomposes complex spatial and semantic relationships into interpretable steps, addressing the "last mile" challenge in RES. To validate the effectiveness of attribute prompting, we conducted ablation studies showing significant advantages over conventional prompting on ReasonSeg:

We show details of standard prompt, attribute prompt, and corresponding generated descriptions in supplementary materials, Section 6.1 and Figure 9. In addition, we also contribute a challenging new dataset whose characteristics deviate significantly from RefCoco, a dataset most prior methods may have overfit towards. Our new dataset is also catered to real-world application scenarios, e.g., eCommerce.

Affordance-based or action-grounded expression involving human pose

Although we did not explicitly evaluate on any affordance-based or action-grounded benchmarks such as TOIST and the mentioned ICLR paper, both ReasonSeg and our proposed ABO-ARES benchmark contain substantial affordance-based and action-grounded queries that demonstrate our method's capability in this domain. Specifically, ReasonSeg includes expressions such as "people cross over obstacles," "object used by people to sit down," "where people wash their hands," and "people catch," which directly involve human-object interactions and spatial affordances. Similarly, ABO-ARES contains functionality-oriented queries like "easy pouring," (in Figure 8) "foldable design," and "cushion support" that explicitly reference object affordances and their intended human interactions. The superior performance of our method on both datasets demonstrates effective handling of these scenarios, validating our approach's applicability to affordance-based reasoning. Our training-free framework can be naturally extended to affordance benchmarks by incorporating embodied attributes such as graspability, stability or manipulation properties into the attribute prompting mechanism, allowing seamless adaptation to action-grounded scenarios without further fine-tuning. We will include these related papers in the final version of the paper.

Efficiency and cost

Our current inference speed (12.1s) is not significantly behind the fastest alternative CaR (about 5.3s), which suffers from low accuracy (35.2% vs. ours 74.6% on ReasonSeg). There is an obvious tradeoff between speed and accuracy. A gap of 6.8s is not significant enough to warrant using a more error-prone method. Ultimately, the errors must be corrected in an application and the time it would take to correct a segmentation mask most likely far exceeds 6.8s on average.

Finally, our current method has not been optimized for inference speed. With parallelization on candidate mask generation from SAM, reference text generation, candidate text generation and similarity comparison between the target and candidates, the RESAnything time can be reduced to 3s (on multiple 4090 GPUs), as shown below: max(SAM, reference text generation) ($\approx$2s) + candidate text generation ($\approx$0.5s) + max(MLLM similarity computation, CLIP similarity computation) ($\approx$0.5s)

This optimization leverages the fact that candidate mask generation and reference text generation can be executed in parallel, and similarity computations using MLLM and CLIP can also run concurrently. Furthermore, inference speed could be improved even more by replacing SAM with FastSAM for mask generation.

Handling special queries

“The black chair” is a good example. This actually reflects a general misunderstanding of referring expressions in this task. In linguistics, a referring expression is any noun phrase whose function in discourse is to identify some individual object. When an expression like "the black chair" is ambiguous and refers to multiple valid targets, both objects can legitimately be segmented, as the ambiguity lies in the expression itself rather than representing a failure case. However, for referring expressions that intentionally target multiple objects or parts (e.g., "standing people", "all people", or “all persons except the kid in white”), our proposed grouping and selection mechanism effectively handles these scenarios. We demonstrate this capability through evaluation on the GRES benchmark containing such expressions in our supplementary materials Section 9.3. which specifically includes such multi-target cases and validates our method's robust performance on these linguistically complex expressions. Figure 16 in the supplementary materials shows several multi-part target examples and specifically, if we change the expression from 'lady middle pink' to 'all players' correctly segments both male players in the image, and similarly, segment 'two kids' instead of 'the mom' appropriately selects both kids. Our method's flexibility allows for different grouping settings and configurations, and we acknowledge that future work can further optimize the grouping mechanism to enhance multi-object performance.

Thank you for your feedback!

Let us first present quantitative results on affordance-based RES tasks as requested. Then we shall come back to your question on clarification of systematic prompting framework, since our answer to this question refers to the quantitative results.

We have conducted evaluation on the COCO-Tasks benchmark to validate our method's performance on affordance-based and action-grounded scenarios.

All compared methods are fully-supervised and fine-tuned on COCO-Tasks training data, while RESAnything operates in a zero-shot manner. As seen above, our method surpasses TOIST and TaskCLIP, demonstrating its strong generalization capability across affordance-based scenarios.

Why our framework goes beyond prompt design

Our core contribution is the attribute prompting and the comprehensive pipeline designed around it. This goes beyond simply designing prompts $q^{ref}$ and $q^{can}$ to encompass a complete multimodal framework for multi-metric mask proposal selection (Section 3.2), including comparison between reference and candidates (text-text similarity, text-mask similarity, grouping and selection). We refer to this integrated pipeline as our systematic prompting framework, which provides a generalizable architecture that operates consistently across diverse referring expression types, from simple object queries to complex part-level and affordance-based scenarios. Our ablation studies between standard prompts and attribute prompts demonstrate that the framework itself is the critical component. In addition, this framework is orthogonal to prompt design: when we incorporate task-specific prompt optimization, e.g., adding “human-object interaction” attributes, the performance improves from 51.2 to 54.6, approaching CoTDet. This demonstrates how prompt engineering can enhance the performance of our framework.

We are happy to refine our wording to better reflect our scope and contribution.

[1] Sawatzky, J., Souri, Y., Grund, C., & Gall, J. (2019). What object should i use?-task driven object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7605-7614).

[2] Chen, H., Huang, W., Ni, Y., Yun, S., Liu, Y., Wen, F., ... & Imani, M. (2024, September). Taskclip: Extend large vision-language model for task oriented object detection. In European Conference on Computer Vision (pp. 401-418). Cham: Springer Nature Switzerland.

[3] Tang, J., Zheng, G., Yu, J., & Yang, S. (2023). Cotdet: Affordance knowledge prompting for task driven object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3068-3078).

Dear reviewer T2sg, thank you for your thorough review and helpful suggestions! We appreciate you mentioning that our paper is well-written, the proposed training-free method saves the annotation cost for this task, and the experiments show promising results. We address your concerns regarding the identified weaknesses below.

Novelty, Robustness and Generalizability

For most problems covering diverse test cases, the biggest challenge is tackling the “last mile,” and RES is no exception. Our method is designed for, and excels at, tackling such a challenge under complex cases involving implicit queries and complex part-level relations. To achieve this, we propose the novel idea of attributing prompting, with the emphasis on reasoning about part attributes. Combining SAM and MLLM is not our novel claim, attribute prompting is! To further validate its effectiveness, we conducted an ablation to show advantages over conventional prompting: quantitative results on ReasonSeg test set with different prompts highlight the contribution of our attribute prompting mechanism:

We show details of standard prompt, attribute prompt, and corresponding generated descriptions in supplementary materials, Section 6.1 and Figure 9. In addition, we also contribute a challenging new dataset whose characteristics deviate significantly from RefCoco, a dataset most prior methods may have overfit towards. Our new dataset is also catered to real-world application scenarios, e.g., eCommerce.

Regarding robustness and generalizability, Table 5 demonstrates consistent performance across diverse MLLM backbones (Pixtral 12B, Claude 3.5, Qwen2-VL), validating cross-model reliability. While prompt sensitivity exists, this characteristic actually provides valuable flexibility rather than a limitation: users can adapt the attribute specifications within prompts to optimize performance for specific domains or categories when building unlabeled datasets / in applications. In addition, our training-free approach allows seamless integration of newer, more powerful MLLMs as they become available, providing immediate performance improvements without retraining.

For multi-object scenarios, our selection mechanism employs comprehensive decision-making analyzing both text-text and text-image similarity rather than single CLIP scores (detailed in supplementary Section 6.2). Section 6.2 and Table 6 in the supplementary further show ablation studies on the effectiveness of this multi-metric mechanism to improve robustness. In addition, we extensively evaluated multi-object performance on the GRES task (target containing multiple objects), as shown in the supplementary materials, Section 9.3. The proposed grouping method actually demonstrates promising results among pretrained/zero-shot approaches, empirically confirming robust handling of multi-object scenarios. Figure 16 in the supplementary materials illustrates our capability for grouping and combining multiple parts of the same object. Additionally, our method successfully adapts to different expression granularities. For example, changing the expression from 'lady middle pink' to 'all players' correctly segments both male players in the image, and similarly, segmenting 'two kids' instead of 'the mom' appropriately selects both kids. Our method's flexibility allows for different grouping settings and configurations, and we acknowledge that future work can further optimize the grouping mechanism to enhance multi-object performance.

Performance gap with supervised methods

Regarding the performance gap with supervised methods, we acknowledge this limitation on the RefCOCO series, which represents a fundamental and well-established benchmark in the field. We recognize the continued importance of RefCOCO for evaluating core RES capabilities. However, we would like to highlight that our zero-shot approach is specifically designed to excel in scenarios where supervised methods face limitations. In particular, when labeled training data is scarce or unavailable, or dealing with complex referring expressions that extend beyond traditional benchmark coverage. While RefCOCO provides essential evaluation for common referring expressions involving humans and daily objects, our method demonstrates particular strength in handling sophisticated expressions involving explicit attributes, part-level descriptions, materials, and functionality. These areas represent emerging challenges in real-world applications. Notably, our zero-shot approach achieves state-of-the-art performance on ReasonSeg and our proposed ABO-ARES benchmark, demonstrating superior performance on the sophisticated referring expressions that represent genuine research challenges beyond the scope of traditional benchmarks. We believe the value of our contribution lies in providing an effective solution for domains and applications where extensive supervised training data is unavailable, while simultaneously advancing the field's capability to handle the increasingly complex referring expressions encountered in real-world scenarios beyond traditional benchmark settings.

Efficiency

Our current inference speed (12.1s) is not significantly behind the fastest alternative CaR (about 5.3s), which suffers from low accuracy (35.2% vs. ours 74.6% on ReasonSeg). There is an obvious tradeoff between speed and accuracy. A gap of 6.8s is not significant enough to warrant using a more error-prone method. Ultimately, the errors must be corrected in an application and the time it would take to correct a segmentation mask most likely far exceeds 6.8s on average.

Finally, our current method has not been optimized for inference speed. With parallelization on candidate mask generation from SAM, reference text generation, candidate text generation and similarity comparison between the target and candidates, the RESAnything time can be reduced to $\approx$3s (on multiple 4090 GPUs), as shown below: max(SAM, reference text generation) ($\approx$2s) + candidate text generation ($\approx$0.5s) + max(MLLM similarity computation, CLIP similarity computation) ($\approx$0.5s).

This optimization leverages the fact that candidate mask generation and reference text generation can be executed in parallel, and similarity computations using MLLM and CLIP can also run concurrently. Furthermore, inference speed could be improved even more by replacing SAM with FastSAM for mask generation.

Others

Regarding the mask filtering, we set a minimum mask region area threshold of 0.4% of the total image area to remove tiny boundary masks commonly produced by SAM. In addition, we apply a prediction confidence threshold of 0.9 to filter out low-confidence duplicate part segmentation masks, as SAM typically generates masks for both whole objects and their constituent parts with varying confidence levels.