DeiSAM: Segment Anything with Deictic Prompting

We thank the reviewer for the fruitful feedback and for acknowledging that the paper is well-written and that combining first-order logic with prompting is interesting.

We would like to address concerns next.

Motivation is not new. Performing reasoning segmentation via LLM is not new. There several previous works exploring the same settings [1]-[5].

We disagree with this. Our claim is to use something other than LLMs for reasoning, as they often hallucinate on complex prompts. Instead, we propose using logic for reasoning. We integrate LLMs and SAM with differentiable reasoners for neuro-symbolic inference on various deictic prompts. To the best of our knowledge, this approach has not been attempted in previous studies.

Most importantly, we demonstrated that LISA [1] significantly degrades its performance on abstract prompts in Tables 3 and 4, indicating a divergence from our problem setting. Our focus does not encompass referring expressions or commonsense reasoning, which differ from high-level abstract reasoning.

The difference lies on the author adopt more visual cues (scene graph) into LLM before sending the language features to SAM.

Not quite. There is a factual error: we do not feed scene graphs to LLMs, nor do we utilize LLMs directly for reasoning. In DeiSAM, LLMs generate logic programs to reason with scene graphs on the abstract level.

The technical novelties are limited. The proposed method just combines previous differentiable logic reasoners with LISA-like segmenter (LLM + SAM) architecture.

As Reviewer YJtK pointed out, our major findings are as follows:

• Existing strong baselines using transformers for reasoning perform poorly on the segmentation tasks with abstract complex prompts.
• This indicates that the current transformer-baed models are limited in high-level abstract visual reasoning.
• DeiSAM is the first framework to successfully extend segmentation models using differentiable reasoning with scene graphs to gain reasoning ability over abstract representations.

I find no extra insights or advantages for end-to-end reasoning learning.

We address the trade-off between logic-based reasoning and model adaptability: while logic provides faithful reasoning capabilities, it forces the model to be deterministic and unlearnable. For instance, employing off-the-shelf logic libraries such as Prolog ensures reasoning but impedes gradient-based learning due to their non-differentiable inference. The key insight of our approach is that it enables the segmentation model to function as both a faithful reasoner and an adaptive learner, as demonstrated in Table 5.

Moreover, the method needs extra scene graphs as inputs, which introduce more complexities to the pipeline.

Not quite. There is indeed a trade-off between the complexity of the input and the reasoning models employed. Scene-graph representations make the reasoning module highly parameter-efficient. For instance, in DeiSAM, the forward reasoner with $N$ rules requires only $N$ parameters, with one weight per rule. In contrast, purely neural models typically require a large number of parameters in the reasoning model (e.g., experiments in [5] employ LLaVA-v1.5-7B as the LLM backbone) to perform reasoning on pixels.

4, Performance and benchmarks. There are only refcoco+ and proposed benchmark results. Several important benchmarks are missing (Please refer to [1]).

In our manuscript, on lines 90-95, we explicitly discuss the reasons for not conducting experiments on the benchmark [1]. The suggested benchmark [1] primarily focuses on low-level commonsense reasoning, whereas our objective is to address high-level abstract reasoning segmentation. This rationale also extends to other referring-expression segmentation benchmarks.

Moreover, the performance is not strong. This result is misleading since the authors miss lots of previous works for fair comparison.

We respectfully disagree with the suggestion that additional baselines are necessary to validate our experiments. We consistently used the LISA [1] model, a state-of-the-art segmentation method, as the baseline for our experiments. None of the works suggested by the reviewer specifically address high-level abstract reasoning. Therefore, we believe that incorporating these additional baselines is not essential to validate our experimental results.

Missing detailed ablation studies for each component. It is hard to know the real benefits on the combined logical reasoner and effects of input scene graph.

We refer to the general remark. To answer this, we conducted additional experiments on abstract visual scenes with more complex prompts and demonstrated the gain by having the logical reasoner with scene graphs.

Missing lots of works on referring segmentation [3]-[5].

We agree with the reviewer that papers [3]-[5] are relevant to our study. Here, we discuss these works and clarify our paper's contribution.

GRES [3] proposes two distinct attention mechanism to address addresses multi-target segmentation. R-RIS [4] proposes RefSegformer, a transformer-based model that includes a language encoder, a vision encoder, and an encoder-fusion meta-architecture for handling incorrectly described textual prompts. See, Say, and Segment [5] propose SESAME, an LMM designed to "see" whether objects are present, "say" to interact with users, and "segment" target objects.

All these methods rely on transformers (or attentions) as their core reasoning pipeline, and they would inherit the reasoning limitations inherent to purely neural models. In contrast, DeiSAM explicitly encodes logical reasoning processes to guarantee accurate and faithful interpretation of abstract and complex prompts. We will include these discussions in the final version.

Thank you once again for your valuable feedback. We hope our response adequately addresses your concerns, and we would be pleased to answer any further questions you may have.

Dear reviewer,

thanks for your participation in the NeurIPS peer review process. We are waiting for your response to the rebuttal. You gave a reject rating (3), with 5 weaknesses.

Is the response from authors satisfactory?

• If yes, are you planning to increase your score?
• If no, could you help the authors understand how they can improve their paper in future versions?

Thanks, AC

We thank the reviewer for the fruitful feedback and for acknowledging that the proposed framework is novel and effective and the empirical results are valid.

We would like to address the concerns raised by the reviewer.

The proposed method should also be evaluated on other referring image segmentation benchmarks, such as RefCOCO, RefCOCOg, and RefCOCO+

We already reported this in the paper. Table 3 shows the result on the RefCOCO+ benchmark. We observed that DeiSAM can handle the referring expression task, which is the common benchmark suggested by the reviewer.

Moreover, in Table 4, we demonstrated that neural baselines (LISA and GroundedSAM) significantly degraded their performance if we modified the textual input to the abstract form, e.g., “kid wearing navy shirt” is modified to “an object that is wearing navy shirt.” This indicates the lack of the ability to use abstract high-level reasoning in neural baselines.

What components are tuned during the training stage? Annotate the tuned modules.

The rule weights in the differentiable forward reasoner. On line 321, we explicitly stated: “We minimize the binary cross entropy loss with respect to rule weights $w_1$ and $w_2$.” To enhance clarity, we added annotations to the figure as the reviewer proposed. Thank you for the suggestion.

Pre-training Limitations: The Scene Graph Generator appears pre-trained on Visual Genome, which annotates only a limited set of objects (e.g., persons, boats, cars).

We agree with the reviewer that good scene graph representations are the key to the proposed DeiSAM framework. Our finding is that if good scene graphs (generators) are available, the segmentation quality on abstract prompts can be much improved by combining logical reasoning.

As scene graphs can be noisy and low-quality, DeiSAM performs embedding-based vocabulary matching and learning scene graphs in combination. It is a vital open question to handle scene understanding for unseen data in a zero-shot manner.

Will the proposed method work on images containing objects not annotated in Visual Genome?

Yes, it works. We conducted additional experiments using the CLEVR environment [1] that contains abstract 3D visual scenes, which is largely different from visual genome. We show that DeiSAM can segment abstract 3D objects given more complex textual prompts. Moreover, our result on the RefCOCO+ dataset also validates that it is not limited to only visual genome images.

[1] CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR 2017

If not, the title "Segment Anything with Deictic Prompting" may be an overstatement.

As the premise is false (we clarified above), we believe this is not an overstatement.

What are the inputs and outputs of the Semantic Unifier?

The input is a scene graph and a set of rules generated by LLMs from textual prompts. If a term in the rules is absent in the scene graph, the semantic unifier investigates the most likely matching between these two representations using term embedding. We added these technical details to the final version of our manuscript Thank you for pointing this out.

What constitutes the "background knowledge" in forward reasoning?

It is a set of rules and facts described in first-order logic. It is common for logical reasoners to accept background knowledge from users. For DeiSAM, we provided no background knowledge throughout the experiments.

What is learned in the forward reasoning process?

In our experiments, rule weights are learned for composing a mixture of scene graphs to maximize segmentation performance.

To clarify, our forward reasoner implements forward-chaining reasoning in first-order logic, a method used to deduce all facts given known rules and known premises. This reasoning process is inherently deterministic. The differentiable version of the forward reasoner introduces weighted rules, which can be optimized through backpropagation. Our approach seamlessly integrates forward reasoning with neural modules, thereby enhancing the overall learning performance.

Thank you once again for your valuable suggestions. We hope our response addresses your concerns, and we are happy to answer any further questions you may have.

Dear reviewer,

thanks for your participation in the NeurIPS peer review process. You indicated that you are leaning towards accepting the paper. Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

• If yes, are you planning to increase your score?
• If no, could you help the authors understand how they can improve their paper in future versions?

Thanks, AC

We thank the reviewer for the thoughtful comment and for acknowledging that the paper addresses limitations in neural segmentation models and is well-structured and easy to read. We address the concerns next.

Over-simplified Assumptions: How would the method handle a prompt to segment the white cat using relative descriptions?

Our method can handle such cases. Let us describe how to do that. Given prompt: “Segment a white cat”, an LLM can generate rules in our format:

cond1(X):-type(X,Y),type(Y,white_cat).
target(X):-cond1(X).

Then, the semantic unifier will consider the embedding of the term white_cat and match it to an entity in the scene graph representing a white cat. The embedding-based semantic unification can address the visual-language alignment. If the scene graph misses entities of a white cat and a black cat in the first place, it would be hard for DeiSAM to perform reasoning correctly. A crucial unresolved issue is how to generate scene graphs in a zero-shot manner, akin to large visual-language models.

Moreover, the dataset size is small (10k), making large models prone to overfitting.

The DeiVG dataset is proposed to evaluate segmentation models, and thus we believe that the dataset with 10k examples is not extremely small for this usage.

Inaccurate scene graphs can lead to incorrect segmentation results, which is a critical limitation for practical applications.

We argue that segmentation models must understand the scenes to be faithful reasoners. Without this assumption, models can learn to shortcut, i.e., learn to pretend to answer via an incorrect scene understanding and reasoning [1]. This does not essentially solve the complex reasoning prompts that we aim to address in the paper.

[1] Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts, NeurIPS 2023

the large performance gap indicated in Table 5 highlights this dependency.

We agree with this. The quality of scene graphs significantly impacts the segmentation quality, but DeiSAM demonstrates resilience to noise in scene graphs. If the quality of scene graphs is extremely low, the resulting segmentation quality will also be low, as the reasoning process relies on the scene graph representation. However, DeiSAM manages noise in scene graphs through embedding-based vocabulary matching and by learning a mixture of scene graphs, as demonstrated in Table 5.

Unclear Philosophy: The rationale for using a differentiable forward reasoner instead of continuing to use an LLM is not well-justified.

The primary advantage is that segmentation models transform into reliable reasoners. LLMs often hallucinate on complex abstract prompts and scenes. To demonstrate this, we conducted additional experiments on 3D visual scenes with more complex prompts. For more details, refer to the general remark.

The necessity and advantage of having a "differentiable" reasoner are unclear. How does this reasoner fundamentally differ from previous neural-based methods like Grounding DINO or LISA?

The advantage of the resulting model lies in its ability to learn through gradients, thereby enhancing its performance. In contrast, using an off-the-shelf discrete logic system (e.g., Prolog) for a segmentation model would render the entire system static by blocking the gradient flow. This static nature would severely limit its applicability, as it would necessitate expert intervention to rewrite rules whenever they are imperfect. To reconcile explicit logical reasoning with gradient-based learning, we employ a differentiable logic reasoner. As illustrated in Table 5, DeiSAM obviates the need for expert efforts to improve segmentation performance.

The paper lacks evaluations on more challenging scenarios, such as multiple objects of the same category or prompts with more complicated relations.

Refer to the general remark.

There is no novel architecture designed for the prompt-based segmentation. It appears more like a technique report for an engineering project instead of being a research paper suitable for a top-tier conference like NeurIPS.

We disagree with the premise. DeiSAM addresses segmentation tasks involving abstract, complex prompts by integrating logic reasoners. Experimental results substantiate its effectiveness, demonstrating that it outperforms strong neural baselines.

We believe that utilizing existing modules and a well-defined computational architecture should not be viewed as a disadvantage. We contend that the criteria for acceptance at a conference should be based on the overall contribution of the paper rather than solely on the introduction of a "novel architecture."

Thank you for your insightful suggestions. We hope our response effectively addresses your concerns, and we are happy to answer any further questions you may have.

Dear reviewer,

thanks for your participation in the NeurIPS peer review process. We are waiting for your response to the rebuttal. You gave a borderline rating (4). Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

• If yes, are you planning to increase your score?
• If no, could you help the authors understand how they can improve their paper in future versions?

Thanks, AC

We thank the reviewer for the detailed and fruitful feedback and for acknowledging that our findings are insightful, the experimental setups are meaningful, and the paper is well-written.

We address the concerns next.

W1: many examples from the constructed dataset are unnatural by removing all nouns from the deictic expressions with "stuff" or "object". It is more common for references to consist of categorial nouns instead of object names (E.g., that toy on top of the chair) instead of teddy bear

We agree with this. Using categorical names (e.g. toy, food, deice) would be very promising in covering more real-world applications of referring expressions. This could be achieved by having an LLM while dataset generation to convert each specific name to a categorical name.

However, we do not conduct experiments on this as our primary focus is abstract reasoning in which object and categorical names are absent in the prompt, i.e. the models need to deduce the object name from a given relational specification.

To mitigate, we reported on Table 10 in the Appendix of performances by making DeiVG datasets with a more referring-expression style, e.g. we provide "Person on the boat" instead of “An object on the boat”. The results indicate that neural baselines gain a lot by having object names. We expect that if we provide the categorical names in the prompt, neural baselines will gain some performance but not as much as with object names.

W2: Reliance on Scene graphs: The proposed work relies on access to scene graph prediction model outputs.

We answer this later for Q1.

W3: Shortcut of in the DeiVG dataset. E.g. "wearing a hat" alone might be enough to decode the object given prompt: tall, wearing a hat, walking on the sidewalk.

We agree with the reviewer that this is often the case in our dataset. We have analyzed how often it happens in the generated DeiVG dataset.

For example, with a prompt “An object wearing a hat walking on the sidewalk", we consider the following variations of ways to match objects:

• Relation: We consider only relations to detect matching between objects in the scene, i.e., "wearing" or "walking on," disregarding the object names.
• Object: We consider only objects to detect matching between objects in the scene, i,e., "hat" or "sidewalk," disregarding the relations.
• Either: We consider either a partial match from Relation or Object.
• Complete: We consider complete counterexamples, i.e., "wearing hat" or "walking on the sidewalk", which is exactly articulated by the reviewer.

The table below shows the proportion of prompts involved with such shortcuts.

The result indicates that the shortcut often happens in the dataset. In DeiVG$_3$, 60% of the prompts are involved with such a shortcut, i.e., no other objects in the scene share the same 3 relations. We included this result to the appendix of our manuscript. Thank you for the suggestion to improve the paper.

Moreover, to explore more cases with similar objects without shortcuts, we conducted an additional experiment on an abstract 3D environment with more complex prompts. For more details, refer to the general remark.

Q1: Can we use scene graphs for other baselines? E.g., "A red furry object" --> Select all objects that are red and furry from scene graph --> Grab the names of those objects -> use the name instead of deictic expression when prompting SEEM or LISA.

This approach would not be fair. While it is an interesting suggestion to use scene graphs for neural baselines, the proposed method involves providing an external program for preprocessing the scene graphs so that neural models can process them. This would result in a composite model consisting of a scene graph preprocessor and a separate large segmentation model. Such an approach severely limits further extensions aimed at enhancing scene understanding and reasoning segmentation because it is not straightforward to propagate segmentation errors back to improve scene graph generation.

In contrast, DeiSAM treats scene graphs as an internal primal representation. It performs segmentation through differentiable reasoning, allowing it to backpropagate errors to the scene graphs. This capability is crucial for enabling the model to learn to segment more accurately from data and to generate explanations using gradients, indicating which parts of the scene were essential for the segmentation.

An alternative comparison could be to encode the scene graphs as text and provide them to the neural baselines.

Q2: What is the effect of the quality of scene graphs on the final results of the proposed method. Is it resitant to mistakes in the SGG prediction?

It significantly impacts the segmentation quality, but DeiSAM demonstrates resilience to noise in SGGs. If the quality of scene graphs is extremely low, the resulting segmentation quality will also be low, as the reasoning process relies on the scene graph representation. However, DeiSAM manages noise in scene graphs through embedding-based vocabulary matching and by learning a mixture of scene graphs.

Thank you for your valuable suggestions. We hope our response satisfactorily addresses your concerns, and we are happy to answer any additional questions you may have.

Dear reviewer,

thanks for your participation in the NeurIPS peer review process. You indicated that you are leaning towards accepting the paper. Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

• If yes, are you planning to increase your score?
• If no, could you help the authors understand how they can improve their paper in future versions?

Thanks, AC