SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation

Thank you for the thoughtful review. We would like to first thank you for identifying the methodological novelty and efficiency of our Saccade–Fixation design—an assessment that echoes 8mtA’s praise for its robust alignment and jD3V’s recognition of its accuracy and speed, while Wg1a also noted the results to be promising.

Below, we address each of your comments point‑by‑point and clarify how our revisions and additional experiments resolve the concerns you raised.

R7DT-W1 && R7DT-Q1: What are the fundamental differences between SaFiRe and reasoning segmentation approaches like LISA?

R7DT-W1-A && R7DT-Q1-A: We first emphasize that SaFiRe and LISA tackle fundamentally different tasks. LISA brings up a novel task beyond RIS, while our SaFiRe focuses on bringing current RIS task into a more practical usage. We compare Existing RIS, our persuade on Ideal RIS and ReasonSeg as the following table.

In few words: (1) LISA focus on what other RIS models can NOT do, while (2) we focus on what other RIS models should do but fail.

(1) LISA focus on what other RIS models can NOT do.

The ReasonSeg task proposed in LISA aims to response queries requiring complex reasoning or world knowledge. However, complex reasoning is an ability emerged from LLMs' large corpus pertraining; while world knowledge also requires large diversity of training data and large parameter size for memorize. LISA utilizes LLaVA (7B) to achieve this. Obviously, these abilities are out of the ability of a standard RIS architecture (typically with Swin+Bert as backbone).

(2) We focus on what other models should do but fail.

We focus on the referential ambiguity problem in RIS task: dealing with object-distracting or category-implicit expression. The cases in our aRefCOCO are still in RIS’s scope and an ideal RIS model should complete these cases perfectly. However, existing RIS models tends to fail due to their inherent limitations: they models RIS as a "key word matching problem". Our work shows that by jumping out of "key word matching" paradigm, we’re able to solve these hard cases.

As for LISA, through it brings novel idea and solution about reasoning ability, they still fails on our "should do but fail" cases. Since they utilize SAM-style architecture for mask producing, which is still inside “key word matching” paradigm.

(3) Evaluating LISA on our aRefCOCO benchmark.

Here we provide the evaluation results of LISA and LISA++ on aRefCOCO. Notably, while LISA++ is designed to outperform LISA on ReasonSeg tasks, it actually performs worse than LISA on aRefCOCO. This indicates that, pursuing stronger reasoning ability may harm the model's performance on ambiguous expression, further highlights the necessity of introducing such a benchmark.

We appreciate the reviewer's comments and will add above analysis to the revision for better clarity.

R7DT-Q3: Benchmarking SaFiRe on the Reasoning Segmentation task

R7DT-Q3-A: As explained above, SaFiRe is not designed for reasoning segmentation, we believe it is unfair and unreasonable to evaluate SaFiRe on the ReasonSeg benchmark, as it does not have access to world knowledge.

R7DT-Q2: Failure Case Insights: While ambiguous expressions are addressed, what patterns emerge in failure cases (e.g., pronoun chains, conflicting attributes)? Are there categories of ambiguity where SaFiRe still struggles?

R7DT-Q2-A: Thank you for your question. We will add detailed visualization of failure cases in revised version. Here, we summarize below the main patterns observed in SaFiRe’s failure cases:

• Fine-grained attribute ambiguity. Failures often occur when multiple candidates share broadly similar attributes but only one matches the intended semantics of the description. In such cases, even subtle differences in attributes (e.g., hand positioning) can carry decisive referential meaning, yet the model may overlook them due to visual similarity. Example: “He crosses his hands while talking to the other person.” In the image, one man rests one hand over the other, while another interlaces his fingers. Although both are hand-related gestures and might visually resemble “crossed hands,” only the latter accurately captures the intended meaning. The model, however, selects the former, revealing its difficulty in grounding fine-grained action attributes to textual descriptions.

• Long spatial/relational chain ambiguity. Another common failure mode arises in texts involving long, compositional chains of spatial or relational references. These expressions require the model to perform multi-step understanding with frequent spatial shifts, jumping between different objects across the scene. A typical symptom is that the grounding process breaks down at one or more intermediate links, leading to cumulative error. Example: “the mug on the tray beside the plate nearest the chef.” In the image, there are multiple mugs, trays, and plates spread across a crowded table, and the chef is positioned off to one side. There is no obvious shortcut to the target—only by following the full relational chain can the correct object be identified. The model, however, misidentifies an object linked to an incorrect plate early in the chain, revealing its difficulty in maintaining spatial consistency and grounding excessive relational dependencies to textual descriptions.

We will add the above analysis into revision to clarify the potential future directions.

R7DT-W3: Benchmark: The availability of aRefCOCO as a test-only dataset could limit its utility within the community. Providing a clear train/val split would enhance its usefulness for future benchmarking efforts.

R7DT-W3-A: Thank you for your suggestion. The primary goal of our paper is to provide a robust and challenging benchmark for evaluating model performance under referential ambiguity, rather than to construct a large-scale training dataset. We fully agree that establishing a training split for aRefCOCO could further benefit the community and enhance future research. We will consider construct train/val splits in the future.

R7DT-W4: Quantification of Speedup: While the proposed method's speedup is specified, quantifying it in relation to computational requirements (GFLOPS) would be beneficial for the community.

R7DT-W4-A: Thank you for your valuable suggestion. We have added GFLOPs results, including both LISA and LISA++ in our comparison. Furthermore, we present the speed comparison in terms of FPS (image-text pair/s). As shown in the table below, SaFiRe achieves a much lower computational cost and faster inference speed compared to LISA and LISA++, which both require near 5000 GFLOPs (SAM- and LLM-driven) and are much slower. At the same time, SaFiRe delivers significantly better segmentation accuracy on the RIS task.

Thank you for your time on our submission. We would like to first thank you for acknowledging the promising experimental results and the suitability of state‑space modeling for RIS—results similarly lauded by 8mtA and jD3V, with R7DT additionally emphasizing the methodological novelty behind them.

Below, we address each of your comments point‑by‑point and clarify how our revisions and additional experiments resolve the concerns you raised.

Wg1a-W1: The motivation of the paper is kind of vague and the explanation of SaFire’s design is questionable. Even though the authors claim that their Saccade and Fixation come from cognitive science, it can not convince me that such operation aligns with their original aspects.

Wg1a-W1-A: First, we would like to clarify that our main goal is to resolve referential ambiguity, a fundamental challenge in real-world RIS scenarios. Previous methods struggle in such scenarios due to their reliance on keyword-matching modeling. Instead, we jump out of such paradigm and propose our Saccade-Fixation mechanism to address the issue.

Second, we are inspired, rather than conducting a bio-faithful simulation to the original concepts. Our Saccade–Fixation mechanism is a computational abstraction: Saccade performs global scanning for coarse understanding , while Fixation executes focused, token-level binding and language alignment.

Additionally, our motivation has also been recognized by Reviewer jD3V as "well-motivated and grounded in practical RIS limitations" and by Reviewer R7DT as "well-conceived."

Further, as already cited in our paper, two well-established findings ([17], [18]) support the scan–then–inspect pattern that inspires our approach. Below, we provide a detailed table that includes direct citations, explanations, and a mapping of each theory to its concrete implementation in SaFiRe, showing how we are inspired.

We appreciate the reviewer's comments and will enrich the introduction part to better clarify our motivation in our technical design.

Wg1a-W2: The novelty of the proposed framework is questionable, as it borrows largely similar designs from VMamba, like the scanning and folding techniques in other SSM works.

Wg1a-W2-A: We believe there are some misunderstandings.

VMamba is designed for single-modal vision tasks. We use the SS2D block serves as a fundamental processing unit in our framework, similar to how attention mechanism functions in other architectures. Our novelty lies in the tailored design of Saccade-Fixation mechanism for tackling multimodal RIS tasks: First, in Saccade operation, the visual features are modulated by the overall meaning of the text and then are scanned to grasps a quick glimpse of image-text correspondence; Second, the Fixation operation follows a carefully designed "group-scan-recover" pattern to achive detailed examination of cross-modal alignment. We have conducted comprehensive ablation studies (see Tables 4 and 5) and visual comparisons (see Fig. 3) to validate the effectiveness of our approach.

Additionally, we clarify that our goal is to better address the RIS task in referial ambiguity, rather than merely designing a new version of Mamba architecture.

Wg1a-W3: It is unclear whether the samples selected for the aRefCOCO are valid or not when including VLM (i.e., Qwen-VL-Max) for curation due to the absence of further discussion of it. This would also cause a potential cherry-picking by selecting challenging samples from RefCOCO where the proposed method outperforms and the baselines do not.

Wg1a-W3-A: Sorry for any confusion. The original samples from RefCOCO are randomly selected and re-annotated using Qwen via its API. Our SaFiRe model is NOT involved during the entire dataset construct process. For more details about the dataset construction, please refer to 8mtA-W2-A to Reviewer 8mtA. We will release the code and dataset upon publication.

Thank you for the careful review. We would like to first thank you for valuing the paper’s clarity and our method’s novelty，accuracy and speed—strengths also stressed by R7DT and supported by the promising results noted by Wg1a, while 8mtA further commended the strong gains on ambiguous benchmarks.

Below, we address each of your comments point‑by‑point and clarify how our revisions and additional experiments resolve the concerns you raised.

jD3V-Q1: How is the inference speed compared to ReMamber?

jD3V-Q1-A: We have added a inference speed comparison (by fps) with ReMamber as follows. Our model achieves 1.46× speedup over ReMamber. That's because its Twister design involves two operations with relatively large-scale computation. As a result, although we use Swin transformer (vs ReMamber's Mamba-B) as visual backbone, our model outperfom it in inference speed.

jD3V-Q2 && jD3V-W3: What are the failure cases of the proposed method?: The paper could benefit from more analysis on failure cases of the proposed method. This would help clarify its limitations, especially in handling complex descriptions

jD3V-Q2-A && jD3V-W3-A: Thank you for your question. We will add detailed visualization of failure cases in revised version. Here, we summarize below the main patterns observed in SaFiRe’s failure cases:

• Fine-grained attribute ambiguity. Failures often occur when multiple candidates share broadly similar attributes but only one matches the intended semantics of the description. In such cases, even subtle differences in attributes (e.g., hand positioning) can carry decisive referential meaning, yet the model may overlook them due to visual similarity. Example: “He crosses his hands while talking to the other person.” In the image, one man rests one hand over the other, while another interlaces his fingers. Although both are hand-related gestures and might visually resemble “crossed hands,” only the latter accurately captures the intended meaning. The model, however, selects the former, revealing its difficulty in grounding fine-grained action attributes to textual descriptions.

• Long spatial/relational chain ambiguity. Another common failure mode arises in texts involving long, compositional chains of spatial or relational references. These expressions require the model to perform multi-step understanding with frequent spatial shifts, jumping between different objects across the scene. A typical symptom is that the grounding process breaks down at one or more intermediate links, leading to cumulative error. Example: “the mug on the tray beside the plate nearest the chef.” In the image, there are multiple mugs, trays, and plates spread across a crowded table, and the chef is positioned off to one side. There is no obvious shortcut to the target—only by following the full relational chain can the correct object be identified. The model, however, misidentifies an object linked to an incorrect plate early in the chain, revealing its difficulty in maintaining spatial consistency and grounding excessive relational dependencies to textual descriptions.

We will add the above analysis into revision to clarify the potential future directions.

jD3V-W1: The Saccade-Fixation design is novel; the rest of the model relies heavily on standard components (Swin-B, BERT). The innovation is primarily in the fusion mechanism rather than in the overall pipeline, which may limit its extensibility to other tasks or modalities.

jD3V-W1-A: First, we sincerely thank you for recognizing the novelty of our Saccade–Fixation design. We would like to clarify that our choice of Swin-B and BERT as the backbone is primarily to ensure a fair comparison with mainstream methods. Meanwhile, it is a convention to focus on fusion mechanism designing in this field, as many other methods share the same backbones in Table 1 in our paper.

Second, the Saccade–Fixation mechanism is a general-purpose fusion method; however, this work focuses on the RIS task, and we leave broader exploration to future work. For example, SaFiRe have the potential to be adapted to other task or modalities like Open Vocabulary Segmentation (OVS) by changing the task head with minimal changes, or Audio Visual Segmentation (AVS) by changing the backbones. We will mention such potential way of exploration in the conclusion as future work.

jD3V-W2: Although the introduced benchmark aRefCOCO increases ambiguity, it is still generated by a VLM, which could inadvertently introduce biases, even with meticulous filtering. The paper lacks evaluations on real-world, human-annotated ambiguous datasets or interactive grounding scenarios, limiting its external validity.

jD3V-W2-A: We acknowledge this bias concern and have already claimed it in our limitations section (Appendix B). Since this study mainly focuses on the techinical innovation in designing a model to handle the referential ambiguity, we mainly verify our method in the common benchmark and our initial small set, aRefCOCO. For broader evaluations on real-world, and human annotated datasets as well as interactive grounding scenarios, we will conduct the continuous verification and extension in our future explorations.

Thank you for the detailed review. We would like to first thank you for highlighting the novelty and performance gains of our Saccade–Fixation framework—benefits that R7DT and jD3V likewise praised, with even Wg1a calling the experimental results “promising.”

Below, we address each of your comments point‑by‑point and clarify how our revisions and additional experiments resolve the concerns you raised.

8mtA-W1: Concerns regarding comparative evaluation: (1) "Unfair" comparison with ReMamber due to backbones difference; (2) "Marginal" gains over Magnet; (3) Missing recent SOTAs

(1) Comparing with ReMamber

We include ReMamber not for a direct comparison of performance, but rather as a reference baseline, since it is the only published Mamba-based RIS method. We will revise our tables to clearly indicate that ReMamber is a full-Mamba-based architecture, to avoid any misunderstanding regarding the fairness of the comparison. Meanwhile, as other methods, ReMamber still approaches RIS through keyword-matching modeling and cannot effectively address the referential ambiguity problem that we aims to solve.

(2) Comparing with MagNet

SaFiRe demonstrates consistent gains across all datasets of RefCOCO series. Our advantage on RefCOCO is kind of marginal because this dataset is relatively simple. However, as the dataset becomes more challenging, our model's advantage becomes evident, with strong improvements on the most difficult RefCOCOg dataset (+1.3 oIoU on average).

Additionally, we here provide more comparisons with MagNet, including mixed‑dataset training (which better reflects the linguistic variety than single-dataset) and inference speed. Results are averaged over each split, with the numbers in parentheses indicating the performance gap relative to ours. SaFiRe surpasses MagNet in both accuracy and speed.

(3) Comparing with recent SOTAs

First, we want to clarify that the main evaluation metric reported in [1]-[4] is mIoU, which differs from the widely-used oIoU in this field. mIoU is typically numerically higher than oIoU (see results in [2],[4]). The distinction between the two is shown in LAVT(CVPR2022). Here, we provide the comparison using mIoU and oIoU. Note that their main results are under mixed-dataset setting (similar with Tab.2 in our main paper). Results are averaged over each split, with the numbers in parentheses indicating the performance gap relative to ours. Compared with these methods, SaFiRe still achieves the best accuracy.

(ft denotes a further fine-tuning is performed on the specific dataset.)

Second, since C3VG and EEVG provide codes and weights, we further test them on aRefCOCO; SaFiRe keeps a clear lead on all metrics.

Third, given C3VG’s good accuracy, we also compare inference speed: SaFiRe runs 3.3× faster, confirming its advantage in application.

8mtA-W2: Inadequate Transparency in aRefCOCO Dataset Generation. The reliability of aRefCOCO, including VLM prompts, filtering and validation steps.

• 8mtA-W2-A: Here we provide details of this process. Codes and datasets will be released upon publication.

(1) Annotation Prompting

In order to ensure the richness and diversity of object descriptions while adhering to the fundamental requirement of single-object localization in RIS tasks, we used a prompt template for Qwen-VL-Max as below:

"### Role\n"
"You are a professional assistant, proficient in object recognition and description. Your task is to provide a clear and focused description of the main object inside the red box in the image. You are given the category information of the main object inside the red box as '{target}'. The description should be based on the provided target information, and you should generate a more detailed description with your visual analysis of the image. Please note that there is no visible box in the actual image.\n\n"

"### Target Category Information\n"
f"targt category information: '{target}'\n\n"

"### Skills\n"
"1. [Important!] Please only focus on the main object, i.e., '{target}', and ignore the background or irrelevant details.\n"
"2. [Important!] Do not assume or include any information that is not present in the image content.\n"
"3. [Important!] Provide 5 independent English sentences, each of which can independently identify '{target}' without relying on other sentences. Each sentence should describe a unique identifying feature.\n" 
"4. [Important!] Use the subject '{target}' from the targt category information, and ensure you are describing a single object '{target}'! Others should be able to locate only one object based on your sentences.\n"
"5. [Important!] Use singular language to ensure the description focuses only on one main object. Do not use plural words like 'they' as the subject, but you are allowed to use pronouns as the subject.\n"
"6. [Important!] If you need to describe based on objects around '{target}', please keep the subject as '{target}'.\n\n"

(2) Filtering

We first pre-screen all descriptions with GPT 4o for the two patterns:"object-distracting" (i.e., containing multiple nouns) and "category-implicit" (i.e., where the subject is a pronoun). Items that match either pattern are retained. Then, to verify the remaining pairs meet the basic rules we set for annotation, GPT 4o is involved as an evaluator. The prompt is as follows:

Role: 
You are a professional visual-language evaluator. Your task is to verify whether each given description sentence is a faithful, grounded, and rule-abiding description of the target object **inside the red box** in the provided image.

Information:
The target object belongs to the category: {target}.

The description was generated under the following constraints:
1. It must describe only the main object ('{target}'), ignoring the background or other irrelevant elements.
2. It must not include objects that are not in the whole image, but objects out of box are allowed.
3. Each sentence must independently identify the target.
4. It can contains objects beyond the target.
5. It must use singular phrasing, and refer to the object using either its category label ('{target}') or singular pronouns.

You will be given 5 sentences. Your task is to verify, for each sentence, whether it satisfies **all** the above requirements based solely on the image content.

Sentences to Verify:
{Sentences}

Instructions:
1. For each sentence, respond with **"yes"** or **"no"**. if "no", follow it by a **brief reason**.
2. If any rule is violated or the content is ungrounded, answer "no" and specify which rule(s) are broken.
3. Do not use external knowledge or assumptions. Base your judgment strictly on what is visible in the image.
4. Provide one answer per sentence, in order. Each answer must be on its own line.
5. Do not add commentary outside the answers. Return only the answers.
6. Ensure the number of answers equals 5.

(3) Validation

Manual review was conducted by at least two annotators with RIS domain expertise. Each pair was independently validated to confirm (1) whether the description uniquely refers to a single object in the image, and (2) whether the expression obeys our referential ambiguity criteria while remaining image-groundable. In borderline cases, annotators reached consensus through discussion, and a third reviewer was consulted when needed.

8mtA-Q1: Given the experimental limitations, state the primary scientific concern that SaFiRe aims to address

8mtA-Q1-A: First, we hope the experiments and analysis in the above response (to 8mtA-W1) can resolve the reviewer's concern about experimental limitations. Our method surpass all existing RIS models, demonstrating its superiority.

Second, we want to clarify that our goal is to resolve referential ambiguity, a fundamental challenge in real-world RIS scenarios. Previous methods struggle in such scenarios due to their reliance on keyword-matching modeling. Instead, we jump out of such paradigm and propose our Saccade-Fixation mechanism to address the issue. To further demonstrate this point, a more challenging benchmark aRefCOCO is proposed besides traditional benchmarks.

8mtA-Q2: clarify whether the first SFLayer uses the final-layer features from Swin and BERT, and whether use LAVT’s cross-modal fusion in the backbone stage

8mtA-Q2-A: Yes, the first SFLayer uses only the final-layer features from both the Swin Transformer and BERT. We also adopt LAVT’s cross-modal fusion in the backbone stage as CGFormer (CVPR23), SADLR (AAAI23), and ReLA (CVPR23) do. We will clarify these details in the revision.