SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning

We sincerely thank the reviewer for the insightful and constructive comments. We have carefully addressed each point as follows:

1. Missing related work discussion

Thank you for your valuable suggestion. In the revised version, we will include a dedicated discussion on the task-generic, promptable segmentation setting, particularly focusing on methods that aim to infer image-specific answers based on generic prompts and produce corresponding segmentations. Specifically, we will incorporate the three cited papers into this discussion to better contextualize recent advances in this area.

2. Insufficient clarification of innovation

Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets with explicit reasoning chains, which are costly and time-consuming to construct. Some recent methods have explored reinforcement learning for segmentation, such as Seg-Zero, but they often pre-process segmentation tasks into object detection form and fail to leverage task-specific, fine-grained reward signals.

In contrast, we propose a novel formulation where SAM is directly employed as a reward provider in the reinforcement learning loop, requiring no special pre-processing of the segmentation dataset. This design enables us to use the predicted segmentation masks as supervision, providing direct, pixel-level feedback to guide reasoning. To complement this, we modify the GRPO optimization objective to improve exploration ability and interpretability, allowing the model to follow complex multimodal instructions and accurately localize the segmentation target. These innovations collectively allow SAM-R1 to achieve fine-grained perceptual reasoning with a simplified, end-to-end training pipeline.

3. Reward threshold design and sensitivity

Thank you for these detailed questions regarding our reward function.

The core motivation for our tiered reward design is to provide a stable and informative learning curriculum. A single, fixed threshold creates a challenging dilemma in RL-based training:

●A high threshold makes rewards too sparse, especially in early training. Most predictions receive no reward, hindering efficient learning and exploration.

●A low threshold leads to quick reward saturation. Once the model can consistently achieve a coarse segmentation, it receives no further incentive for fine-grained refinement.

Our tiered reward strategy (as defined in Equation 5) elegantly resolves this trade-off. By discretizing IoU into multiple levels, it creates a hierarchical learning path, guiding the model progressively from coarse localization (earning a small reward) to precise segmentation (earning the maximum reward).

The effectiveness of this tiered approach is empirically validated by our ablation study, which we include here for your convenience:

As the results clearly demonstrate, the tiered strategy consistently and significantly outperforms all fixed-threshold alternatives. On the challenging ReasonSeg dataset, it yields a substantial +1.6 to +4.0 point gain in gIoU and a +2.4 to +3.5 point gain in cIoU compared to the single-threshold methods. This provides clear evidence that the hierarchical reward signal is crucial for achieving superior performance.

4. Clarification of improvements over standard GRPO

While GRPO provides a solid foundation, our method is not a direct application. We introduced significant, task-specific adaptations that are essential for enabling a model to perform complex reasoning for segmentation.

Our main technical contributions can be summarized in three key areas:

●Task-Specific Reward Engineering: Standard GRPO is designed for general text generation. We engineered a completely new reward function tailored for vision-language grounding. This includes:

A multi-level, IoU-based reward (Eq. 5) to provide a granular learning signal for segmentation accuracy.

A supplementary format-based reward to enforce correct structural output (i.e., reasoning text followed by segmentation coordinates).

●Core Algorithmic Enhancements: We modified the GRPO optimization objective itself to improve training stability and performance on this unique task. Our key changes are token-level advantage normalization and an asymmetric clipping mechanism (Clip higher). These are not part of the standard GRPO implementation and are vital for balancing the needs of generating fluent reasoning with precise, numerical coordinate outputs.

●Unified End-to-End Training: Unlike prior works that separate reasoning and segmentation modules, we train our system jointly using RL. This allows the model to learn a direct mapping from visual input and natural language instructions to segmentation masks, using pixel-level feedback to refine its reasoning process.

The necessity and impact of our algorithmic enhancements are empirically validated in our ablation study. We present the key results from Table 4 of our paper below:

As the table clearly shows, the standard GRPO baseline yields the lowest performance. Adding our proposed token-level normalization or asymmetric clipping individually provides noticeable gains. When combined, our full model (Ours) achieves a substantial +3.1 cIoU gain on ReasonSeg over the baseline.

This result directly demonstrates that our modifications are not minor tweaks but rather crucial components that collectively enable the success of our approach. We will ensure this clarification is prominently featured in the revised manuscript.

We thank the reviewer for the thoughtful and constructive feedback. We address the raised concerns as follows:

1. On technical novelty and architectural contributions

We acknowledge that our work leverages open-source components such as Qwen2.5-VL and SAM. However, we emphasize that the novelty of SAM-R1 lies not in introducing entirely new architectures, but in designing a novel reinforcement learning framework that enables these powerful models to collaborate effectively on fine-grained reasoning and segmentation tasks.

While prior works have mainly focused on model architecture or segmentation backbones, we propose a unified system-level approach that includes:

●Directly using SAM as a reward provider, introducing pixel-level supervision into RL without requiring any special preprocessing or conversion of segmentation data;

●Modifying the GRPO reward objective to suit multimodal fine-grained reasoning tasks, allowing for interpretable and stable optimization signals;

●Designing an end-to-end training pipeline that jointly optimizes reasoning and segmentation under natural language instructions.

This represents a contribution in system design rather than architecture alone, and we believe it offers more practical value for advancing instruction-based fine-grained vision understanding.

2. On the feasibility of reward computation and dependence on fine-grained annotations

We appreciate the reviewer’s concern regarding the reliance on fine-grained annotations. We would like to clarify that our method does not require manually annotated fine-grained segmentation masks.

The primary goal of the RL stage in SAM-R1 is to help the VLM understand the target referred to in the instruction via reasoning and accurately pass that semantic information to SAM for segmentation. The model's fine-grained understanding emerges from internal reasoning rather than from external labels.

Our reward function provides feedback based on the IoU between the predicted segmentation mask and the ground-truth object-level mask. While standard datasets typically label whole objects, this still provides meaningful supervision — the goal is not part-level semantic correctness but rather alignment between the referred object and the predicted mask. Even in the absence of detailed part annotations, IoU-based feedback is sufficient to drive learning of semantic localization and perceptual alignment. Furthermore, we believe SAM-R1’s design actively mitigates the challenge of lacking fine-grained annotations. By delegating reasoning to the VLM and segmentation to SAM, the model learns to bridge language and vision without requiring costly, dense labels. Our results on standard datasets like RefCOCOg and ReasonSeg demonstrate that this design is both effective and practical in real-world settings.

3. On fairness of comparison in Table 1

We sincerely thank the reviewer for raising these important concerns regarding experimental fairness. Addressing this point allows us to clarify the rigor of our evaluation.

We want to state unequivocally that the comparison between our SAM-R1 and Seg-Zero is entirely fair and controlled. Both models are built upon the exact same backbone, Qwen2.5-VL 7B.

Furthermore, you correctly note that we reproduced the Seg-Zero results ourselves. This was a deliberate and necessary step to ensure a true comparison. We found that the results reported in their original paper were generated using different model weights for different metrics, which prevented a reproducible comparison from a single, unified model. To address this, we used their official, publicly released weights to generate all Seg-Zero results under an identical evaluation protocol.

This meticulous approach ensures that the observed performance gains are directly attributable to our proposed training paradigm—the RL-based reasoning and segmentation framework—rather than any differences in backbone or evaluation setup.

We agree with the reviewer that other baselines listed in Table 1 utilize different, and often weaker, MLLM backbones. Our primary goal with this table was to position our work within the broader landscape of existing methods.

To provide full transparency and address your concern directly, we will update Table 1 in the revised manuscript to include a dedicated column specifying the backbone model used by each method.

We thank the reviewer for the valuable feedback. Below, we address each point in detail:

1. On novelty and distinction from Seg‑Zero

Thank you for highlighting the perceived similarity to Seg‑Zero. We clarify that SAM‑R1 differs fundamentally from Seg‑Zero in its training paradigm. Although both methods leverage the SAM model, Seg‑Zero does not integrate SAM or any segmentation feedback during training. Instead, it preprocesses the segmentation task into an object detection problem and trains its vision–language model accordingly; the segmentation module itself is never part of a closed‑loop optimization. Consequently, Seg‑Zero cannot benefit from pixel‑level feedback, limiting its ability to learn fine‑grained visual detail.

In contrast, SAM‑R1 introduces a true reasoning–segmentation–reward loop during training. Our key innovations include:

●SAM as an RL reward provider: We retain the SAM module in the RL loop and compute rewards based on the IoU between SAM’s output mask and the ground‑truth mask. This preserves spatial detail and lets the model learn directly from pixel‑level supervision without converting to a detection formulation.

●GRPO objective modifications for fine‑grained multimodal tasks: We adapt the GRPO reward structure to include multi‑level, token‑wise feedback, increasing exploration and providing richer signals for reasoning chains.

●Closed‑loop training for joint generalization and expressivity: Our reward design not only assesses segmentation quality but also serves as a semantic alignment signal for the reasoning path. This encourages the model to abstract the referred object from complex instructions and precisely localize it, driving co‑evolution of language understanding and spatial perception.

Thus, while both methods utilize SAM, SAM‑R1 is the only approach that jointly optimizes reasoning and segmentation in a fully closed loop during training, representing a fundamental shift in paradigm.

2. On backbone model size fairness

Thank you for this important question. We want to assure the reviewer that ensuring a rigorous and fair comparison was our top priority.

All comparisons between SAM-R1 and Seg-Zero in our paper use the identical backbone: Qwen2.5-VL 7B.

We took the step of running their officially released model ourselves for a crucial reason: during our initial review, we observed that the results reported in their paper were generated using different model weights for different metrics. This prevented a direct and consistent reproduction across all benchmarks from a single model.

Therefore, to establish a truly controlled and fair comparison, we used their public weights to generate all Seg-Zero results with the same backbone and under the same evaluation protocol. This approach ensures that the performance differences we report are attributable to the methods themselves, not variations in experimental setup. We will emphasize this specific motivation in the revised manuscript to ensure full transparency.

3. On making the segmenter trainable

We did consider integrating a trainable segmenter for end‑to‑end learning, as in PixelLM. However, we opted against this for two main reasons:Scalability and modularity: SAM is a powerful, promptable segmentation model validated across many tasks. By keeping it as a separate module and driving it with a reasoning-based prompt, we maintain flexibility to adapt to diverse downstream tasks without the overhead of fine-tuning the entire segmenter.

Training stability and efficiency: Credit assignment in RL is already challenging. Incorporating the segmenter into the RL loop would introduce additional noise from unstable segmentation outputs, demanding far more data and hyperparameter tuning to converge effectively. By focusing RL optimization on the VLM alone, we achieve more stable training and faster convergence while substantially improving reasoning and expressive capabilities.

1. Regarding the comparison with other RL strategies

Thank you for this insightful question. Our decision was based on a principled evaluation of efficiency and stability, particularly within the resource-intensive context of VLM fine-tuning. In this setting, GRPO presents clear advantages over standard approaches like PPO:

●Computational and Resource Overhead: PPO and its mainstream implementations typically rely on a separate value network to estimate state values and use Generalized Advantage Estimation (GAE) to reduce variance. This not only increases the number of parameters but also leads to significant computational and memory overhead. In contrast, GRPO cleverly omits the value network, estimating advantages solely through the normalization of group-wise responses. This allows it to achieve high sample efficiency with lower resource consumption, making it more suitable for fine-tuning large models like VLMs.

●Stability and Hyperparameter Tuning: To prevent excessively large policy updates that can lead to training collapse, PPO introduces complex mechanisms like Clipped Probability Ratios and/or KL divergence penalties. The design of GRPO's loss function is more elegant, as it naturally integrates these stabilization mechanisms into a single objective function. This not only simplifies hyperparameter tuning but also further enhances overall training stability.

We will add a detailed discussion in the Methods section to articulate these points, thereby strengthening the justification for our deliberate algorithmic choice.

2. Regarding data efficiency and scale

Thank you for your excellent question concerning the model's scalability. To investigate this, we conducted additional experiments by increasing the training data size from 3k to 10k. The results clearly show that our method is highly data-efficient, with performance saturating at 3k samples. We are pleased to present the direct comparison below. On the ReasonSeg dataset, performance remains stable with no significant changes:

As shown in the table, increasing the data to 10k results in negligible fluctuations on ReasonSeg. The test gIoU slightly moves to 60.8 (+0.6) while the cIoU shifts to 53.9 (-0.4), strongly indicating that performance has already plateaued. On the RefCOCO benchmarks, we observe only marginal gains, all under 1 percentage point:

These results lead to a clear conclusion: our method's core performance saturates at 3k samples. Given the substantial increase in training cost versus the minimal performance returns, we deliberately chose 3k samples as the optimal trade-off point for demonstrating our method's capabilities. This finding is a strong testament to the excellent data efficiency and cost-effectiveness of our SAM-R1 framework, which we consider a core advantage of our work.

3. On failure case analysis

Thank you for this important suggestion, which greatly enhances the rigor of our paper. To provide a comprehensive view of our method's capabilities and limitations, we have already included an analysis of "Ablation Failures" in the submitted appendix. For instance, we show that:

●Removing the KL constraint leads to training instability.

●An attempt to encourage the model to generate "negative reference points" was not effective. This existing analysis validates the rationale behind our current framework design.

As we cannot include new figures during the rebuttal phase, we have completed a qualitative analysis and commit to adding a dedicated "Failure Case Analysis" section in the final paper. We will focus on typical cases such as:

Reasoning‑Execution Disconnect: For instance, given the instruction “segment the apple closest to the camera,” the model’s reasoning chain correctly identifies that there are three apples and that the largest bottom one is nearest, yet the final segmentation mask erroneously includes an adjacent apple. This highlights remaining challenges in precise spatial localization and the conversion of textual reasoning into geometric masks.

By integrating our ablation findings with these new output failure analyses, we will offer readers a more honest and comprehensive evaluation of SAM‑R1. Thank you again for this valuable recommendation.