ReGRAF: Training free Prompt Refinement via Gradient Flow for Segmentation

Thank you for the insightful comments and suggestions. Your thorough review and valuable feedback have guided us in enhancing both the precision and presentation of our work. In addressing your questions, we carefully conducted additional experiments, which allowed us to further validate and refine our findings. We trust these efforts have strengthened the overall rigor of our study.

• The effectiveness of ReGRAF? To better demonstrate the effectiveness of ReGRAF, we presented average mIoU segmentation results of each dataset as well as the oracle results in the appendix D. The oracle results are obtained by selecting the most accurate segmentation masks across the maximum iterations (T=5) based on a comparison with the ground truth, rather than simply choosing the mask from the specific iteration. This approach closely resembles real-world usage of ReGRAF, as it allows selecting the most suitable segmentation result from different iterations.
  As time was limited, we first performed the oracle test on PerSAM/HQ-PerSAM. We will update this oracle results for the other baselines (i.e., Matcher, HQ-Matcher) as soon as possible.
  Our results demonstrate that while the optimal iteration may vary across samples, ReGRAF consistently refines segmentation masks effectively, yielding significant performance improvements in practical applications. (For example, PerSAM achieves at least a 4.26% improvement in segmentation accuracy, with a maximum improvement of 15.62%.)

• Additional experiment result for fine-grained segmentation dataset? To show the performance of ReGRAF on the segmentation of fine-grained datasets, we validated our method on the DIS5K dataset ([3]). We chose DIS5K over the other datasets suggested by the reviewer for the following reasons:

  • The data used in [1] is not publicly available,
  • We argue that datasets with less distinct separation between foreground and background are better suited for evaluating more challenging cases,
  • DIS5K is publicly available, features poor separation between foreground and background, and includes class metadata for few-shot guidance.

  We recorded the segmentation results without further hyper-parameter tuning (We applied the same hyperparameter settings for ReGRAF as those used in part segmentation.) Each element from iter1 to iter5, as well as the oracle, represents the mIOU gain.
  Although the progression of average mIoU gains suggests challenges in selecting the optimal maximum iterations T, the oracle results demonstrate that our method effectively enhances segmentation quality, even for this difficult task (This table is also attached in Appendix E).

  Baselines	Baseline	iter 1	iter 2	iter 3	iter 4	iter 5	Oracle
  PerSAM-F	27.47	0.00	0.24	0.33	0.02	-0.24	6.33
  HQ-PerSAM-F	52.62	0.00	-0.12	0.06	0.17	0.11	5.55
  Matcher	46.77	-0.29	0.14	-0.19	-0.23	0.14	10.87
  HQ-Matcher	58.39	0.09	0.58	-0.06	0.18	0.52	10.65

We sincerely appreciate the insightful feedback. Your thoughtful comments and questions have played a key role in improving the clarity and quality of our work. Through our revisions, we aimed to carefully address your suggestions and further strengthen the presentation of our study.

• Comparing HQ-Matcher and Matcher results on COCO-20$^i$ dataset? HQ-SAM was fine-tuned using prompts (visual clues) such as bounding boxes surrounding the true mask and random points within it. This fine-tuning likely introduced a bias toward the labels in the training dataset. Additionally, while Matcher's auto-prompting was specifically tailored for SAM, HQ-Matcher incorporated HQ-SAM into ReGRAF without further tuning to emphasize the broad applicability of our approach. As a result, HQ-SAM’s HQ features may have been derived from incomplete prompts, leading to an out-of-distribution (OOD) scenario that likely constrained HQ-Matcher's improvements over Matcher.
• Additional explanation of equation (6), (12) and Theorem 1. This is because the density ratio in Equation (12) is derived from the approximation introduced in Equation (6). Specifically, the exact density ratio is represented as $\rho_t / \mu$, where $t$ is intractable. To address this, in Equation (6), we approximated the density ratio as $\rho_0 / \mu$ (e.g. $\rho_t \approx \rho_0$ ), which is valid for small values of $t$ but not for large $t$. However, this approximation raises questions about the stability of ReGRAF. To resolve this, we proved that, given the near-optimal latent vector of the visual foundation model, ReGRAF converges at an exponential rate (Theorem 1). This proof ensures the stability of our algorithm, liberating it from reliance on the initial approximation.
• Better evaluation criteria? In the appendix D, we presented average mIoU segmentation results of each dataset as well as the oracle results. The oracle results are obtained by selecting the most accurate segmentation masks across the maximum iterations ($T=5$) based on a comparison with the ground truth, rather than simply choosing the mask from the final iteration. This approach demonstrates practical use of ReGRAF, as it allows selecting the most suitable segmentation result from different iterations.
  As time was limited, we first performed the oracle test on PerSAM/HQ-PerSAM. We will update this oracle results for the other baselines (i.e., Matcher, HQ-Matcher) as soon as possible.
  Our results demonstrate that while the optimal iteration may vary across samples, ReGRAF consistently refines segmentation masks effectively, yielding significant performance improvements in practical applications. (For example, PerSAM achieves at least a 4.26% improvement in segmentation accuracy, with a maximum improvement of 15.62%.)
  Furthermore, we further validated our method on DIS5K dataset [1]. DIS5K is specifically designed for segmentation tasks where fine-grained objects are difficult to distinguish and require more accurate segmentation masks. We recorded the segmentation results without further hyper-parameter tuning (We applied the same hyperparameter settings for ReGRAF as those used in part segmentation.) Each element from iter1 to iter5, as well as the oracle, represents the mIOU gain.
  Although the progression of average mIoU gains suggests challenges in selecting the optimal maximum iterations T, the oracle results demonstrate that our method effectively enhances segmentation quality, even for this difficult task (This table is attached in Appendix E). | Baselines | Baseline | iter 1 | iter 2 | iter 3 | iter 4 | iter 5 | Oracle | |-------------|----------|----------|----------|----------|----------|----------|-----------| | PerSAM-F | 27.47 | 0.00 | 0.24 | 0.33 | 0.02 | -0.24 | 6.33 | | HQ-PerSAM-F | 52.62 | 0.00 | -0.12 | 0.06 | 0.17 | 0.11 | 5.55 | | Matcher | 46.77 | -0.29 | 0.14 | -0.19 | -0.23 | 0.14 | 10.87 | | HQ-Matcher | 58.39 | 0.09 | 0.58 | -0.06 | 0.18 | 0.52 | 10.65 |

[1] Qin, X., Dai, H., Hu, X., Fan, D. P., Shao, L., & Van Gool, L. (2022, October). Highly accurate dichotomous image segmentation. In European Conference on Computer Vision (pp. 38-56). Cham: Springer Nature Switzerland.

We are grateful for the detailed and valuable feedback from the reviewers. Your insightful questions and suggestions have encouraged us to refine our work and strengthen its contributions. We trust that our revisions adequately address your concerns and further enhance the quality of this study.

• More clear motivation and objective of ReGRAF? Our method was not meticulously designed to address the issues that you suggested, but to exploit the gradient flow to draw out the hidden potential of the Visual Foundation Model without incurring additional expenses in terms of training, dataset, parameter, etc. ReGRAF's value in segmentation refinement lies in its ability to improve segmentation quality as a training-free method. It requires minimal additional computation time (as highlighted in Table 3 of Section 4.2.3) and only minor code modifications for implementation. This makes ReGRAF broadly applicable to auto-prompting frameworks.
• Failure cases? Yes, in some cases, performance declines after refinement. Our method is based on the strong assumption that the Vision Foundation Model (VFM) obtains latent vectors for query images that are already near-optimal. However, in real-world applications (E.g., out-of-distribution settings) the VFM sometimes fails to capture a near-optimal latent vector. In these scenarios, refinement may lead to poorer segmentation results (Please refer to the appendix C for the failure cases.).
• Sensitivity of iteration times? We undertook a sensitivity analysis on the step size $\eta$ and the iterations $T$ in the appendix F. We randomly sampled 100 examples from the set-aside dataset (derived from the COCO-20$^i$ training dataset) and we evaluated the mIoU over iterations, testing up to 30 iterations with step sizes of $10^{-2}, 10^{-3}, 10^{-4}$ and $10^{-5}$). Figure 16 in the appendix F shows a moving average with a window size of 2 to smooth fluctuations and better reveal overall trends.
  The results emphasize the critical role of the step size ($\eta$) in segmentation performance and stability. $\eta=10^{-3}$ consistently delivered the best results, balancing rapid convergence and long-term stability. In contrast, $\eta=10^{-2}$ improved quickly but degraded over time, while smaller values ($\eta=10^{-2}, \eta=10^{-5}$) showed slower convergence and higher variability.
  This underscores the importance of tuning the step size, however ReGRAF can be easily tuned and applicable to various scenarios. As shown in Table 3, incorporating ReGRAF into the baselines introduces minimal additional running time. Furthermore, Table 4 and Table 5 demonstrate that even with hyperparameters tuned on 10 randomly sampled instances from COCO-20$^i$, ReGRAF effectively improves the baseline's segmentation performance across diverse datasets.

We deeply appreciate the reviewers for their careful evaluation and valuable feedback. Your constructive suggestions and thoughtful questions have significantly enhanced the clarity and quality of our work. We sincerely hope our detailed responses and improvements meet your expectations and contribute positively to your overall assessment.

• Sensitivity of hyperparameters? Since we assumed that the visual foundation model yields a near-optimal segmentation, we did not take much time on tuning the hyper-parameters, using a single set-a-side dataset for the entire experimental settings (10 random samples from COCO-20i set-aside dataset). Depending on the purpose of refinement, we employed different step size . Specifically, we used $10^{-3}$ when understanding overall visual concept would be important, e.g., semantic segmentation, and $10^{-4}$ (which is relatively small) when the segmentation would require capturing fine details in a scene, e.g., part segmentation.
• Additional explanation of equation (6), (12) and Theorem 1. This is because the density ratio in Equation (12) is derived from the approximation introduced in Equation (6). Specifically, the exact density ratio is represented as $\rho_t / \mu$, where $t$ is intractable. To address this, in Equation (6), we approximated the density ratio as $\rho_0 / \mu$ (e.g. $\rho_t \approx \rho_0$ ), which is valid for small values of $t$ but not for large $t$. However, this approximation raises questions about the stability of ReGRAF. To resolve this, we proved that, given the near-optimal latent vector of the visual foundation model, ReGRAF converges at an exponential rate (Theorem 1). This proof ensures the stability of our algorithm, liberating it from reliance on the initial approximation.
• The trends of results? For segmenting small objects or fine details, using a smaller step size is advantageous to prevent abrupt changes in the prompt. However, when the visual semantics between the reference image and the target image differ significantly (4th rows in the appendix C.), prompt refinement becomes challenging, potentially leading to performance degradation.
  In addition, when the visual instructions in the reference images are difficult to interpret— such as when distinguishing between specific parts of a bicycle is challenging, or when the reference depicts a general tray while the segmentation target is the tray's edge—ReGRAF struggles to refine the prompts effectively (2nd, 3rd, and 5th rows in the appendix C).

We presented the complete updated oracle results in Appendix D.

Dear Reviewers,

We sincerely hope that our rebuttal has addressed your concerns and clarified our work based on your feedback.

We greatly value your comments, and if you have any additional questions or need further clarification, please let us know before the PDF revision period ends (Nov. 27 23:59 (AoE)). We look forward to contributing to the ICLR and the broader ML/CV community through the publication of this paper.