Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation

We thank the reviewer for valuable feedback and for appreciating the idea of our method, strong justification of each component of our framework and extensiveness of our experiments. Following are the responses regarding your concerns.

Inconsistent Performance: In scenarios where hallucinations are highly inaccurate, the performance of ProMaC could degrade significantly.

• Hallucinations are used solely to gather task-relevant information from the image and are effectively filtered by the Visual Contrastive Reasoning module and the subsequent Mask Semantic Alignment model to eliminate the negative impact of task-irrelevant information. This ensures the generated instance-specific prompts are accurate and not influenced by irrelevant hallucinations, maintaining robust performance even when hallucinations are unreliable.

• To elaborate, as stated in Lines 143-150, by dividing the input image into patches at various scales, MLLM induces hallucinations based on varying object visibility. These hallucinations use prior knowledge to explore connections between the image data and the associated task, gathering task-relevant information. Simultaneously, the original image is also fed directly into the MLLM to obtain task-relevant candidates without relying on hallucinations. This ensures that even when hallucinations are highly inaccurate, the collected information still includes relatively reliable content.

• Visual Contrastive Reasoning module then selectively reduces irrelevant semantics to remove irrelevant influences while benefiting from relevant insights obtained from pre-trained knowledge (hallucinations). When hallucinations are completely inaccurate, the information derived from them is discarded. Additionally, the Mask Semantic Alignment module further ensures that the generated masks align with the task semantics.

Dependence on MLLM Quality.

As we explained in the limitations section, the quality of MLLMs affects our model's generalization ability, which is a potential direction for our future research.

Computational overhead and implementation challenges.

• Computational Overhead: Traditional methods often require multiple GPUs for extensive training, leading to higher computational and memory demands. In contrast, as we discussed in Line 281, our method performs iterative test-time adaptation on a single 40G A100 GPU without the need for training, which is computationally efficient. Compared to GenSAM [19], which also uses test-time adaptation, our method achieves better performance with fewer iterations. As shown in Tab. 6(a), ProMaC reaches higher performance by the second iteration than GenSAM does after six iterations, further demonstrating the efficiency of our approach.

• Implementation Challenges: Although our approach uses multiple modules, all module parameters are fixed and do not require training, making it easier to reproduce. In Tab. 7, we conducted three sets of experiments randomly and calculated the mean and variance. The results show that our training-free method exhibits very low variance, further proving its robustness. Additionally, we provide implementation details in the PyTorch Implementation Details section (Line 271-281) of the main text and Line 551-581 of the appendix. Our code is also available in the supplementary materials, ensuring that our method is easy to understand and implement.

The initial setup still requires careful selection of task-generic prompts. Ensuring that these prompts are effective across diverse tasks without additional fine-tuning could be challenging.

• Task-generic prompt $P_g$ is decided by the task name rather than our manual selection, and has minimal impact on experimental results. Using the camouflaged animal detection task as an example, we selected "camouflaged animal" as $P_g$ because the task is named "camouflaged animal detection". Similarly, "polyp" is selected as $P_g$ because the corresponding task is named as "polyp detection".
• To explore the impact of different $P_g$ prompts, we used ChatGPT to suggest two synonyms for "camouflaged animal": "hidden animal" and "disguised animal" as possible $P_g$ candidates. We evaluated the effect of different $P_g$, as shown in Tab. 2 of the attached PDF file. Although different $P_g$ prompts cause slight fluctuations in the outcomes, the overall performance remains stable. This demonstrates that as long as $P_g$ effectively describes the task semantics, our method can achieve relatively consistent segmentation performance.

Transferability: The approach might not generalize well to entirely new tasks or domains where the hallucinations generated by MLLMs do not align well with the actual task requirements.

• Because our ProMaC can be easily applied to different MLLMs, we can ensure robust performance across various tasks and domains by selecting the most appropriate MLLM for each specific task.

• PoMaC is designed to be flexible and can be adapted to different MLLMs that are more suitable for specific tasks or domains. By selecting an MLLM that has been pre-trained on data relevant to the target domain, we can improve alignment with task requirements and enhance performance.

• For example, as shown in Tab. 1 in attached pdf, in medical imaging segmentation, ProMaC based on LLaVA performed modestly. However, when we used LLaVA-Med [1], which has stronger medical characteristics, as the base, ProMaC significantly outperformed both LLaVA-Med combined with SAM and LLaVA-based ProMaC. This demonstrates that by applying our ProMaC method to a more suitable base MLLM, we can achieve better performance, thereby ensuring transferability across various tasks.

[1] Li C, Wong C, Zhang S, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day[J]. Advances in Neural Information Processing Systems, 2024, 36.

We thank the reviewer for the positive evaluation of our work and for acknowledging the originality of the proposed method as well as its significance for future research. We are glad that the reviewer finds the results impressive and the idea of this paper is quite interesting.

It would be better to involve more examples from different segmentation tasks to illustrate the motivation comprehensively, especially general segmentation like COCO and Pascal VOC. I am curious whether this phenomenon would still occur when the objects are more visible.

We have also included additional samples for the polyp dataset and general object dataset in the attached PDF (Fig. 1) to illustrate our motivation. It is evident that our method, although specifically designed for segmentation tasks where visual cues are weak or ambiguous (e.g., hidden/camouflaged foreground objects in visually similar backgrounds), can also utilize hallucinations as prior knowledge to identify potential categories in complex images in general task, such as those in the VOC dataset (Fig. 1(b) in the attached PDF). However, when visual cues are strong, obvious, or distinct, the benefit of scene understanding through exploring hallucinations becomes less critical. In such situations, there is less likelihood of confusion between foreground and background, and less need for reasoning about all plausible backgrounds versus foregrounds, in such cases, our method may not provide significant improvements. (Fig. 1(c) in the PDF).

Some terms are unclear and need explanation. For example, I’m not very clear about the meaning of the term ‘prompt’ used in the paper. Does it refer to text prompts or spatial prompts like boxes or points? From the method section, it seems that P_B and P_A are text prompts, and the instance-specific prompts refer to bounding boxes? Also, what does query P refer to? If its meaning differs from P_g, P_B, and P_A, it would be better to clarify this at the beginning of the method section. Since the method is based on prior work [19] and some terms are borrowed directly from [19], it is necessary to include a preliminary section to explain the background and terms. This would make the method section easier to follow.

The term 'prompt' used in the paper includes both text prompts and spatial prompts (bounding boxes and points).

• As explained in Line 152-155, in each task, the input text prompt consists of two parts: prompt for bounding box prediction ($P_B$) and prompt for class prediction ($P_A$):

1. Prompt for bounding box prediction ($P_B$) instructs: "This image is from the $P_g$ detection task, output the bounding box of the $P_g$."

2. Prompt for class prediction ($P_A$) states: "Output the name of the $P_g$ and its environment in one word."

Here, $P_g$ is a test-generic prompt that is consistent across tasks. For example, for camouflaged animal detection task, $P_g$ is "camouflaged animal", for polyp detection task, $P_g$ is "polyp".

• $P_B$ and $P_A$ are fed into MLLM to infer instance-specific spatial prompt bounding box and instance-specific text prompt class name. This class name is then mapped into anther instance-specific spatial point prompts using spatial CLIP. These instance-specific spatial prompts (both point prompts and bounding box prompts) are subsequently fed into SAM to guide the segmentation process.

• We refer to the queries inputted into the MLLM as P.

• The terms borrowed from [19] will be explained in more detail in the final version.

Question about the proposed Multi-scale Chain of Thought Prompting. From my understanding, chain-of-thought prompting aims to achieve complex reasoning capabilities through multiple intermediate reasoning steps. Could the author please explain how this idea is reflected in the proposed method?

As we explained in Line 143-190, our MCoT uses multiple intermediate reasoning steps on various chains to infer accurate instance-specific prompts from task-generic prompts.

• First, as we mentioned in Line 146-156, each chain includes two intermediate reasoning steps: 1. MLLM first captions the image and 2. Based on this captions, MLLM infers the names and backgrounds of task-relevant objects. Without this initial image captioning step, the inferred instance-specific prompts would be inaccurate.
• Second, The input image scales differ across various chains, leading to variations in the positions and completeness of task-relevant objects in each chain. This difference results in distinct intermediate reasoning paths, ensuring diverse information extraction across chains. This variation allows the MLLM to fully leverage prior knowledge from hallucinations to uncover potential task-relevant information.
• Finally, the visual contrastive reasoning module aggregates effective information from different chains, eliminating the influence of task-irrelevant hallucinations. This multi-chain, multi-step reasoning process ultimately produces accurate instance-specific prompts.

We thank the reviewer for the positive evaluation of our work and for acknowledging the value and insight of the proposed method.

Could they provide comparison results between the predictions of the ProMaC method and the ground truth across iterations? This would offer a more visual representation of the method's performance.

On Page 16, in Fig. 6, we provide visualizations of segmentation results across different tasks as iterations progress. These results demonstrate that our method shows consistent performance improvement with each iteration across various tasks. Additionally, the quantitative experimental results in Tab. 6 further corroborate this finding, illustrating the incremental enhancement in performance as the iterations proceed.

The proposed method's computational and memory requirements aren't clearly discussed, which might be significantly higher due to the iterative nature of the Prompt-Mask Cycle and multiscale chain of thought prompting. This could limit its applicability in resource-constrained environments.

Traditional methods often require multiple GPUs for extensive training, leading to higher computational and memory demands. In contrast, as we discussed in Line 281, our method performs iterative test-time adaptation on a single 40G A100 GPU without the need for training, which is computationally efficient. Compared to GenSAM, which also uses test-time adaptation, our method achieves better performance with fewer iterations. As shown in Tab. 6(a), ProMaC reaches higher performance by the second iteration than GenSAM does after six iterations, further demonstrating the efficiency of our approach.

Dependence on Initial Prompts: While the iterative refinement process is beneficial, the initial quality of prompts still plays a crucial role. Poor initial prompts might lead to suboptimal starting points, impacting the overall effectiveness of the adaptation process. I would like to see some discussion on this.

The format of the initial prompt is fixed, with the only modifiable part being the task-generic prompt $P_g$. However, $P_g$ has minimal impact on experimental results. Using the camouflaged animal detection task as an example, we selected "camouflaged animal" as $P_g$ because the task is named "camouflaged animal detection". To explore the impact of different $P_g$ prompts, we used ChatGPT to suggest two synonyms for "camouflaged animal": "hidden animal" and "disguised animal" as possible $P_g$ candidates. We evaluated the effect of different $P_g$ on the results, as shown in Tab. 2 of the attached PDF file. Although different $P_g$ prompts cause slight fluctuations in the outcomes, the overall performance remains stable. This demonstrates that as long as $P_g$ effectively describes the task semantics, our method can achieve relatively consistent segmentation performance.

The authors attribute this to the inherent generalization limitations of LLaVA. Recently developed methods (e.g., GPT4o) have shown better generalization capabilities in medical imaging. Could the authors conduct experiments using these more robust MLLMs on specific sub-tasks to substantiate this perspective?

In Tab. 8 on Page 16, we compare the performance of our method with different baseline methods using various MLLMs across different tasks. It is evident from the comparisons among the different baselines that the generalization capability of an MLLM directly impacts its performance on different tasks, which supports our explain. Furthermore, we evaluate an MLLM model based on LLaVA-Med [1] on the Polyp Image Segmentation task, as shown in Tab.1 in attached pdf file, which also support this claim.

[1] Li C, Wong C, Zhang S, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day[J]. Advances in Neural Information Processing Systems, 2024, 36.

We thank the reviewer for the careful reading and insightful comments. Following are the responses regarding your concerns.

The authors should clearly explain how this paper utilizes the hallucinations.

As we explained in Line 143-150, we utilize hallucination to bootstrap a scene-understanding of each test image. By dividing the input image into patches at various scales, the MLLM induces hallucinations based on varying object visibility in diverse patches. These hallucinations use prior knowledge to explore connections between the image data and the associated task, gathering task-relevant information before eliminating irrelevant semantics in our reasoning process. This is critical when visual cue is weak / ambiguous for segmentation. We then selectively reduce irrelevant semantics in order to remove irrelevant influence whilst benefit from relevant influence obtained from pre-trained knowledge (aka hallucination). This is in contrast to existing methods which all aim to remove hallucination blindly regardless its usefulness in helping scene-understanding in an image, therefore improving segmentation when visual cues are weak / ambiguous.

The iterative process and the integration of various techniques (e.g., MLLM, SAM, CLIP, and inpainting) add complexity, which may hinder the ease of implementation and understanding.

• Traditional methods often require multiple GPUs for extensive training, leading to higher computational and memory demands. In contrast, as we discussed in Line 281, our method performs iterative test-time adaptation on a single 40G A100 GPU training-free, meaning all module parameters are fixed from the start. This eliminates the need for training and parameter tuning, and therefore reducing implementation complexity.

• We also provide implementation details in the PyTorch Implementation Details section (Line 271-281) of the main text and Line 551-581 of the appendix. Our code is available in the supplementary materials, ensuring our method is easy to understand and implement.

Compared with GenSAM [19]， What is the main innovation of the Mask Generator?

• As we explained in Line 212-213, the key innovation of our mask generator is the proposed Mask Semantic Alignment (MSA) module, which ensures that the generated masks align with task semantics—an achievement that GenSAM cannot match.

• In GenSAM, the mask generator uses spatial CLIP to convert instance-specific text prompts into positive/negative point prompts and directly feeds them into SAM to obtain masks. However, since SAM is trained without category labels, it lacks label prediction capabilities. This leads to masks that do not meet task requirements when the point prompts fed into SAM are not accurate.

• Our MSA module addresses this limitation by using similarity-weighted masks aggregation to ensure that the output masks are aligned with the task semantics. The effectiveness of our MSA module is demonstrated in the last two rows of the ablation experiment in Tab. 4 in original paper.

At the first iteration, since masks have not yet been generated, what are the visual markers in this process?

As we explained in Line 199-201, in the first iteration, Multi-scale Chain of Thought Prompting (MCoT) infers predicted bounding boxes for multiple different patches. We use the union of these bounding boxes as the visual marker for the first iteration to guide the generation of the contrastive sample $X^{'}$.

From my perspective, this task should be evaluated more on general datasets, such as COCO, to demonstrate the method's generalizability rather than on specific domain tasks (e.g., camouflage). Some comparison methods in this paper, such as X-Decoder and SEEM, are all generalized methods. Comparing with them, I believe, is unreasonable. I hope the author can discuss this issue with me.

• In Tab. 3(b), we have conducted experiments on three general datasets: COCO Object, PASCAL VOC, and Pascal Context, showing that our method achieves good performance on general segmentation tasks comparing to other unsupervised/weakly supervised methods.

• As for generalized methods like X-Decoder and SEEM, it is ideal to fairly compare X-Decoder and SEEM on both general and specific tasks. However, these two methods are trained on general segmentation datasets with semantic labels, while our pre-trained model only trains with image mask pairs without semantic labels. Therefore, it is unfair to directly compare ProMaC with these methods on general segmentation tasks. Meanwhile, since neither these methods nor our ProMaC have been trained on specific domain datasets, comparing them on these tasks is fair.

List the genetic task prompts used for different tasks.

As we explained in Line 273-276, the text generic prompt $P_g$ for the camouflaged animal detection task is “camouflaged animal”. For the two sub-tasks in medical imaging, the text generic prompts are “polyp” for polyp image segmentation and “skin lesion” for skin lesion segmentation. For the transparent object detection task, the text generic prompt is “glass”.