Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection

Thanks for your detailed review! We will organize our feedback very soon!!

(1) Response to weakness2: Thank you for highlighting this potential ambiguity. We agree that "medical detection" may cause confusion, particularly for readers from a medical science background. To clarify, we will replace this term with more precise phrases, such as "disease detection" or "object detection in medical images," in the camera-ready version to ensure clarity and avoid misinterpretation.

(2) Response to weakness3: We apologize for the oversight, which occurred during manuscript preparation. The correct sentence should read: "$B^{i}$ denotes the representation of the prompt bank at the $i^{\text{th}}$ layer." This has been corrected in the revised manuscript.

(3) Response to weakness4: We appreciate your thoughtful suggestion. While the equations in their current form explicitly describe the mutual selection process for clarity, we agree that simplifying them using argmax or argsort would improve readability. To minimize any potential misunderstanding or confusion among other reviewers during the rebuttal phase, we will retain the current equations for now and implement these updates in the camera-ready version.

(4) Response to weakness7: We fully agree that pure zero-shot detection settings (Fig. 2(a) & (b)) have limited practical utility in clinical environments, as clinicians require high levels of accuracy and reliability. However, this setting serves as an important experimental setup to evaluate the effectiveness of our proposed methods in incorporating prompts to enhance zero-shot detection performance.

To address practical clinical needs, we also introduce the zero-shot enhancement setup (Fig. 2(c) & (d)), where StructuralGLIP incorporates additional appearance attributes of the target object using prompts after the base VLM model has been fine-tuned on domain-specific medical data. This hybrid approach demonstrates its practical value by achieving significant performance improvements over standard supervised models such as RetinaNet and Faster R-CNN. This setup has been thoroughly discussed in Sec. 4.3.

Thank you for this insightful comment. Yes, we use BLIP as our Vision Question Answering (VQA) model, as mentioned in Line 367-371. We appreciate your suggestion to further validate the robustness of our approach by exploring the impact of different VQA models.

To address this, we conducted an ablation study to compare the performance of various VQA models, including BLIP, LLaVa-7b, and Qwen2-VL-7b. These models were used to generate prompts for both instance-level and category-level detection tasks on the CVC-300 dataset. The results are summarized in Table S6:

Table S5: Ablation on the choice of VQA model with CVC-300 dataset

As shown in Table S6, our method demonstrates robust performance across different VQA models. The results indicate that the selection of the VQA model has a relatively small impact on the final detection performance, confirming the generalizability of our approach.

Thank you for your thoughtful assessment of our methodology. We appreciate your detailed summary of our approach and the recognition of its differences from standard contextual prompt methods. While we acknowledge that the operations themselves may not appear fundamentally novel, we believe our method introduces key insights and contributions that address overlooked challenges in the field, setting it apart from prior work.

1. Overcoming the domain adaptation limitations of prompt-based methods: Existing prompt-based methods typically input both the target object and prompts as equivalent linguistic contexts into the VLM's language encoder (Fig. 1(a)). However, this approach fundamentally alters the linguistic representation of the target object, making the combined representation a mix of the target and the prompt. This introduces significant challenges during domain fine-tuning:

• During fine-tuning, only the target name is used as linguistic input, resulting in a linguistic representation optimized solely for the target.
• During inference, the introduction of prompts causes a distribution shift in the linguistic representation, leading to degraded performance.

As you correctly pointed out in Weakness 7, zero-shot detection methods alone may not be reliable in clinical scenarios, and domain-specific fine-tuning is often essential. Unfortunately, the aforementioned limitations of prompt-based methods significantly hinder their practical application in fine-tuned VLMs.

StructuralGLIP effectively addresses this challenge by isolating prompt encoding in a separate auxiliary branch. The linguistic knowledge from prompts is incorporated into the main branch via residual connections, allowing the target object representation to remain consistent. This design ensures that our method is compatible with fine-tuned VLMs without introducing distribution shifts. As demonstrated in Sec. 4.3 (StructuralGLIP facilitates further improvement on fine-tuned models), our method achieves significant performance improvements even when applied to fine-tuned models, making it both robust and practical.

2. Addressing the overlooked impact of noisy prompts:

Most existing works [1, 2, 3, 4] assume that all generated prompts are beneficial and treat prompts as static, context-expanding knowledge to enhance vision-language model (VLM) performance. However, this assumption neglects the potential negative impact of noisy or irrelevant prompts, which can misguide the detection process, especially in medical applications. Our approach is unique in addressing this challenge by:

• Modeling prompts as a latent knowledge bank that decouples prompt encoding from the target representation-
• Introducing a mutual selection mechanism that dynamically filters and selects only the most relevant prompts for each input.

To validate the robustness of the mutual selection mechanism, we conducted additional experiments on the BCCD dataset, which includes red blood cells, white blood cells, and platelets. We introduced noisy prompts by mixing attributes from unrelated categories and evaluated StructuralGLIP. The results are summarized in Tables S4 . Here (X, X) means we only employ prompts for X when detect X. (X, Y) means we mix the prompts for Y when detection X (X $\neq$ Y).

Table S4: StructuralGLIP’s AP@50 performance with noisy knowledge under zero-shot detection setup.

Table S4-2: GLIP’s AP@50 performance with noisy knowledge under zero-shot detection setup.

Our findings reveal that StructuralGLIP maintains its performance when noisy knowledge of other category is added to the prompt. This robustness arises from the mutual selection mechanism, which effectively filters out irrelevant information and retrieves only the most relevant prompts for the input image.

Therefore, we believe StructuralGLIP provides novel perspectives and solutions to challenges in prompt-enhanced vision-language models. By addressing issues such as domain adaptation, distribution shifts, and noisy prompts, our work represents a significant step forward in the practical application of prompt-based methods for medical imaging.

References

[1] Zero-Shot Nuclei Detection via Visual-Language Pre-trained Models. MICCAI.

[2] Medical image understanding with pretrained vision-language models: A comprehensive study. ICLR, 2023

[3] Multiple prompt fusion for zero-shot lesion detection using vision-language models. MICCAI, 2023

[4] Zero-Shot Nuclei Detection via Visual-Language Pre-trained Models, MICCAI, 2023

I largely understand your new concerns regarding the comparison and novelty. As for novelty, we think there are still some misunderstanding and we will soon prepare our clarification. And for VQA comparison, we will test our method and AutoPrompter on ClinicDB and CVC-300 with different VQA model very soon.

Response to Weakness 6

We have now provided results comparing StructuralGLIP and AutoPrompter across both a high-performance dataset (CVC-300) and a lower-performance dataset (ClinicalDB) using different VQA models (Table S8-1 & Table S8-2). As can be seen, StructuralGLIP consistently outperforms AutoPrompter in most cases, even on the lower-performance dataset (ClinicalDB).

Table S8-1: StructuralGLIP's AP@50 Performance with instance-level prompt generated with different VQA models.

Table S8-2: AutoPrompter's AP@50 Performance with instance-level prompt generated with different VQA models.

Regarding your request to compare the category-level prompt performance, we must point out that existing baseline methods do not currently support this type of prompt (see Line 389-415). As such, conducting an ablation study for VQA with category-level prompts under these circumstances would lead to an unfair comparison. Instead, we have provided additional evidence in Table S9 to further support our argument.

Table S9: AP Performance comparison of different methods when applied category-level prompt (BLIP + GPT4).

As shown in Table S9, StructuralGLIP gains an obvious improvement when using category-level prompt. This is due to we embed prompt as knowledge bank with auxiliary bank, and introduce a mutual selection mechanism to select prompts relevant to the current sample during inference. A more detailed explanation for this can be found in Line 52-76, and Line 389-415.

Response to Weakness 7

As shown in Fig.2 and Line 173, zero-shot enhancement refers to the process in which we apply prompts to a model that has already been fine-tuned on the target task. Additionally, during fine-tuning, only the target name (e.g., the label of the target object or class) is fed into the text encoder of the VLM, without any prompt included. We hope this clears up any confusion regarding the difference between zero-shot enhancement and model fine-tuning.

Thanks for your reply. It seems there may have been some misunderstandings, and I hope to clarify these aspects.

1. On Novelty and Innovation

In our last reply, we only acknowledge that our design is based on ``standard structures'' like cross attention, and would like to emphasize core innovation lies in the fine-grained alignment of visual and textual information through a structural representation (which has been highlighted in our title). This is a critical aspect of our approach and is central to its ability:

1. Robustness to noisy prompts
2. Effective prompt enhancement on fine-tuned models.

This fine-grained structural representation is essential because it addresses key challenges that have not been explored in previous prompt-enhanced works, particularly in the context of domain adaptation and prompt noise management. Thus, we believe our approach makes a novel and valuable contribution to the field.

2. Domain Shift and Residual Connection

There seems to be a misunderstanding. Our discussion of domain shift was meant to explain why StructuralGLIP is still effective when it is applied to fine-tuned GLIP. In our experiments, we observed that previous prompt-enhanced models, such as AutoPrompter and MIU-VL, do not consistently improve performance across all benchmarks after fine-tuning (see Table 4). The key reason for this is that during fine-tuning, only the [target-name] is fed into the text encoder to optimize performance. However, during prompt enhancement, both “[target-name, prompt]” are fed into the fine-tuned GLIP’s text encoder. This inconsistency in input format during fine-tuning and prompt-enhanced stages can lead to performance degradation, as the model is exposed to a format it was not trained on. We hope this clarification helps in better understanding our approach.

3. Prompt Selection and Experimental Design

We would like to clarify the setup of our experiments. We do not cross-use prompts between different categories, as you mentioned. Instead, we mix prompts from different categories to evaluate the impact of prompt noise on model performance. Specifically, when detecting Red Blood Cells, we use a mixture of prompts for both Red Blood Cells and White Blood Cells as noisy prompt, rather than using White Blood Cells' prompts directly for Red Blood Cells detection.

This experimental setup is crucial in demonstrating StructuralGLIP's robustness when it is exposed to noisy or mixed prompts. As shown in Table S4 and Table S4-2, the performance of StructuralGLIP remains relatively stable even when noise is introduced, while GLIP suffers a marked performance decline. This highlights the effectiveness of our mutual selection mechanism in filtering out irrelevant prompt information.

Table S4: StructuralGLIP’s AP@50 Performance with Noisy Knowledge under Zero-Shot Detection Setup. Here (X, X) means we only employ prompts for X when detect X. (X, Y) means we concat prompts for both X and Y when detection X (X ≠ Y).

Table S4-2: GLIP’s AP@50 Performance with Noisy Knowledge under Zero-Shot Detection Setup. Here (X, X) means we only employ prompts for X when detect X. (X, Y) means we concat prompts for both X and Y when detection X (X ≠ Y).

We hope this clears up the misunderstanding and further demonstrates the effectiveness of our approach in managing noisy prompts.

Actually, we have provided the analysis of the selected prompts of StructuralGLIP in Appendix E. In our previous response, we did not directly address this question because we felt it was more important to first resolve the key misunderstandings. With the most primary misunderstanding coming out, we further provide the prompts selected by StructuralGLIP as the supplement of Appendix E. To make our results more representative, we have included statistics on the words selected by StructuralGLIP at different layers across the ClinicDB and BCCD datasets. Additionally, please note that if a word is preceded by “##,” it indicates that the tokenizer has split the word into tokens.

Table S10-1 The selected prompts of StructuralGLIP with different layers on ClinicalDB datasets. We have categorized the prompts into color, shape, texture, location 4 types.

Table S10-2 The selected prompts of StructuralGLIP with different layers on BCCD datasets. We have categorized the prompts into color, shape, texture, location 4 types.

We hope that after reading our previous explanation to Weakness 1, most of your concerns will be addressed. However, to provide a more direct and detailed response, we would like to elaborate further.

Response to Question 1

There is still some misunderstanding regarding this setting. All the prompt-based methods discussed in our paper, including StructuralGLIP, are built upon GLIP. In the case of the CVC-300 dataset for zero-shot enhancement experiments, we first fine-tune the GLIP model using the labeled data from the dataset. During fine-tuning, no prompts will be employed. After this fine-tuning, we then built various prompt-based methods for the experiment, as described in Line 173. We will make this point clearer in future revisions. Therefore, the zero-shot enhancement setting means building the prompt methods with a fine-tuned GLIP. This GLIP is fine-tuned on the target dataset with the labeled data without prompt. As the prompt methods may improve the fine-tuned GLIP's performance, we call it "enhancement", and the building of prompt methods does not require further supervised labels, we call it "zero-shot".

Response to Question 2

We would like to emphasize that StructuralGLIP is indeed training-free. Specifically, the language encoder in the auxiliary branch shares its model parameters with the language encoder in the main branch, meaning there is no additional training or fine-tuning involved for the auxiliary branch (see Line 210). To clarify, in the zero-shot enhancement setting, only the GLIP model is fine-tuned on the target dataset (e.g., CVC-300). After this fine-tuning, we introduce the auxiliary branch with its own text encoder to process the prompt. However, the model parameters for the auxiliary branch’s text encoder are directly shared with those of the main branch's GLIP text encoder, so no additional training or fine-tuning is required for the auxiliary branch.

We hope this clears up the misunderstanding and provides a clearer explanation of our method and training process.

Response to Question1

We hypothesize that the higher performance of AutoPrompter + Qwen2-VL-7B (75.0 on CVC) may be attributed to the inherent alignment between certain vocabulary and the CVC-300 dataset’s images. Specifically, for the GLIP model, some words may naturally align with the images in the dataset, enabling good performance without the need for StructuralGLIP's mutual selection mechanism. In contrast, the mutual selection structure in StructuralGLIP only selects a subset of visual and textual tokens, which, while helping with denoising, may result in some information loss.

Furthermore, we believe it is normal for such performance variations to occur on individual datasets. The true effectiveness of a method should be assessed based on its performance across a range of datasets. In Table S10, we provide a comparison of the performance of StructuralGLIP and AutoPrompter with Qwen2-VL-7B as the VQA model on additional datasets, which shows a more comprehensive evaluation.

Table S10: AP and AP@50 performance of StructuralGLIP and AutoPrompter with Qwen2-VL-7B as VQA model.

Response to Question2

You observe that prompt-based methods are inherently dependent on the VQA model. However, this is a standard assumption in research aimed at enhancing the performance of vision-language models in medical domains using prompts [1, 2, 3, 4]. As long as the same prompts are used across different methods, the comparisons remain fair and consistent.

Response to Questino3

Finally, regarding your inquiry about why we report AP instead of AP50, we simply chose to present the AP metric without any particular implication or hidden intention. Additionally, Table S9 already addresses why we do not evaluate category prompts with different VQA methods. Here we add the AP@50 performance towards Table S9 as follows:

Table S9-2: AP and AP@50 Performance comparison of different methods when applied category-level prompt (BLIP + GPT4).

Reference

[1] Zero-Shot Nuclei Detection via Visual-Language Pre-trained Models. MICCAI.

[2] Medical image understanding with pretrained vision-language models: A comprehensive study. ICLR, 2023

[3] Multiple prompt fusion for zero-shot lesion detection using vision-language models. MICCAI, 2023

[4] Zero-Shot Nuclei Detection via Visual-Language Pre-trained Models, MICCAI, 2023

It appears that the primary misunderstanding has come to light.

Question1

We would like to clarify that StructuralGLIP is indeed training-free. Specifically, the language encoder in the auxiliary branch shares its model parameters with the language encoder in the main branch, meaning there is no additional training or fine-tuning involved for the auxiliary branch (see Line 210). We will further emphasize this point for better clarity.

In the zero-shot detection setting (Fig. 2 (a)~(b)), we directly employ the pretrained GLIP to initialize the parameters of the vision encoder and text encoder in the main branch. Thus, in this setting, the auxiliary branch's text encoder also shares the same weights with the pretrained GLIP's text encoder.

In the zero-shot enhancement setting (Fig. 2 (c)~(d)), we employ a GLIP thas has been fine-tuned on the target dataset to initialize the parameters of the vision encoder and text encoder in the main branch. Thus, in this setting, the auxiliary branch's text encoder also shares the same weights as the fine-tuned GLIP's text encoder.

With this clarification, we believe that your subsequent questions regarding Table S4/4-2 and the second question in Weaknss7 have also been addressed.

Question2

We would like to clarify the details regarding domain shift from the model’s perspective.

1. StructuralGLIP Construction in the Zero-shot Enhancement Setting

Our discussion of domain shift was meant to explain why StructuralGLIP is still effective when it is applied to fine-tuned GLIP (zero-shot enhancement setting). To clarify how StructuralGLIP is constructed in the zero-shot enhancement setting: for the CVC-300 dataset, the zero-shot enhancement process begins by fine-tuning the GLIP model using its labeled data (all prompt-based methods will then build after the fine-tuning). During fine-tuning, the text encoder of GLIP receives only the target name (i.e., [target name]) as input. After fine-tuning, we build StructuralGLIP by introducing an auxiliary branch with its own text encoder to process the prompt (Except the weights we use are the fine-tuned GLIP now, other setups can be found in the response to Question 1). However, it is important to emphasize that the model parameters for the auxiliary branch’s text encoder are directly shared with those of GLIP's text encoder, and no additional training is required.

2. What is Shifted in the Model’s View in the Zero-shot Enhancement Setting?

In the zero-shot enhancement setting, GLIP is first fine-tuned with [target name] as input to the text encoder, after which prompt-based methods are built on the fine-tuned GLIP. During fine-tuning, we feed the CLS token of the text representation of [target name] into the RPN network. Contextual prompt-based methods, however, introduce the prompt by feeding both [target name; prompt] into the GLIP text encoder. This modification causes a shift in the feature distribution fed into the RPN network because the input to the RPN now consists of the CLS token from the concatenated sentence [target name; prompt], which alters the textual feature distribution.

3. What mechanism in StructuralGLIP Can Address the Domain Shift?

StructuralGLIP addresses this issue through its dual-branch design. Specifically, the main branch’s text encoder only receives the [target name] input, while the auxiliary branch’s text encoder handles the [prompt]. The textual representation from the main branch is then passed to the RPN network, ensuring that the RPN uses the textual features corresponding solely to the target name. The prompt only influences feature fusion in the main branch and is incorporated into the visual encoder. This strategy minimizes the shift in the textual feature distribution at the RPN input compared to methods that directly concatenate [target name; prompt].

4. Clarification on Figure E

Regarding Figure E, we would like to clarify that it represents the distribution of average attention values calculated from the attention matrices of different layers in the main branch of StructuralGLIP (see Appendix G). This figure is not related to the distribution of features fed into the RPN network but instead shows the interaction between visual and language representations at the attention level.

Question3

We have provided it in the next comment.

(1) Response to weakness1. We acknowledge that our method introduces additional computational costs. However, StructuralGLIP mitigates this by supporting category-level prompts, which are fixed across images of the same class and can be pre-encoded. This significantly reduces runtime costs, avoiding per-image prompt generation during inference. The auxiliary branch’s GPU memory overhead is also minimal (from 6082 MB to 7428 MB with image batch size=2).

(2) Response to weakness2. We believe our responses to Question 1&2 comprehensively address this concern, demonstrating how the mutual selection mechanism ensures prompt utility and how CLIP-Score can validate prompt quality.

(3) Response to weakness3. Please refer to our response to Question 3, which includes an ablation study demonstrating the robustness of StructuralGLIP across different LLMs.

(4) Response to weakness4. We understand the concern about transparency in clinical applications. But actually, StructuralGLIP can explicitly log the selected prompts and their corresponding visual patches during inference. This allows practitioners to trace how prompts are utilized in the decision-making process. Appendix D includes examples of visualized prompt selection, showing which attributes (e.g., "red color" or "irregular shape") were emphasized during detection. We hope this additional analysis addresses your concern about model interpretability.

(5) Response to weakness5. We apologize for the formatting error in Table 5, which resulted from a LaTeX rendering artifact. This has been corrected in the revised manuscript.

To assess the impact of different LLMs on performance, we evaluated StructuralGLIP using category-level prompts generated by GPT4, LLaVa-7b, and Qwen2-VL-7b. Table S8 compares their performance across multiple datasets.

Table S8: Ablation on the choice of LLMs when generating category prompts for zero-shot enhancement (AP@50).

From the experimental results, it is evident that the choice of LLM has a considerable impact on the final performance. In contrast, as shown in Table S6 and Table S7, the selection of different VQA models has relatively minimal influenceon the quality of generated prompts. This is because the LLM-based prompt expansion process primarily relies on the model's internal knowledge and memory of the appearance attributes related to the target lesion, rather than utilizing example images.

Therefore, when employing LLMs for prompt expansion, it is crucial to select high-performing models to ensure the generation of reliable prompts. Additionally, for detecting rare diseases, the prompts generated through LLM expansion may not be applicable, as the models might lack sufficient domain knowledge for these less common conditions. This experiment provides deeper insights into the scenarios in which LLM-based prompt expansion can be effectively utilized.

The quality of instance-level prompts is ensured through two complementary approaches:

(1) Attribute-Specific Prompt Generation: The VQA model (e.g., BLIP) generates prompts focusing on attributes directly related to the target's visual appearance (e.g., color, shape). This approach minimizes the reliance on domain-specific knowledge, allowing the method to generalize effectively across datasets.

(2) Quantitative Evaluation with CLIP-Score: Inspired by prior works in text-to-image generation [1, 2], we evaluate the semantic alignment between cropped disease regions and their corresponding prompts using CLIP-Score. Table S6 and S7 summarize the results, demonstrating a strong correlation between higher CLIP-Score and better detection performance.

Table S6: CLIP-Score and detection performance under zero-shot enhancement setting on TBX-11k.

Table S7: CLIP-Score and detection performance under zero-shot setting on CVC-300.

The strong correlation between CLIP-Score and detection performance demonstrates that high-quality prompts directly contribute to better detection results, thereby validating the effectiveness of the VQA-generated instance-level prompts. From another perspective, the quality of prompts can also be ensured by analyzing the CLIP-Score of prompts generated by different VQA models. A higher CLIP-Score indicates better semantic alignment between the generated prompts and the visual content, serving as a reliable indicator for selecting the most suitable VQA model.

References

[1] Li Y, Liu H, Wu Q, et al. Gligen: Open-set grounded text-to-image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[2] Hao Y, Chi Z, Dong L, et al. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 2024.

Thank you for raising this insightful question. Ensuring valuable prompts for unseen classes is a challenge, especially when relying on automated generation methods like VQA and LLMs. We address this potential issue through two key mechanisms in StructuralGLIP:

(1) Mutual Selection Mechanism: During inference, StructuralGLIP dynamically filters and retrieves only the most relevant prompt tokens from the knowledge bank. This ensures that noisy or irrelevant prompts do not interfere with detection. The mutual selection mechanism is fundamental to our method’s robustness against distracting information.

(2) Experimental Validation: To evaluate the robustness of this mechanism, we conducted an experiment on the BCCD dataset, where noisy prompts (e.g., unrelated category attributes) were introduced alongside the relevant prompts. The results in Table S4 demonstrate that StructuralGLIP maintains its performance even in the presence of noise on the BCCD dataset. Here (X, X) means we only employ prompts for X when detect X. (X, Y) means we mix the prompts for Y when detection X (X $\neq$ Y).

Table S4: StructuralGLIP’s AP@50 performance with noisy knowledge under zero-shot detection setup.

Table S4-2: GLIP’s AP@50 performance with noisy knowledge under zero-shot detection setup.

The results highlight that the mutual selection mechanism effectively filters irrelevant information, ensuring the prompts contribute positively to the detection process.

Thank you for your thoughtful and detailed review of our work. We deeply appreciate your acknowledgment of our paper’s strengths, particularly in addressing key challenges in medical image understanding and introducing StructuralGLIP. Below, we address your questions, suggestions, and noted weaknesses in detail.

Thank you for this insightful suggestion. To test the robustness of the mutual selection mechanism, we conducted additional experiments on the BCCD dataset, which includes red blood cells, white blood cells, and platelets. For this test, we added noisy prompts by mixing attributes of unrelated categories and evaluated StructuralGLIP. Results are displayed in Table S4. Here (X, X) means we only employ prompts for X when detect X. (X, Y) means we mix the prompts for Y when detection X (X $\neq$ Y).

Table S4: StructuralGLIP’s AP@50 performance with noisy knowledge under zero-shot detection setup.

Table S4-2: GLIP’s AP@50 performance with noisy knowledge under zero-shot detection setup.

Our findings reveal that GLIP suffers significant performance degradation when noisy knowledge is introduced, while StructuralGLIP maintains high accuracy. This robustness stems from the mutual selection mechanism, which effectively filters irrelevant information and retrieves the most relevant prompts for the given image.

(1) Response to weakness 1: Thank you for pointing this out. The incomplete sentence on Line 234 was a formatting issue during manuscript preparation. The corrected sentence reads: "$B^{i}$ denotes the representation of the prompt bank at the $i^{\text{th}}$ layer."

(2) Response to weakness 2: We appreciate your suggestion to incorporate a reparameterization trick like Gumbel-Softmax to make the Top-K selection process differentiable. While our current focus is on leveraging zero-shot methods without fine-tuning, this enhancement could improve the fine-tuning capability of our model, opening new avenues for research. We plan to explore this direction in future work.

However, it is important to note that the non-differentiable selection mechanism does not hinder our model’s performance in zero-shot scenarios. StructuralGLIP achieves state-of-the-art results across multiple datasets, showcasing its robustness and effectiveness even without fine-tuning.

You raise an excellent point regarding the quality of prompts generated by VQA, especially on datasets where domain adaptation is lacking (e.g., radiology). To evaluate the quality of generated prompts, we have devised two complementary approaches:

(1) Fine-tuning VLM on radiology datasets for evaluation While the original GLIP model performs poorly on radiology datasets like TBX-11k due to a lack of adaptation, we conducted additional experiments using a fine-tuned GLIP model to better evaluate the prompt quality. Table S1 shows the results of using instance-level and category-level prompts under the zero-shot enhancement setting (see Figure 2 (c)-(d)). Notably, the category-level prompts, which are more comprehensive, consistently improved performance over instance-level prompts, validating their quality.

Table S1: Zero-shot enhancement results on radiology datasets.

(2) Quantitative evaluation using CLIP-Score Inspired by text-to-image generation works [1, 2], we evaluated prompt quality using CLIP-Score, which measures the cosine similarity between the embeddings of cropped regions (e.g., disease regions) and their corresponding prompts. We extended this analysis to compare the prompts generated by different multi-modal vision-language models (MLLM), e.g., BLIP, LLaVa-7b, Qwen2-VL-7b, where the MLLMs are used as VQA model. Table S3 summarizes the results, comparing both instance-level prompts and category-level prompts.

Table S3: CLIP-Score and detection performance under zero-shot enhancement setting on TBX-11k.

There is a clear positive correlation between CLIP-Score values and the detection performance (AP@50), validating the effectiveness of CLIP-Score as a metric for evaluating prompt quality. This analysis reinforces the utility of CLIP-Score as an additional evaluation metric for VQA-generated prompts.

Reference

[1] Li Y, Liu H, Wu Q, et al. Gligen: Open-set grounded text-to-image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[2] Hao Y, Chi Z, Dong L, et al. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 2024.

Thanks for your careful review! We will try our best to reply to your concern regarding our method soon!

Question2: Thank you for raising this point. In addition to quantitative metrics such as AP and AP@50, we conducted a qualitative user study to better understand the practical value of our method. Specifically, we randomly selected 18 test images from the ColonDB dataset and generated detection results using MIU-VL, AutoPrompter, and StructuralGLIP. A group of participants (N=49) was tasked with ranking the methods based on the perceived accuracy of detection boxes. Participants evaluated factors such as precise localization of the target and avoidance of false positives (e.g., misidentifying shadows or irrelevant regions). The results are summarized in Table S2:

Table S2: Human evaluation results.

The feedback from participants confirms that our method not only achieves fewer false positives but also outputs higher-confidence predictions, leading to a clear preference for StructuralGLIP over baseline methods.

For future work, we plan to incorporate clinical feedback and consider additional metrics, such as clinical usability assessments, to evaluate the model’s real-world applicability.

Question3: Yes, our proposed method can also be effectively applied to natural image datasets. We would like to clarify why we chose to focus on the medical domain for prompt enhancement. Our primary motivation was to demonstrate how vision-language models can leverage linguistic information to provide additional object priors, thereby improving detection performance. The medical domain, with its relatively simpler and more focused detection tasks, serves as an ideal environment for validating this idea.

For natural images, given that GLIP has already undergone extensive pre-training, we deliberately selected less common natural image datasets—such as Hematite, Cosmos, and Peony—to assess the generalization capabilities of our method. We have included visualizations in Appendix H to demonstrate how our method performs on natural images.

We appreciate this insightful question. The generalization of our method can be discussed from two perspectives:

Generalization to other medical imaging modalities:

While this work primarily focuses on endoscopy and microscopy datasets, we indeed plan to extend our method to other modalities, such as CT scans and radiology images. However, as GLIP was originally designed for natural RGB images, our preliminary experiments show limited zero-shot performance on these modalities without further adaptation. To address this limitation, we plan to explore two solutions: (a) using vision-language models pre-trained on modality-specific data (e.g., chest X-rays or CT), or (b) fine-tuning our method on medical imaging datasets. For example, Table S1 below shows the effectiveness of our method when applied to radiology (chest X-ray) data after fine-tuning in a zero-shot enhancement setting (Fig. 2 (c)-(d)). All comparisons were conducted with consistent prompts.

Table S1: Zero-shot enhancement results on radiology datasets.

Generalization to less common diseases:

Our instance-level zero-shot setup (Fig. 2 (a)) is specifically designed to address situations with limited training data. From an evaluation perspective, this setup inherently assesses performance in data-scarce scenarios. From a methodological perspective, our prompt generation pipeline uses VQA and LLMs to generate descriptive prompts covering appearance-related attributes such as color, shape, and texture (Appendix F). However, we acknowledge that for extremely rare diseases, LLMs may not always provide accurate descriptions. In such cases, the prompts can be generated solely using VQA models, bypassing LLMs. The instance-level comparisons in Table 1 already evaluate our method under such conditions, demonstrating its applicability to less common diseases.

(1) Computational complexity. We acknowledge that introducing an auxiliary branch adds architectural complexity. However, this does not always result in increased computational overhead. First, the GPU memory usage increases due to the auxiliary branch is marginal, from 6082 MB (GLIP) to 7428 MB (StructuralGLIP) with image batch size=2, allowing the model to run efficiently on a standard NVIDIA 3060 GPU. Moreover, our method supports category-level prompts, which are fixed for all images of the same target. This eliminates the need to generate instance-specific prompts for each image during inference, thereby reducing overall computational costs and increasing inference speed (Table 2 (b) & (d)).

(2) Data imbalance. This concern has been addressed in our response to Question 1. Our method inherently handles data-scarce scenarios through its instance-level zero-shot setup and flexible prompt generation pipeline.

(3) Interpretability. We understand your concerns about interpretability. However, we believe StructuralGLIP enhances interpretability compared to prior methods. By decoupling prompt encoding into the auxiliary branch, the selection mechanism makes the utilized prompt words visible to users. This transparency allows practitioners to analyze which prompts were used during inference. For example, in Appendix D (Figure D and Table A), we analyze the selected prompts across different tasks, revealing the attributes the model prioritizes during detection. This provides insights into the model’s reasoning process.

(4) Misalignment in atypical cases. We acknowledge that misalignment may occur in atypical presentations. While this limitation is partially mitigated by fine-tuning on specific datasets, as shown in Table S1, we plan to explore anomaly detection techniques and adaptive learning strategies to better handle rare or atypical cases in future work.