Learning Visual Prompts for Guiding the Attention of Vision Transformers

Thank you for your feedback. As outlined in the introduction, existing methods for redirecting the attention of ViTs to specific locations either require fine-tuning or rely on manually placed markers on the input image. The former is computationally expensive, while the latter introduces human bias, making it far from an optimal solution.

. We are the first to achieve a visual marker capable of directing the attention of ViTs to specific locations using a self-supervised learning strategy. Unlike previous markers, which are manually engineered, our approach represents a significant step forward. Not only does it demonstrate superiority and innovation by learning the optimal marker, but it is also model-agnostic, making it applicable to all types of transformer-based vision encoders. Moreover, our method operates in a fully self-supervised manner, eliminating the need for costly annotated data. This further emphasizes its strength in terms of transferability across diverse datasets. We would appreciate a deeper explanation of why such a groundbreaking approach might be viewed as less innovative compared to manual, heuristic methods, as this perspective seems to overlook the clear advantages of our contribution to manipulating the attention of vision-transformers.

. While the manual markers may appear straightforward, the learning strategy we propose is far from simplistic. It employs a carefully designed self-supervised loss function to bridge two distributions: the Gaussian attention map and the averaged attention values of the CLS token. This is a delicate process, as the values in the pixel space are not directly updated but instead rely on a neural prior (as illustrated in Figure 2 and ablated in Figure 5) and a random noise initialization patch. We would greatly appreciate more detailed feedback on how the reviewer perceives our approach as simple, particularly in comparison to previous methods for location-controlled visual markers. These earlier methods use heuristic techniques to draw markers directly on the input image, which contrasts sharply with the principled and sophisticated framework we present.

. Currently, no learning-based method exists for optimizing visual markers in Vision Transformers (ViTs). We are the first to propose such a method. In contrast, all prior approaches heavily rely on manually engineered markers derived from human intuition (see Shtedritski et al., 2023, and Yang et al., 2024). To address this gap, we thoroughly compare our method against the strongest engineered marker to date—the blurring strategy (Yang et al., 2024)—in Table 3. Further, for the three models aligned with text encoders (CLIP-large, CLIP-base, and SigLip), we report accuracy based on the number of correctly identified body parts, demonstrating the clear advantage of our learned markers in Table 3. For the remaining vision encoders (DeiT, DINOv2-Large, and DINOv2-Base), which lack aligned text encoders, we propose to look at he improved attention on the pointed region. We highlight the effectiveness of our learned marker through its measurable influence on attention values. Figure 6 explicitly shows how the attention values across transformer layers improve when our learned prompts are applied. Moreover, we demonstrate that markers optimized for one model (e.g., CLIP-large) are not optimal for another (e.g., DINOv2), underscoring the specificity and adaptability of our method.

. In addition to this, Tables 1 and 2 present detailed ablations on marker size and shape (vanilla, frame, and ring) with varying thickness factors. Figure 4 further illustrates the effect of applying learned markers interchangeably between models, revealing their direct impact on attention values across transformer layers. In Figure 5, the significance of the neural prior component is ablated. Importantly, this work is unique and the first to examine how attention values evolve through ViT layers when applying visual markers. Figure 7 provides compelling evidence through attention heatmaps, showing how our learned markers dynamically redirect attention to specific objects in an image. Finally, Figure 8 demonstrates the transformative power of our prompts in Visual Question Answering (VQA) tasks. Without prompts, the language model produces incorrect answers; with our learned prompts, it returns accurate responses. We request the reviewer to address the shortcomings in our work. If these extensive experiments, ablations, and evaluations are still considered insufficient, we respectfully request specific, actionable feedback detailing what is missing. Given the breadth and depth of our study, we find it essential to understand how our approach, with its significant innovation of proposing a self-supervised optimization strategy for learning visual markers, could be deemed inadequate compared to heuristic, manually engineered methods. We hope this explanation clarifies our decisions and approach.

Thank you for your thoughtful feedback. We respectfully disagree with the assertion that the use of labeled datasets for evaluation contradicts the self-supervised nature of our training approach. It is important to clarify that using a self-supervised training method does not inherently require a self-supervised evaluation. While self-supervised evaluation can be valuable in scenarios where labeled evaluation data is unavailable, performance metrics derived from labeled, annotated data are often more reliable and interpretable. This is especially true for benchmarking against widely recognized datasets like CUB (for keypoint detection) and RefCOCO (for object localization), which offer standardized comparisons across methods. Furthermore, we do address the scenario of label-scarce environments in our work. As demonstrated in Figure 6, we propose a self-supervised evaluation metric that measures attention gain, providing an alternative means of validating the effectiveness of our method without relying on labeled data. This approach showcases the versatility and practicality of our method in both labeled and unlabeled contexts.

We acknowledge the importance of evaluating methods in diverse and challenging settings. In our work, we strive to include a wide range of evaluation scenarios to test the robustness of our approach. Specifically, we assess performance on single-object test datasets like CUB, multi-object annotated datasets such as RefCOCO, and sampled data from the COCO dataset with randomly assigned target locations. These settings collectively represent both controlled and less structured visual contexts. Moreover, the use of the Gaussian target map in our self-supervised training ensures that the model focuses on the area surrounding the prompt's location, regardless of the presence or type of object in the image. This design inherently encourages generalization by making the model robust to variations in object appearance and context. While more evaluations on completely unlabeled, complex natural scenes are valuable future directions, we believe our current setup demonstrates the method's applicability across a range of scenarios, balancing complexity and practicality. Prior to this work, other studies have proposed manually crafted prompts, claiming these prompts are universally applicable across all vision transformer (ViT) models. However, our results demonstrate that the effectiveness of such prompts heavily depends on the training procedures (e.g., datasets and objectives) of the tested ViTs, and they fail to generalize effectively to newer models like DINO and DeiT (as shown in Figure 6). This indicates that these so-called "universal" prompts are not as broadly applicable as previously claimed. The primary goal of our work is to learn prompts tailored to each individual model, removing reliance on prior biases stemming from the specific training procedures of these models. Furthermore, as shown in Figure 5, learning a single universal prompt for multiple models leads to performance degradation across a selection of ViTs when compared to individually learned prompts. While such a universal learned prompt still significantly outperforms manually crafted prompts based on human intuition, it highlights the challenge of generalizing prompts across diverse architectures. For each model, we only need to train.

Lastly, about the comparison with the CNN architectures mentioned in the questions, we clearly indicate in the title that this work is only proposed because we are presenting a self-attention mechanism that does not exist in the CNN methods! We hope this explanation clarifies our decisions and approach, and we thank you again for your valuable feedback.

Thank you for your insightful feedback. We appreciate the opportunity to address your points and clarify our decisions. First, regarding the computational requirements, fine-tuning a ViT foundation model is significantly more computationally demanding than learning a limited pixel-space patch and a lightweight, 3-layer CNN-based neural prior. For this reason, we did not consider an ablation study on fine-tuning to be particularly insightful for inclusion in the paper. As noted in the Introduction, we emphasize the high computational cost of fine-tuning, but since our approach circumvents this need, we did not find it necessary to include additional experiments on this topic in the Results section.

We also acknowledge the suggestion to explore multiple markers on a single image as a potential future direction and will consider this as an additional ablation in a subsequent version of the submission. However, it is important to note that this was not the primary goal of our study. Our focus was on redirecting the attention of a ViT to a specific location. The objective function of our method was explicitly designed around this goal, targeting a single attention point rather than multiple markers. In Figures 4, 5, and 6, we compare our approach with the red circle prompt, studying the attention gain across all model layers. As stated earlier in the paper, the red circle has been identified as the optimal prompt for the CLIP-Large model. In Figure 6, we include the red circle alongside all the learned prompts for various models to ensure a comprehensive comparison.

While LoRA is indeed a popular PEFT method, it still involves significantly more parameters than our approach, which is limited to a 3-layer neural prior and a token-sized learnable pixel patch. The primary goal of our research is to discover the optimal patch while leveraging frozen ViTs. This optimal patch not only provides computational efficiency but also enhances the interpretability of ViTs by shedding light on their attention mechanisms. Our learned prompt successfully manipulates the model’s attention to a specific location without requiring any structural changes or fine-tuning of the model itself, maintaining its frozen state. We hope this explanation clarifies our decisions and approach, and we thank you again for your valuable feedback.

We appreciate your comments and feedback. In the Methodology section, we have clearly and meticulously detailed the components of our proposed framework: the initialization of random noise, the neural prior, the shape mask, the randomized insertion of the learned marker onto images, and the use of a frozen vision encoder as the main model. Furthermore, we explicitly describe our self-supervised KLD loss function, which aligns the attention heatmap with the Gaussian map defined by the random location of the learnable marker. These steps are thoroughly documented. In the Experiments section, we dedicate a full paragraph to outlining the training datasets. For evaluations, the relevant test sets are explicitly stated alongside the respective figures and tables. For instance, in Figures 4, 5, and 6, we clarify that attention gain values were calculated on a random subset of MS COCO, demonstrating the generalizability of our learned prompt trained on ImageNet. Tables presenting accuracy results clearly specify the CUB or RefCOCO family as the evaluation datasets, leveraging their ground truth text annotations. Additionally, the Keypoint-to-Name task is a well-known benchmark, and due to space constraints, we deemed an extended explanation unnecessary.

It is deeply surprising and disappointing that the reviewer has overlooked these details and dismissed our work as an incremental contribution. Our framework introduces a self-supervised learning strategy to direct the attention of transformer-based vision encoders (e.g., CLIP, DeiT, DINO) to specific image locations—an innovation that extends well beyond prior studies. In the Introduction, we emphasized the contrast with previous works, such as the "red circle" manual markers, by proposing an entirely learnable framework. Furthermore, we included a comparison with image cropping in the appendix. However, it is unfair to remove major contextual parts of an image in the cropping baseline while our marker allows the full image context to be preserved. Comparisons with the “red circle” baseline were extensively evaluated through our self-supervised metric, attention gain, as demonstrated in Figures 4, 5, and 6. These results highlight that our learned markers, tailored specifically to each model, outperform generic manual markers. Notably, while prior works were limited to the CLIP family, our contribution extends beyond to include SigLip, DeiT, and DINOv2—a significant step forward that should not be trivialized. This work presents a clear advance in self-supervised learning for vision encoders, offering a novel method to optimize attention redirection across diverse transformer-based models. We respectfully urge the reviewer to reconsider the depth and scope of our contribution.

Finally, while the LLaVa + Mixture-of-Features model by Tong et al. (2024) requires location information due to the MMVP dataset’s task-specific nature, this dependency is not methodological. We explicitly acknowledge this in the paper’s limitations and future work sections. These points were transparently presented, and any misinterpretation of our approach is unfounded. The experiment on the MMVP dataset (Figure 8) demonstrates the utility of visual markers for next-generation VLMs. It shows that applying a prompt to a specific location enhances language output by redirecting the vision encoder’s attention, enabling the language decoder to produce more accurate answers.

The true comparison, as depicted, lies between the outputs with and without prompts. We acknowledge the reviewer’s suggestion and will include statistics on successful cases in future versions. We argue that manually engineered markers like "red circles," though effective, may succeed by chance, with no evidence of their optimality for directing CLIP’s attention. Previous studies have focused solely on CLIP, with no exploration of other ViT-based models. In contrast, our framework removes biases from manual engineering, leveraging optimization to control attention mechanisms across models like CLIP, DeiT, and DINO. Regarding marker placement, during training, markers are inserted randomly, ensuring a self-supervised approach independent of annotated data for both location and text. In testing, evaluations were conducted in supervised and unsupervised settings. The supervised approach requires location and object-specific text for accuracy measurements, as shown on CUB and RefCOCO datasets, which we explicitly acknowledge as a limitation. The unsupervised evaluation, based on attention gain, demonstrates independence from text annotations or object locations, with experiments on MS COCO (Figures 4, 5, and 6) highlighting the flexibility of our approach in the absence of labeled data. We hope this explanation clarifies our decisions and approach, and we thank you again for your valuable feedback.