Thank you for your thoughtful review and for highlighting the strengths of our approach, particularly regarding our novel use of explainability methods to enable region-specific focus while maintaining global context information. We appreciate your constructive feedback and questions, which we address below:

1. Multi-object scenarios:

We appreciate the reviewer's feedback regarding multi-object scenarios. While our presentation may have understated this capability, MaskInversion naturally handles multiple objects, as demonstrated quantitatively in our experiments on semantic segmentation datasets like PascalContext (Table 2), where masks frequently encompass multiple instances of the same object class.

To further illustrate this capability qualitatively, we have added visualizations using λ-ECLIPSE diffusion model in Figure 6. of the Annex of the current paper version (please see the Annex written in blue). The figure shows how MaskInversion effectively captures multiple objects within a single mask, preserving their individual characteristics. For instance, when the mask covers multiple characters (as shown in the rightmost examples), the generated images accurately reflect the group composition while maintaining contextual relationships. This demonstrates that our method can effectively encode complex, multi-object arrangements without requiring explicit object-level separation. The explainability maps (top row) further validate this, showing how attention is appropriately distributed across multiple objects within the masked region. While quantitative evaluation of multi-object scenarios remains challenging due to the lack of standardized metrics, these qualitative results strongly suggest that MaskInversion successfully handles complex, multi-object compositions.

2. Global context capture and visualization:

To address your question about global context capture, we conducted additional experiments analyzing how the regularization parameter $\alpha$ influences the balance between local and global information (please see Table 6 written in blue in the updated paper). We observe that:

• Lower $\alpha$ values in [0,1] result in highly focused embeddings that primarily capture the masked region
• Medium $\alpha$ values in [2,5] incorporate relevant contextual information while maintaining region specificity
• Higher $\alpha$ values (>10) progressively approach the behavior of the global [CLS] token We have added visualizations in the paper (Figure 5) showing explainability maps and corresponding captions across different $\alpha$ values, demonstrating how this parameter effectively acts as a "slider" for controlling the local-global information balance. This is particularly valuable for tasks like referring expressions where contextual understanding is crucial.

3. Performance on RefCOCO+:

Regarding the performance difference between MaskInversion and Masked Crop on RefCOCO+ with ViT-B/16, this is an interesting observation that reveals important characteristics of the dataset. RefCOCO+ was specifically designed to focus on appearance-based descriptions rather than relational or contextual ones. Therefore, methods that isolate the target region (like Masked Crop) can perform well on this particular dataset. However, MaskInversion shows superior performance to Masked Crop across all other datasets and larger model architectures, particularly in scenarios requiring contextual understanding (PhraseCut and RefCOCO). We will clarify this dataset-specific characteristic in the paper.

Minor correction:

Thank you for catching the typo ("maks"). We will correct this in the final version.

We hope our detailed responses have addressed all your concerns. If you have no further questions, we would greatly appreciate if you could consider increasing the score.

We thank the reviewer for their thorough and constructive feedback. We particularly appreciate the recognition of the paper's clear writing, comprehensive experimental evaluation, and the advantages of our zero-shot approach that leverages pre-trained models effectively.

Response to Weaknesses:

1. Innovation and LeGrad Description

We understand the concern about the extensive description of LeGrad potentially overshadowing our contributions. We included this detailed explanation to ensure the paper is self-contained and accessible. We emphasize that our key innovation lies not in the use of LeGrad itself, but in the novel approach of using explainability maps to guide the learning of localized embeddings without any fine-tuning (Sec. 3.2). As a second contribution, we also propose a gradient decomposition strategy for efficient computation (Sec. 3.3). Following the reviewer’s comment, we offer to move the detailed LeGrad description to the supplementary material upon request, while maintaining only the essential components in the main text.

2. Regularization Loss Analysis

Thanks for raising this topic. We want to point out that we see the regularization as a special feature that can be useful if there is a reason to include more global information in the LET (localized embedding token), which so far seems to be only relevant in cases of referential expressions. To further analyze the influence of $\alpha$, we conducted a respective ablation study on the RefCOCO dataset (please see Table 6 in Annex of the current version of the paper). We further provide a qualitative example of the effect of alpha on the explainability map and the generated caption in the supplementary of the current version in Figure 5. The added figure illustrates this effect through generated captions for different $\alpha$ values. When $\alpha=0$, the model generates descriptions focused strictly on the masked region (e.g., "woman in a boat"), while increasing $\alpha$ progressively incorporates more contextual information(e.g., "produce" or "vegetables")

The results of both the quantitative and qualitative analysis (please see the updated annex written in blue) show that $\alpha$ acts as a "slider" controlling the balance between local and global information. When $\alpha$ is very small (≈0), the model focuses strictly on the masked region, potentially missing contextual cues. As $\alpha$ increases, the model incorporates more spatial context, with optimal performance around $\alpha=5.0$. At very high values ( $\alpha$>7.5), the performance slightly decreases as the representation becomes too similar to the global [CLS] token.

3. Performance depends on the quality of input masks

We acknowledge this important practical consideration and have thoroughly investigated it in Sec. 4.5 (Table 4) of our paper. Our analysis shows that: Even using just bounding boxes instead of masks only results in a modest performance drop (44.7% to 42.9% on MSCOCO for the Class Retrieval Task). Automatically segmenting bounding boxes with SAM and using the resulting masks as input to our method achieves comparable performance to inputting ground-truth masks (45.0% vs 44.7%). Hence, in combination with SAM, our method works very well given cheap bounding-box input from the users, a practical application scenario. Our method is more sensitive to under-specification (erosion) of masks (42.7%).

4. Failure Cases

Thank you for highlighting the need for deeper analysis of failure cases. We assume the main scenario where MaskInversion may underperform might be in case of resolution limitations. Namely, as the method's effectiveness is bounded by the grid size of the underlying foundation model, if masks, e.g., of very small objects, are below the regular grid sampling size, we will not be able to exactly recover the localized embedding. We have conducted an additional experiment where we disentangle the performance of MaskInversion depending on the size of the mask. The reported numbers are the retrieval accuracy on the COCO dataset obtained using ViT-B/16.

The table shows that MaskInversion performance is mainly bounded by the performance on small objects. We will add a respective discussion in the main paper.

Response to Questions

Your suggestion about incorporating mask-guided feature capture during training is interesting. We believe this could be implemented as a form of self-distillation during the pre-training phase to enhance the model's fine-grained perception capabilities. This represents an exciting direction for future work that could potentially improve the foundation model's local feature capture abilities directly. We appreciate this suggestion and will explore it in future research.

We thank you for your constructive feedback and have addressed all points in detail. If you have no further questions, we kindly ask you to consider increasing the score.

We sincerely thank the reviewer for their thorough and constructive feedback. We particularly appreciate the suggestion to compare with MaskCLIP, SCLIP, and CLIPSurgery, which are indeed relevant baselines and strengthen the paper.

1. Missing Baselines:

We thank the reviewer for bringing attention to these important baselines. Following your suggestion, we have conducted comprehensive experiments comparing MaskInversion with MaskCLIP [1], CLIPSurgery[2], and SCLIP[3] across all evaluation datasets. To compare the respective performance, we used the training-free pipelines to compute respective patch token representations and average pool all patch tokens inside the mask to get the localized embedding for that mask. Using the official implementation provided by the authors, we ran each method on our evaluation suite without any retraining, as these methods are designed to be training-free. We keep the same evaluation pipeline as in the paper and only change the localized embedding tokens used. The results are summarized in the table below. We report the top-1 Accuracy for all datasets extending Table 1 and Table 2 in the original paper:

Our results demonstrate that MaskInversion consistently outperforms these baselines across nearly all datasets and backbone sizes, with particularly strong improvements for larger backbones. It shows that only in one case (ViT-B/16) ClipSurgery is able to outperform our method on OpenImages V7. More importantly, it shows that MaskInversion consistently outperforms these baselines and profits from larger backbones by showing increased performance. In contrast, the performance of the evaluated training-free methods starts to degrade.

While this shows the capabilities of MaskInversion, we want to emphasize that we consider MaskInversion and training-free methods as two different valuable lines of work that overlap in this task, but might be used in different contexts.

[1] https://github.com/chongzhou96/MaskCLIP [2] https://github.com/wangf3014/SCLIP [3] https://github.com/xmed-lab/CLIP_Surgery

2. Complexity compared to training-free methods:

We think that it is difficult to directly compare the trade-off between complexity and performance of MaskInversion compared to training-free methods as we think of them as two independent lines of work. e.g., while our method operates on the model outputs without requiring architectural modifications, ClipSurgery needs the modification of the forward-pass over several layers. If the reviewer has any suggestions on how to explore this topic further, please let us know.

3. Gradient Decomposition Significance:

We appreciate the concern about the gradient decomposition's practical utility. While the speed improvements become significant only with >10 masks per image, which might not be common in e.g., human interactions, we believe this contribution is rather valuable for an automatic processing scenario. Indeed, recent methods such as SAM and open-world objectness detectors typically generate 100-250 masks per image. Being able to efficiently process large amounts of masks might allow converting such image regions into meaningful embeddings. We consider this to be an interesting direction for future work.

4. Minor Comments:

Thank you for catching the typo regarding Table 4.5 which is actually Table 5. This will be corrected in the final version.

We believe the results and clarifications we provide in this rebuttal address the main concerns while demonstrating the significant advantages of our approach. We will update the paper to include these comparisons and clarify the practical significance of the gradient decomposition contribution.

We hope that our comprehensive response, particularly the extensive experimental comparisons with MaskCLIP, SCLIP, and CLIPSurgery, has adequately addressed your concerns about missing baselines and demonstrated the significant advantages of our approach. Given these new results showing consistent improvements across datasets and backbone sizes, we kindly ask if you would consider revising your assessment of our paper, or if you have any additional questions we could address.