From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection

Q1: Experimental details in Table 3.

A1: Thanks. The results of baselines in Table 3 follow WCA protocol: Tuning-based methods are 16-shot source-trained and target-evaluated for OOD generalization. Notably, our method requires no fine-tuning, operating in a zero-shot manner on both source and target datasets and even surpasses fine-tuned methods.

Regarding your concern, we include in the table below the results of Coop fine-tuned directly on target data. Since our method functions as a plug-and-play module without requiring fine-tuning, integrating it with Coop further enhances performance.

Q2: About negative impact of DINO.

A2: Thanks.

1. DINO establishes a new contrastive learning framework that learns superior visual representations. Its exceptional attention maps, which have gained prominence for reliably pinpointing primary objects across diverse datasets, provide the foundation for our feature selection mechanism. And also a lot of works [2][3] using DINO for assistance. This reliability in object localization motivates our strategic adoption of DINO's attention maps to guide our selection process.
2. We observe that DINO's attention maps consistently localize semantically relevant object regions in multiple datasets. While occasional inaccuracies may occur, our adaptive crop_ratio enhances diversity in the cropped regions to mitigate such cases. Furthermore, our soft matching strategy effectively downweights irrelevant image crops, thereby reducing potential negative impacts from DINO's misalignments.
3. We observe that different attention heads in DINO may focus on distinct regions. In our paper, we simply average all attention heads. To further mitigate potential negative effects from DINO, we use attention maps from heads with higher variance, ensuring the selection maintain strong focus on salient objects. The following table shows the results demonstrate nearly identical performance to our original method. This suggests that ABS already achieves low error rates in attention map selection, making further error reduction less noticeable.

Regarding more explicit filtering of negative attention maps, we will conduct further research.

ref:

[2] Zero-guidance segmentation using zero segment labels. CVPR 2023.

[3] Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. ECCV 2024.

Q3: About inference time overhead.

A3: Thanks. When processing and encoding images, we do need to augment a single image into 2N crops. However, our hyperparameter sensitivity experiments reveal that the results are not sensitive to the num_crop. Therefore, we can reduce the value of N to lower time costs, while achieving average improvements of approximately 2% on Zero-shot classification and 5% on OOD datasets compared to CuPL, both of which represent significant gains. Additionally, we can follow the approach of WCA by pre-storing image features prior to inference. This ensures that the final similarity computation stage incurs comparable time costs to CuPL, while delivering substantially enhanced performance.

Q4: About dimension in Equation 7.

A4: Thanks. The specific process of interpolation please refer to our answer A2 to Reviewer dz4u and the pseudo-code is as follows:

# s_p: sampled patches from DINO attention
raw_crops = []
fea_crops = []
mid_fea = CLIP(x, layer=l-1)
for p in s_p:
    c_size = random(α, β)
    c_img = crop_img(x, p.center, c_size)
    raw_fea = CLIP(c_img)
    raw_crops.append(raw_fea)

    c_fea = crop_fea(mid_fea, p.center, c_size)
    r_fea = interpolate(c_fea, original_size)
    f_fea = CLIP.final_layer(r_fea)
    fea_crops.append(f_fea)
com_fea = concatenate(raw_crops + fea_crops)

Q5: About negative impact of DINO.

A5: Thanks. Please refer to A2.

Q6: Comparison with other weighted methods.

A6: Thanks. FILIP [1] proposes a token-level matching approach for VLM pretraining. However, since its pretrained model is not publicly available, we cannot directly apply its token-level matching to VLMs that were not pretrained with this method. As an alternative, we experiment with the weight-based matching approach proposed in WCA and common entropy weighted methods and compared them with our soft matching. The results demonstrate the superior performance of our soft matching, indicating its enhanced capability to select more semantically aligned image-text pairs for effective matching.

Q1: Claims And Evidence.

A1: Thanks. The purpose of employing cropping is to enable the model to focus on local features of objects, thereby achieving better alignment with certain LLM descriptions. However, random crop exhibits two inherent limitations: When using smaller crop sizes, it introduces randomness and compromises global semantic coherence; while with larger crop sizes, although uncertainty diminishes (as you mentioned), this comes at the cost of sacrificing the ability to concentrate on local features, instead providing more global information similar to using full images.

Our method addresses this by enabling the model to eliminate randomness while preserving global semantics when focusing on local and pivotal features. Through ablations in the table below, we observe that our approach does not exhibit monotonically increasing accuracy with larger crop sizes. Conversely, it achieves superior results with smaller crop sizes, whereas WCA demonstrates an incremental pattern with increasing crop sizes. This contrast underscores the significance of our method's dual capability in eliminating randomness and maintaining global semantic integrity during local feature focusing, ultimately enhancing the model's optimal performance.

Q2: Clarification of feature crop operation.

A2: Thanks. In our method, feature selection is performed before the forward of the final transformer layer. Given input features with dimensions [bs, 197, 768], we first separate the [CLS] token ([bs, 1, 768]) from the remaining tokens ([bs, 196, 768]). The remaining tokens are reshaped into 2D size [bs, 14, 14, 768]. Based on DINO’s attention map, we then select N crops from this feature map. Each crops is interpolated to the original feature map size, and concatenated with the [CLS] token, reconstructing the feature into [bs, N, 197, 768]. This modified feature is fed into the final transformer layer. During this layer's forward, the [CLS] token interacts with the crops to capture diverse local features enriched with global semantic information. We will supplement this detail in subsequent versions of our paper.

Q3: Integrating with other VLMs.

A3: Thanks. As shown in the table below, we apply our method to a broader range of VLMs (e.g. ALIGN, AltCLIP and GroupViT) in ImageNet and achieved superior performance compared to other approaches.

Q4: The impact of different components of our method.

A4: Thanks. Please refer to our answer A1 to Reviewer C2Mc.

Q5: Contribution of the work.

A5: Thanks. Please refer to our answer A1 to Reviewer bJ6A.

Q6: About other attention-based strategy methods.

A6: Thanks. [2] employs heatmaps to localize target regions. However, it relies on dynamically updating bounding boxes during training, which is incompatible with our zero-shot scenario. Furthermore, while [2] can successfully locate target regions, it fails to address the preservation of global information after cropping, which may lead to inter-class confusion in the model.

[3] combines features from VFMs with CLIP through Proxy Attention, which enhances the prominence of primary objects. [4] proposes to balance global and local contexts within CLIP's attention layers by analyzing attention values to estimate region-wise saliency. However, these methods fail to focus the model on local object features and lack the capability to focus on localized features of individual objects. In contrast, our approach not only highlights local object characteristics but also preserves crucial semantic information, offering a more comprehensive solution.

We implement the core modules of [3] and [4] within our framework and conduct experiment on the ImageNet dataset. As shown in the table below, our method demonstrates superior performance, validating the advantage of our approach in simultaneously focusing on local features while preserving global semantic information. A comprehensive comparison with methods [2]-[4] will be presented in the subsequent version of our paper.

Q7: The clarity regarding this paper's position.

A7: Thanks. The key insight of ABS compared to previous attention-based methods lies in two aspects: not only do we leverage attention-guided cropping in the raw space to focus on local regions while eliminating randomness, but more importantly (and to the best of our knowledge) we are the first to implement attention-guided feature space selection to complement global semantics. This dual-space cropping enables the crops to better align with LLM descriptions.

Q1: About contribution.

A1: Thanks. Our core contribution resides not in proposing incremental adjustments to established frameworks, but rather in advancing systematic methodologies that demystify stochastic factors while enhancing holistic semantic comprehension for this research domain.

Firstly, the adoption of attention-based mechanisms to mitigate cropping randomness plays a pivotal role in zero-shot classification task, enabling consistent and robust performance improvements.

Moreover, our key insight lies not merely in employing attention-based selection to focus on critical regions, but also in being the first (to the best of our knowledge) to perform feature cropping that complement raw-space crops by integrating global semantic information. This dual mechanism proves crucial for aligning with LLM descriptions, while also establishing a novel paradigm for future research in multimodal alignment.

Additionally, our approach demonstrates high flexibility, enabling seamless integration with various advanced VLMs (please refer to our answer A2 to Reviewer C2Mc and A3 to Reviewer dz4u) to enhance their performance through plug-and-play adaptation.

Q2: Table 6 is not properly cited.

A2: Thanks for pointing it out. We will fix it in the future version of our paper.

Q3: Maintain “cos” consistency.

A3: Thanks for pointing it out. We will fix it in the future version of our paper.

Q4: Capitalization consistency.

A4: Thanks for pointing it out. We will fix it in the future version of our paper.

Q5: About hyperparameters set.

A5: Thanks. Our method involves three critical hyperparameters: crop_ratio, num_crops, and top-k, which we consistently set to 0.5, 60, and 20 respectively across all datasets. This unified configuration serves dual purposes: First, it demonstrates our approach's capability to achieve consistent performance improvements without dataset-specific tuning, and second, it facilitates convenient application in industrial deployments. As evidenced by the hyperparameter sensitivity analysis presented in the table below or in the Figure5 in our paper, our method exhibits remarkable robustness to parameter variations. In fact, alternative configurations (e.g. crop ratio=0.6, num_crops=30 and top-k=30) might potentially yield superior results to those reported in our paper. Nevertheless, we maintain fixed parameters throughout all experiments to ensure fair comparative evaluation and rigorous empirical validation of the proposed methodology.

Q6: About soft matching.

A6: Thanks. The soft matching mechanism is proposed to addresses the inherent mismatch issue in CLIP's original matching strategy, specifically the "description-crop misalignment" phenomenon where textual descriptions partially deviate from cropped image regions. This problem arises because our selection process emphasizes local object features - compared to using full images, these focused crops may induce partial mismatches with LLM descriptions during semantic alignment. The soft matching resolves this through adaptive weighting: it suppresses irrelevant LLM descriptions while amplifying semantically aligned ones. As demonstrated in the table below, the ablation studies confirm that removing soft matching causes performance degradation, verifying its critical role in mitigating mismatch effects. Notably, applying soft matching to WCA further improves accuracy, empirically proving its general effectiveness in filtering noisy descriptions.

Q1: The concerns in Claims And Evidence.

A1: Thanks.

1. The table below compares applying soft matching alone vs. combined two selections to the baseline. Combined with the ablations in our paper, it shows that individual components yield improvement when used independently, but integration works best. It is noteworthy that the combined two selections outperforms using soft matching alone, proving these key modules significantly enhance overall performance.
2. The small improvement observed when using the raw space and feature selection individually because: Using either selection alone may lead to excessive focus on either local or global information. Moreover, employing cropping will focus on specific local regions. While this enables better alignment with LLM descriptions that match the currently focused regions, it weakens the alignment for those unrelated to these regions. Consequently, without soft matching to filter irrelevant descriptions, many unrelated descriptions would adversely affect the current crop's alignment. This phenomenon is also observed in the ablations in WCA, demonstrating that filtering or weighted integration of crops is crucial.
3. We replace the components in WCA with ours in the table below. The results prove that our components deliver performance gains compared to the original WCA, proving superiority of our design. ||ImageNet|DTD|ImageNetv2|$\Delta$Avg |-|-|-|-|- |Baseline|69.61|50.53|63.27| |Baseline w/ soft m.|70.03|52.89|63.68|+1.06 |Baseline w/ two sele.|70.32|52.66|64.34|+1.30 |Ours|71.92|54.26|66.19|+2.98 |WCA|71.08|52.79|64.71| |WCA w/ soft m.|71.37|53.60|65.67|+0.69 |WCA w/ two sele.|71.42|53.76|65.83|+0.81

Q2: More advanced VLP model.

A2: Thanks, as your suggestion, we employ the Blip-2 and compare with other baselines. The table below shows that ABS outperform all others across two datasets, demonstrating our effectiveness and adaptability. More advanced VLMs (e.g. ALIGN, AltCLIP and GroupViT) please refer to our answer A3 to Reviewer dz4u.

Q3: The comment of Broader Scientific Literature.

A3: Thanks. Please refer to our answer A1 to Reviewer bJ6A.

Q4: About attention‐based cropping methods.

A4: Thanks. [1] generates attention maps to crop and randomly erasing regions to force the model to focus on key areas. Although their method focus local regions, it fails to address the subsequent loss of global context. In contrast, our approach compensates by using feature selection. We experiment with [1] on our task (as shown in the table below), ABS demonstrate superior performance. This advantage stems from our global semantic compensation for cropped regions.

Although [2] and [3] are both attention-related works, [2] uses geographical information, making it only applicable to specific scenarios. [3] requires training for its modules, which contradicts zero-shot tasks. Moreover, neither method considers supplementing global information. In the revision, we will discuss and compare with these methods.

Q5: About technical contribution.

A5: Thanks. Please refer to A1 and our answer A1 to Reviewer bJ6A.

Q6: Concern about Food101.

A6: Thanks. ABS demonstrates performance improvements across multi-object or complex backgrounds datasets including Place365, only with the exception of Food101.

The Food101 differs from conventional multi-object datasets, it contains inherently multi-label images but provides only single-label annotations. For instance, an image labeled as "french fries" may actually contain multiple objects (e.g., fries, steak, and salad), where non-target objects could occupy a larger visual proportion than the labeled subject (We show several examples in our anonymous link: https://anonymous.4open.science/r/Submission_4487-1B78). Because all these objects fall within the predefined label set, the single-label assignment introduces ambiguity. These inherent properties could cause our method to identify unlabeled object categories within the images.

WCA’s randomness may inadvertently benefit from this properties. However, our experiments confirm substantial improvements in most of classification tasks including multi-object or complex backgrounds datasets.

Q7: About prompt and pseudocode.

A7: Thanks.

1. For the generation of LLM descriptions, we simply follow CuPL, utilizing their publicly released JSON files. The specific prompts can be found in the CuPL.
2. For details and pseudocode of the alignment process, please refer to our answer A2 to Reviewer dz4u and A4 to Reviewer iUpr.

We will supplement these in future versions of our paper.

Q8: Weakness 4&5.

A8: Thanks. We will revise our paper thoroughly.

Q9: Other comments and questions.

A9: Thanks. Please refer to the answer above.