Locality Alignment Improves Vision-Language Models

The following table shows the relevant results from the probing benchmark:

Can you give more context on the significance of the VLM benchmark improvements? Some of the relative improvements on Figure 5 are so small that they seem like noise.

Thanks for raising this point. We set the axis scales for our radar charts following the approach from Karamcheti et al. (2024): we considered the pool of all VLMs we evaluated and set axes using the mean +/- 3 times the standard deviation for each benchmark. This leads to reasonable scaling for most benchmarks, but we agree that there are a couple whose differences seem magnified due to low performance variability (POPE and GQA). We thought it best to keep our scaling method consistent between benchmarks rather than set these differently by hand.

Based on your comment, we’ve made sure this point is explained in the paper. Section 5.1 now contains the following sentence: “Following prior work (Karamcheti et al., 2024), we scale each benchmark’s y-axis based on the mean and standard deviation within our pool of models.” We also explain this point in more detail in Appendix D, including the specific pool of models used for axis scaling.

Thank you for taking the time to read our paper and provide helpful feedback! We respond below to the points raised in your review.

Ablation studies could be more comprehensive [...] Exploring a broader range of datasets, such as CC3M (diverse) versus IN1k or SAM images (multi-object), could have offered more insightful findings on generalizability.

Thanks for raising this point. Investigating the role of data scale/diversity is an important direction for future work, and we made sure to mention it in our conclusion. We didn’t prioritize data ablations here because our main goal was to show that locality alignment works, and we found that it does even with relatively small datasets (by modern standards) like IN1k and IN21k. It may turn out to work even better with other datasets, but investigating a range of options is difficult given our computational resources and the short rebuttal period, so we are deferring this study to future work. In addition to CC3M and the SAM dataset, we are also interested in whether locality alignment can benefit from training longer on larger, more diverse datasets like DataComp-1B and DFN-2B.

Limited ablation of the loss function. For example, testing reconstruction of only unmasked tokens rather than the entire embedding sequence could provide valuable insights into the role of different token types in the loss.

Thanks for the suggestion, we’re happy to include additional loss function ablations. Reconstructing either just the masked tokens or just the unmasked tokens is a reasonable idea because methods like MAE/BEiT do this, but it seems inadvisable here because all of the token reconstruction targets are non-trivial so ignoring any of them will only make the task easier. We tested this and found that both versions degrade local probing performance compared to a loss that considers all tokens (using our benchmark from Section 4).

Our results are shown in the table below, where all experiments are conducted with IN1k ViT-B models for consistency with our other ablations. We also included a comparison with cosine similarity loss and L1 loss, neither of which outperforms our choice of MSE. We will include these results in the revised paper.

Comparison with alternative masking strategies is missing. While the rationale for masking in the current way is sound, comparing with an approach like dBOT—where the student rather than the teacher is masked—could have strengthened their case, as dBOT follows a nearly identical pipeline and has shown strong spatial feature learning.

Thanks for the suggestion. Reviewer ARqQ also requested a comparison with dBOT, and we provided a longer discussion there of our findings and attempts to use the dBOT codebase. Briefly, the main points are:

1. dBOT is another masked image modeling (MIM) method like MAE, BEiT, MaskFeat and EVA02, and these have important differences with our proposal. The key differences between MaskEmbed and MIM methods as a whole are 1) we mask the student model at its output layer rather than its input layer, 2) we mask the teacher as well to create reconstruction targets that vary with the mask $m$. These differences encourage MaskEmbed to learn specific features associated with each patch, whereas MIM teaches a model to impute features for missing patches. We’ve clarified these differences in our revised Appendix C.1 under a section called “Relationship with masked image modeling,” and we also have a new extended related work section in Appendix A.

2. We tested dBOT’s random teacher checkpoint in our probing benchmark, and found that it’s slightly better than MAE, but not competitive with language-supervised models. This is dBOT’s main proposal, and it leaves a lot of room for improvement on this evaluation.

3. We tried to use dBOT’s CLIP teacher checkpoint, but we were unable to get reasonable results despite our best attempts to follow the authors’ code. Fortunately, because dBOT’s teacher bootstrapping technique didn’t work for CLIP (as described in their Appendix C), their checkpoint uses standard single-stage MIM, and a reasonable surrogate is to compare to other CLIP MIM models. We consider EVA02 ViT-B from Fang et al. (2023), which was trained using MIM with a large EVA-CLIP 1B teacher, and should therefore have an advantage over our locality-aligned CLIP ViT-B. While EVA02 ViT-B outperforms the pre-trained CLIP ViT-B, our locality-aligned version has stronger performance at both local and global probing. Interestingly, locality alignment also leads to a significant improvement on EVA02’s local probing performance, showing that MIM does not saturate its performance at this task.

As far as I know, the majority of current VLM methods employ a multi-stage training approach, and the mentioned Llava-1.5 does as well, which gives it better performance than the one-stage baseline in this paper.

Thanks for raising this point. Our training runs used the approach recommended in the Prismatic VLMs codebase (Karamcheti et al., 2024), which is essentially a Llava-1.5 replication that carefully studied key hyperparameters. One of their experiments tested the value of an initial adapter training stage, and they found that it slightly degraded the final model compared to a one-stage approach (see their Figure 4). We found this surprising but confirmed it in our own early experiments. Based on this, we proceeded with the simpler one-stage training approach. We’re aware that this is somewhat uncommon in the literature, but it didn’t seem justifiable to use a worse-performing setup just to be consistent with Llava/Llava-1.5. In any case, this choice doesn’t seem like it would interact strongly with the choice of vision backbone.

To my knowledge, the latest method in the Llava series, Llava-OneVision[3], fine-tunes the vision encoder part simultaneously during VLM training. Many other recent methods[4][5] also adopt this setting, making the claim in lines 52-54 somewhat Inadequate. I wonder whether simultaneously fine-tuning the vision encoder could lead to VLMs gaining local semantic understanding, potentially diminishing the advantages of the proposed method.

Thanks for raising this point. It’s true that some recent VLMs fine-tune the vision backbone during training, including DeepSeek-VL, Qwen2-VL, PaliGemma and Llava-OneVision. We didn’t do so because it appears to be harmful when training with our scale of multi-modal data (~1M examples), and our choice is consistent with the Llava/Llava-1.5/Prismatic setups that all train at our data scale. However, it’s true that the effect of our backbone could be different when performing full fine-tuning. We reason that it would still help to start from a better initialization (results like this are shown for SigLIP vs DINOv2 in Cambrian-1), but we’re unable to perform the large-scale experiments to verify this. Based on your comment, we’ll be sure to acknowledge this limitation in the paper. Our conclusion contains the following text: "The benefits of locality alignment may change with end-to-end fine-tuning, but we did not explore this because it is unhelpful with our quantity of multi-modal training data (Karamcheti et al., 2024). An important direction for future work is to test locality alignment in other VLM training approaches, with larger LMs, and to evaluate how it composes with other techniques that enhance visual features."

Thank you for taking the time to read our paper and provide helpful feedback! We respond below to the points raised in your review.

The authors need to include comparisons and discussions with more methods, such as dBOT[1] and UMG-CLIP[2]. dBOT[1] employs a distillation strategy similar to the method presented in the paper, so it is necessary to discuss the differences with this method and provide performance comparisons.

Thanks for pointing us to dBOT, we’ve carefully reviewed the paper and will be sure to cite it in our work. dBOT is another masked image modeling (MIM) method like MAE, BEiT, MaskFeat and EVA02, and has important differences with our proposal. The key differences between MaskEmbed and MIM methods as a whole are 1) we mask the student model at its output layer rather than its input layer, 2) we mask the teacher as well to create reconstruction targets that vary with the mask $m$. These differences encourage MaskEmbed to learn specific features associated with each patch, whereas MIM teaches a model to impute features for missing patches. Based on your comment, we’ve clarified these differences in our revised Appendix C.1 under a section called “Relationship with masked image modeling,” and we also have a new extended related work section in Appendix A.

We’re also happy to include results for dBOT. We’ll separately describe our attempts to use dBOT’s checkpoints trained with the random teacher and CLIP teacher below:

• dBOT’s random teacher checkpoint was straightforward to load from the official GitHub repository, and we evaluated it using our probing benchmark from Section 4. The results are slightly better than MAE, but this backbone is not competitive with language-supervised models like CLIP. This shows that dBOT’s main proposal leaves large room for improvement on this benchmark. With locality alignment, we can start from a strong model like CLIP and make it even stronger with a short fine-tuning phase.

• dBOT’s CLIP teacher checkpoint would have been more interesting, because MIM and locality alignment are competing interventions to improve a strong pre-trained model like CLIP. Unfortunately, we were unable to get reasonable results with the model provided by the authors (described in more detail below). However, because dBOT’s proposal of bootstrapping the teacher didn’t work for CLIP (as shown in their Appendix C), their CLIP-teacher checkpoint used standard single-stage MIM, and a reasonable surrogate is to compare with other MIM methods that use CLIP teachers. One such model is EVA02 from Fang et al. (2023), which has the advantage of using a stronger EVA-CLIP 1B teacher. EVA02 was already included in our experiments (Table 5), and we find that it’s indeed better than the pre-trained CLIP ViT-B, but locality alignment leads CLIP to exceed EVA02 at both local and global probing. Interestingly, we also find that locality alignment also significantly improves EVA02’s local probing performance, suggesting that MIM alone does not saturate its performance at this task.

The following table shows the relevant results from the probing benchmark:

Re: loading dBOT CLIP-teacher checkpoint. We loaded the checkpoint using the authors’ beit branch, which provides separate modeling code for pre-training and fine-tuning, with checkpoint loading code in their modeling_finetune.py script. The probing benchmark results were worse than the dBOT random teacher checkpoint, so we suspected an issue and took several steps to identify it. We ensured that 1) the model was initialized with the correct arguments, 2) all relevant keys were loaded into the model, 3) there were no clear inconsistencies in the pre-training and fine-tuning modeling code, 4) the forward_features outputs were identical when the checkpoint was loaded into the pre-training and fine-tuning models. We couldn’t identify the problem with the checkpoint, but we believe our EVA02 comparison provides reasonable evidence that MIM with a pre-trained teacher is not a substitute for locality alignment.

Thank you for taking the time to read our paper and provide helpful feedback! We respond below to the points raised in your review.

I suggest the authors to conduct a thorough analysis of MaskEmbed's sensitivity to hyperparameters. This includes varying mask sizes, patch sampling strategies, and the influence of different reconstruction targets.

We included a set of ablations that helped guide hyperparameter selection, please see Appendix B.1 in our updated submission. These include all the experiments you suggested, including sampling masks of varying sizes and with different patch sampling strategies (block-structured and random/unstructured), and the choice of reconstruction target. They also include ablations for the use of stronger augmentations, the training dataset and training duration, and we ran new loss function ablations requested by reviewer XvKA (discussed in their response). Finally, we also ran MaskEmbed with a variety of teacher models, and these results are shown in Figure 4 and Table 5 (including IN1k classifiers, CLIP, SigLIP, DFN, EVA02, MoCov3, etc).

Besides, including additional evaluations that specifically test the impact of MaskEmbed on global semantic understanding tasks may help to validate whether the method indeed compromises the model's ability to capture long-range dependencies.

Thanks for this suggestion, it’s a good idea to include experiments for global semantic understanding. We’ve added new results where we fine-tune models for IN1k classification before and after locality alignment, and we verified that performance does not degrade but instead improves after locality alignment. This is likely because when learning patch-specific features, the model is also trained to retain its original prediction: the all-ones mask $m = 1$ corresponds to reconstructing the original teacher output $f(x)$, so preserving global understanding is built into our objective.

Our IN1k fine-tuning results for CLIP models of different scales are shown in the following table, and we’ll include them in our revised submission. Our hyperparameters are chosen to match those in OpenCLIP. In addition to achieving better final results, locality-aligned models also trained significantly faster in the early epochs, suggesting that they learn strong features that are easy to adapt with fine-tuning.

This approach, while innovative, raises concerns about its impact on the encoder's ability to model global interactions and capture long-range dependencies, which are crucial for tasks requiring global semantic understanding.

Thank you. Please note that in addition to our new IN1k results, our previous results also showed that locality alignment generally improves models’ global probing performance (Figure 4 and Table 5), and this improvement is consistent with our gains on VLM benchmarks (Figure 5). Overall, our results consistently show that loss of global semantic understanding is not an issue.

Thank you for taking the time to read our paper and provide helpful feedback! We respond below to the points raised in your review.

However, pre-training methods like DINO, despite using image-level supervision, exhibit strong locality, to the point where their features can even be directly used for semantic segmentation maps. This suggests that the initial assumption may be flawed.

Thanks for giving us the opportunity to clarify. A few models like DINO, DINOv2 and MAE exhibit relatively good locality, but this is because they’re trained with some form of dense supervision. Note that the dense supervision for MAE and DINOv2 is explicit, but for DINO it’s due to multi-crop training with random sub-images, see Caron et al. (2021) for more details. Unlike these models trained with dense supervision, other models trained with only image-level supervision have clear room for improvement in their ability to extract localized features, as we demonstrate by boosting their performance via locality alignment: our procedure uses only the model itself as supervision yet reliably improves local probing accuracy, and we show this for models including IN1k classifiers, CLIP, SigLIP, DFN, OpenCLIP and Moco v3 (see Figure 4 and Table 5).

We believe our focus on these models trained with image-level supervision is warranted, because language-supervised models like CLIP/SigLIP are widely adopted in VLMs, whereas MAE and DINOv2 are not as effective (see Karamcheti et al., 2024; Tong et al., 2024). For these models that don’t excel at extracting localized semantics, we successfully address this shortcoming with locality alignment.

Additionally, regarding the claim about minimal inductive biases, is the paper referring to the lack of inductive biases in ViT architectures? If so, would using a convolution-based structure like ConvNext or a hierarchical structure like Swin Transformer resolve this issue?

Thanks for raising this point. Yes, we’re referring to the lack of inductive bias in ViTs. We believe our focus on ViTs is justified given that they’re used for all recent state-of-the-art models and adopted in nearly all VLMs. However, you’re correct that a lack of local semantics could potentially be resolved by architectural solutions, such as reintroducing inductive biases like those in ConvNext or Swin Transformer, or even stronger options like an architecture that processes patches independently. It’s a reasonable idea that we also considered, but we’re more interested in solutions based on the training objective, because that lets us retain maximal expressive power and improve the best current models.

Based on your comment, we’ve made sure this point is reflected in the paper. Our related work section contains the following sentence in the context of ViTs lack of localized features: “Some have proposed hybrid ViTs that reintroduce inductive biases from CNNs (Liu et al., 2021, Wu et al., 2021, d’Ascoli et al., 2021), but we improve the original ViT’s local feature extraction without sacrificing expressive power.”

This approach could weaken the encoder's ability to model global interactions, potentially limiting its capacity to capture long-range dependencies. Such a strategy could harm performance on tasks requiring global semantic understanding.

Thanks for this suggestion, it’s a good idea to include experiments for global semantic understanding. We’ve added new results where we fine-tune models for IN1k classification, and we verified that performance does not degrade but instead improves after locality alignment. This is likely because when learning patch-specific features, the model is also trained to retain its original prediction: the all-ones mask $m = 1$ corresponds to reconstructing the original teacher output $f(x)$, so preserving global understanding is built into our objective.

Our IN1k fine-tuning results for CLIP models of different scales are shown in the following table, and we’ll include them in our revised submission. Our hyperparameters are chosen to match those in OpenCLIP. In addition to achieving better final results, locality-aligned models also trained significantly faster in the early epochs, suggesting that they learn strong features that are easy to adapt with fine-tuning.

Now, we'll support this perspective with some experiments. Our arguments above suggest that MIM with a pre-trained teacher $f$ results in a model $h$ that’s perhaps a bit better than the original, but that locality alignment on the MIM version $h$ should lead to a much better model $g$. We conducted this comparison in our codebase with IN21k training data, $f$ as a CLIP ViT-B, and $h$ trained using MIM with a couple small improvements (fine-tuning vs training from scratch, using our $p(m)$ from Section 4.2, and using a loss that considers all patches). We then applied locality alignment to both the CLIP model $f$ and MIM model $h$, and and evaluated all the models on our probing benchmark. Here are the results:

The key observations are:

1. MIM improves upon the CLIP teacher, but not as much as locality alignment
2. When we apply locality alignment to the MIM teacher, we get the best overall results: the best local probing accuracy by a solid margin (consistent with our arguments above), and nearly the best global probing performance (roughly matching locality alignment applied directly to the CLIP teacher)

To summarize, MIM seems less like an alternative to locality alignment than an ideal precursor. Doing this can add an extra step to our pipeline, unless the model is pre-trained with random masking; a simple alternative is to apply locality alignment directly, as we do in our experiments. If anything, this means we've understated the effectiveness of locality alignment in our current results.

Overall, we hope this resolves the questions about the similarity between these ideas. They’re different procedures with different aims, but they’re complementary when applied together. We’ll be sure to clarify this in the revised paper, including a recommendation that researchers consider applying an initial MIM stage to maximize the benefits of locality alignment.