Enhancing Vision-Language Model with Unmasked Token Alignment at Scale

Thank you for your detailed review. We address the concerns below.

Q1: Motivation on unmasked token alignment and comparison with MIM.

R1: We share the similar spirit with MIM works (EVA) by using the partial inputs (masked images) to predict the CLIP features obtained by full inputs. However, MIM is a too difficult pretext task which hinders the learning of representations. In addition, using full inputs (without mask) to predict the CLIP features is too easy. Our method balances the learning difficulty by proposing unmasked tokens alignment. Compared to MIM, our method 1) is training-efficient by dropping the masked tokens and 2) can converge more quickly by only aligning the unmasked tokens. We show that UTA can match MIM with only 60% epochs of training. We summarize the results in the table below.

Q2: The method is similar to the simple distillation process which drops tokens randomly.

R2: Simplicity should be the advantage of our method instead of disadvantage. In our paper, we show that our simple method can already outperform SOTA method on a broad of uni- and multi-modal tasks while being much more training-efficient.

Q3: The claim “It causes training-finetuning inconsistency and makes the trained ViT unable to perform zero-shot classification without fine-tuning” is not accurate because the trained ViT could be transferred to unseen domains.

R3: The zero-shot classification refers to the evaluation method proposed by CLIP instead of transferring the pre-trained model to unseen domain. As in CLIP, zero-shot evaluation is to use text encoder to encode the classnames, vision encoder to encode images, and match them with cosine similarities. The output features of MIM pre-trained models are not aligned with CLIP text encoder. Therefore, those pre-trained models are unable to perform zero-shot classification without fine-tuning. We also verify the claim in experiments by conducting zero-shot classification with the pre-trained EVA model which uses [MASK] tokens. We only obtain 0.1% accuracy on ImageNet-1K, which is the same as random guess.

Q4: In terms of technical novelty, this paper lacks some careful design.

R4: Our method is designed to outperform SOTA EVA-CLIP and Open-CLIP with simple modifications. Simplicity should be considered as a part of technical novelty but not a disadvantage.

Q5: Weak connection between the part of the ablation study (Tab. 7) and motivation.

R5: The ablation study enhances our method by showing that our approach is robust in terms of position embedding. We speculate that MIM with pre-trained CLIP can be seen as a combination of (1) unmasked token alignment as our approach does (2) performing interpolation of the unmasked tokens’ features to predict the masked tokens. So we believe that if we remove the position embedding, i.e., break down the (2) part, MIM will perform worse, which is shown in Table 7.

Q6: Give some details about the advantages or disadvantages of the method you proposed in our experiments.

R6: Our experiments show that our method has strong performance and is training efficient. Our ablation studies show the impact of different design choices. We present those results with tables and detailed descriptions, which we believe are comprehensive. I hope the reviewer can specify which advantage or disadvantage we need to add. We are willing to include it in the revised version.

Q7: Why use LLaVA-Bench for testing?

R7: Large Multi-modal Model (LMM) is a highly regarded research direction in computer vision, and the latest LMM are all based on SOTA EVA-CLIP/Open-CLIP models. To have a comprehensive comparison with those SOTA CLIP models, we test our model’s performance on LLaVA-Bench.

Q8: Why UTA-g/14 model did not beat that of L/14?

R8: Note that the overall performance of UTA-g/14 (83.1) is better than L/14 (82.0). The overall performance averages the performance of the other 3 categories, which can be a more comprehensive indicator of performance.

Q9: Minor improvement compared to EVA-02.

R9: Note that our method performs better than EVA-02 on all tested tasks. We believe the comprehensive improvements over SOTA method are not trivial. It is worth noting that our method achieves a 0.5 mIoU improvement on ADE20K, as well as a 0.6 increase in $AP^{box}$ and 0.9 in $AP^{mask}$ on LVIS, which are substantial improvements. Additionally, our method is training-efficient and can converge faster as stated in R1.

Thank you for your detailed review. We address the concerns below.

Q1: Motivation of reducing pre-training cost is not convincing as UTA relies on a pre-trained CLIP model at the first place.

R1: Even with a pre-trained CLIP model, our total cost is still lower than EVA-CLIP or Open-CLIP. Compared to EVA-02-CLIP-B, we use the same training schedule but require much less training cost. Note that EVA-02-CLIP-B is obtained by contrastive fine-tuning from EVA-02-B. In the pre-training stage, we use the same dataset (ImageNet-21K), teacher model (EVA-01-CLIP-g), and training epochs (150) as in EVA-02-B. However, UTA significantly reduces training FLOPs by 40% by not processing masked tokens, a key distinction from the approach of EVA-02-B. For the contrastive fine-tuning, we require fewer image-text pairs (2B vs. 8B in EVA-B-CLIP) but achieve better results (77.0 acc. vs. 74.7 acc.). We break down the training samples and show the performance in the table below.

Compared to Open-CLIP-g (34B pairs, 78.5 acc.), our results are UTA-g (0B pair, 79.3 acc.) and UTA-CLIP-g (2B pairs, 81.5 acc.). Considering that the teacher CLIP model uses 11B pairs, our method still requires fewer image-text pairs and obtains better results than CLIP models that train from scratch.

Q2: Performance gains come from contrastive fine-tuning.

R2: The SOTA EVA-CLIP also used contrastive fine-tuning. Taking the EVA-02-CLIP-B as an example, they first use a ViT-B model to perform masked image modeling with EVA-01-CLIP-g as the teacher, and then use the pre-trained ViT-B model to perform contrastive fine-tuning with 8B image-text pairs, which obtains 74.7 zero-shot accuracy. In comparison, we use the same teacher model to performance UTA pre-training, and further perform contrastive fine-tuning with 2B image-text pairs. Our pre-trained model obtains 76.0 zero-shot accuracy. Our fine-tuned model obtains 77.0 zero-shot accuracy. The training setting is a little complicated. However, we use the same setting in EVA-02 to ensure fair comparison. Note that EVA-02 is an improved version of original EVA, and we focus on the comparison with EVA-02 in our paper. We updated Tables 1 and 2 in our revised version for clarification.

Q3: Missing baseline that fine-tunes the EVA-CLIP on the same dataset.

R3: As stated in R1 and R2, the results of EVA-CLIP are also obtained by extensive contrastive fine-tuning.

Q4: Cite FLIP.

R4: Thanks for the suggestion. FLIP is cited in the revised version.

Q5: Is the CLIP teacher-model always the giant EVA-CLIP?

R5: By default, we use giant EVA-CLIP as the teacher model to align the setting in EVA-02. We ablate the CLIP teacher model of different sizes in Tab.8, we find that using stronger CLIP model can largely improve the results on classification and semantic segmentation, while this is not the case for object detection or instance segmentation.

Q6: Provide the CLIP teacher results in Table 3, 4, & 5.

R6: Thanks for the suggestion. The results are added in the revised version.

Q7: Provide reference that uses second-stage contrastive fine-tuning.

R7: As stated in R1 and R2, the EVA-CLIP utilized second-stage contrastive fine-tuning.

Q8: Is fine-tuning helpful in Table 4 & 5?

R8: Thanks for the suggestion. We tried such experiments and found that contrastive fine-tuning is not helpful for those tasks. The conclusion is similar to other MIM works that the task-agnostic pre-training is better than task-specialized models when fine-tuning on other tasks. We summarize the results below and have added the results in our revised version.

Q9: Is masking strategy applied in the fine-tuning stage.

R9: No.

Thank you for your detailed review. We address the concerns below.

Q1: Will the new module bring extra training cost?

R1: Our overall training cost is lower than SOTA method EVA-02. As shown below (Table 6 in the paper), we achieve stronger results on all tasks with fewer training FLOPs.

Q2: Unfair comparison with MAE as UTA uses extra teacher.

R2: It is important to note that more advanced variants of MAE, such as BEiT-v2, EVA, and EVA-02, also utilize additional teachers in their training methodologies. In our study, we explicitly address this concern by providing a comprehensive comparison with these advanced variants in Tables 4 and 5.

Thank you for your detailed review. We address the concerns below.

Q1: Contribution and performance gap are small compared to Feature Distillation.

R1: We appreciate the reviewer's attention to the comparison between our proposed method (UTA) and Feature Distillation (FD).

1. Note that the results of FD in Table 6 are our revised version by incorporating negative cosine loss. To ensure a fair comparison, we employed the same dataset and architecture for training. The original FD uses smooth L1 loss for training, and therefore the pre-trained vision encoder cannot be used for zero-shot evaluation because the embeddings of pre-trained vision encoder and CLIP text encoder are not aligned in the normalized embedding space. In comparison, the pre-trained vision encoder of UTA can be directly applied for zero-shot evaluation and obtain decent zero-shot performance after the unmasked alignment.
2. Our method outperforms FD across all tasks while reducing the pre-training FLOPs by a significant 40%. Specifically, on COCO, our improvements include 0.5 $AP^{box}$ and 0.4 $AP^{mask}$. On more challenging LVIS, we observe improvements of 1.0 $AP^{box}$ and 0.9 $AP^{mask}$. Additionally, on ADE20K, we improve FD by 0.7 mIoU. We emphasize that these performance enhancements are substantial, considering our utilization of a strong baseline setting. While acknowledging that the improvement on ImageNet is relatively modest, we attribute this observation to the comprehensiveness of the training dataset (ImageNet-21K), which effectively covers ImageNet.

Q2: Unfair to compare UTA against Open-CLIP / EVA-CLIP as UTA uses ImageNet-21K.

R2: The SOTA EVA-CLIP also used ImageNet-21K for pre-training. To ensure strictly fair comparison with SOTA EVA-CLIP, we use the same setting for pre-training, including dataset and architecture. Note that the EVA-CLIP models are fine-tuned from EVA-02 models, which also use ImageNet-21K for pre-training. As the settings of Open-CLIP and EVA-CLIP are very different, we choose the stronger EVA-CLIP as the baseline setting.