VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions

We thank Reviewer Sser for the constructive feedback and insightful comments. We hope to address the concerns of the Reviewer with the responeses below.

W1. More Evaluations on Compositionality Benchmarks.

We will ensure these results are featured prominently in the main body of the revised paper. To better address the reviewer's valid concern and provide a more comprehensive analysis, we have expanded our evaluation in the table below:

As evidenced by the results, our CLIP-IN framework consistently outperforms baseline models across a diverse range of compositional benchmarks:

Consistent Improvement over Baselines: Our method significantly surpasses the original OpenAI CLIP (ViT-L/224), with notable gains of +5.5% on ARO, +1.9% on SugarCrepe, and +4.0% on SPEC. This trend persists across nearly all benchmarks when using other backbones.

Superiority over SigLIP2: Our framework enhances the compositional reasoning capabilities of the robust SigLIP2 model, boosting performance across all four benchmarks when applied to its backbone, which underscores the broad applicability and effectiveness of our approach.

W2. More Evaluations on CLIP-Benchmark.

We agree with the reviewer that it is crucial to provide more evaluations on CLIP-Benchmark and we include more results in the following table. It shows the our proposed method obtains consistent improvement over baseline in the zero-shot image classification task.

W3. More Ablation Studies.

While our original manuscript (Table 5 and Table 6) included an ablation study on the impact of our two primary data sources (instruction editing data and long caption data), we agree that more detailed ablations on the core technical components of our framework are beneficial. In response, we have conducted two new component-level ablation studies to validate the effectiveness of our proposed symmetric hard negative loss and our RoPE-based text encoder distillation.

1) Ablation on RoPE-based Text Encoder Distillation.

To evaluate the benefits of our RoPE-based text encoder distillation, we compared it to a robust baseline utilizing absolute positional encoding interpolation, commonly adopted by LongCLIP. We initialized the second training stage with LongCLIP, a text encoder pre-trained with positional interpolation, and contrasted it with our RoPE-based model, keeping all other settings constant.

The results show that our RoPE-based approach is a more effective method for extending the text encoder's context length. It significantly outperforms the interpolation baseline on general classification (IN-1K: +1.2%), long-text retrieval (+6.0%).

2) Ablation on Symmetric Hard Negative Loss (L_HN​).

To evaluate the effectiveness of our symmetric hard negative contrastive loss (L_HN), we compared it with two other hard negative mining strategies: NegCLIP Loss, which uses only the image-to-text hard negative term, and Triplet Loss, which employs separate image-to-text and text-to-image triplet losses but without a symmetric formulation.

Using our primary ViT-L/14@336 model for this ablation, the results clearly demonstrated the superiority of our symmetric loss design, consistently and significantly outperforming both alternatives across all evaluated benchmarks. Compared to the Triplet loss, our symmetric loss achieved an impressive +2.8% improvement in ImageNet-1K accuracy and a 2.0% boost in average short-text retrieval scores. The performance gap is even more pronounced when compared to the one-directional NegCLIP loss, with gains of +3.1% on ImageNet-1K and +2.4% in average retrieval.

Q1.

That was a typo in our draft. We will revise it in the final version.

Q2. Comparisons to FG-CLIP.

The identified gap in long caption retrieval performance appears to stem from the disparity in training data scale (18M pairs vs. FG-CLIP's 1B pairs). To address this, we expanded our training dataset from 18M to 30M pairs. The results demonstrate our framework's scalability, as shown in the table below:

Our updated model, trained on 30M pairs, achieves a long caption retrieval accuracy of 85.1%, surpassing FG-CLIP's performance of 81.8% by +3.3%. This highlights our framework's capability to excel with significantly less training data (over 30 times less data). Despite the improvements in retrieval performance, our classification results remain similar to those of FG-CLIP, ensuring a balanced overall performance across various metrics.

We thank Reviewer RrCr for the constructive feedback and insightful comments. We hope to address the concerns of the Reviewer with the responses below.

W1. Novelty and Contributions

While it is true that our framework focuses on enhancing both long-text understanding and fine-grained perception, the essence of our contributions and innovation lies in the specific methodologies employed to achieve these objectives, and most importantly, in the synergistic integration of these components.

1) Novel Hard-Negative Sourcing

Our primary contribution is being the first to repurpose instruction-editing datasets as a novel source of hard negatives for contrastive learning. Unlike prior work that synthetically generates hard negatives via text-to-image models (TripletCLIP) or rule-based text perturbations (NegCLIP), our approach uses data that provides high-fidelity hard negatives from real, minimally-edited image pairs, enabling a more effective and symmetric contrastive loss. By demonstrating a new, effective use for these datasets beyond their original purpose, our work contributes a valuable insight to the community.

2) Novel Synergistic Combination

We are the first to propose a framework that synergistically combines these two complementary forms of fine-grained information:

• Instruction Editing Data (Image-centric): Provides fine-grained information for the visual representation.
• Long Caption Data (Text-centric): Provides fine-grained information for the textual representation.

This fusion of two complementary supervisory signals is the core of our framework. Our ablation studies (Tables 5 & 6 in manuscript) confirm this synergy, showing that the combination of both data types achieves the best performance and significantly outperforms models trained on either one alone.

W2. Ablation on RoPE-based Text Encoder Distillation

To isolate the benefit of our RoPE-based text encoder distillation, we compare it against a strong baseline that uses absolute positional encoding interpolation, as popularized by LongCLIP. We initialize our second stage of training with LongCLIP, a text encoder pre-trained using positional interpolation and compare it to our proposed RoPE-based model. All other settings are identical.

The results show that our RoPE-based approach is a more effective method for extending the text encoder's context length. It significantly outperforms the interpolation baseline on general classification (IN-1K: +1.2%), long-text retrieval (+6.0%).

W3. Capability of Long Caption Data

In response to the reviewer's query, we provide the following analysis and new experimental data to clarify the necessity of the long-text component within our synergistic framework.

• The reviewer's observation regarding the drop in MMVP performance when using only long-text data (as shown in our initial ablation study in Table 6) highlights the central focus of our work. It demonstrates that the components are synergistic, and their true potential is realized only when they are combined. While a single data source in isolation may not improve every metric, the final model combining both instruction data and long captions showed a +3.3% improvement on the MMVP MLLM benchmark over the baseline.

• To directly test the reviewer's valuable hypothesis that a simpler model (using 18M original short captions and hard negatives) could enhance fine-grained perception, we conducted a new experiment. The results are detailed below, comparing the simpler model ("w/o long captions") against our full framework.

The results from this new experiment provide a definitive answer:

• Fine-Grained Perception: The data shows the simpler model does not enhance fine-grained perception; in fact, it achieves an MMVP score of only 20.7. Our full framework, with the long-text component, reaches a substantially higher score of 26.1.

• Retrieval Performance: Our full model also demonstrates significantly stronger performance across all retrieval tasks, outperforming the simpler model on both short-text retrieval (73.1 vs. 65.45) and long-text retrieval (76.35 vs. 69.0).

• Classification: The full framework also maintains a stronger ImageNet-1K classification score (77.0 vs. 76.3).

This experiment confirms that the long-text capability is a necessary component. The synergistic combination of long-caption context and hard-negative training is essential to achieve the reported gains in fine-grained visual perception and overall model performance.

W4. Model for Long Caption Generation

We selected InternVL2 because it was one of the state-of-the-art open-source vision-language models available at the time of our research. Our method is not dependent on this specific choice and could easily incorporate detailed captions generated by other current or future state-of-the-art models.

We thank Reviewer mmoh for the constructive feedback and insightful comments. We are grateful for the acknowledgment of our work's strengths, including the clever repurposing of instruction-editing datasets, the well-structured framework, and its potential to benefit the community. We hope to address the concerns of the Reviewer with the responses below.

W1. Scaling Long Caption Data

We agree with the reviewer that demonstrating the scalability of our framework is crucial. To directly address the reviewer's question about scalability, we have conducted a new experiment where we increased the amount of long caption data for training CLIP-IN from 18M to 30M pairs. The results, compared with our initial model and FG-CLIP, are presented below.

Doubling the long caption data from 4M to 8M results in a dramatic improvement in the Long Retrieval Average from 76.4% to 85.1% (+8.7 points). Performance also improves on both Short Retrieval (+1.8 points) and the fine-grained MMVP benchmark (+1.3 points). This clearly demonstrates that our framework's performance scales effectively with more data. With this new data, our model now outperforms FG-CLIP on the primary Long Retrieval metric (85.1% vs. 81.8%) and also surpasses it on Short Retrieval. This is achieved while using vastly less data (30M vs. 1B), highlighting the data efficiency of our synergistic training approach. While there is a slight trade-off in general classification, our ImageNet-1K score remains competitive with FG-CLIP.

Q1. Ablation on RoPE and the Proposed Hard Negative Loss

1) Impact of Rotary Positional Embeddings (RoPE)

To isolate the benefit of our RoPE-based text encoder distillation, we compare it against a strong baseline that uses absolute positional encoding interpolation, as popularized by LongCLIP. We initialize our second stage of training with LongCLIP, a text encoder pre-trained using positional interpolation and compare it to our proposed RoPE-based model. All other settings are identical.

The results show that our RoPE-based approach is a more effective method for extending the text encoder's context length. It significantly outperforms the interpolation baseline on general classification (IN-1K: +1.2%), long-text retrieval (+6.0%).

2) Impact of the Symmetric Hard Negative Loss (L_HN​).

To validate our proposed symmetric hard negative contrastive loss (LHN​), we compare it against two other hard negative mining strategies:

NegCLIP loss: Uses only the image-to-text hard negative term.

Triplet loss: Uses separate image-to-text and text-to-image triplet losses but without the symmetric formulation.

We use our main ViT-L/14@336 model for this ablation. The results below confirm the superiority of our symmetric loss design.

Our approach consistently and substantially outperforms both alternative strategies across all evaluated benchmarks. Compared to the Triplet loss, our symmetric loss achieves a remarkable +2.8% improvement on ImageNet-1K accuracy and boosts the average short-text retrieval score by 2.0%. The performance gap is even more pronounced when compared to the one-directional NegCLIP loss, where our method shows gains of +3.1% on ImageNet-1K and +2.4% on average retrieval.

Q2. Sensitivity to Instruction Editing Data.

1) The Effect of Scale. The framework's performance scales positively with the amount of instruction editing data. As detailed in the original manuscript's ablation study, increasing the dataset size from 1.4 million to 4 million samples improved ImageNet-1K accuracy by 0.6% while maintaining high performance on retrieval tasks. This indicates that the model benefits from more data.

2) The Effect of Selection and Variety. To analyze the impact of data variety and potential biases, we performed two controlled experiments using a 1.4 million sample subset as a baseline. We compared the baseline, which was randomly sampled, against one curated subsets: balanced by the most frequent visual concepts (objects and colors). By curating the dataset to ensure a balanced distribution of core visual concepts, the model achieves substantial performance gains on tasks that require detailed understanding. This strategy leads to a significant improvement in both short-text retrieval (+2.4) and, critically, fine-grained visual perception on MMVP (+2.0).

We thank Reviewer Kftn for the constructive feedback and insightful comments. We hope to address the concerns of the Reviewer with the responeses below.

Q3. Contributions, w.r.t. prior works.

1) Novel Hard-Negative Sourcing Our primary contribution is being the first to repurpose instruction-editing datasets as a novel source of hard negatives for contrastive learning. Unlike prior work that synthetically generates hard negatives via text-to-image models (TripletCLIP) or rule-based text perturbations (NegCLIP) , our approach uses data that provides high-fidelity hard negatives from real, minimally-edited image pairs, enabling a more effective and symmetric contrastive loss. By demonstrating a new, effective use for these datasets beyond their original purpose, our work contributes a valuable insight to the community.

2) Novel Synergistic Combination We are the first to propose a framework that synergistically combines these two complementary forms of fine-grained information:

Instruction Editing Data (Image-centric): Provides fine-grained information for the visual representation.

Long Caption Data (Text-centric): Provides fine-grained information for the textual representation.

This fusion of two complementary supervisory signals is the core of our framework. Our ablation studies (Tables 5 & 6 in manuscript) confirm this synergy, showing that the combination of both data types achieves the best performance and significantly outperforms models trained on either one alone.

To illustrate the benefits of our synergistic approach over methods that solely focus on long-text capabilities, we compare our model with LongCLIP. While LongCLIP extends the text encoder, our method complements this with instruction-editing data and uses RoPE for more effective long-text processing. The results below demonstrate the superiority of our approach.

Our model significantly outperforms LongCLIP in both classification and retrieval, demonstrating that our synergistic approach leads to more robust and capable representations. We hope this clarifies the scope and significance of our primary contributions, which we will better stress in the introduction of the paper.

Q1. Comparions to NegCLIP [a].

As suggested, we provide a more direct comparison with prior works like NegCLIP[a] on compositional reasoning benchmarks. We ablate our proposed hard negative loss with the NegCLIP loss.The results indicate that the symmetric LHN​ provides a more robust and well-rounded improvement to the model's capabilities. On standard benchmarks, the symmetric loss shows significant advantages, outperforming the NegCLIP loss by +3.1% on ImageNet-1K classification and +2.4% on the short-text retrieval average. This enhanced performance extends to complex compositional reasoning benchmarks like ARO and Winoground, where the proposed loss also achieves higher scores. While the NegCLIP loss scores higher on the SugarCrepe benchmark, which heavily focuses on text-based object and attribute swaps, our proposed symmetric loss demonstrates stronger overall performance. This suggests that the symmetric LHN​ fosters a more comprehensive fine-grained understanding by learning from both image and text mismatches, rather than specializing in a single type of textual perturbation.

Q2.Evaluations on Compositional Benchmarks.

We would like to gently point out that our original manuscript already includes a detailed evaluation on three compositional reasoning benchmarks—SugarCrepe, Winoground, and SPEC in the supplementary materials. To better address the reviewer's valid concern and provide a more comprehensive analysis, we have expanded our evaluation in the table below:

As evidenced by the results, our CLIP-IN framework consistently outperforms baseline models across a diverse range of compositional benchmarks:

Consistent Improvement over Baselines: Our method significantly surpasses the original OpenAI CLIP (ViT-L/224), with notable gains of +5.5% on ARO, +1.9% on SugarCrepe, and +4.0% on SPEC. This trend persists across nearly all benchmarks when using other backbones.

Superiority over SigLIP2: Our framework enhances the compositional reasoning capabilities of the robust SigLIP2 model, boosting performance across all four benchmarks when applied to its backbone, which underscores the broad applicability and effectiveness of our approach.

W4. Transferability to real-world datasets.

We would like to offer two key pieces of evidence from our paper that demonstrate our method's strong transferability to real-world data:

1) Grounded in Real Images: Unlike methods that generate images from scratch, the UltraEdit dataset starts with real images (i.e., high-quality image-caption pairs from diverse real image datasets like COCO) and applies localized, instruction-guided edits. As we state in our paper, this "mitigates domain biases of purely generated data". This grounding makes the learned representations more robust.

2) Strong Zero-Shot Performance: The most direct evidence of transferability is our model's performance on standard, real-world benchmarks. As shown in Table 1 of original manuscript, our CLIP-IN model achieves significant zero-shot performance gains on ImageNet-1K, Flickr30K, and MS-COCO. For instance, with a ViT-L/14 backbone, our method improves average short-text retrieval accuracy from 62.5% to 73.1% and ImageNet-1K accuracy from 76.6% to 77.0% over the strong OpenAI CLIP baseline. These consistent improvements on diverse, real-world datasets shows that our training strategy generalizes effectively and is not limited to the training distribution.