Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions

To Reviewer 4 (gRGm)

We thank the Reviewer gRGm for the detailed review and constructive suggestions. We appreciate your acknowledgement that our method achieves strong result, our ablation studies are valuable, and our evaluation is solid and thorough. Please find our answers to your comments and questions as follows.

[R4-1] This is more of a minor issue, but some parts of the method section, particularly the mathematical formulation, are difficult to follow. The notation is heavy, with many variables introduced at once, which can make the equations hard to parse on a first read. A clearer presentation of the method, possibly with simplified or modular notation, would improve readability.

Thank you for highlighting this point. To address this concern, we plan to improve the clarity in the revised manuscript by:

• Modularizing the notation around core conceptual units (e.g., alignment vs. reconstruction),
• Adding intuitive summaries between mathematical blocks to guide the reader through key steps.

We will ensure that these revisions preserve mathematical precision while significantly improving readability. Thank you again for this constructive suggestion.

[R4-2] Another potential issue is that, on a first read, the method may appear to require the generation of multiple captions per image, which could be perceived as a limitation. It would be helpful to clarify early in the paper, either in the Introduction or Related Work, that datasets like MS-COCO already exist and provide multiple captions per image, making the method more accessible. Additionally, when MS-COCO is mentioned in Section 4.1, a brief description of its structure would help readers who may not be familiar with the dataset.

We thank the reviewer for the thoughtful comment.

In the revised manuscript, we will clarify early in the paper (Introduction or Related Work) that datasets such as MS-COCO already provide multiple human-written captions per image. This will help readers understand that our method does not require any additional caption generation. Furthermore, when introducing MS-COCO in Section 4.1, we will include a brief description of its structure to better support readers who may be less familiar with the dataset.

[R4-3] Clarification on the paraphrasing process: The paper mentions paraphrase generation through augmentation, but the details remain unclear. Did you use a specific LLM to generate paraphrases? If so, which one, and under what settings? Or are you using a VLM to do this, conditioning on both text and image? If a VLM was used for paraphrasing, this could raise a concern about circularity, i.e., relying on VLM, that you criticized for poor compositional reasoning, to improve compositional reasoning of other VLMs. Clarifying this aspect would strengthen the methodological transparency of the paper.

Thank you for raising this important point.

In our experiments, paraphrasing was performed using a text-only LLM, not a vision-language model. As mentioned in supplementary material, to generate paraphrased captions for our sentence-level alignment loss, we use the gpt-4o-mini-2024-07-18, and details are as follows:

• Each original caption was provided as text-only input.
• No image or visual features were used during generation.
• We used a simple prompt (shown in Appendix Fig. 6), with temperature = 1.0 and all other decoding parameters left at default settings.
• For every caption in the COCO dataset, we generated one paraphrased variant offline before training.

We will include these details in the revised manuscript.

[R4-4] Highlighting the originality of the contribution: While the reconstruction and alignment losses are motivated and well implemented, their novelty may not be immediately evident to readers who are less familiar with the literature. I recommend emphasizing more explicitly how your approach differs from prior work using reconstruction losses, specifically, by explaining that although similar techniques exist for general-purpose encoder training, they have not been applied to the problem of improving compositional reasoning in contrastive VLMs. Making this distinction more visible could better communicate the originality of your work.

Thank you for this helpful suggestion. As you rightly pointed out, the core novelty of our work lies in introducing token-level reconstruction and sentence-level alignment objectives into the contrastive training of vision-language models (VLMs)—not for general-purpose encoding, but explicitly to improve compositional reasoning.

While text reconstruction and text alignment objectives have appeared in prior work (e.g., for general-purpose language model training), to the best of our knowledge, our work is the first to adapt these textual supervision for improving compositional generalization in VLMs trained with contrastive learning like CLIP.

We acknowledge that this distinction may not be immediately clear to readers unfamiliar with the relevant literature. Accordingly, we will revise the manuscript to emphasize this point more explicitly in the Introduction and Related Work sections.

[R4-5] More qualitative examples: The qualitative examples presented in the main paper are insightful. However, I suggest including additional qualitative examples, especially failure cases and edge scenarios, maybe in the appendix. This would help readers better understand the behavior of the model under different conditions and strengthen the empirical analysis.

Thank you for this thoughtful suggestion. As per the reviewer's comment, we will augment the appendix in the revised manuscript with additional qualitative examples, including:

• Cases where both READ-CLIP and other baselines fail, highlighting shared limitations in compositional reasoning.
• Cases where READ-CLIP fails but other baselines succeed, which can reveal specific weaknesses unique to our method.

Reviewer gRGm, please engage in the discussion period. I understand that the review period started over a weekend, but we only have a few days remaining in the (now slightly extended) discussion period. The authors have provided a thoughtful response to your review and you are obligated to respond to it. You should share with the authors if they addressed questions or concerns you had, and seek clarification about any questions or concerns that remain. Please post your response as soon as you can so that there is time for the authors to follow up and discussion to progress as needed.

To Reviewer 2 (a9tz)

We thank the Reviewer a9tz for the detailed review and constructive suggestions. We appreciate your acknowledgement that the paper is well motivated, easy to read, and analyzed well. Please find our answers to your comments and questions as follows.

[R2-1]: This paper focuses on CLIP-like models, given that VLMs cover both auto-regressive and ITM, it should be good to make the distinction explicitly.

Thank you for the helpful observation. As the reviewer correctly points out, vision-language models (VLMs) can be broadly categorized into three classes:

1. Image–Text Contrastive (ITC) Models – e.g., CLIP, which learn via contrastive objectives over image–text pairs using dual encoders.
2. Auto-Regressive Generation Models – which generate textual descriptions or answers conditioned on images.
3. Image–Text Matching (ITM)-Based Models – which treat alignment as a classification task over paired inputs.

Our work focuses specifically on the first category (ITC), targeting CLIP-style dual-encoder architectures.

We acknowledge that this distinction was not made sufficiently clear in the submitted manuscript. We will clarify this distinction in the revised manuscript.

[R2-2]: This paper explores only T5 text encoder. While this is ok, relevant prior work is not further discussed. As an example, VQAScore for Evaluating Text-to-Visual Models explicitly fine-tunes a T5 text encoder and explains the rationale behind this decision. Is this something the authors of READ also leverage, thus the positive results?

Thank you for raising this important point.

First, we would like to clarify the difference in the role of T5 between VQAScore and READ-CLIP. VQAScore explicitly fine-tunes the T5 text decoder, whereas READ-CLIP fine-tunes the dual encoders while making use of a frozen T5 text decoder. In our method, the T5 text decoder remains completely frozen throughout training and is only used to provide sentence-level reconstruction supervision for the text encoder.

Our rationale for using T5 is closely tied to the core idea of our work. One of the key motivations of READ is to improve the training of the encoder-only CLIP architecture by introducing a token-level reconstruction loss. This, however, requires a high-quality pre-trained text decoder. T5, as a widely adopted encoder-decoder model with a strong pre-trained decoder and cross-attention capabilities, is a natural fit for this purpose. Importantly, it allows us to achieve the reconstruction objective without modifying the CLIP backbone, aligning with our design goal of enhancing alignment in a minimally invasive way.

We will clarify this design choice in the revised manuscript to avoid any potential confusion.

As per the reviewer's suggestion, we will further include relevant prior work.

[R2-3]: Evaluation only limited to relatively old benchmarks: it is highly encouraged to report results on more recent/more challenging/realistic benchmarks, e.g.: SugarCrepe++, Winoground, ConMe.

Thank you for this suggestion.

1. On SugarCrepe++

We would like to note that SugarCrepe++ is already included in our evaluation, as shown in Tables 1–4 and Figures 4–5 of the main paper. This benchmark is central to our compositional evaluation suite.

READ-CLIP outperforms other baselines on SugarCrepe++, as shown in Table 1 of the submitted manuscript.

2. On Winoground and ConMe

Winoground and ConMe are important benchmarks. However, Winoground go beyond compositionality alone and often requires advanced capabilities, as stated in a related work:

“..., we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images” — Diwan et al. 2022 [Diwan'22]

Additionally, ConMe is designed to evaluate large-scale generative VLMs (e.g., Flamingo, PaLI, GPT-4V) and is not used as a standard benchmark for studies on contrastive CLIP fine-tuning.

Recall that our focus was to assess compositional reasoning of vision-language models trained with contrastive learning. While testing on Winoground/ConMe is an interesting direction for future exploration, we believe their exclusion does not detract from the core objectives of this study.

We hope this clarification addresses the concern, and we thank the reviewer again for the constructive feedback.

References:

[Diwan'22] Diwan, A., Berry, L., Choi, E., Harwath, D., & Mahowald, K. (2022, December). Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2236–2250). Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.

[R2-4]: How sensitive is READ to paraphrase quality (noisy paraphrases)?

Thank you for this insightful question.

We interpret the reviewer’s concern as asking how robust our method is when some of the paraphrases used for sentence-level alignment are not true paraphrases which serve as semantic noise.

To resolve the reviewer's question, we conducted an additional experiment where we injected noise into the paraphrase data by randomly replacing a fraction of paraphrase captions with unrelated captions from the dataset. Specifically, we ran experiments with 10% and 20% of the paraphrase pairs being replaced with mismatched captions.

The results are shown below:

As expected, increasing noise leads to performance degradation. However, the drop is relatively modest—only 1.2 to 2.0 points in average score—indicating that READ is reasonably robust even in the presence of moderate paraphrase noise.

We will include this analysis and discussion in the revised manuscript.

[R2-5]: Does READ hurt zero-shot generalization on standard CLIP tasks (e.g., ImageNet, etc)?

Thank you for raising this important question. To address the reviewer's question, we ran additional experiments on two standard CLIP tasks: cross-modal retrieval and image classifications tasks.

From the results shown in the below tables, one can confirm that READ-CLIP maintains or even improves the performance of CLIP on cross-modal retrieval benchmarks, while some performance drop is observed on image classification tasks when READ is applied to CLIP.

1. Zero-shot Cross-modal Retrieval (Flickr30K)

The above result shows that applying READ improves Recall@1 (text-to-image) by +10.1%, demonstrating that READ effectively enhances text representations in line with its design goals. Note that READ-CLIP is fine-tuned only on COCO dataset, which implies that the improvement on Flickr30K is a strong indicator of READ’s cross-dataset generalization ability.

2. Zero-shot Image Classification

The above result shows that applying READ incurs performance drop on image classification tasks. We note that this slight degradation on classification tasks is a common trade-off observed in prior work on CLIP-based compositional fine-tuning.

We will report these results in the revised manuscript.

[R2-6]: Why not use the same loss objectives with another text backbone?

Thank you for this thoughtful question. We believe your question refers to our choice of using CLIP encoders (backbone), and we would like to share our thoughts in more detail below.

Although we applied READ to vanilla CLIP backbone in the submitted manuscript, our proposed objectives are also applicable to any vision-language model (VLM) that uses the dual-encoder architecture, as the reviewer pointed out. These VLMs include a variety of off-the-shelf CLIP variants with different text encoders or backbone modifications.

We will revise the manuscript to explicitly state that READ can be directly applied to other pre-trained dual-encoder models in a plug-and-play fashion during fine-tuning.

First, I would like to thank the authors for their responses. However, now I have more questions and concerns, which I summarize below (my apologies for a bit late response, I had to re-read the paper and re-read prior works):

• T5, as a widely adopted encoder-decoder model with a strong pre-trained decoder and cross-attention capabilities, is a natural fit for this purpose.: I would like the authors to further expand on this. As far as I know, OpenAI CLIP, EVA-CLIP uses a GPT-2 style decoder transformer; ALIGN and SigLIP uses a BERT-base encoder, OpenCLIP uses a RoBERTa-base encoder, M-CLIP uses a XML-R encoder. On the other hand, generative models like SimVL, CoCa, BLIP-2 leverage a T5-like encoder-decoder.

• Furthermore, VQAScore authors find that "FlanT5 benefits from bidirectional image-question encoding and extensive training on challenging QA datasets."; this brings again the question of not using other text backbones that does not benefit from bidirectional image-question encoding and extensive training on challenging QA dataset.

• T5 is pre-trained to reconstruct arbitrary spans that have been masked out, so the encoder learns bidirectional context while the decoder learns generative semantics; this seems to give the main strength in the proposed approach, not the proposed READ method. However, without empirical evidence, it is hard to say whether this is true or not. Using a different text backbone could have easily clarify this.

• Winoground results: this dataset is particularly small and easy to run; results should be available within minutes. It is particularly unfortunate that Diwan et al. 2022 [Diwan'22] was able to avoid testing on this dataset, since this particularity "commonsense reasoning or locating small, out-of-focus objects in low-resolution images" are in fact, some of the core principles for compositionality and realistic settings.

• It also seems evident that READ hurt zero-shot generalization on standard CLIP tasks, typically, results on 21 datasets are reported. These results are important, and the evaluation of zero-shot generalization after fine-tuning yields important insights about the task, showing both effectiveness and robustness of the proposed method.

• Taking a closer look, and thanks to the discussion with other reviewers (Zdd4), prior work has also explored using paraphrases or extensions of the textual inputs generated by other models ([Doveh’23-1]), however, in the Related Work section, it seems the authors refer to this work as "approaches typically generate hard negative captions" (L87-88). This is not accurate, since some of the cited work generate both expansions/paraphrases, as well as contrastive captions.

I would like the authors to further clarify on these questions before adjusting my final score.

Reviewer a9tz, please engage in the discussion period. I understand that the review period started over a weekend, but we only have a few days remaining in the (now slightly extended) discussion period. The author's have provided a thoughtful response to your review and you are obligated to respond to it. You should share with the authors if they addressed questions or concerns you had, and seek clarification about any questions or concerns that remain. Please post your response as soon as you can so that there is time for the authors to follow up and discussion to progress as needed.

To Reviewer 1 (UjDy)

We thank the Reviewer UjDy for the detailed review and constructive suggestions. We appreciate your acknowledgement that the paper is well-written, with clear logic and comprehensive experiments. Please find our answers to your comments and questions as follows.

[R1-1]: Although the technical details differ, the proposed algorithm shares a similar motivation with two existing CLIP-based papers, FG-CLIP and FineCLIP. Both of these aim to enhance the fine-grained representation capability of the CLIP model by leveraging fine-grained image-level and region-level multimodal contrastive learning. Please compare the proposed algorithm with these works in terms of algorithmic details and performance (if possible)

We appreciate the reviewer’s thoughtful feedback. While READ shares the high-level motivation of improving CLIP's fine-grained compositional understanding, our approach is distinct in scope, algorithmic design, and empirical setting. Below, we clarify these differences in depth and provide quantitative comparisons to support our claims.

1. Overall Distinction

• FG-CLIP and FineCLIP enhance visual representations by explicitly aligning image regions and parts with fine-grained textual cues using large-scale region-level annotation or dynamic distillation strategies.
• In contrast, READ introduces lightweight, text-centric auxiliary losses to structurally regularize the text encoder:
  • (i) a token-level reconstruction loss, and
  • (ii) a sentence-level alignment loss using LLM-generated paraphrases.

The main differences between FG-CLIP [Xie'25], FineCLIP [Jing'24], and READ-CLIP are summarized in the table below:

[Table R1] Conceptual Differences of READ-CLIP and baselines

2. Performance Comparison on Compositional Reasoning Benchmarks

[Table R2] Performance Comparison between READ-CLIP and baselines

Table R2 shows that, even with the coarse granularity (i.e., large patch size and lower spatial resolution) of ViT-B/32 relative to ViT-B/16 and modest training data scale (as shown in Table R1), READ-CLIP consistently outperforms FG-CLIP and FineCLIP across all compositional reasoning benchmarks, highlighting the effectiveness of its text-encoder-centric approach for enhancing compositional reasoning capability in CLIP.

We hope this clarifies the detailed difference between READ, FG-CLIP and FineCLIP, and provides a clear message that READ is a simple and effective alternative.

References:

[Xie'25] Xie, C., Wang, B., Kong, F., Li, J., Liang, D., Zhang, G., Leng, D., & Yin, Y. (2025). FG-CLIP: Fine-grained visual and textual alignment. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).

[Jing'24] Jing, D., He, X., Luo, Y., Fei, N., Yang, G., Wei, W., Zhao, H., & Lu, Z. (2024). FineCLIP: Self-distilled region-based CLIP for better fine-grained understanding. In Advances in Neural Information Processing Systems, 37 (NeurIPS 2024).

[R1-2]: The training dataset used in this study is COCO, which contains many images featuring similar objects and collected from similar scenes. How can it be ensured that fake negative pairs do not adversely affect the model's training?

Thank you for raising this important concern. We fully agree that the semantic false negative problem—i.e., negative pairs that are semantically similar to positives—is a key challenge in contrastive learning.

Nonetheless, MS-COCO remains the standard for CLIP-style contrastive fine-tuning, and is widely used not only in recent compositional representation learning studies such as NegCLIP [Yuksekgonul'23] and FSC-CLIP [Oh'24], but also in recent multi-modal representation learning such as M^3-Mix [Oh'23] and CLIP-Refine [Yamaguchi'23]. These prior works have shown that, with COCO datasets, contrastive objectives leads to meaningful performance gains, indicating that the practical effect of false negatives is limited in this setting. Thus, we believe that our experimental setup is both standardized and empirically sound.

We appreciate the reviewer’s thoughtful question and hope this clarifies our position.

References:

[Yuksekgonul'23] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2023). When and why vision-language models behave like bags-of-words, and what to do about it? Proceedings of the 11th International Conference on Learning Representations (ICLR 2023).

[Oh'24] Oh, Y., Cho, J. W., Kim, D.-J., Kweon, I. S., & Kim, J. (2024). Preserving multi-modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).

[Oh'23] Oh, C., So, J., Byun, H., Lim, Y., Shin, M., Jeon, J.-J., & Song, K. (2023). Geodesic multi-modal mixup for robust fine-tuning. Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

[Yamaguchi'23] Yamaguchi, S., Feng, D., Kanai, S., Adachi, K., & Chijiwa, D. (2025). Post-pre-training for modality alignment in vision-language foundation models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025).

[R1-3]: Please provide additional details regarding the paraphrased captions. How are the captions generated, and which tool is used for generating them?

Thank you for your question regarding the generation process of paraphrased captions used in our method.

As described in Appendix A.1 of the supplementary material, the sentence-level alignment loss in READ requires semantically equivalent yet lexically distinct captions for each image–caption pair. To obtain these, we generate paraphrases using a structured LLM-based procedure:

• Model: gpt-4o-mini-2024-07-18 via the OpenAI API
• Prompt format: Shown in Appendix Figure 6
• Generation parameters: temperature = 1.0 (all other settings default)
• Timing: Paraphrases are generated offline, prior to training
• Quantity: One paraphrase per original caption

These synthetic captions are used only for supervising the sentence-level alignment loss. For the token-level reconstruction loss, we instead leverage the multiple original captions provided by MS-COCO, without any LLM-based augmentation. These details are in the Appendix of the submitted manuscript.

We will include these details in the main body of the revised manuscript.

[R1-4]: If the granularity of the captions is further enhanced, would the performance improve accordingly?

Thank you for your insightful question. We agree that the granularity of captions can affect model performance. Indeed, prior work on compositional reasoning—such as FSC-CLIP [Oh'24], DAC [Doveh'23-1], and TSLVC [Doveh'23-2]—has demonstrated that replacing generic captions from datasets like CC3M with fine-grained, VLM-generated descriptions can lead to consistent improvements.

In our case, READ is designed to promote understanding relationship between words through both reconstruction and alignment objectives. As such, we believe that incorporating more fine-grained captions would further enrich the training signal and potentially enhance model performance.

We appreciate this valuable suggestion and will incorporate a discussion on the potential impact of caption granularity on READ's performance in the revised manuscript.

References:

[Oh'24] Oh, Y., Cho, J. W., Kim, D.-J., Kweon, I. S., & Kim, J. (2024). Preserving multi-modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).

[Doveh'23-1] Doveh, S., Arbelle, A., Harary, S., Herzig, R., Kim, D., Cascante-Bonilla, P., Alfassy, A., Panda, R., Giryes, R., Feris, R., Ullman, S., & Karlinsky, L. (2023). Dense and aligned captions (DAC) promote compositional reasoning in VL models. In Advances in Neural Information Processing Systems, 37 (NeurIPS 2023 Spotlight).

[Doveh'23-2] Doveh, S., Arbelle, A., Harary, S., Panda, R., Herzig, R., Schwartz, E., Kim, D., Giryes, R., Feris, R., Ullman, S., & Karlinsky, L. (2023). Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023).

Reviewer UjDy, please engage in the discussion period. I understand that the review period started over a weekend, but we only have a few days remaining in the (now slightly extended) discussion period. The author's have provided a thoughtful response to your review and you are obligated to respond to it. You should share with the authors if they addressed questions or concerns you had, and seek clarification about any questions or concerns that remain. Please post your response as soon as you can so that there is time for the authors to follow up and discussion to progress as needed.

To Reviewer 3 (Zdd4)

We thank the Reviewer Zdd4 for the detailed review and constructive suggestions. We appreciate your acknowledgement that our idea is innovative, the contribution is noteworthy, and the paper is easy to follow. Please find our answers to your comments and questions as follows.

[R3-1] The evaluation on the subsets of the benchmarks is missing.

Below we provide a comprehensive per-subset performance. Due to the space limitation, we compare with CLIP and FSC-CLIP (the latest baseline).

READ-CLIP consistently outperforms CLIP across most subsets, not just in average scores. Notably:

• On SugarCrepe and SugarCrepe++, READ-CLIP shows up to 20 percentage point improvements in replace and swap tasks.
• On VALSE, READ-CLIP has strong gains in existence (69.3 → 80.8), plurals (57.9 → 73.8), and relations (66.0 → 75.0), all of which require understanding fine-grained semantic variation.
• On WHATSUP, READ-CLIP yields improvements across all subcategories.
• On CREPE, significant performance improvement is observed on Atom (18.9 → 37.8) and Swap (17.3 → 61.2), while performance drop is observed on Negate.

We will include the complete table as well as the above discussion in the revised manuscript.

SugarCrepe

• att: attribute, obj: object, rel: relation

SugarCrepe++ (ITT)

VALSE

WHATSUP

CREPE

[R3-2] The term "paraphrase" in the token reconstruction loss is misleading.

As the reviewer correctly observes, captions in the COCO dataset describe different aspects of the same image and are not necessarily semantically equivalent. We used such alternative captions for the reconstruction loss.

In the submitted manuscript, we referred to these alternative captions as “paraphrases,” but this term is misleading as pointed out by the reviewer. More precisely, our method samples caption variants—captions associated with the same image that describe complementary or overlapping content.

In the revised manuscript, we will revise the terminology from “paraphrase” to “caption variant” or “alternative caption” to better reflect their nature and avoid confusion.

We also agree with your interpretation that the performance peak at $K = 1$ in Table 3 may stem from this potential semantic difference among captions. We will explicitly include this explanation in the revised manuscript to clarify the notation and interpretation of our experimental result in Table 3.

[R3-3] LLM-Generated Paraphrases are Reliable?

Thank you for raising this important question.

First, we did not apply any post-processing or filtering to the LLM-generated paraphrases used during training. This decision was motivated by recent studies that demonstrate the effectiveness of using LLM-generated captions in contrastive learning pipelines, even without filtering [Fan'23, Doveh’23-1, Doveh’23-2].

As per the reviewer's comment, we ran additional experiments validating the reliability of the LLM-generated captions. To be specific, we randomly sampled 1,000 paraphrase pairs (original vs. LLM-generated) from our training set and assessed their semantic consistency using two stronger LLMs:

• GPT-4.1-2025-04-14
• Gemini-2.5-Flash

Each model is independently prompted to determine whether the two captions are semantically equivalent using a strict boolean format, where the detailed prompt is shown below:

Prompt (used in both models):

<task>
Given two image captions, decide whether they are semantically equivalent (i.e., paraphrases).

Return the result **only** as a strict JSON object with one boolean field named "paraphrase".

Example formats:
{"paraphrase": True}
{"paraphrase": False}

No additional text, explanations, or metadata.
</task>

**Input**:
Caption 1: {original_caption}
Caption 2: {synthetic_caption}

Result:

These results indicate that more than 98% of our paraphrases were judged to preserve the original meaning, providing strong empirical support for the semantic reliability of our LLM-generated captions.

To further demonstrate the reliability of LLM-generated captions, we conducted additional experiments analyzing how the choice of different LLMs (GPT-4o-mini, GPT-4.1, Gemini-2.5-flash) for caption generation impacts the compositional reasoning performance of READ-CLIP, averaged out over 5 benchmarks used in the manuscript. As shown in the result below, the performance of READ-CLIP remains consistently stable regardless of which LLM is used to generate captions.

We will include this discussion in the revised manuscript.

References:

[Fan'23] Fan et al. Improving CLIP training with language rewrites. NeurIPS 2023.

[Doveh'23-1] Doveh et al. Dense and aligned captions (DAC) promote compositional reasoning in VL models. In NeurIPS 2023.

[Doveh'23-2] Doveh et al. Teaching structured vision & language concepts to vision & language models. In CVPR 2023.

[R3-4] Performance on other downstream tasks?

As per the reviewer's questions, we ran additional experiments on two downstream tasks -- cross-modal retrieval and image classification.

Due to the space limitation, please check the detailed result in our response to [R2-2] -- one can confirm that READ-CLIP maintains or even improves the performance of CLIP on cross-modal retrieval benchmarks, while some performance drop is observed on image classification tasks when READ is applied to CLIP. We note that this slight degradation on classification tasks is a common trade-off observed in prior work on CLIP-based compositional fine-tuning [Oh'24].

We will report these results in the revised manuscript.

[Oh'24] Oh et al. Preserving multi-modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. In EMNLP 2024.

[R3-5] Why applying READ to the fine-tuned CLIP model (Table 4: average performance of CLIP+READ = 58.2) underperforms compared to applying READ to the pre-trained CLIP model (Table 1: CLIP+READ = 64.1)?

Let us clarify the implication of the results in Tables 1 and 4. Let us focus on three methods which are fine-tuned versions (using MS-COCO) of the same pre-trained CLIP model, where different fine-tuning loss is used for each method:

• Method 0 (Table 1, last row): Uses our full READ loss as in Equation (8), where the contrastive component $\mathcal{L}'_{I \rightarrow T}$ includes hard negatives.

• Method 1 (Table 4, Row 1): Standard CLIP baseline, fine-tuned with only the vanilla contrastive loss (average of losses in Equation (2)), without using hard negatives.

• Method 2 (Table 4, Row 2): Uses the full READ loss from Equation (8), but replaces $\mathcal{L}_{I \rightarrow T}^{**\prime**}$ with the standard contrastive loss $\mathcal{L}_{I \rightarrow T}$ in Equation (2), without using hard negatives.

As such, the READ-CLIP in Table 1 and the READ-CLIP in Table 4 refer to two different variants of our method, where the former includes hard negatives in the loss, while the latter does not. The performance difference of two methods (64.1 for method 0 and 58.2 for method 2) implies the importance of including hard negatives in the fine-tuning loss.

We acknowledge that this distinction may not have been sufficiently clear in the initial submission and could understandably cause confusion. To address this, we will update the revised manuscript to make these differences clear.