Reviewer yyfG

We thank the reviewer for the valuable time in the evaluation.

Absence of Image-Text Pairs and Caption-Quality Metrics

Although one of the contributions of this paper is the theoretical demonstration that a well-designed recaptioning process can yield high-quality data, the goal of our experiments is not to re-justify this insight, as it has already been extensively validated by empirical evidence in prior works [1,2,3]. Instead, we directly leverage their models and datasets to support our theoretical analysis. For example, Figure 4 and Figure 6 in [1] present image-text pairs, Appendix A of [2] provides further examples, and Appendix C of [3] includes numerous captions generated by LLMs.

Moreover, Table 1 and Figure 4 in [2] empirically shows that higher cosine similarity between synthetic captions and image features correlates with improved ImageNet accuracy, thereby serving as a proxy caption-quality metric. Figure 2 in [2] further demonstrates the effectiveness of using a mixture of raw and synthetic captions filtered by cosine similarity. In addition, Section 4.3 of [1] discusses the diversity of synthetic captions and their benefits in pretraining.

T-SNE not direct

We agree that t-SNE visualization (influence from the choice of hyperparameters) is not direct, and does not provide statistical evidence for the separation quality between different methods. To address this limitation, we adopt the Silhouette Score (SS) with cosine distance to quantitatively and statistically assess feature embedding quality. We did a new experiment to compare vanilla CLIP (trained on original captions) with the LaCLIP model [3], where the model has the same architecture and the same training data with the vanilla CLIP, while the only difference is that a fraction of the captions are replaced with synthetic captions generated by LLM in training LaCLIP.

Tables 1, 2, and 3 show the comparison between vanilla CLIP and LaCLIP on CIFAR-100, CIFAR-10, and Caltech-101 datasets, respectively. The results show that LaCLIP consistently achieve higher Silhouette Scores than their CLIP counterparts. Since we use cosine distance, a higher Silhouette Score indicates that feature embeddings within the same class are more aligned with high cosine similarity, and embeddings from different classes are more orthogonal with low cosine similarity. This provides quantitative evidence for Theorem 4.3 and 4.5, which show that the neurons can learn purified representations better when some captions are replaced with synthetic captions.

Table 1: Comparison of CLIP and LaCLIP on CIFAR-100

Table 2: Comparison of CLIP and LaCLIP on CIFAR-10

Table 3: Comparison of CLIP and LaCLIP on Caltech-101

Errorbar of the cosine similarity

BLIP-generated captions demonstrate higher semantic alignment with a mean cosine similarity of $0.2633$ ($σ=0.0373$) compared to raw captions at $0.2467$ ($σ=0.0472$).

References:

[1] Li et al., BLIP: Bootstrapped Language-Image Pretraining, 2022

[2] Chen et al., Improving Multimodal Datasets with Image Captioning, 2023

[3] Fan et al., Improving CLIP Training with Language Rewrites, 2023

We thank the reviewer for the evaluation.

General Response 3: New experiments

Quantitative Results on Silhouette Score

We agree that t-SNE visualization does not provide statistical evidence for the separation quality between different methods. To address this limitation, we adopt the Silhouette Score (SS) with cosine distance to quantitatively and statistically assess feature embedding quality. A higher score indicates better intra-class alignment and inter-class orthogonality, reflecting more purified representations. We first calculate the SS in the simulated experiments of Figure 1 in the main paper. As shown in Table 1, when $C_s$ decreases, SS increases, and both the number of neurons learning purified features and the corresponding classification accuracy increase. This empirically supports our theoretical insight that reducing spurious and misaligned data encourages more neurons to learn purified representations, thereby improving the quality of learned embeddings. Moreover, training with a mixture of synthetic and raw data consistently yields significant improvements compared to using raw data alone.

Table 1: Comparison between Raw and Synthetic Data under varying $C_s$

Vanilla CLIP vs CLIP with synthetic text

We appreciate the reviewer's concern regarding architectural alignment between theory and experiments. To address this, we did a new experiment to compare vanilla CLIP (trained on original text) with the LaCLIP model [1], where the model has the same architecture and the same training data with the vanilla CLIP, while the only difference is that a fraction of the text are replaced with synthetic text generated by LLM in training LaCLIP.

Tables 2 show the comparison between vanilla CLIP and LaCLIP on CIFAR-100 (Results for CIFAR-10 and Caltech-101 can be found in our response to Reviewer yyfG.). The results show that LaCLIP consistently achieves higher Silhouette Scores than its CLIP counterparts. Since we use cosine distance, a higher Silhouette Score indicates that feature embeddings within the same class are more aligned with high cosine similarity, and embeddings from different classes are more orthogonal with low cosine similarity. This provides quantitative evidence for Theorem 4.3 and 4.5, which show that the neurons can learn purified representations better when some text is replaced with synthetic text.

Table 2: Comparison of CLIP and LaCLIP on CIFAR-100

No theoretical analysis of synthetic text generation

• We quantitatively prove that synthetic text with filtering exhibits better feature alignment with images than raw text, and that contrastive learning dynamics on such data lead to improved representation learning.
• To demonstrate the existence of such synthetic text with better feature alignment, we focus on a simplified text generation model in (10) and theoretically prove in Appendices F and G that this even such a simple text decoder can generate synthetic text that is better aligned with images (see GR2 in Reviewer iZBY).

Absence of direct text-quality metrics

See Section Absence of Caption-Quality Metrics in Reviewer yyfG.

Essential References Not Discussed:

We thank the reviewer for pointing out relevant papers. We will cite and discuss them in the revision. However, we would like to clarify that our contributions are fundamentally different.

• We believe the concern stems from a misunderstanding of our theoretical results. Beyond analyzing how feature misalignment affects contrastive learning, we rigorously prove that retexted data contains fewer spurious features and more task-aligned features, which in turn improve representation learning (see GR2 Reviewer iZBY). This theoretical guarantee is not provided in the referenced works.
• Regarding the role of data quality in generalization, our work offers a detailed analysis of the training dynamics of multimodal encoders $f$ and $h$ with nonlinear activations, which is absent from the mentioned papers. Both Xue et al. and Saunshi et al. directly assume access to a pretrained optimal encoder without analyzing the nonconvex training problem.

References: [1] Fan et al., Improving CLIP Training with Language Rewrites

We thank the reviewer for the valuable time in the evaluation.

General Response

GR1: Clarification of modality misalignment

• Our feature misalignment model includes both spurious correlation and less informativeness in the raw text. The latter means that synthetic text adds relevant features that raw text misses, but the former is not only missing from the text but also wrongly described as unrelated text. Therefore, less informativeness is merely a simple and special case within our misalignment analysis. We apologize that Assumption 3.5 only reflects spurious correlations. A more complete form of Assumption 3.5 appears in (67)–(68), where $P_1$ is the probability of spurious features and $1 - P_2$ is the probability of missing relevant features. The improvement lies in reducing both $P_1$ and $1-P_2$ through the generation of synthetic text. This improvement is rigorously proved, not assumed. (152) and (151) show that synthetic texts reduce spurious features and retain relevant ones, explaining their benefit in producing more informative content.
• Assuming a single spurious feature is a simplification for presentation that was made for ease of presentation in the proof and can be extended to a more general setting without altering the underlying insights. If each feature $j$ has $K{-}1$ spurious correlates, (38) becomes a $2K{\times}2K$ matrix, and $N_i=\{j,j'\}$ in the last sentence of Theorem 4.3 contains $j$ and other $K-1$ features. Our analysis relies on the total spurious feature probability (bounded by $C_s$), not the number of correlated features, so as long as the sum of all spurious feature probabilities is upper bounded by $C_s$, the core mechanism and insights of the theorem remain unchanged.

GR2: Clarification of synthetic text generation

Regarding the concern of implicitly assuming spurious-free synthetic text, we clarify that our work is not a simple comparison of text quality. Theorem 4.5 does not assume that synthetic text is better.. Instead, we formally analyze the synthetic text generation process, proving that the generated text is of high quality. As shown in Appendices F and G, we prove that after synthetic captioning and filtering, the resulting text contains fewer spurious features and more relevant features than raw text. In particular, the probability of spurious features can be reduced from a constant to $\frac{1}{d}$, while the probability of retaining all relevant features increases from $\frac{1}{2}$ to $1-\frac{1}{d}$ as shown in (151) and (152). We apologize for not including these results more prominently in the main text due to space constraints.

Weaknesses1:

• We think there may be some confusion of $f(x)=g(y)$ and $z_x=z_y$. Although we assume the latter, we do not mean the former holds. In contrast, our analysis shows that $f(x)$ captures information about $z_x$, and $g(y)$ about $z_y$. Although $z_x=z_y$, this does not imply $f(x)=z_x$ or $g(y)=z_y$.
• $z_x=z_y$ does not imply perfect alignment between image and text due to the presence of noise $\xi$, which can be order-wise larger than the signal itself in Assumption 3.3(d).
• This modeling assumption $z_x=z_y$ is standard in contrastive learning analysis, such as in [2].

Weaknesses2:

Please see GR1.

Weaknesses3:

We believe this is a misunderstanding from our imprecise statement in section 2.1, where we say "We use the high-quality data pairs in $S_h$ to train an image-grounded text decoder." Here, what we should have written is that the image-grounded text decoder is FINE-TUNED on $S_h$. We consider a simplified image-grounded text decoder, which is initialized from the weights $\overline{\mathbf{W}}$ and $\overline{\mathbf{V}}$ learned in stage 1 using a mixture of low-quality and high-quality image-text pairs. The decoder is then fine-tuned on high-quality pairs. Our setup is consistent with many real-world systems such as BLIP, LLaVA, and GIT, where text decoders are initially trained on noisy datasets like LAION and subsequently fine-tuned on curated data such as COCO. We also add the comparison of vanilla CLIP and LaCLIP [Fan et al, Improving CLIP Training with Language Rewrites], further validating our theoretical framework (see GR3 in Reviewer C7ye).

Weaknesses4:

We do not consider training on synthetic text only. In Section 2.1, we adopt a partial replacement strategy, where only detected noisy text is replaced with synthetic text, while the rest of the original dataset remains unchanged. In stage 4, the model is retrained using a mixture of raw and synthetic text.

Questions For Authors:

Following the previous comment, our paper exactly focuses on how fusing alt-text pairs with synthetic text affects the representation learning.

References:

[1] Liang et al., Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

[2] Chen et al., Understanding Transferable Representation Learning and Zero-Shot Transfer in CLIP

We STRONGLY recommend the reviewer to first read the General Responses 1 and 2 provided in Reviewer iZBY's rebuttal because of the space limit.

We thank the reviewer for the valuable time in the evaluation.

Oversimplified model architecture and loss function

• The training dynamics analysis of one-hidden-layer neural networks with non-linear activation function is the SOTA for contrastive learning. As shown in Table 1 in the main paper, the model we consider is already the most advanced for theoretical analysis among existing theoretical works regarding modeling fidelity. However, other works are limited to linear models or focus solely on training a single encoder in the VLM setting.
• Our paper aims to provide theoretical explanations for the advantage of synthetic captions within the CLIP-style learning framework. We initially chose BLIP and ALBEF as testbeds to empirically verify the theoretical benefits of synthetic captions and took necessary steps to minimize the differences between the two models (see details in the next response). To further strengthen our findings, we have added new experiments using vanilla CLIP and LaCLIP (trained with partially synthetic captions) (see GR3 in Reviewer C7ye).

Continued architectural and methodological mismatches between theory and experiments

• We fully agree with the reviewer that comparing vanilla CLIP with CLIP pre-trained on the same dataset with captioner and filter would be valuable. Thus, We have added such a new experiment (see GR3 in Reviewer C7ye).
• We initially considered BLIP and ALBEF because, at that time, publicly available vanilla CLIP models and CLIP models trained with a captioner and filter were not known to us. Consequently, we selected BLIP and ALBEF, as both use the same 14M pre-training dataset—with and without the captioner and filter, respectively (as reported in BLIP Section 4.1). We acknowledge the differences of model architecture and loss functions as pointed out by the reviewer. As an attempt to reduce the impact of these differences, in the paper, we focus exclusively on the Image-Text Contrastive (ITC) pathway in these two models, avoiding cross-attention, fusion, or decoding modules, as the ITC pathways of these two models share the same architecture.

Modeling over oversimplified assumptions:

We thank the reviewer for raising these concerns. However, we believe they arise from misunderstandings due to unclear presentation in our submission rather than fundamental weaknesses in our work. To clarify, we do not assume that synthetic captions are free of spurious correlations. Instead, we formally prove that synthetic captions help reduce such correlations (see GR2 in Reviewer iZBY). Furthermore, our model of image-text feature misalignment considers not only spurious correlations but also missing features, where raw captions often lack detailed and descriptive content, while synthetic captions provide richer information (see GR1 in Reviewer iZBY).

Regarding Figure 1: Figure 1 in the main paper is consistent with our theoretical results. Theorems 4.3 and 4.5 require $C_s$ less than 1/2, as indicated in Assumption 3.5, meaning that the probability of misalignment cannot be too large. This is consistent with the resulting in Figure 1 that the performance degrades as $C_s$ increases.

Comparison with Prior Theoretical Work

• We rigorously prove that recaptioned data contains fewer spurious features and more task-aligned features, which in turn improve representation learning. Specifically, the probability of spurious feature activation is reduced from a constant to $\frac{1}{d}$, while the probability of retaining all relevant features increases from $\frac{1}{2}$ to $1-\frac{1}{d}$. These theoretical guarantees are not provided in prior works.
• Our work provides a detailed analysis of the training dynamics of multimodal encoders $f$ and $h$ with nonlinear activations. Unlike most previous studies, we consider the realistic case where both encoders are jointly trained under a non-convex objective with multi-weight interactions. For example, [1] only considers a single encoder, while [2] only analyzes a linear model.
• Both [3] and [4] assume access to a pretrained optimal encoder, without analyzing the optimization dynamics of contrastive learning.

Difficulty in Finding Theorem Proofs in the Appendix

We will provide brief proof sketches in the main paper to guide the reader toward the detailed derivations in the appendix.

References:

[1] Wen et al., Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning

[2] Li et al., Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data

[3] Saunshi et al., Understanding Contrastive Learning Requires Incorporating Inductive Biases

[4] Xue et al., Investigating Why Contrastive Learning Benefits Robustness Against Label Noise