Contrastive Learning with Data Misalignment: Feature Purity, Training Dynamics and Theoretical Generalization Guarantees

We thank the reviewer for the valuable time in the evaluation.

Response to Weakness 1: Clarification on the specific setting and semi-supervised learning

We clarify the data model adopted in this paper is standard and not designed for our specific setting. Both our learning framework and theoretical analysis are different from those in semi-supervised learning and cannot be regarded as extensions of existing work in that literature.

The sparse-coding model we use is standard in theory and widely used to represent images and texts [R1, R2, R3]. Shallow networks remain SOTA for analyzing training dynamics and generalization [R4, R5, R6], as deeper models are harder to study. The one-hidden-layer model is still SOTA for theoretical analysis of single-modal contrastive learning.

Our contrastive setting follows the four-stage structure used by many frameworks. E.g., BLIP applies the recaption–filter cycle iteratively; LaCLIP does it once. We do not aim to propose a new training method but instead give theoretical foundations for these existing strategies. This paper focuses on theory, not algorithmic novelty.

Our setting differs from semi-supervised learning [R7, R8, R9], which assumes labeled data is noiseless or nearly clean, and uses unimodal classifiers. Our “clean” pairs are well-aligned but can be highly noisy (lines 162–169). We clarify that "clean" means aligned features, not noise-free. So standard semi-supervised theory cannot provide guarantees in our setting. In addition, contrastive learning aligns two modalities without labels, unlike semi-supervised learning with true or pseudo labels. This alignment requires analyzing coupled encoders under a non-convex contrastive loss, which is far harder than the single-encoder supervised case. As far as we know, no existing work gives full dynamic analysis in this multimodal setting. Should the reviewer request it, we are happy to provide further comparison or elaboration in the next stage of discussion.

Response to Weakness 2: Clarification of the Key Step in Removing Spurious Features

To avoid confusion in terminology: in our paper, recaptioning means generating synthetic text using $G(x)$, and filtering refers to replacing some raw text with this synthetic text.

Both recaptioning and filtering are essential. Appendix B.1 (line 626) shows that fine-tuning the decoder $G(x)$ on clean data $S_h$ amplifies alignment with true features exponentially while keeping alignment with spurious features negligible. Thus, the generated captions retain relevant semantics and suppress spurious correlations.

Filtering uses these purified captions to replace misaligned ones, further eliminating spurious content. As shown in Theorem 4.3, the resulting dataset contains significantly fewer spurious features and more well-aligned pairs than the original data.

Response to Weakness 3: On Embeddings from Noisy vs. Filtered Data

Training only on $S_h$ reflects limited supervision in practice—$S_h$ is small and high-quality, while $S_w$ provides scale. High-capacity VLMs cannot be trained effectively on $S_h$ alone. Empirically, using only $S_h$ yields downstream accuracy 0.86, far below the $>0.90$ when including $S_w$ (Fig. 1(c)), so we exclude this baseline.

For the setting $S_h \cup \{(x, \hat{y}) \mid x \in S_w\}$, our theory directly applies. As shown in Lines 1053–1054, we compare cosine similarity between filtered ($\langle z'_x, \hat{z}_y \rangle$) and fully replaced ($\langle z'_x, z'_y \rangle$) captions, selecting the higher similarity. We prove filtering is at least as effective as full replacement, and empirically performs better.

Prior work ([R10], Fig. 2) studies how varying the ratio of synthetic/raw captions affects performance. Rather than repeating such ablations, our focus is to explain theoretically why recaptioning plus filtering improves feature purity and downstream results.

Response to Question 1: Noise Dominates Signal

The assumption that noise dominates the signal (line 160) is not necessary for our main theorems, but it helps to demonstrate that our main theorems work even in some extreme high-level noise cases. Our results remain valid in the low-noise regime without any significant modification.

We highlight the high-noise case to emphasize that even under severe noise, our one-hidden-layer contrastive model can still learn useful features. The lower bound in Eq(160) is only to show robustness in this worst-case regime.

This assumption is motivated by prior work on supervised learning [R6], to justify the improved performance of nonlinear networks over linear models under high-level noise. Since our goal is not to compare linear and nonlinear models, we do not expand on this in the main text. To avoid confusion, we will remove the lower bound in (160) and add a footnote that our guarantees apply in both low- and high-noise regimes.

Response to Question 2: Clarification on Whether the Results Depend on the Relative Size of $S_h$ and $S_w$

In our theoretical analysis, we characterize the required batch size $B$ and the required number of training iterations for the model to converge. As we assume independence of the samples in different batches, the sample size requirement is at least $T \times B$ (iterations times batch size).

(i) Pretraining ITCP with contrastive loss (Theorem 4.1, line 241) uses $T = \Theta(d^2 \log d)$ and $B = \Omega(d)$. This is Stage (S1) on raw data (line 102) and Stage (S4) on filtered data (line 118) with $S = S_h \cup S_w$ and $\tilde{S} = S_h \cup \tilde{S}_w$.

(ii) Finetuning the decoder $G$ for recaptioning (Theorem 4.3, line 259) uses $T = \Theta(d \log d)$ and $B = \Omega(d)$. This is Stage (S2) (line 107), where only pairs in $S_h$ are used to finetune the image-grounded decoder.

VLM pretraining typically needs far more samples than decoder finetuning, so we model $|S_w| \gg |S_h|$ to reflect practice, where human-annotated $S_h$ is small and web data $S_w$ is abundant.

The results will depend on sample size; specifically, larger samples will lead to faster convergence and smaller generalization error. Our intention was to study the worst-case scenario, aiming to identify the minimum number of samples needed for successful training. Therefore, we did not specifically analyze cases with varying sample sizes.

Response to Question 3: Assumption 3.4

When $z_{x_p}^j = 0$, $z_{y_p}^j$ can still equal 1 if a different feature $j' \neq j$ is active in the image and spuriously mapped to index $j$ in text. So:

$\Pr(z_{y_p}^j = 1 \mid z_{x_p}^j = 0) = \Pr(z_{x_p}^{j'} = 1)\\Pr(z_{y_p}^j = 1 \mid z_{x_p}^{j'} = 1) = \frac{C_z}{d}C_s$

where $C_z/d$ is the activation probability of any other feature $j'$ in the image and $C_s$ is the spurious alignment rate. Thus,

$\Pr(z_{y_p}^j = 0 \mid z_{x_p}^j = 0) = 1 - \frac{C_z}{d} C_s$

We assume a single spurious feature $j'$ per $j$ (forming $d/2$ disjoint pairs) only to simplify presentation. This can be generalized to $K-1$ spurious correlates per $j$, making Eq. (34) a $2K \times 2K$ matrix (line 701) without altering the insight.

Theorem 4.2 and its guarantees rely only on the total spurious mass (bounded by $C_s$), not on how many $j'$ exist per $j$. So the core mechanism is robust to such generalization.

Both $z_x$ and $z_y$ are $d$-dimensional sparse latent vectors (line 161). Spurious misalignment can occur on multiple dimensions, not just one pair. If $z_x$ has $k$ nonzeros, then $z_x$ and $z_y$ can differ on up to $2k$ coordinates.

Response to Question 4: Clarification on Initialization at $\bar{W}, \bar{V}$

Although $\bar{W}, \bar{V}$ are learned on the noisy dataset $S = S_h \cup S_w$, Theorem 4.2 shows that each neuron pair learns a mixed representation containing both relevant and spurious features. Despite the contamination, such weights preserve partial structure and are substantially better than random initialization.

When fine-tuning the decoder $G$ on clean data $S_h$, initializing from $(\bar{W}, \bar{V})$ allows the model to purify relevant features, yielding accurate synthetic captions. This practice is standard in state-of-the-art VLMs such as BLIP [R11], where the decoder is initialized from a pretrained VLM and then fine-tuned on human-annotated image-text pairs.

It is possible to use standard random initialization, but that would require a larger set of clean data to achieve the same level of representation purity of the learned decoder.

References

[R1] Yang et al., "Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification," CVPR 2009.
[R2] Yang et al., "Robust Sparse Coding for Face Recognition," CVPR 2011.
[R3] Arora et al., "A Latent Variable Model Approach to PMI-based Word Embeddings," TACL 2016.
[R4] HaoChen et al., "Provable Guarantees for Self-Supervised Contrastive Learning," NeurIPS 2021.
[R5] Wen and Li, "Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning," ICML 2021.
[R6] Allen-Zhu and Li, "Feature Purification: How Adversarial Training Performs Robust Deep Learning," FOCS 2022.
[R7] D.-H. Lee, "Pseudo-label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks," ICML 2013.
[R8] Zhang, Shuai, et al., "How Does Unlabeled Data Improve Generalization in Self-Training? A One-Hidden-Layer Theoretical Analysis," ICLR 2022.
[R9] Wei, Colin, et al., "Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data," ICLR 2021.
[R10] Nguyen, et al., "Improving Multimodal Datasets with Image Captioning," NeurIPS 2023.
[R11] Li, Junnan, et al., "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation," ICML 2022.

We thank the reviewer for the valuable time in the evaluation.

Response to Weakness 1: The theoretical model uses a simplified 2-layer network and does not directly extend to real-world architectures like transformers

While the model studied in this paper is a simplified version of real-world applications, we believe that analyzing the training dynamics of a one-hidden-layer network with nonlinear activation represents the most advanced setting currently allowing a comprehensive theoretical analysis of contrastive learning. Table 1 in the paper shows that our two-encoder architecture achieves the highest modelling fidelity among existing theoretical works published in recent top ML conferences:

Contrastive-learning theory
[R3] analyses only linear encoders.
[R4] extends to one hidden layer but still studies a single encoder for one modality.
Our work is the first to handle two nonlinear encoders, misaligned image–text multi-modal data, and zero-shot evaluation.

Existing theory for transformers is limited to supervised tasks or linear approximations:
[R1, R2] studies model editing in one-hidden-layer transformers under supervised objectives.

In summary, we believe our work has addressed more complicated and realistic models compared to prior studies. Extending the analysis beyond one-hidden-layer ReLU networks would require fundamentally different tools to handle the highly non-convex loss landscape. Therefore, we leave this direction for future works.

Empirical validation on transformer VLMs
Notably, our empirical experiments are all conducted using 12-layer ViT-based encoders, and their training behavior aligns well with the theoretical predictions, providing strong empirical support for the theoretical insights:

1. Recaption improves performance. LaCLIP achieves higher accuracy and silhouette scores than CLIP (Table 2).

2. Purified neurons enhance performance. Lowest-similarity neurons yield best accuracy; highest-similarity worst (Figure 2a–c,e–g).

3. Misalignment reduces purity. Caption shuffling increases neuron similarity proportionally, matching Theorem 4.2 (Figure 2d,h).

These observations show that conclusions proved for a simplified network transfer qualitatively to state-of-the-art transformer VLMs such as CLIP and LaCLIP.

Extending our analysis to deeper or transformer-based architectures would require substantial technical advances. Beyond the nonlinearity introduced by ReLU activations, such extensions must address the multiplicative interactions of the attention mechanism and the highly nontrivial behavior of the softmax function. These components introduce dense, dynamic coupling across features, posing significant challenges for any rigorous analytical framework.

Response to Weakness 2: Recaptioning mechanism is not theoretically modelled or analysed

We theoretically analyze a simplified recaptioning mechanism in Appendix G and prove it enhances feature purity. A simplified recaptioning mechanism $G(x)$ is actually theoretically analyzed in Appendix G, beginning at line 973. We could not include that in the main paper due to space limit, resulting in this confusion. We actually theoretically prove that fine-tuning the decoder on clean data $S_h$ can make the neurons become aligned with true features, i.e., their alignment is increased exponentially see Eq.(130), while spurious alignments remain negligible see Eq.(132). This theoretical analysis of a simplified captioning step serves as a prototype explanation, offering insight into the underlying mechanisms of recaptioning in practice. It supports the effectiveness of $G(x)$ in preserving relevant features and suppressing spurious ones, thereby enhancing feature purity.

Response to Weakness 3: The notion of spurious alignment (Cₛ) is theoretically assumed but not estimated or measured in real datasets.

Same as Response to Question 5.

Response to Weakness 4: Additional Experiment - Similarity-Based Shuffling

Random shuffling may exaggerate misalignment. We therefore ran a more realistic variant: for each image we located the ten captions with highest similarity and randomly selected one of them for shuffling. Feature purity was measured by average absolute cosine similarity of all neuron pairs, identical to Figure 2(d). Results are shown below:

Although similarity-based shuffling yields lower absolute cosine similarity than fully random shuffling, feature purity still degrades as $C_s$ increases. Hence realistic misalignment produces the same qualitative trend, reinforcing our theoretical conclusions.

Response to Question 1: Extension to Deeper/Transformer Architectures

Same as Response to Weakness 1.

Response to Question 2: Clarification on Assumptions and Structure

Sorry for the confusion. The perceived omission of Assumption 3.1 appears to stem from a misunderstanding. Due to the latex numbering system used in this paper, Definitions and Assumptions are numbered together. Assumption 3.2 appears after Definition 3.1, thus obtaining this numbering. No assumption has been omitted.

We will provide a clearer separation in the indexing and organization of the theoretical framework and empirical validation to improve the paper's clarity.

Response to Question 3: Feature Purity – Monitoring and Conceptual Clarity

We thank the reviewer for raising this point. The formal definition for measuring feature purity can be found in line 340. Specifically, we compute the absolute cosine similarity between every pair of learned neurons $v_j$ and $v_{j'}$ as $|\langle v_j, v_{j'} \rangle| / (\|v_j\| \|v_{j'}\|)$ for all $j, j' \in \{1,2,\dots,512\}$. A lower average value means neurons are more orthogonal whereas a larger value indicates feature mixing.

Figures 2 apply this metric: (a–c,e–g) show that selecting high-purity (lower average absolute cosine similarity) neurons maximises zero-shot scores, whereas (d,h) confirm that higher shuffle rates reduce purity (average absolute cosine similarity increases) as theory predicts.

Response to Question 4: Impact on Fine-tuning and Transfer Learning

Purified representations, which emphasise true semantic features and suppress spurious correlations, offer a stronger starting point for gradient-based adaptation. In principle, this lets fine-tuning specialise relevant features more efficiently instead of spending capacity to correct entangled ones, thereby reducing sample complexity and accelerating convergence.

A systematic empirical study of fine-tuning and domain-adaptation benefits is beyond the scope of the current work; we leave a detailed exploration of these transfer-learning effects as future research.

Response to Question 5: Estimability and Practical Relevance of $C_s$

While estimating $C_s$ in real datasets remains challenging, our theory provides meaningful qualitative insights across a broad regime. The spurious alignment rate $C_s$ plays a central role in our theoretical framework by quantifying the likelihood that an image feature appears spuriously in low-quality image-text pairs. While our theorems assume that $C_s \in \left( \omega(1/\log d), 1/2 \right)$, they remain valid throughout this broad regime—from low spurious alignment levels $\omega(1/\log d)$, up to moderate noise levels approaching $1/2$. Across this range, our analysis guarantees that recaptioning significantly improves alignment and downstream performance.

That said, we acknowledge that estimating or approximating $C_s$ in real-world datasets remains a challenging open problem. It may be possible to develop proxy metrics based on semantic similarity scores or caption relevance to approximate the degree of spurious feature alignment, but we leave this for future empirical work. Despite the absence of a direct estimator, the qualitative implications of our theory remain meaningful and broadly applicable in practical VLM training scenarios.

Response to Limitation 1: Real-world robustness

Our theory considers the worst-case noise of totally irrelevant misalignment because semantic ambiguity is difficult to model analytically, and, to our knowledge, no existing theory fully addresses this form of ambiguity.

Despite this simplification, we validated the key predictions on real datasets such as CC12M and LAION-400M and LaCLIP, which naturally contain ambiguous captions. The empirical results in Table 2 and Figure 2 align with our theory.

Response to Limitation 2: Estimating spurious alignment rate

Same as Response to Question 5.

References

[R1] Li et al., "When Is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers," NeurIPS 2023.
[R2] Huang, Yu, et al. "A Theoretical Analysis of Self-Supervised Learning for Vision Transformers," ICLR 2025.
[R3] HaoChen et al., "Provable Guarantees for Self-Supervised Contrastive Learning," NeurIPS 2021.
[R4] Wen and Li, "Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning," ICML 2021.

We thank the reviewer for the valuable time in the evaluation.

Response to Question 1: How do contrastively pre-trained VLMs align modalities, extract feature representations, and achieve zero-shot capabilities? How does text recaptioning on noisy image-text pairs provably enhance generalization performance?

During contrastive pretraining, stochastic gradient descent (SGD) initially causes neurons to align with both true and spuriously correlated features, due to comparable gradient magnitudes (Appendix D). As training progresses, this spurious alignment is reinforced—neurons entangled with mixed features amplify both components simultaneously (Appendix E). Without intervention, the final representation encodes multiple features per neuron, preventing clean separation (Appendix F, Theorem 4.2).

To improve the pre-trained model, a decoder is fine-tuned on clean data to generate synthetic captions. Theorem 4.3 (Appendix G) proves that neurons aligned with true features enhance their alignment exponentially, while suppressing spurious activations to negligible levels. Appendix H further shows that replacing original captions with synthetic ones filters out spurious correlations. Consequently, purified alignment is restored, and Theorem 4.5 (Appendix J) establishes that the zero-shot error drops to $o(1)$, demonstrating improved downstream generalization. A more detailed description can be found in Appendix B.1 Proof Scratch.

Response to Question 2: How well do the theoretical insights extend to deeper or transformer-based encoders commonly used in VLMs?

Extending our analysis to deep or transformer-based encoders would require major technical advances.

Attention and softmax complexity
Transformers rely on multi-head attention, where each layer multiplies token similarities by a softmax weight matrix. This induces dense, data-dependent coupling across all feature dimensions, far beyond the local interactions in a one-hidden-layer ReLU network. Formal convergence proofs must track the evolution of these weights and the resulting non-linear dynamics, a problem still open even in supervised settings.

Tokenized input and pairwise structure
Vision–language transformers operate on sequences of image patches or subword tokens. A rigorous treatment would need a mathematical model for tokenization and a similarity metric for positive pairs that captures spatial or linguistic structure. Such formulations are not yet available in a form amenable to SGD dynamics analysis.

Current theoretical frontier
Existing theory for transformers is limited to supervised tasks or linear approximations: [R1, R2] studies model editing in nonlinear transformers under supervised objectives. For contrastive learning, the most expressive theoretical results remain at the one-hidden-layer level, as in [R3].

Thus our two-encoder, one-hidden-layer nonlinear contrastive model represents the state of the art for provable training dynamics in VLMs. Generalising to full transformers is an important direction for future work.

Empirical verification on large transformer VLMs
Our theoretical insights, derived from a one-hidden-layer model, are borne out by experiments on deep ViT-based systems:

1. Purified neurons enhance generalisation. LaCLIP, which incorporates the recaption–filter stage predicted to raise feature purity, attains higher accuracy and a larger silhouette score than its CLIP counterpart (Table 2).

2. Data misalignment reduces feature purity. When we shuffle captions to increase misalignment, the average neuron cosine similarity grows in proportion to the shuffle rate, matching Theorem 4.2.

These observations show that conclusions proved for a simplified network transfer qualitatively to state-of-the-art transformer VLMs such as CLIP and LaCLIP.

Response to Question 3: Motivation for the Symmetric ReLU Activation in Definition 3.1

We adopt the symmetric ReLU $f_i(x)=\sigma(\langle w_i,x\rangle-b_i)-\sigma(-\langle w_i,x\rangle-b_i)$. This form lets us take $b_i>0$ without loss of generality and greatly simplifies analysis. The positive bias makes a neuron activate when $\langle w_i,x\rangle>b_i$ and also when $\langle w_i,x\rangle<-b_i$, yielding $f_i(x)=-f_i(-x)$. With this symmetry a single indicator function covers both signs of the inner product, streamlining gradient–alignment calculations while preserving the expressive power of the standard ReLU. All results carry over to the usual ReLU $\max(0,\langle w_i,x\rangle)$ after a linear rescaling of weights and biases.

The same symmetric activation is common in recent theory [R4, R5].

Response to Question 4: Could correlated or overcomplete text dictionaries break the guarantees?

Practical incoherence suffices
This is an interesting question. Since orthogonal or incoherent bases are standard assumptions in learning theory, we follow existing works in adopting them for our analysis. Exact orthogonality is not required; our theoretical guarantees remain valid even if the text dictionary is somewhat correlated, as long as distinct features are sufficiently incoherent—meaning their pairwise inner products are upper bounded by a smaller value. This bounded incoherence limits cross-feature interference, allowing our convergence and feature-purity results to hold. The weaker incoherence yields larger residual terms $r_i$ in Eq.(12) and requires more iterations to achieve convergence.

On the other hand, if the dictionary is highly coherent, it can lead to non-unique representations of the data. This makes it difficult to clearly define positive pairs and learned features, posing challenges that fall outside the scope of our current analysis. We leave a deeper exploration of this case for future work.

Extension to overcomplete bases
An overcomplete dictionary can be reduced to a complete basis by a suitable linear transformation. The latent code $z_x$ therefore need not be limited to the discrete set $\{0,\pm1\}$; any real-valued sparse representation suffices. Allowing arbitrary magnitudes in $z_x$ changes only the scale of the coefficients $\alpha_{i,j}$ and $\alpha_{i,j'}$ in Eq.(12), but the qualitative conclusions regarding feature mixing and eventual purification are unchanged.

Standard theoretical assumption
Orthogonal or incoherent bases are standard in neural-network theory, such as transferability analysis in CLIP embeddings [R6] and sparse-modelled contrastive learning dynamics [R3]. Sparse coding is in fact a standard modelling choice for both images and texts and is widely used in recent neural-network theory:

1. Image evidence
   [R7] shows that Caltech101 images admit sparse codes over learned dictionaries and that sparse coding outperforms vector quantisation.
   [R8] demonstrates that face images can be reconstructed by a small set of dictionary atoms and that sparsity improves recognition accuracy.

2. Text evidence
   [R9] models word co-occurrence statistics with sparse latent factors, providing a theoretical basis for skip-gram embeddings.

3. Sparse coding in neural-network theory
   [R6] analyses sparse text features in multimodal models such as CLIP.
   [R4] proves adversarial robustness under a sparse latent image model.
   [R3] tracks how contrastive learning separates true and spurious features using a sparse data model.

These studies confirm that assuming sparse codes for both modalities is a common and practical basis for rigorous analysis.

References

[R1] Li et al., "When Is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers," NeurIPS 2023.
[R2] Huang, Yu, et al. "A Theoretical Analysis of Self-Supervised Learning for Vision Transformers," ICLR 2025.
[R3] Wen and Li, "Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning," ICML 2021.
[R4] Allen-Zhu and Li, "Feature Purification: How Adversarial Training Performs Robust Deep Learning," FOCS 2022.
[R5] HaoChen et al., "Provable Guarantees for Self-Supervised Contrastive Learning," NeurIPS 2021.
[R6] Chen et al., "Understanding Transferable Representation Learning and Zero-Shot Transfer in CLIP," ICLR 2023.
[R7] Yang et al., "Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification," CVPR 2009.
[R8] Yang et al., "Robust Sparse Coding for Face Recognition," CVPR 2011.
[R9] Arora et al., "A Latent Variable Model Approach to PMI-based Word Embeddings," TACL 2016.

We thank the reviewer for the valuable time in the evaluation.

Response to Weakness 1: Model simplicity

While the model studied in this paper is a simplified version of real-world applications, we believe that analyzing the training dynamics of a one-hidden-layer network with nonlinear activation represents the most advanced setting currently allowing a comprehensive theoretical analysis of contrastive learning. Table 1 in the paper shows that our two-encoder architecture achieves the highest modelling fidelity among existing theoretical works.

Contrastive-learning theory
[R1] analyses only linear encoders.
[R2] extends to one hidden layer but still studies a single encoder for one modality.
Our work is the first to treat two nonlinear encoders, misaligned image–text multi-modal data, and zero-shot evaluation.

Supervised and adversarial theory
[R3] establishes feature purification with one hidden layer under adversarial training.
[R4] studies nonlinear transformers, but only for supervised model editing, and did not consider contrastive or multimodal objectives.

Semi-supervised pseudo-labeling
[R5] and [R6] analyse self-training with one-hot class pseudo labels. Generating a single class label is far simpler than producing a full synthetic caption, and pseudo-label methods involve a single encoder with explicit labels. By contrast, our setting requires two nonlinear encoders, high-dimensional text generation, and a non-convex contrastive loss, making the theoretical analysis substantially more challenging.

Response to Weakness 2: The sparse-coding assumption may be overly strong

Thanks for the observation. Sparse coding is in fact a standard modelling choice for both images and texts and is widely used in recent neural-network theory, rather than a special assumption of our work.

Image evidence
[R7] shows that Caltech101 images admit sparse codes over learned dictionaries and that sparse coding outperforms vector quantisation.
[R11] demonstrates that face images can be reconstructed by a small set of dictionary atoms and that sparsity improves recognition accuracy.

Text evidence
[R8] models word co-occurrence statistics with sparse latent factors, providing a theoretical basis for skip-gram embeddings.

Sparse coding in neural-network theory
[R9] analyses sparse text features in multimodal models such as CLIP.
[R10] proves adversarial robustness under a sparse latent image model.
[R2] tracks how contrastive learning separates true and spurious features using a sparse data model.

These studies confirm that assuming sparse codes for both modalities is a common and practical basis for rigorous analysis.

Response to Question 1: Why is $f_i(x)$ defined as in Equation (2)?

We adopt the symmetric ReLU $f_i(x)=\sigma(\langle w_i,x\rangle-b_i)-\sigma(-\langle w_i,x\rangle-b_i)$. This form lets us take $b_i>0$ without loss of generality and greatly simplifies analysis. The positive bias makes a neuron activate when $\langle w_i,x\rangle>b_i$ and also when $\langle w_i,x\rangle<-b_i$, yielding $f_i(x)=-f_i(-x)$. With this symmetry a single indicator function covers both signs of the inner product, streamlining gradient–alignment calculations while preserving the expressive power of the standard ReLU. All results carry over to the usual ReLU $\max(0,\langle w_i,x\rangle)$ after a linear rescaling of weights and biases.

The same symmetric activation is common in recent theory [R3, R2].

Response to Question 2: Supplementary ImageNet Results

The LaCLIP variants consistently surpass their CLIP counterparts on both Top-1 and Top-5 accuracy. Higher silhouette scores further indicate cleaner feature separation after recaptioning, in line with our theoretical predictions.

References

[R1] HaoChen et al., "Provable Guarantees for Self-Supervised Contrastive Learning," NeurIPS 2021.
[R2] Wen and Li, "Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning," ICML 2021.
[R3] Allen-Zhu and Li, "Feature Purification: How Adversarial Training Performs Robust Deep Learning," FOCS 2022.
[R4] Li et al., "When Is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers," NeurIPS 2023.
[R5] D.-H. Lee, "Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks," 2013.
[R6] Zhang, Shuai, et al., "How Does Unlabeled Data Improve Generalization in Self-Training? A One-Hidden-Layer Theoretical Analysis," ICLR 2022.
[R7] Yang et al., "Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification," CVPR 2009.
[R8] Arora et al., "A Latent Variable Model Approach to PMI-based Word Embeddings," TACL 2016.
[R9] Chen et al., "Understanding Transferable Representation Learning and Zero-Shot Transfer in CLIP," ICLR 2023.
[R10] Allen-Zhu and Li, "Feature Purification: How Adversarial Training Performs Robust Deep Learning," FOCS 2022.
[R11] Yang et al., "Robust Sparse Coding for Face Recognition," CVPR 2010.