Q1: Computational overhead.
A: SVD in our method is performed only once before inference and can be cached, resulting in negligible overhead during test time. The per-image OT computations are parallelizable and efficient in practice. Overall, the additional cost remains moderate—for example, 8 minutes on ImageNet, 1 minute on Caltech101, and 0.5 minutes on DTD. While fine-tuning-based methods have similar inference speed to CLIP, they require substantial training overhead and thus fall outside the scope of our training-free setting.

Q2: Concerns over entropy-based weighting.
A: We agree that confidently misclassified views may receive lower entropy and thus higher weights. To examine this concern, we conducted an empirical analysis on the ImageNet dataset and found that the average weight assigned to such views is only 0.29, indicating that while these cases exist, they are non-dominant in the overall distribution. More importantly, the OT solver computes a global transport plan across the entire set, guided by the pairwise similarity structure between image and text distributions. Therefore, a single view cannot dominate the result due to OT’s normalization constraints. Entropy has been widely adopted in test-time adaptation and prompt tuning [56, r1, r2], as it directly reflects model confidence by quantifying output certainty. To assess its impact, we have compared entropy-based weighting with uniform weighting. As shown below, entropy-based weights consistently yield better robustness across datasets.

[r1]Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS 2022
[r2]Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning, CVPR2023

Q3: Impact of using a single prompt.
A: The subspace projection step is relatively insensitive to prompt richness, as class names already encode essential semantics. However, OT operates on distributions, when only a single prompt is used ($M=1$), the textual side collapses to a point mass, and the OT objective no longer captures meaningful distributional alignment. Consequently, the transport plan becomes degenerate and fails to effectively model semantic matching across multiple views. We conducted an experiment using a single prompt and observed a significant performance drop.

Q4: Empirical validation of perturbation decomposition.
A: We have conducted an empirical analysis to validate the assumption that adversarial perturbations primarily lie outside the text subspace. Specifically, we decompose each perturbation into parallel ($\boldsymbol\delta_\parallel$) and orthogonal ($\boldsymbol\delta_\perp$) components with respect to the semantic subspace and compute their magnitudes. Results consistently show that $|\boldsymbol\delta_\perp| \gg |\boldsymbol\delta_\parallel|$ across datasets and attacks. For example, on ImageNet under PGD-1/255, we observe: $||\boldsymbol\delta_\perp|| / ||\boldsymbol\delta|| \approx 75.4$. This confirms that most adversarial perturbations lie in the orthogonal subspace, supporting our assumption.

Q5: Generalization beyond simplifying assumption $\boldsymbol\delta_1 = \boldsymbol\delta_2$.
A: We adopt the simplifying assumption $\boldsymbol\delta_1 = \boldsymbol\delta_2 = \boldsymbol\delta$ in Appendix A.1 to simplify the derivation and highlight the key insight: projection reduces cosine similarity distortion under adversarial perturbations. Importantly, this conclusion still holds in the general case where $\delta_1 \ne \delta_2$. The distortion before projection is approximated as:

\approx \left| \mathbf{x}_1^\top \boldsymbol{\delta}_2 + \mathbf{x}_2^\top \boldsymbol{\delta}_1 - s(\boldsymbol{x}_1^\top \boldsymbol{\delta}_1 + \mathbf{x}_2^\top \boldsymbol{\delta}_2) + \boldsymbol{\delta}_1^\top \boldsymbol{\delta}_2 \right| + O(\epsilon^2)$$ After projection onto the semantic subspace via matrix $\mathbf{\Pi}$:

\Delta_\Pi = \left| \cos(\mathbf{\Pi}(\mathbf{x}_1 + \boldsymbol{\delta}_1), \mathbf{\Pi}(\mathbf{x}_2 + \boldsymbol{\delta}_2)) - s \right| \leq |s - s_\Pi| + \epsilon \left( \frac{1}{|\mathbf{\Pi}\mathbf{x}_1|} + \frac{1}{|\mathbf{\Pi} \mathbf{x}_2|} \right)

$$This leads to:$$

\frac{\Delta_\Pi}{\Delta} \lesssim \frac{2\epsilon}{2(1 + |s|)\epsilon} = \frac{1}{1 + |s|} < 1

where $s = \cos(\mathbf{x}\_1, \mathbf{x}\_2)$ and $s\_\Pi = \cos(\mathbf{\Pi} \mathbf{x}\_1, \mathbf{\Pi} \mathbf{x}\_2)$. Thus, projection consistently reduces cosine distortion even when the adversarial perturbations differ.

Q1: Applicability beyond classification tasks.
A: Our method is specifically designed to enhance the adversarial robustness of CLIP in classification settings, where the text labels (i.e., class names) are the predefined categories used to classify images at inference time. While our current framework is designed for classification tasks with a predefined class set, we acknowledge the importance of extending it to open-ended settings. A promising direction is to extract pseudo-labels from captions via clustering or noun phrase parsing, enabling the construction of a latent text feature space without explicit labels. This would allow projection and OT-based alignment in more general scenarios, which we consider a compelling direction for future work.

Q2: Label influence on feature extraction fairness.
A: We would like to clarify that CLIP-based classification is a closed-set problem, where all class names are predefined and available during inference. Our method projects image features into the subspace spanned by all class text embeddings, without using the ground-truth labels of test samples, thereby avoiding any label leakage. Moreover, the baselines we compare against (e.g., TTC, TeCoA) also rely on class text embeddings during inference, ensuring that all methods are evaluated under consistent and fair settings.

Q3: Analysis of perturbation reduction and validation of orthogonality assumption.
A: We thank the reviewer for raising this important point. We provide both theoretical justification and empirical validation:

• Perturbation reduction after projection.
  As shown in Equation (12) and formally proven in Appendix A.1, we demonstrate that projecting adversarial features onto the text subspace reduces semantic distortion:

  $$\Delta_\Pi \leq \Delta,$$

  where $\Delta$ and $\Delta_\Pi$ denote the cosine deviation before and after projection, respectively. This is further supported by Figure 4, which shows that projected features exhibit significantly higher similarity to text embeddings than the adversarial ones.

• Validation of the orthogonality assumption.
  We decompose adversarial perturbations into components parallel and orthogonal to the text subspace and find that the orthogonal component consistently dominates. For example, under PGD-1/255 on ImageNet, $\left\| \boldsymbol\delta_\perp \right\| / \left\| \boldsymbol\delta \right\| \approx 75.4\%,$ This confirms that most adversarial perturbations lie outside the semantic subspace.

Q4: Implementation details for TeCoA and FARE.
A: We appreciate the reviewer’s suggestion. In our experiments, we did not re-implement TeCoA or FARE, but utilized the fine-tuned models released by TTC [51] to extract features. We have added the specific attack settings (including perturbation size, attack type, and evaluation protocol) used in the original TeCoA and FARE papers to the revised version.

Q5: $\boldsymbol\delta_1 = \boldsymbol\delta_2 = \boldsymbol\delta$ assumption.
A: The assumption $\boldsymbol\delta_1 = \boldsymbol\delta_2 = \boldsymbol\delta$ was made for simplicity, to emphasize the core insight: projection suppresses cosine similarity distortion under adversarial perturbations. We have extended the analysis in the appendix to cover the general case where $\boldsymbol\delta_1 \ne \boldsymbol\delta_2$. Specifically, under first-order approximation, the cosine distortion before projection becomes:

$$$$

and after projection:

$$\Delta_\Pi \leq |s - s_\Pi| + \epsilon \left( \frac{1}{\|\mathbf{\Pi} \mathbf{x}_1\|} + \frac{1}{\|\mathbf{\Pi} \mathbf{x}_2\|} \right).$$

This yields the bound:

$$\frac{\Delta_\Pi}{\Delta} \lesssim \frac{1}{1 + |s|} < 1,$$

The result confirms that projection consistently mitigates cosine similarity distortion even under mismatched perturbations, with $s = \cos(\mathbf{x}\_1, \mathbf{x}2)$ and $s\_\Pi = \cos(\mathbf{\Pi} \mathbf{x}\_1, \mathbf{\Pi} \mathbf{x}\_2)$.

Q6: Figure labeling errors.
A: We thank the reviewer for pointing out the labeling errors. We have corrected them in the revised version.

Q7: Why was ε=1/255?
A: We follow prior works such as TTC and TeCoA in using ε = 1/255 to ensure a fair and consistent comparison. We additionally conduct experiments under ε = 2/255 with ViT-B/16. Results are reported below and again confirm the superiority of our approach.

Q1: Framework diagram or pseudo-code.
A: Thank you for the helpful suggestion. In the revised version, we have added a framework diagram in Section 3 to clearly illustrate the main components of our method, including subspace projection and OT-based alignment. We have also included pseudo-code in the appendix to describe the inference procedure in detail.

Q2: Extending to more complex models or tasks.
A: Our current work focuses on CLIP-based classification tasks, which rely on computing similarity scores between visual features and predefined textual class embeddings. Tasks like image captioning involve language generation, which falls outside the design scope of our OT-based framework. We have conducted additional experiments using SigLIP and EVA-CLIP，two recent and stronger CLIP variants. As shown in the table below, our method consistently improves robustness under adversarial attacks, demonstrating its effectiveness across different vision-language model backbones.

Q3: Labeling in Figure1(d).
A: We thank the reviewer for pointing this out. The legend in Figure 1(d) was mistakenly swapped between "Original CLIP" and "Attacked CLIP". We have corrected this labeling error in the revised version.

Q4: Results of autoattacks.
A: We have conducted additional experiments using AutoAttack on ViT-B/16. As shown in the table below, our method consistently improves robustness on both 9-datasets and ImageNet, demonstrating effectiveness beyond specific attack types. We have included these results in the revision.

Q1: Computation time analysis and comparisons.
A: We have conducted additional experiments on running time, as shown in the table below. These results indicate that our method (COLA) not only requires less time compared to TTC but also achieves better performance.

Q2: Computational cost for projection and optimal transport.
A: The projection step involves simple matrix multiplication between a $d$-dimensional input feature and the class-specific projection matrix, with a per-input cost of $\mathcal{O}(d^2)$, which is negligible in practice (e.g., <0.01s per sample). For OT, its complexity is generally $\mathcal{O}\left(n^2 \cdot \log\left(\frac{1}{\varepsilon}\right)\right)$, where $\varepsilon$ is the regularization coefficient and usually set to 0.01. Importantly, the OT computation is performed in parallel across all classes, resulting in efficient runtime. As reported in the table in response to Q1, the time costs are 8 minutes on ImageNet, 1 minute on Caltech101, and 0.5 minute on DTD.

Q3: Explanation for OT.
A: To the best of our knowledge, we are the first to apply OT for improving the adversarial robustness of CLIP. Prior works have used OT in other tasks such as distribution calibration [17] and few-shot learning [22].
Regarding the cost metric, cosine similarity is commonly used in multimodal OT-based alignment. Prior works such as PLOT [8], AWT [56], and ALIGN [44] adopt it to measure semantic distances between visual and textual representations. We have revised the manuscript accordingly.

Q4: Details in Table 8.
A: The time reported in Table 8 includes the PCA computation. OT computation takes approximately 8 minutes on ImageNet, 1 minute on Caltech101, and 0.5 minute on DTD, as shown in the table provided in response to Q1.

Q5: Citation for CLIP modality misalignment.
A: We have revised Section 1 and Section 3.2 to include relevant citations. Li et al. [25] and Mao et al. [30] demonstrate that adversarial perturbations and low-level visual biases can disrupt the alignment between visual and textual features. More recent studies, such as [r1] and [r2], further characterize the modality gap in CLIP.
[r1] Stabilizing Modality Gap & Lowering Gradient Norms Improve Zero-Shot Adversarial Robustness of VLMs, KDD 2025
[r2] Mitigate the Gap: Improving Cross-Modal Alignment in CLIP, ICLR2025

Q6: Impact of dynamic prompt generation.
A: In CLIP-based classification, class names are fixed, and prompts are generated using simple templates. For example, CLIP uses “a photo of a [class name],” which is shared across all test samples. Thus, prompt generation incurs only a one-time cost per class and does not affect per-image inference time. While dynamic prompts per image could introduce overhead, such cases are rare and fall outside the scope of CLIP-based classification. We have included this limitation in the revision.