Response to Reviewer Jf2T

We appreciate that Reviewer Jf2T finds our method novel, our experimental results significant, and the topic interesting.

Response to the questions

Answer for Q1

Thank you for this valuable suggestion. While we have already cited the FARE paper in the introduction (Line 41), we agree that a clearer comparison between LORE and FARE at the outset would better contextualize our contribution. In the revised version, we will explicitly highlight the key distinctions and motivations for proposing LORE in contrast to FARE within the introduction.

Answer for Q2

In the initial version of the paper, due to space limitations, we kept the Related Work section concise and focused on maintaining a coherent narrative rather than elaborating on each individual reference. Additionally, we would like to clarify that our primary focus is on improving the robustness of visual encoders, rather than VLMs as a whole. While we acknowledge the growing body of work on adversarial robustness in VLMs, we have deliberately centered our discussion on methods most relevant to image encoder fine-tuning.

That said, we appreciate the suggestion and in the camera-ready version, where we are allowed an additional page, we plan to expand this section and provide deeper analysis and categorization of relevant literature, particularly those highlighting VLM vulnerabilities and defenses.

Answer for Q3

Thank you for your comment. To clarify, Figure 1(a) is not related to varying the hyperparameter $\rho$, $\rho=0.1$ was used for this experiment. If the comment is instead referring to Figure 1(b), then as explained in Section 6.1, $\rho$ defines the threshold for the proximity constraint (see Eq. 5) and directly controls the trade-off between nominal (clean) accuracy and adversarial robustness. Varying $\rho$ adjusts the allowed deviation from the reference encoder, enabling a principled way to navigate this trade-off.

We included Figure 1(b) in Section 4 to motivate the LORE framework by showcasing its empirical advantage over naïve regularization methods, even before formally introducing the full formulation. Additionally, we explicitly reference and explain the meaning of varying $\rho$ in Lines 229–231, where we describe how LORE achieves controllability through this parameter. The theoretical and empirical roles of $\rho$ are then discussed in full detail in Section 6.1 and are further supported by our robustness suboptimality analysis.

We hope this clears up the confusion, and we are happy to provide further clarification if needed.

Answer for Q4

The models we fine-tune, such as CLIP ViT-B/32, DINOv2, and ConvNeXt-B, are large, pre-trained foundation models, not simple architectures. The relatively low number of fine-tuning epochs (2 for ImageNet and 5 for ImageNet-100) is not due to model simplicity; it signifies the efficiency of this approach for fine-tuning on large-scale datasets.

The number of epochs was selected based on training convergence behavior. Specifically, for the large-scale ImageNet dataset, 2 epochs were sufficient to reach strong convergence, consistent with the FARE paper, and yielded robust performance in both in-domain and zero-shot evaluations. Similarly, given the smaller size and diversity of ImageNet-100, we found that 5 epochs were adequate. Additionally, we provide training curves for several experiments in Appendix G.2 (e.g., Figure 10), where the model's convergence can be clearly observed.

Answer for Q5

Thank you for your comment. The '0' values for the original CLIP model (rows labeled "CLIP") under adversarial attacks in Table 3 are expected and reflect the severe vulnerability of the base model. This underscores the motivation for robust fine-tuning methods like LORE, as CLIP without adversarial training exhibits negligible robustness. As shown in the table, LORE consistently outperforms FARE on average across different settings, further validating its effectiveness. Regarding other '0' values in Table 3 (e.g., under evaluation at $\epsilon=4$), these correspond to models that were trained with lower perturbation strengths (e.g., $\epsilon=1,2$ for FARE1 and LORE2). As expected, models trained with smaller perturbations typically do not generalize well to stronger attacks—highlighting a common limitation of fixed-$\epsilon$ training.

As for high-intensity perturbations (e.g., $\epsilon=8$), we have indeed conducted those evaluations. The corresponding results are reported in Tables 9 and 10 of Appendix J.4. Due to space constraints in the main manuscript, these extended results are included in the appendix, as noted in Line 188 of the Results section.

Regarding further experimental validations, in addition to the results presented in Section 6, we conducted additional evaluations in Appendix J, including Square Attack, a black-box adversarial attack, and evaluation under Gaussian noise corruption.

Further experimental validation

1. Validating our choice of sample-specific margins.

To validate the design choice between sample-specific and sample-invariant tolerance settings, experiments were conducted using ViT-B/32 CLIP models. These models were fine-tuned for 5 epochs on ImageNet-100. For sample-invariant settings, $\rho=0.8$ was empirically chosen as the best-performing fixed tolerance among tested candidates. The baseline for comparison was a prior LORE setup with $m(x)=\|\phi_{\theta_0}(x)\|^2_2$ and $\rho=0.1$, which represents the sample-specific adaptive tolerance.

The table below presents the results:

Fixed tolerances show limitations: 0.6 is too restrictive, hurting robustness, while 1.5 is too permissive, mimicking FARE. Finding a universally optimal fixed tolerance is difficult. The sample-specific margin (e.g., LORE$^2$+$m(x)$ and LORE$^4$+$m(x)$ with $\rho=0.1$) is superior, adapting to embedding scale, consistently offering better robustness across various perturbations and training conditions while maintaining clean accuracy and stability.

2. Extentions to supervised finetuning

We also demonstrate that LORE's proximity constraints can be extended to supervised adversarial fine-tuning, applying it to TeCoA [1]. We introduce two variants: TeCoA + $l_2$, which constrains the $l_2$ distance between fine-tuned and pretrained image embeddings, and TeCoA + KL, which uses KL divergence to constrain output probability distributions. Experiments with ViT-B/32 CLIP models on ImageNet-100 demonstrate that both regularized methods consistently enhance robustness and clean accuracy compared to baseline TeCoA, across various perturbation strengths. This proves LORE's broad applicability and the stabilizing effect of proximity-based constraints in both embedding and probability spaces for supervised adversarial fine-tuning. For more details, refer to our comment to Reviewer CYtr.

---

[1] Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023.

We appreciate that Reviewer TnAL appreciates the superior trade-offs and stability of our method, and finds it important for adversarial robustness.

Response to Questions

In your method, you use a sample-specific tolerance margin, which is related to the norm of the original feature embedding. What would the results be if we used a sample-invariant tolerance?

Thank you for the insightful question. Using a sample-invariant tolerance instead of a sample-specific margin indeed offers an alternative, but it is challenging because it requires extensive, model-dependent hyperparameter tuning that may not generalize. Therefore, the authors chose a sample-specific margin using $m(x)=\|\phi_{\theta_0}(x)\|^2_2$, which provides an adaptive, scale-invariant tolerance tied to the pre-trained embedding's norm, reducing manual tuning and preserving semantic structure. additionally this relates the constraints geometrically to the cosine similarity of embeddings.

Experimental Comparison

To validate the design choice between sample-specific and sample-invariant tolerance settings, experiments were conducted using ViT-B/32 CLIP models. These models were fine-tuned for 5 epochs on ImageNet-100. For sample-invariant settings, $\rho=0.8$ was empirically chosen as the best performing fixed tolerance among tested candidates. The baseline for comparison was a prior LORE setup with $m(x)=\|\phi_{\theta_0}(x)\|^2_2$ and $\rho=0.1$, which represents the sample-specific adaptive tolerance.

The table below presents the results:

The results show that a fixed tolerance of 0.6 is overly restrictive, degrading robustness, while 1.5 is too loose, making LORE behave like FARE. Although fixed values can be tuned per setting, finding a universal optimum is difficult. In contrast, the sample-specific margin ($\text{LORE}^2$+$m(x)$, $\text{LORE}^4$+$m(x)$, $\rho=0.1$) adapts to embedding scale, consistently improving robustness under various perturbations while maintaining clean accuracy and offering greater stability across training $\epsilon$ values and model types.

Response to the Strengths And Weaknesses:

We appreciate your feedback on Section 4 effectively motivating our method and your recognition of its ability to achieve robust results with minimal clean accuracy degradation across diverse settings.

In the experiment part, the authors use the image classification task as the measurement to verify the robustness. Since the CLIP representation has been used in various applications such as detection and VLM, could the authors validate the robustness on those tasks?

This is an insightful point that has also been raised by another reviewer, and we fully agree on its importance. While our initial evaluation focused on the foundational aspect of visual encoder robustness through image classification, zero-shot capabilities, and out-of-distribution generalization (relevant to VLM applications), we acknowledge that the utility of models like DINOv2 and CLIP extends to various downstream tasks, such as segmentation.

During the rebuttal period, we conducted additional experiments to assess LORE's effectiveness on these more diverse downstream tasks for robust models. We found that LORE-hardened models maintain their strong performance even when evaluated on tasks beyond classification. However, due to time constraints, these evaluations are not as comprehensive as our previous experiments and focused specifically on the ViT-S/14 model, which was fine-tuned for 2 epochs on ImageNet for these new task-specific assessments. The results of these evaluations are presented in the table below, further demonstrating the generalizability and practical utility of LORE. This required dedicated experimental setups for each task, involving specialized datasets and metrics, which we are now able to include. The table shows the results of DINOv2 ViT-S/14 on the ADE20k dataset.

There are many other related methods proposed to improve the robustness of visual encoders [1,2], so I suggest you compare your method with more baselines rather than only comparing with FARE.

We appreciate your suggestion. Reference [1] is the original paper introducing the FARE method, which we have used as our primary benchmark and discussed in detail throughout our paper. Regarding [2], unfortunately, we could not include it in our comparison as the paper is not publicly available (there isn't on arXiv and also it isn't accessible through other academic sources).

It is also worth noting that TeCoA [3] was compared directly in the FARE paper, where FARE demonstrated competitive or superior performance over TeCoA. Since our method, LORE, consistently outperforms FARE across multiple metrics and setups, this indirectly highlights its strength relative to TeCoA as well. Due to space constraints, we did not replicate TeCoA in our paper, but we acknowledge it as a relevant supervised baseline.

[3] Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023.

The novelty of this work is limited. There has already been the unconstrained version of the adversarial optimization problem. And the difference between the proposed loss function Eq. (6) and the regularized loss function Eq. (3) only lies in the trainable weight $\lambda(x)$

Thank you for the comment. We respectfully disagree with the notion that the novelty of our work is limited. Our approach addresses a critical limitation in standard unsupervised adversarial fine-tuning frameworks, particularly in how they handle the trade-off between robustness and nominal (clean) performance.

In the FARE method, the regularization coefficient $\lambda$ is effectively set to 0, meaning there is no mechanism to explicitly control the trade-off between clean and adversarial performance, and no such regularization schemes are introduced in the paper. While Eq. (3) introduces a constant $\lambda$ as a regularizer, we show in Section 4 that using a fixed $\lambda$ leads to suboptimal performance and instability.

In contrast, our method proposes a sample-dependent and adaptive $\lambda$, modeled as a trainable dual network (Eq. 6), which offers several key advantages:

• Formulating the problem as a constrained optimization problem, not just an ad hoc regularization of the original objective.
• It provides principled control over the robustness–accuracy trade-off via constrained optimization (Section 6.1).
• Stabilizing the network during training, whenever needed throughout training, only activated when the model is losing its nominal performance (Figure 10, 11. Appendix H)
• In Figure 1b, we empirically show that by adaptively adjusting $\lambda$ (the dual function), LORE achieves a strictly better Pareto frontier than naively regularized losses with any fixed $\lambda$
• As demonstrated in Section G.1, sample-based regularization performs significantly better than input-independent formulations.
• It improves Out-of-Distribution Robustness, Embedding Interpretability, Image Classification accuracy and by dynamically adjusting the regularization strength based on each input sample (Section 6.2, 6.3).

Despite the structural simplicity of our approach, we emphasize that LORE achieves strong empirical performance across multiple backbones and datasets, as shown in Section 6 and Appendix J.

Additionally, we provide theoretical justification for LORE in:

• Section 6.1, which presents a formal analysis of the controllability of the trade-off via the constraint threshold $\rho$, and
• Appendix B, where we analyze the impact of dual variable parameterization on suboptimality bounds.

Overall, we believe our formulation offers a practical and theoretically grounded improvement over previous methods, resolving a previously overlooked issue in unsupervised adversarial fine-tuning.

In Line 130, the embedding space is denoted as ‘$\mathcal{Y}$’, while ‘$y$’ is used to represent the label of a data point ‘$x$’. It would be better to distinguish these two.

Thanks for the suggestion. We will address it in the final version.

Thank you for your careful reading of our paper and for highlighting both its strengths and areas for improvement. We respond below to your main concerns and detail how we will strengthen the manuscript.

Answer for W1

We agree with your observation and have revised the caption of Figure 2 to remove the term “optimal,” which could misleadingly imply a proven global optimum that we do not formally establish. As shown in our experiments (Section 6), our goal is to present an approach that consistently outperforms the FARE baseline in practice, emphasizing empirical improvements without overstating theoretical guarantees.

Answer for W2

We sincerely appreciate your suggested references. The first paper you mentioned (Kim et al., NeurIPS 2020) is indeed relevant, and we will include it in the extended version of our Related Work section. However, we were unable to locate the last two references as cited. If you could clarify or provide more details, we would be happy to consider them as well.

Answer for W3

The authors acknowledge the importance of evaluating LORE's effectiveness on diverse downstream tasks beyond image classification, such as segmentation, for models like DINOv2. During the rebuttal, they conducted additional, though time-constrained and less comprehensive, experiments focusing on the ViT-S/14 model fine-tuned for 2 epochs on ImageNet. These new evaluations show that LORE-hardened models maintain strong performance on these tasks, demonstrating the general utility of LORE. The table shows the results of DINOv2 ViT-S/14 on the ADE20k dataset. We used EmbedAttack to evaluate robustness, following the approach in [4].

Answer for C1

We thank the reviewer for this insightful comment and fully agree that extending LORE to supervised adversarial fine-tuning would broaden its impact. While LORE was originally developed in an unsupervised setting, its core principle is potentially compatible with supervised adversarial training.

Experimental Extension:

1. LORE in Supervised Adversarial Fine-Tuning

To directly address the reviewer’s suggestion, we extended our work by applying LORE regularization to the supervised adversarial fine-tuning method TeCoA [1], which aligns image-text pairs using supervised losses.

Given an image-label pair $(x, y)$, TeCoA optimizes a contrastive loss to align adversarial image embeddings with text embeddings. The loss function is defined as:

$

L_{sup}(x, y; \theta) = \max_{\delta \in \Delta} - \sum_i \Bigg[y_i \log \frac{\exp(\cos(z, \psi(t_i))/\tau)}{\sum_k \exp(\cos(z, \psi(t_k))/\tau)} \Bigg],

$

where $z = \phi_\theta(x + \delta)$ and $\psi(t_i)$ are the image and text embeddings, respectively. Here, $\psi(t_i)$ corresponds to the text embedding of the i-th class in the dataset, where each class label is inserted into a fixed natural language template, e.g., "This is a photo of {class}".

---

We incorporated our proximity-based regularizer into TeCoA, forming a combined method (TeCoA + $l_2$) as follows:

$$\max_{\omega\in\Omega}\min_{\theta\in\Theta} E_{x,y \sim D} \left[ L_{\text{sup}}(x,y;\theta) + \lambda_\omega(x) \big(\|\phi_\theta(x)- \phi_0(x)\|_2^2 - \rho \|\phi_0 (x)\|_2^2 \big) \right]$$

2. Distribution-Level Constraint

We also explored a probability-distribution variant of LORE for the supervised setting, constraining the KL divergence between the fine-tuned model’s output distributions and those of the pretrained model. This approach requires only class text embeddings, not sample labels. Applied to TeCoA, it yields a second regularized variant: TeCoA + KL.

For classification using probability distributions induced by the image classifier or a contrastive model (outputs logits of softmax of cosine similarities), for both current and original models, We formulate the constrained optimization problem:

$$\min_{\theta \in \Theta} \; E_{x,y \sim D}\big[L_{sup}(x,y;\theta)\big] \quad \text{s.t.} \quad D_{\text{KL}}(p_{\theta_0}(\cdot \mid x)\,\|\,p_\theta(\cdot \mid x)) \leq \rho\cdot m(x) , \quad \text{where} \quad m(x) = H(p_{\theta_0}(\cdot \mid x))$$

And building on this formulation, LORE's final objective function can be computed as before. The framework adjusts the proximity constraint by model confidence: low entropy (high confidence) tightens the constraint, while high entropy relaxes it for greater adaptation flexibility.

Results

Experimental setting: We fine-tuned ViT-B/32 CLIP models for 15 epochs on ImageNet-100 using a learning rate of 1e-5 and perturbation strength ε. The proximity constraint hyperparameter used ρ=0.1, consistent with previous LORE experiments.

As shown above, TeCoA consistently benefits from both LORE and KL regularization, achieving improved robustness while maintaining clean performance. These results demonstrate that LORE is not limited to unsupervised fine-tuning and that proximity-based constraints—whether applied in embedding space or probability space—can effectively stabilize adversarial fine-tuning in supervised frameworks as well.

Answer for Q1

Our primary focus in this work was improving robustness on the vision side, but your suggestion opens an important direction: extending LORE to jointly fine-tune both image and text encoders. Prior findings (e.g., TeCoA [1]) indicate that such joint tuning can cause training instability, especially in zero-shot settings where cross-modal alignment may drift and generalization degrades. Similarly, FARE [2] emphasizes the need to preserve the original embedding characteristics of both clean and adversarial examples in VLM pipelines (e.g., CLIP in LLaVA). Simultaneous fine-tuning risks keeping modalities aligned with each other while drifting away from their pretrained distributions, harming robustness and semantic fidelity. Nonetheless, it could offer benefits in scenarios such as image-only attacks, where updating the text encoder may help realign embeddings with perturbed image features, potentially mitigating attack effects.

On Extending LORE:

In Unsupervised Settings:

Extending LORE to jointly fine-tune both encoders in a fully unsupervised setting is non-trivial. Without access to text supervision (e.g., class labels or prompts), it's unclear how to meaningfully update the text encoder. In this case, LORE would not have the necessary guidance to preserve alignment or constrain both encoders effectively.

In Supervised Settings:

In contrast, supervised settings offer a viable path forward. Since textual labels and their embeddings are available, the text encoder can be updated effectively. For instance, in our proposed extensions to TeCoA (discussed in response to Comment C1), applying LORE’s proximity-based regularization can help stabilize joint fine-tuning. The constraints act as stabilizers, keeping both encoders anchored to their pretrained representations while allowing controlled adaptation.

Moreover, if the threat model includes perturbations to both modalities, then fine-tuning the text encoder becomes necessary, and LORE-style regularization could be critical for maintaining dual robustness.

Answer for Q2

Both LORE and FARE's effectiveness hinges on the quality of their frozen pretrained reference model, which acts as an anchor. LORE will still converge and improve robustness, as it helps control the nominal-adversarial robustness trade-off by constraining model deviation; however, its performance ceiling (both clean accuracy and robustness) is inherently limited by the reference model's quality. There's no sharp cutoff; LORE's benefits gradually lessen with lower-quality pretrained models. Empirical evidence from Tables 1 and 2 consistently supports this: stronger pretrained models (e.g., higher clean accuracy, larger architectures like ViT-B/16 over ViT-B/32, or DINOv2 ViT-B/14 over ViT-S/14) consistently lead to better nominal and robust results when LORE is applied.

---

[1] Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023.

[2] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML, 2024.

[4] Kowalczuk et al., Benchmarking Robust Self‑Supervised Learning Across Diverse Downstream Tasks, CVPR 2024

Thank you for your answer.

References

I apologize for the earlier incorrect references. Below are the correct ones. My intention was to bring to the authors’ attention a relevant, though somewhat parallel, line of work [1, 2, 3, 4]. I believe it offers useful background for robust self-supervised learning, which is increasingly important given the large volume of work currently being published in deep learning.

[2] Zhang et al., Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness, ECCV 2022
[3] Kim et al., Adversarial Self-Supervised Contrastive Learning, NeurIPS 2020

Other Tasks

This concern was also raised by reviewer TnAL28. I find the authors’ response reasonable given the limited time during the rebuttal. I would kindly suggest expanding the comparison to include additional tasks, such as those in [4] and [5], as this would significantly strengthen the paper.

[4] Kowalczuk et al., Benchmarking Robust Self‑Supervised Learning Across Diverse Downstream Tasks, CVPR 2024

[5] Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, 2024

On Extending LORE

I appreciate the additional discussion and experiments on extending LORE. This clearly strengthens the method in my view.

Summary

Overall, I would like to see the paper accepted. While I am not comfortable with increasing my already strong score to the maximum 6/6, I will actively participate in the discussion and advocate for other Reviewers to consider raising their ratings.

I find that addressing the performance drop on clean data during the early stages of adversarial fine-tuning is an important contribution with practical implications. For instance, it could be applied in the context of Model Merging, where pretrained self-supervised models can be fine-tuned using LORE for just a few steps—as required by Model Merging—to improve robustness without compromising clean accuracy. I strongly suggest that the authors make their code as user-friendly as possible and provide trained checkpoints to increase the impact of their work, given acceptance.