Response to Reviewer zCJ1

We thank the reviewer for constructive feedback.

1. (Weakness-1) Comparison of EUR with adaptive weighting in [1]

We thank the reviewer for pointing this out. While [1] introduces an adaptive weighting based on confidence for weak-to-strong generalization, it focuses solely on single-modal vision models. Our work addresses robust weak-to-strong generalization for multimodal Vision-Language Models (VLMs), and our Entropy-guided Uncertainty Re-weighting (EUR) extends beyond raw confidence by leveraging prediction entropy, theoretically justified via Theorems 1 & 2 (feature norm $\leftrightarrow$ classification margin $\leftrightarrow$ robustness). In contrast, [1] provides only empirical justification.

To quantify the impact, we also implement the adaptive confidence weighting from [1] as a variant and compare it with our EUR below (average over 14 zero-shot datasets):

Our Adv-W2S method with EUR achieves higher robustness while also improving clean accuracy, validating the benefits of entropy-based adaptive scaling. We will incorporate this comparison and clarification into the revision.

[1] Guo, Jianyuan, et al. "Vision superalignment: Weak-to-strong generalization for vision foundation models." arXiv preprint arXiv:2402.03749 (2024).

2. (Question-1) Unique benefits of EUR beyond [1].

The empirical comparison of EUR with the adaptive weighting in [1] is reported above. Here, we emphasize the theoretical contributions unique to EUR:

• [1] balances teacher–student alignment purely via prediction confidence yet lacks a theoretical link to generalization/robustness properties.
• Our EUR is built on prediction entropy, which we theoretically show (Theorems 1 & 2) to correlate with feature norm and classification margin, thereby providing a direct mechanism for robustness improvement. This connection enables a principled adaptive weighting scheme: low-entropy predictions (confident teacher) lead to stronger alignment, while high-entropy predictions defer to student self-refinement, improving robust weak-to-strong generalization.

This theoretical grounding explains why EUR improves both clean and robust performance, as confirmed by the empirical results. We will add this discussion to the revised manuscript.

[1] Guo, Jianyuan, et al. "Vision superalignment: Weak-to-strong generalization for vision foundation models." arXiv preprint arXiv:2402.03749 (2024).

Response to Reviewer m9zw

We thank the reviewer for constructive feedback.

1. (Weaknesses-1) Analysis and impact of the parameter $\beta$ in the CLIP weak-to-strong formulation (i.e., Eq. (4)).

We thank the reviewer for raising this point. The trade-off hyperparameter $\beta$ controls the balance between weak-to-strong teacher-student alignment and self-prediction consistency in Eq. (4) and is similarly applied in our adversarial baseline (Eq. (5)). In our experiments, $\beta$ is selected via 1% of the training data and either fixed or linearly warmed up. To further clarify the role of $\beta$, we report the performance of the baseline adversarial weak-to-strong formulation (Eq. (5)) under different fixed $\beta$ values in the table below:

However, a fixed $\beta$ cannot adapt to varying teacher reliability, particularly for uncertain or hard samples. To address this, our final method introduces Entropy-guided Uncertainty Re-weighting (EUR) (Eq. (6)), which adaptively scales $\beta$ based on teacher confidence, leading to our unified objective in Eq. (13). We report average clean and robust accuracy across 14 zero-shot datasets to quantify this effect:

These results demonstrate that entropy-based adaptive weighting notably improves robustness generalization while also enhancing clean performance. We will incorporate this clarification and the ablation results into the revised manuscript.

2. (Weakness-2) PGR evaluation from Burns et al. [a]

We thank the reviewer for this helpful suggestion. We here estimate the "strong ceiling" by performing robust fine-tuning of the strong model directly on each downstream dataset, which approximates the upper bound of achievable robustness. This provides a clearer perspective on the gap between robust weak-to-strong generalization in the zero-shot setup and task-specific robust training. To better contextualize our results, we consider the Performance Gap Recovered (PGR) [a] in both clean performance and adversarial robustness across diverse tasks. The table below summarizes the PGR results (%) of our method (average classification accuracy against clean/adversarial examples across zero-shot datasets, captioning on the COCO dataset, and VQA on the VQAv2 dataset):

These results indicate that Adv-W2S recovers the majority of the performance gap to the strong ceiling in both clean and robust settings, validating its effectiveness in achieving robust weak-to-strong generalization without task-specific training. We will include this analysis in the revised manuscript.

[a] Burns et al., Weak-to-strong generalization: Eliciting strong capabilities with weak supervision (ICML 2024).

3. (Question-1) Clarifying “superalign” terminology.

We thank the reviewer for this observation. We agree that "superalign" is commonly used in AI safety to describe aligning superhuman-level models with human values. Our work uses the term in the weak-to-strong generalization sense, following Burns et al. [a], where the goal is to transfer robustness from a weaker model to a stronger one — an analogous alignment challenge in multimodal robustness.

• We will revise the introduction to explicitly acknowledge the conventional meaning of "superalign" and clarify our intended usage.
• We will highlight that our method starts from CLIP (a foundational vision-text model) and extends to large-scale VLMs such as LLaVA, emphasizing that our formulation addresses robustness transfer rather than human-value alignment.
• This clarification will be added early in the paper to prevent confusion and situate our contributions within the broader literature.

[a] Burns et al., Weak-to-strong generalization: Eliciting strong capabilities with weak supervision (ICML 2024).

4. (Question-2) Scalability to larger model backbones

To validate scalability, we here extend experiments to larger CLIP backbones (ViT-H/14 and ViT-g/14), which are ~3$\times$ larger than ViT-B/16 and widely used in recent VLMs. Average zero-shot classification accuracy (%) across 14 datasets under perturbation radius $\epsilon=2/255$ are summarized below:

Our method maintains consistent gains on both clean and robust accuracy, with especially strong improvements in robustness as model capacity increases. We will include this analysis and also reference robustness results of other large-scale VLMs in other vision-language tasks reported in Response 2 to Reviewer 7iVd.

Response to Reviewer 7iVd

We thank the reviewer for constructive feedback.

1. (Weakness-1) (Question-1) Generalization beyond perturbation-based visual adversarial attacks (e.g., text perturbation attacks, text prompt attacks).

We thank the reviewer for this question. While our main experiments focus on perturbation-based attacks on the visual encoder (dominant in multimodal robustness studies), our approach is not restricted to this setting.

• Connection to text-based perturbations: Work [1] studies continuous embedding-level perturbations for LLMs, while [2] addresses discrete text modifications to achieve prompt-based attacks. Importantly, both types of text-based adversarial attacks are applicable for evaluating the robustness of our method as well as other adversarial fine-tuning approaches.
• Extended evaluation: To demonstrate generality, we additionally evaluate robustness against text perturbation [1] and prompt-based attacks [2]; results are summarized below:

These results confirm that our Adv-W2S generalizes beyond image-level perturbations and remains robust under diverse modalities. We will add this discussion and cite [1,2] in the revised manuscript.

[1] Kim, Jaehyung, et al. "RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training." EMNLP (Findings). 2023.

[2] Xu, Xilie, et al. "An LLM can Fool Itself: A Prompt-Based Adversarial Attack." ICLR. 2024.

2. (Weakness-2) (Question-2) Evaluation on recent VLMs (LLaVA 1.6 (next), Qwen VL)

We thank the reviewer for this helpful suggestion. To address it, we extend our evaluation to recent stronger VLMs on Qwen VL 7B and LLaVA 1.6 (Next) 7B, both using the ViT-L vision encoder for fair comparison. Results on COCO captioning (CIDEr) and VQAv2 (VQA accuracy) are summarized below. Adv-W2S continues to provide significant robustness gains while maintaining competitive clean performance, indicating the method generalizes well to recent VLM architectures.

These results show that Adv-W2S retains its robustness advantage on newer, stronger VLM backbones, scaling effectively while maintaining competitive clean accuracy. We will add this discussion and highlight that the robustness improvements persist even as model capacity increases.

Esteemed Reviewer,

We sincerely thank you for engaging with our rebuttal, and for your valuable comments. Rest assured we will incorporate all your suggestions in the draft. Meantime, kindly let us know if there is anything else we can answer, clarify or improve.

Best regards,
Authors

Response to Reviewer uQC8

We thank the reviewer for constructive feedback.

1. (Weakness-1) Intuitions of theoretical results for clarity.

Absolutely. We will include clearer and intuitive explanations in the revision. Our theoretical results are intended to explain why entropy-guided re-weighting and inverse adversarial refinement improve robustness transfer. Intuitive remarks will be added after each theorem:

• Theorem 1 shows that low prediction entropy (high certainty prediction) enforces a larger vision feature norm, whereas norms of features for high entropy of prediction can be low. Therefore, the model demonstrably is able to filter high-confidence samples from low-confidence samples. This is somewhat similar to the organization of feature space in hyperbolic encoders, where a low norm of features indicates high uncertainty of predictions, and a high norm indicates low uncertainty of predictions [a].
• Theorem 2 builds on Theorem 1, showing that larger feature norms lead to wider classification margins, improving robustness since small perturbations are less likely to change predictions.
• Theorem 3 explains that the loss between adversarial student and inverse adversarial teacher upper bounds that of the clean student, implying that improvements on adversarial samples also benefit clean predictions.

These clarifications directly link the theory to our design choices, improving readability without sacrificing rigor.

[a] Nickel et al., Poincare embeddings for learning hierarchical representations, NeurIPS 2017

2. (Weakness-2) Intuition of maximising teacher-student gap (Table 11).

The rationale for maximizing the teacher–student prediction gap is to align adversary generation with weak-to-strong generalization:

• Target worst-case disagreements: Perturbations focus on where student predictions deviate most from the teacher, forcing learning from challenging cases.
• Consistent optimization: This keeps inner (adversary generation) and outer (parameter optimization) steps aligned toward reducing the gap rather than optimizing different objectives.
• Selective guidance: The student aligns with reliable teacher predictions while refining uncertain ones. In other words, rather than fully imitating the teacher, the student should maximize agreement with the teacher on stable predictions while relying on their own refinement when the teacher is uncertain. Optimizing adversaries that maximize the teacher–student gap directly exposes these disagreement regions, allowing the student to correct them and inherit both natural performance and robustness more effectively than student-only perturbations, thereby improving transfer performance.

We will add this explanation in the revised manuscript for clarity.

3. (Weakness-3) Theorem 2 provides an $\ell_2$ robustness bound, but evaluations use $\ell_\infty$ attacks.

The $\ell_2$ bound in Theorem 2 was presented for notational simplicity, yet the Lipschitz continuity argument is norm-agnostic and directly extends to $\ell_\infty$. Specifically, robustness bounds can be converted via standard norm inequalities (e.g., $\lVert\delta\rVert_\infty\leq\lVert\delta\rVert_2$), yielding $\lVert\delta\rVert_\infty < \frac{\gamma(x)}{L\max_{c\neq y} \lVert \Psi_c - \Psi_y \rVert_2}$. Therefore, our theoretical framework is fully compatible with the $\ell_\infty$-based robustness. We will clarify this generality in the revision to ensure consistency between theory and $\ell_\infty$-based evaluations.

4. (Weakness-4) Computational cost analysis.

We report the average accuracy (%) across 14 datasets with the training time per epoch in the table below. We also show that reducing the adversary generation step to 5 further enhances the efficiency while preserving most of the natural performance and robustness.

This demonstrates that Adv-W2S can maintain state-of-the-art robustness with substantially lower computational overhead and scales well to efficiency-sensitive settings.

5. (Question-1) How was the one‑hot label in Eq. 4 derived from strong model predictions?

In Eq. (4), $M(\cdot)$ denotes a simple argmax operation that converts the student’s soft prediction into a one-hot label by selecting the index of the maximum logit. No additional threshold is used. The one-hot label directly corresponds to the predicted class with the highest probability. We will clarify this notation in the revision.

6. (Question-2) Robustness Beyond $\ell_\infty$ Attacks (larger $\epsilon$, $\ell_2$-norm attacks, or text-prompt-based adversaries)

We report average accuracy (%) of larger $\ell_\infty$ perturbation $\epsilon$ in the table below:

We also validate the robustness against $\ell_2$ perturbations ($\epsilon=128/255$)

Robustness against text-prompt-based adversaries can be found in Response 1 to Reviewer 7iVd. We show that our Adv-W2S method consistently outperforms others under diverse perturbation types/scenarios.

7. (Question-3) What happens when teacher predictions are close to uniform?

When teacher predictions are uniform (high entropy), the weighting function in Eq. (6) naturally degrades toward a constant weighting (0.5) across all instances. In this case, the teacher cannot provide informative guidance. If teacher predictions are close-to-uniform but contain useful information, weights can be post-processed by sigmoid with sharp slope to boost the teacher's signal. In our experiments, the weak teacher is adversarially fine-tuned and typically exhibits non-uniform predictions, so this scenario rarely occurs in practice. We will clarify this behavior in the revision.

8. (Question-4) Can Adv-W2S handle open-set classes unseen by the teacher? How is $p_{T}(x)$ defined in that case?

In our current formulation, teacher–student alignment during training assumes a shared label space, where $p_T(x)$ is defined as the teacher’s predictions over this space via the cosine similarity between the image feature of $x$ and its corresponding text feature set of diverse categories. This enables efficient alignment without ground-truth labels, consistent with CLIP-style training.

For an open-set scenario during training, where class candidates may be unknown or unbounded, we envision three possible extensions:

• Large-scale virtual label space: Following the original CLIP pretraining approach, we can construct $p_T(x)$ by computing cosine similarity between the image feature of $x$ and a large set of text embeddings (e.g., descriptive phrases). While feasible, this approach incurs a higher computational cost. Using a subset of image properties can somehow enhance the computational efficiency.
• Prototype-based virtual labels: A more efficient variant is to cluster the text embedding space into representative prototypes, then define $p_T(x)$ via cosine similarity between image features and these prototypes, forming pseudo-label distributions for alignment.
• Feature-level alignment: Alternatively, adversarial weak-to-strong generalization can be adapted to perform directly at the feature level, potentially with a projector to resolve dimensionality differences between weak and strong models.

While our current experiments focus on shared-label training, these strategies highlight that our framework can be extended to open-set training scenarios; we will clarify this point in the revision.

9. (Question-5) Assumption of Global Lipschitzness in Theorem 2.

The global $L$-Lipschitz assumption was given for analytical clarity. Naturally, local Lipschitzness in the vicinity of datapoints ${\bf x}$ is sufficient for our margin–robustness argument. In Eq. (13), we align predictions of perturbed $\hat{\bf x}$ with predictions of ${\bf x}$, which implicitly teaches CLIP student local Lipschitzness on the data, as $w({\bf x})>0$ is in practice. However, as we assume Lipschitzness on the feature rather than the predictions, we verified results by imposing a Lipschitz regularization term with the equation:

$\eta \\,\Big(\lVert\Psi(x)-\Psi(x')\rVert_2 - L\lVert x-x' \rVert_2\Big)^2 + \beta\lvert L \rvert,$

which enforces bounded variations in feature space. This approach directly optimizes the Lipschitz equation and yields similar robustness trends (Table below), confirming that the theoretical assumption is not restrictive.

Moreover, our analysis focuses on explaining the mechanistic relation between margin and robustness rather than requiring exact constants. Although standard CLIP backbones are not explicitly Lipschitz-controlled, we empirically estimated the local gradient norm of ViT-L/14 on 1,000 random ImageNet samples and observed bounded values around 0.22–0.29. These small values indicate local Lipschitz behavior around natural data, making the approximation reasonable and consistent with the robustness trends confirmed by our experiments.

10. (Weakness-5) Writing. (Question-6) Several typos.

We will thoroughly proofread the manuscript and correct all typographical errors to improve overall clarity in the revised version.