Thank you for your review! We are happy to hear that you are “very open to revising [your] score to acceptance if the authors report the results they claim to have done”. Our revised manuscript addresses the lack of clarity that we believe led to this core concern. As we discuss below, all of the discussed experiments do indeed have their results reported in the submission. We would be happy to discuss any new or remaining concerns during the response period.

My understanding of the main claim of this paper is that researchers need to improve inference time compute methods and base model robustness in parallel? I think it is possible to over-read Zaremba as “inference time compute gives you robustness for free”. In contrast, this paper says the two together are stronger because more robust models -> more interpretable perturbations -> more amenable to reasoning benefits.

Exactly! Indeed, our findings push back against the risky misconception that inference scaling alone solves robustness. Going beyond Zaremba et al. (2025), we show that increasing model size or inference compute, without addressing representation robustness, offers limited benefit under adversarial conditions. In contrast, using robust base models enables inference-time reasoning to actively defend against attacks, improving performance even in challenging settings.

Critical Issue: No experimental details on o1 results from Figure 1 and apparently missing the claude results which are discussed in multiple sections, especially section 3.2... Where are the results discussed in lines 191 to 202?... Where are the experimental details for figure 2?

Figure 2 has both o1 and Sonnet results; i.e., “the experimental details for figure 2” are the Sonnet-augmented-reasoning experimental details given on “lines 191 to 202”. Our revision will clarify Figure 2’s o1 and Sonnet experiments, respectively, as follows:

• Sonnet: The Sonnet-augmented-reasoning results in Figure 2 are described by the experimental details “discussed in lines 191 to 202” and are discussed in Section 2 (e.g. Lines 94 and 98). Our revision will use the extra page to create a new experiment subsection in Section 4 that describes the Sonnet experiment in a self-contained manner. Note that we will also take the reviewer’s advice to discuss methodology and experiment in the same area to allow “the reader to maintain an ongoing ‘context’ of each experiment.” Our revision will also keep Section 2’s minimal discussion of Figure 2’s Sonnet results because we believe that they provide critical background for the relatively new problem area that we explore.
• o1: Unfortunately, Zaremba et al. (2025) tests a proprietary model and does not provide extensive experimental details. Our revision will clarify that the o1 results in Figure 2 are copied from Zaremba et al. (2025) and will include the experimental details they do provide, which are:
  • class label information is provided within the prompt,
  • the model predicts the class label given the image,
  • the attacker is considered successful if the model’s prediction is incorrect.

The main technical weakness of the first set of experiments in this paper is the way they instantiate “Inference-Time Compute”

Notably, our submission studied both black-box and white-box attacks, and for black-box attacks we instantiated inference compute via both Sonnet-augmented reasoning (Figure 2) and LLaVA CoT (Table 2).

The reviewer is correct that the white-box attacks involved repeating the correct answer in the model’s context. We took this approach to make the PGD attack stronger: with a static context, the PGD attacker is attacking a fixed function – a fixed model context window – that allows it to make consistent progress over a series of PGD steps. The alternative approach would be to risk interrupting the PGD attacker’s progress by letting the model reason freely such that the context window is constantly changed.

To test the effect of letting the model reason freely, Zaremba et al. (2025) used black-box attacks from the Attack-Bard dataset. We also test the effect of letting the model reason freely with the same black-box attacks; however, we also include the aforementioned white-box experiments.

[limited reproducibility with frontier models]

Our revision will enhance reproducibility by including the specific version of Sonnet that we used (“claude-3-7-sonnet-20250219”), though we agree that exact reproducibility is hard to guarantee with closed-weight models.

Moreover, we now have new and improved results that achieve this “realistic” quality (see below). First, we improved the methodology used in Table 2. Second, we added the requested “visual prompt injection” attack to our white-box methodology.

Fig 3: caption should be more clear about difference between high and low, add more info to make this caption more self contained.

We will update Figure 3's caption to add details about these Sonnet-augmented-reasoning experiments. Notably, given image descriptions from the various LLaVA models, Figures 2 and 3 used Sonnet without reasoning for the “low” setting, and 16,000 token thinking budget in the “high” setting.

Table 1: Would be good to have this experiment duplicated for more attack images? Why is it so few examples.

Our revision (and the below table) increases Table 1’s data by a factor of four, such that we now attack models on four different images, with three unique attack targets per image (i.e., 12 attacks per table cell). The original table used fewer examples due to limited compute time. The updated table maintains the trend observed originally, while reducing the standard errors of the estimated PGD steps (i.e. attack difficulties). In particular, Delta2LLaVA receives the clearest benefits from increasing inference compute, and it is the most robust base model.

Table R1: PGD steps required for a successful attack across models, perturbation budget ε, and inference-compute levels K. Mean (standard error) computed on three attack variations (color, shape, texture) for the Red Ball, Speed Limit Sign, iPod and Sea Urchin images.

Table 2: The very minimal increase in robustness seems to weaken the rich hypothesis substantially.

Our revision improves Table 2's experimental design and finds statistically significant support for the RICH. Previously, CoT didn't provide a significant improvement on clean data (see the first two rows of Table 2). We altered our CoT prompt to illustrate to the VLM how to format its output, which caused all models to show a benefit from CoT on clean data. Given this baseline improvement from CoT, we then computed the benefit of CoT on adversarial data. Consistent with the RICH, we found that only robust models benefitted from CoT on the adversarial data (at the p-value < 0.01 level, using a McNemar Test), supporting the RICH hypothesis.

Table R2: Attack Bard Black Box 30-Way Classification on LLaVA Family

Do you have any theories on how this work may transfer to semantic adversarial attacks? e.g. “visual” prompt injections?

Great question! Our revision will include the following exciting result that further supports the RICH. We inserted a visual prompt injection into a clean Attack-Bard image, then we performed a white-box PGD attack that encouraged the model to repeat the visually injected text instead of describing the image as asked. When we scaled the inference compute, only robust models benefitted, as shown by the increase in the loss of the PGD attacker below.

Table R3: Only robust LLaVA models obtain added robustness (measured with PGD attacker’s loss) from inference compute scaling when faced with visual prompt injections.

Q: experiments with an open weights reasoning model where we can inspect reasoning traces could have been beneficial.

We will add to the revision’s appendix the CoTs used by the open weight models we study. Additionally, we will mention the reviewer’s suggestion as a direction for future work.

have the introduction/analysis of each experiment be contained… E.g Merge 3.1 and 4.1! This allows the reader to maintain an ongoing “context” of each experiment.

Good idea! We will adopt this approach in our revision.

Hello!

Thank you for addressing my concerns, which were mostly about the organization of experimental details. The paper has a bevy of intriguing and thoughtful experiments, which is a key strength of the paper, but I think that makes being explicit about which figures correspond to which experiments extra important. I love the added visual prompt injection experiments.

I believe the author's changes and edits alleviate my concerns and allow the scientific impact of the RICH hypothesis to shine through.

eg, I totally misunderstood that the "o1" experiment was merely reporting the Zaremba et al results and thought that was a novel experiment in this paper.

Accordingly, I have changed my score to 5.

Thank you for your insightful review! We are happy to hear that you believe our work is an "important contribution". Moreover, thank you for emphasizing that visual robustness has been a weakness for vision language models. Yes, our objective is to help address this weakness by providing a hypothesis, and empirical evidence, on the mechanism for when increased inference compute confers robustness benefits. Our findings push back against the common misconception that scaling alone solves robustness: additional inference compute without representation robustness yields little benefit. Thank you for your concern about the lack of model size diversity. We direct you to our expanded results with frontier open-weight models which bolsters the generality of the Robustness-from-Inference-Compute Hypothesis (RICH).

Experiments: it is not clear if there is a difference if the model has a different size… Number of models in experiment could be larger.

Our revision will address both of these concerns by adding more models to our experiments, and by choosing new models that are larger than those we explored. As shown in Table R1 below, LLama-3.2-Vision-High-90B and Qwen-2.5-VL-72B-Instruct do not exhibit substantial, statistically significant gains from CoT on attacked data despite doing so on clean data. This supports the RICH, consistent with our submission’s Table 2 and the updated version of Table 2 below (Table R2).

Table R1 Attack Bard Black Box 1000-way Classification On Large Models

Table R2 Attack Bard Black Box 30-Way Classification on LLaVA Family

Related work: Test-time adaptation (TTA) has never been discussed. This could be interesting… Why TTA was not considered important for this paper?

Thank you for the suggestion! Our submission will discuss TTA as an interesting direction for future work, referencing how TTA can induce adversarial vulnerabilities (Wu et al., 2023). Please let us know if you have any additional suggestions.

Ln:159: What makes PGD novel in this context?

Great question, and we will clarify the novelty in the revised manuscript by including the following explanation.

We use a standard white-box PGD attack on the VLM’s vision input, but before each attack we pre-fill the model’s context window with a different amount of content. This allows us to test the effect of different levels of inference compute – i.e., different amounts of pre-filling – on the difficulty of the attack.

This contribution is important because naively letting the model reason freely (dynamically changing the context window’s contents) during an attack could destroy the progress made by the PGD attacker: an attack that is effective with one model context may be completely benign when the model context is changed. In Zaremba et al. (2025), this only problem did not arise because only static black box attacks were used: the input image does not change in black-box attacks, which allows measurement of the effectiveness of reasoning against a constant input attack as a function of inference compute.

Thus, we provide a white-box attack experiment that tests for the first time (to our knowledge) how inference compute scaling affects PGD attack success. Notably, we also test models using the same black-box setting as Zaremba et al. (2025). Crucially, we observe support for the RICH (our core hypothesis) in both settings.

Ln311: While the insights are interesting to me, it is somehow not explained in the paper how close this could be helpful for real-world defenses? (practability)

We will revise our submission to clarify the real-world practicality as follows.

Our findings have several practical implications. First, they show that using robust base models is critical—reasoning only improves robustness when built on stable representations. Second, we demonstrate that lightweight, post-hoc adversarial finetuning (e.g., FARE-LLaVA) is sufficient to enhance the robustness benefits of reasoning without costly retraining. Third, our results highlight inference-time reasoning as a viable defense mechanism—when paired with robustness. Finally, we challenge the common belief that scaling compute alone improves robustness, suggesting that resources should instead be allocated toward robust training and finetuning strategies. More broadly, robust representations introduce a distinct and underutilized axis for improving performance—complementary to inference compute and large-scale pretraining—by enabling more effective generalization under adversarial and out-of-distribution conditions.

Joint Response for All Readers

We are glad to know reviewers were “very intrigued” (oVEA) by our work, finding it made “an important contribution” (mxf9) to an “important problem” (ZqwH) with “practical significance” (VKwn). To our knowledge, we are the first to replicate a phenomenon previously only observed in proprietary models (Zaremba et al., 2025). Excitingly, our results are consistent with prior work: obtaining robustness from inference compute is indeed a promising direction. However, we go beyond replicating prior work, focusing on the relatively small improvements previously obtained with multimodal reasoning as a defense against adversaries. We propose the Robustness from Inference Compute Hypothesis (RICH), which predicts when scaling inference compute in multimodal settings will provide robustness benefits. Our submission’s experiments, and new experiments we added for reviewers like visual prompt injection defense, consistently support the RICH.

We thank the reviewer for their thoughtful feedback. We would like to reemphasize the broader impact and practical implications of our work: it challenges the prevailing assumption that scaling alone yields robustness, and instead highlights robust representations as a key pathway for improving performance under adversarial conditions. To strengthen our case, we have added new experiments showing: (a) the RICH hypothesis holds for larger open-weight models (LLaMA 3.2 Vision 90B and Qwen2 VL 72B), (b) stronger chain-of-thought prompting significantly improves performance in the adversarial setting for robust models, and (c) RICH applies to semantic adversarial attacks that perform visual prompt injections (see response to Reviewer oVEA). Together, these additions provide further empirical support for RICH and reinforce its practical relevance to building safer, more robust systems. We hope you will champion acceptance of the paper, and we will sincerely appreciate your engagement if there are further questions.

The paper proposes an explanatory hypothesis but does not develop specific improvement methods or new defense strategies based on this hypothesis. The findings remain largely theoretical without practical defensive solutions.

While our paper introduces an explanatory hypothesis, it also offers practical implications grounded in empirical results. Our findings push back against the common misconception that scaling alone solves robustness. We show that increasing model size or inference compute, without addressing representation robustness, offers limited benefit under adversarial conditions. In contrast, using robust base models enhances inference-time compute’s ability to defend against attacks, improving performance even in challenging settings. We also demonstrate that lightweight, post-hoc adversarial finetuning (e.g., FARE-LLaVA from the Robust CLIP paper) can obtain this effect, which has notable practical relevance. Thus, our findings offer easy-to-use, actionable pathways for improving robustness in deployed systems, not just theoretical insights. We will emphasize this practical dimension more clearly in our revision—thank you for the helpful suggestion.

The experiments primarily validate the hypothesis through increasing computation and comparing models with different robustness levels, without exploring more sophisticated experimental approaches.

Please let us know if you have ideas for more sophisticated experimental approaches. Our experiments already greatly augmented those considered by Zaremba et al. (2025). In fact, we developed a novel white-box attack that is (to our knowledge) the first white-box attack that has been applied to test inference-time compute as a defense – the adversarial attacks in Zaremba et al. (2025) only include the black-box attacks in the Attack Bard dataset, which we also studied.

The experiments are limited to relatively small-scale models as acknowledged in the limitations section. This raises questions about whether the findings would hold true for larger, more sophisticated models.

Our revision addresses this by adding new models that are larger than those we explored. As shown in Table R1 below, LLama-3.2-Vision-High-90B and Qwen-2.5-VL-72B-Instruct do not exhibit substantial, statistically significant gains from CoT on attacked data despite doing so on clean data. This supports the RICH, consistent with our submission’s Table 2 and the updated version of Table 2 below (Table R2).

Our revision will mention that future work can adversarially finetune these models to investigate whether the increased robustness from finetuning improves the robustness benefits these models obtain from reasoning.

Table R1 Attack Bard Black Box 1000-way Classification On Large Models

Table R2 Attack Bard Black Box 30-Way Classification on LLaVA Family

The paper lacks ablation studies to isolate the impact of different components of their approach, making it difficult to understand which aspects are most crucial for the observed improvements.

Could you please clarify the different approach components that you’re referring to? Our understanding of our approach is that we induced increases in inference time compute in various ways (e.g. through CoT and through a novel text-repetition technique), and we tested whether these increases led to robustness improvements. We consistently found that, regardless of how inference compute was increased, robustness only improved significantly when the base model was already robust.

How would the proposed hypothesis generalize to other types of attacks beyond vision attacks?

We discuss this in lines 461-467 of our submission, in the appendix, and we will move this discussion into the main text if our paper is accepted (which grants an additional page). This discussion is paraphrased as follows: Our hypothesis suggests that other attacks, like text attacks, may be resisted by increased compute due to the mechanism outlined by the RICH. In particular, since larger LLMs tend to be more robust even without adversarial training (Howe et al., 2024), the large-scale LLMs explored in Zaremba et al. (2025) may have been able to resist text attacks via inference compute through the RICH mechanism. We leave more targeted experiments of this phenomenon in the text modality to future work.

[Request for theoretical analysis.] What are the specific characteristics of "in-distribution" attacks that make them more amenable to reasoning-based defenses?

We would like to emphasize that our primary contribution is empirical, and these findings have standalone value. The link between inference-time compute and robustness across attack types is consistent, reproducible, and practically valuable. Publishing such empirical insights provides a foundation for the community to build on, even if a full theory is not yet in place.

Our revision will suggest modeling perception noise or errors in the DS3 framework (Ellis-Mohr et al., 2025) as a potential way to study the impact of robust perception with inference-time compute. This potential future work warrants a separate investigation due to its complexity and out-of-scope nature.

Regarding the reviewer’s question on in-distribution attacks: our hypothesis is that these attacks often exploit superficial correlations that reside on the data manifold. Reasoning-time defenses can better detect and override such patterns by leveraging more semantically aligned representations in robust models. In contrast, out-of-distribution attacks may operate off-manifold, demanding fundamentally different mechanisms—something we are actively investigating.

How does the computational cost of increased inference-time compute compare to other robustness approaches?

This is a great question, and our revision will include the following discussion to address this. Addressing robustness by increasing inference compute introduces new computational costs. While our work provides evidence for a hypothesis that guides efficient usage of inference compute in adversarial settings, we note that alternative methods (e.g., use of a diffusion-based purification model, Nie et al., 2022) could be more efficient paths to robustness in some circumstances. However, in settings like our white box attack on the red ball image, increasing inference compute is an inexpensive defense. The base instruction is 35 tokens and each inference compute level (1, 2, 3) adds an additional 8 tokens.

Could there be alternative explanations for why robust models benefit more from increased compute beyond the distribution shift hypothesis?

We would be happy to add to our revision alternative explanations, but our tests of alternative hypotheses found no support. For example, our submission suggested an alternative explanation of our results: optimizing a PGD attack against any model may become more difficult as inference compute (context window size) grows, but our experiments found that larger context window sizes did not raise the number of optimization steps required by the PGD attacker – unless the model being attacked had been trained to be robust. We then considered another alternative hypothesis: optimizing a PGD attack against adversarially trained models becomes more difficult as inference compute (context window size) grows. However, our experiments again found this explanation to be insufficient: we found that inference compute could make PGD attacks more difficult in models that had not been adversarially trained – if the attacker’s budget was smaller (epsilon = 16). In this case, we did indeed find that (even for non-robust models), inference compute has robustness benefits. In summary, despite considering alternatives, we only found evidence for the idea that inference compute provides benefits when the attacked data is near enough to the attacked model’s training distribution.

Thank you for your thoughtful review! We are happy you identified that “robustness and inference compute interact synergistically rather than additively, offering clear guidance for how to prioritize robustness efforts.” Indeed, our findings challenge the current notion that scaling alone solves robustness. Our work in this new area suggests a guiding principle for future adversarial defenses: additional inference compute is wasted without representation robustness. Regarding the scope of our experiments, please note that we have expanded our results with frontier open-weight models and a new open-ended setting that bolsters the generality of evidence for the Robustness-from-Inference-Compute Hypothesis (RICH).

Limited scope of models and tasks: All experiments are confined to 7 B-class LLaVA derivatives and image-classification prompts; it remains unclear whether RICH generalises to larger frontier models or open-ended chat scenarios.

As we discuss below, our revision will broaden our analysis to include frontier multimodal open-weight models (Llama 3.2 Vision 90B and Qwen 2 VL 72B), extending our black box transfer experiments. Additionally, we study the RICH in the context of visual prompt injection attacks that mirror open-ended real-world settings.

First, we add Llama 3.2 Vision 90B and Qwen 2 VL 72B to the Attack-Bard experiments (shown in Table R1 below). Given that these models have not been adversarially trained, their robustness to adversarial image attacks is limited. The Robustness from Inference Compute Hypothesis (RICH) – which predicts that reasoning on non-robust data representations is less beneficial than reasoning on robust representations (i.e. from data that is more in-distribution) – would therefore be supported if these models had less improvement from inference compute on adversarial data than on clean data. Comparing the relative accuracy of chain-of-thought to no reasoning, we find that raising inference compute clearly helps larger models on clean data. However, raising inference compute provides no statistically significant (p-value < 0.01) benefit on adversarial data within each model family.

In contrast, when we use a robust base model in the same setting (Table 2 and its updated version Table R2, below), raising inference compute improves both clean and adversarial data performance, with adversarial data performance benefitting the most. This underscores our key finding: scaling model size or inference compute alone won’t address robustness. Raising awareness of this limitation can avoid potentially inefficient (or ineffective) scaling.

Table R1: Attack-Bard Black Box 1000-Way Classification on Frontier Models

Table R2: Attack-Bard Black Box 30-Way Classification on LLaVA Family

Second, we find evidence for the RICH in a setting other than image classification using non-robust and robust models from the LLaVA series. We inserted a visual prompt injection into a clean Attack-Bard image, then we performed a white-box PGD attack that encouraged the model to repeat the visually injected text instead of describing the image as asked. When we scaled the inference compute, only robust models benefitted, as shown by the increase in the loss of the PGD attacker below. This extends the generality of our findings and broadens RICH support to additional attack methods.

Table R3: Only robust LLaVA models obtain added robustness (measured with PGD attacker’s loss) from inference compute scaling when faced with visual prompt injections.

Compute overhead unreported in the main text: High-compute settings allow very long “thinking” traces yet provide no latency, throughput, or cost analysis, leaving real-world viability uncertain.

Our revision will include the following discussion on the cost implications of increasing robustness via inference compute. Addressing robustness by increasing inference compute introduces new computational costs. While our work provides evidence for a hypothesis that guides efficient usage of inference compute in adversarial settings, we note that alternative methods (e.g., use of a diffusion-based purification model, Nie et al., 2022) could be more efficient paths to robustness in some circumstances. In settings like our white box attack on the red ball image, increasing inference compute is a simple and computationally inexpensive defense. The base instruction is 35 tokens and each inference compute level (1, 2, 3) adds an additional 8 tokens. This suggests that robust base models can have their defenses bolstered cheaply in some scenarios.

Small adversarial corpus: The black-box Attack-Bard set contains only 200 images, which may limit statistical power and diversity.

Please note that the experiments in Zaremba et al. (2025) only use Attack-Bard for their adversarial multimodal experiments. We use the same dataset to follow their experiment methodology. Additionally, we go beyond their experiments by introducing new data and methodologies (e.g. our novel white-box attack setup).

However, our revision will address the concern about statistical power. We have updated our reporting of the black-box experiment results in Table 2 such that p-values are now visible (we also modified the CoT prompt to improve its effectiveness on clean data). The updated table R2 is above. Despite only using 200 images (like Zaremba et al., 2025), we find statistically significant benefits (p<0.01) of CoT on the Attack-Bard task when the model is robust or the data is clean – if the data is attacked/unclean and the model is not robust, then the benefit of CoT is not statistically significant. These findings are consistent with and statistically support the RICH.

Additionally, please note that our submission’s white-box experiments reported standard errors to provide statistical support for our claims in that setting (see Table 1).

Reproducibility gaps: Critical implementation details (exact prompt templates, PGD step sizes, per-ε results) are pushed to appendices, making replication from the main paper alone difficult.

We appreciate the reviewer’s emphasis on reproducibility and agree that critical implementation details are essential for replication. We would like to clarify that all such details (e.g., exact prompt templates, PGD step sizes, per-ε results) are fully included in the appendix and are explicitly referenced in the main text. While page limitations currently preclude their inclusion in the main body, we plan to move them into the main text upon acceptance, when an extra page becomes available.

Incremental originality of idea: While applying inference-compute scaling to vision is novel, the core intuition—that extra compute can improve robustness—extends earlier text-only studies rather than introducing a fundamentally new mechanism.

We disagree about how original our work is and the core intuition: we do not simply extend earlier studies to the vision domain and show “that extra compute can improve robustness”. Instead, we introduce a novel hypothesis and explanatory mechanism that indicates when inference compute will benefit robustness. Further, we show that our predictions are consistent with experiments: models predicted to not significantly benefit from inference compute do not, and models predicted to significantly benefit from inference compute do. In summary, beyond overcoming the limited benefits to multimodal robustness achieved via inference compute by Zaremba et al. (2025), we provide an explanation for why these benefits are achieved and leverage it to obtain our superior results.