AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization

• Eqn. 4 is basically the loss as in FARE (see Eqn. 3 and discussion in section 3.3 in [1]) - and this discussion is missing from the text. This seems misleading, since this equation does the heavy-lifting in generating adversarial examples that make AdPO robust.

• The fact that AdPO leads to more robust models is not surprising since the method finetunes the whole LVLM (LLM is frozed but the loss/gradients are propogated through) unlike FARE and TECOA which finetune just the vision encoder. The comparison's thus seem not fair as the settings are different. There should have been standardization of FARE and TEcoA at the same level of compute-time (could be done with more epochs/more data etc for FARE/TECOA ) as AdPO.

• Overall time complexity of the method seems very high (relative to baselines) and is not described in the paper. Given the loss is calculated over the whole LVLM, and there is online description generation of the image within the method - it seems significantly higher that TECOA or FARE.

• Table-2 does not add help AdPO's efficacy much, since the number of samples is small and the difference between FARE$^2$ and AdPO$^2$ is only 2%, and there is no difference for larger radii.

• In Figure3, AdPO does not seem better than FARE, for instance: "FARE: A young boy is walking on a grassy field." is subjectively slightly better description than "AdPO: A young boy is playing with a soccer ball in a field." since the boy is walking while holding the ball and not playing with it.

• Lacks other important evaluations: other important aspects of LVLMs like hallucination, robustness to jailbreaks should have been tested with AdPO, does the proposed method also help in these cases? This is important as making models robust should not lead to substantial degradation in performance in other regards - else the model is not usable as before.

[1] Schlarmann, Christian, et al. "Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models." Forty-first International Conference on Machine Learning.

• The method section introduces preferred and adversarial image optimizations; however, the experimental section lacks a comparative analysis to showcase the specific benefits of these optimization methods.

• Joint optimization may add complexity, making training more challenging, particularly in terms of hyperparameter tuning and convergence. The paper could benefit from a discussion on the complexities and potential trade-offs involved in joint optimization.

• Although adversarial robustness is a core focus of this paper, the evaluation under the untargeted attack setting is limited to a single attack method. A more comprehensive evaluation would benefit from employing multiple attack types, including stronger attacks, to thoroughly assess the model's robustness. This approach would provide a clearer and more detailed understanding of AdPO's efficacy across a broader range of adversarial scenarios.

1. The novelty is weak. The proposed method just applies previous DPO method into the LVLM scenarios. The main contribution is to generate prefer and non-prefer pairs for training. All the components are not new. Therefore, the contribution of this paper to the field of LVLM adversarial defense is weak.

2. Missing many related references of LVLM attacks and defenses. There are many latest LVLM attack and defense papers released on Arxiv. There are also many survey papers that summarize and discuss existing LVLM attacks and defenses. However, this paper does not cite, summarize, and discuss these existing works. The author only introduces part of it, thus the discussion is not comprehensive and the comparison is not fair.

3. Experiments are not convincing. First, the authors just simply conduct experiments on two simple LVLM models. More experiments on the general LVLM models (like GPT4, BLIP2) are required. Otherwise, it cannot validate the generalization of the proposed method. Second, the authors just implement two defenses method for comparison. This is not enough. At least 4-5 defense baselines are needed.

4. Since this paper aims to improve the adversarial robustness of existing LVLM models, it should provide in-depth analysis of the defense performance to existing LVLM attackers. However, the authors do not provide any analysis.

5. No complexity and efficiency discussion. It is necessary to provide corresponding analysis compared to existing defense methods.

1. The attack strengths used in adversarial training and evaluation were only 2 and 4; does AdPO still perform better than other baselines at larger attack strengths (e.g., 8)? Robustness evaluation under other attack methods, e.g., unconstrained strong CW attack, adversarial attack with smoothing adversarial images, is also expected.

2. Complexity analysis of AdPO. What is the amount of adversarial training data and the computational cost of adversarial training between AdPO and TeCoA and FARE? If AdPO uses less data or is more efficient, then this is another advantage of AdPO. If AdPO uses more data or requires more computational cost, then a fair comparison should be made.

3. Lack of performance comparison with direct adversarial training. While the cost of adversarial training is high, 32 A100 GPUs can support this training, would AdPO work better than direct adversarial training against LVLM?

4. The clean COCO performance of AdPO2 in Table 1 even exceeds that of CLIP, why might this be?

5. Despite what is stated in line 530, I'm still curious why AdPO is only done on CLIP, if end-to-end AdPO would lead to better improvements.

6. Can you show the results after scaling up? AdPO should be more efficient than adversarial training, can using more high quality DPO data further improve the performance?

7. Many variants of DPO exist nowadays and it is expected that the authors discuss or compare with these DPO variants to further demonstrate the effectiveness of AdPO.