ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the novelty, flexibility, and scalable of ETA as well as the superiority of our work. We believe the constructive feedback will improve the paper and increase its potential impact on the community.

• Weakness 1: Limitations in Pre-Generation Evaluation Using CLIP

Thanks for the constructive suggestions. We found that the original CLIP model already performs strongly in our task even without fine-tuning. This observation aligns with previous works that highlight its strong generalization ability in safety-related downstream tasks [1]. Specifically, we have the following findings:

(i) CLIP score effectively distinguishes between harmful and safe images: As shown in Fig. 3(b), we found that CLIP score effectively distinguishes between harmful and safe images.

(ii) Removing some categories in the prompt does not compromise CLIP’s performance: Our safety-related prompts already include common safety categories such as "physical harm". However, even after removing some of these categories, the CLIP score still effectively distinguishes between harmful and safe images. We believe this is because, as long as the textual input is semantically harmful, regardless of the specific unsafe categories, the CLIP-encoded image associated with harmful content naturally receives a higher CLIP score compared to non-harmful images.

(iii) The goal is to identify whether the image is safe, rather than classify its unsafe categories: Moreover, the purpose of our Pre-Generation Evaluator is simply to assess whether the input image is harmful or not, rather than categorize it into specific harmful categories. As demonstrated in our experiments, the CLIP score is effective in detecting unsafe images through our designed prompt.

• Weakness 2 & Question 1: Potential Instability in Post-Generation Evaluation Using Reward Models (RMs)

Thanks for the valuable suggestions. To address the instability of the reward model in safety evaluation, we have proposed a safety-specific input format, as shown in Appendix A.2. The reward model we used already shows strong capabilities in safety evaluation [2]. We found that this designed input format further enhances the Reward Model’s ability to distinguish between safe and unsafe responses. Specifically, the safety-specific input format significantly increased the KL divergence between the distributions of safe and unsafe responses, from 0.07 to 0.21.

The focus of ETA is not on improving the interpretability of the reward model, but rather on utilizing it to distinguish between safe and harmful responses. ETA can also be combined with judge models capable of generating explanations, and we leave this exploration for future work.

We are also looking forward to the release of more safety-specific reward models or VLM reward models in the future. As a flexible plug-and-play framework, ETA can easily incorporate more powerful reward models, leading to even better performance.

• Question 2: Did you observe any scenarios where the ETA framework failed or produced unexpected results?

As shown in our experimental results, while ETA can serve as a plug-and-play method to significantly reduce the USR compared to baselines, there are still instances where it struggles. We hypothesize that this is mainly due to the performance of VLM baselines. As a plug-and-play method, the alignment phase still relies on the VLM baseline to generate responses. The baseline’s inherent capabilities, such as instruction-following ability and built-in safety measures, will impact the performance of ETA. As shown in the table below, we observed that when ETA is applied to Llama3.2-11B-Vision-Instruct (released in Sep. 2024) a more recent and stronger model, the reduction in USR is significantly greater than that of other baselines. This finding highlights the potential of ETA, as future models with enhanced capabilities are released, ETA can be seamlessly applied to safeguard new VLM baselines.

Reference

[1] Tsai, Wei Lun, Jacob J. Lin, et al. Generating construction safety observations via CLIP-based image-language embedding. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Haoxiang Wang, Wei Xiong, et al. Interpretable preferences via multiobjective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024

We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the well-written and superior performance including the safety and helpfulness of our work. We believe your constructive feedback will improve the paper.

• Weakness 1: Novelty: although the simplicity of the proposed methods, the novelty seems significantly limited, as the proposed methods are based on the simple application of existing models such as CLIP and Reward Models.

While ETA leverages existing models such as CLIP and reward models (RMs), its contribution lies in how these models are uniquely used to address VLM safety, which we will further elaborate on below. In fact, ensuring safety while preserving the original capabilities of VLMs is a very challenging problem, and we believe this is where ETA makes a significant contribution. We summarize the novelty of ETA as follows:

(i) New Perspective on Safety Issues in VLMs: We validate that the primary factor contributing to safety problems in VLM systems is the continuous nature of visual token embeddings.

(ii) First to Introduce Additional Visual Safeguards: The usage of CLIP in our work differs significantly from its application in prior studies. Previous works have primarily leveraged the CLIP score during post-processing stages, such as inference and evaluation, to assess the similarity between output text and input images. In contrast, our paper innovatively utilizes the CLIP model during the pre-processing stage. We demonstrate that the CLIP score can be effectively generalized to evaluate the safety of continuous visual embeddings, which can easily bypass the existing safety mechanisms in VLMs. By utilizing predefined prompts with unsafe semantics, we show that it is generalizable to classify input images for safety directly, providing a novel application of CLIP in ensuring the safety of visual inputs.

(iii) An innovative Strategy to Improve Safety Evaluation Capability of Reward Models: We demonstrate that designing a safety-specific input format for general reward models can significantly enhance their ability to more effectively evaluate the safety of responses. This differs from RMs’ usage in prior studies.

(iv) Harmless and Helpful Responses, Rather Than Generic Refusals: We found that existing methods often provide generic refusals such as “I cannot help you, …”. However, an ideal response should be both harmless and helpful, offering helpful suggestions to users. By leveraging the multimodal evaluator (CLIP+RM) for sentence-level Best-of-N searching, we can effectively identify safer and more helpful generation trajectories. This is also a novel usage of CLIP and RM.

(v) Superior Performance of ETA as a Simple Plug-and-Play Framework: Extensive experiments demonstrate that our plug-and-play approach is applicable to common VLM baselines, significantly enhancing the model’s safety without affecting its general capabilities. Moreover, it improves both the safety and usefulness of the model’s responses in unsafe scenarios.

• Weakness 2 & Questions 1: Clarification of the motivation: While the authors posed the major cause of the VLMs being jailbroken is the continuous nature of the visual embeddings, I think it is due to the visual embeddings not being safety-aligned with RLHF. Also, I'm not sure if the proposed methods correspondingly resolve the motivation behind the visual embedding's continuous nature. & Performance comparison with the safety-aware visual RLHF (if any) would be appreciated to see if the proposed method would rather be the optimal way of safeguard VLMs.

Thanks for your valuable comment.

(i) We believe these two explanations are not contradictory. As shown in Figure 2, we illustrate that continuous visual token embeddings bypass existing safety mechanisms, which were aligned based on discrete text token embeddings. This highlights the need to establish safety awareness specifically for these outlier continuous visual tokens. While applying RLHF to these visual embeddings could be a potential solution, it is resource-intensive. It requires a substantial number of annotators, the collection of difficult-to-obtain multimodal datasets, and significant computational resources for training. Consequently, most open-source VLMs have not undergone safety-aware visual RLHF and lack safety awareness for continuous visual embeddings. Therefore, we propose an alternative solution, ETA, which is an inference-time plug-and-play framework, demonstrating excellent performance across various baselines.

(ii) Our solution correspondingly resolve the motivation behind the visual embedding's continuous nature. We note that these continuous visual embeddings in VLMs are essentially obtained by the pre-trained CLIP vision encoder. CLIP is trained to align vision and text embeddings by maximizing semantic similarity across modalities [1]. This inspired us to leverage CLIP score to evaluate the safety of continuous visual embeddings by measuring their similarity to pre-defined harmful text prompts embeddings. This approach requires only the additional use of the CLIP text encoder to effectively assess whether the input image is safe. Therefore, we enhance VLMs’ multimodal safety awareness by utilizing the CLIP score to assess the safety of visual inputs.

(iii) At the time of submission, no VLMs with safety-aware visual RLHF were known to exist. Now, we have identified a recently released model Llama3.2-11B-Vision-Instruct (released in Sep. 2024), which incorporates safety-aware visual RLHF. Since Llama3.2-11B-Vision-Instruct does not have a corresponding model without RLHF, we choose another widely-used VLM LLaVA-1.5-13B and apply ETA on it to compare ETA and visual RLHF. The results, as shown in the table below, reveal that applying ETA leads to originally weaker safety performance of LLaVA-1.5-13B resulting in a USR that is generally lower than that of Llama3.2-11B-Vision-Instruct. In addition, we also applied ETA to Llama3.2-11B-Vision-Instruct and found that applying ETA further enhances the safety of the baseline aligned with RLHF. This demonstrates the promising application potential of ETA.

• Weakness 3: Pre-generation Evaluator: although the image contains the harmful image, the textual instruction might be normal and benign (e.g., a criminal scene image + "How to prevent such crimes?"). According to the proposed method, it may judge these benign inputs as unsafe, which possibly leaves as a side-effect.

Thanks for the valuable question. In the scenario you described, ETA does not interfere with the model’s normal responses. It is indeed likely that the Pre-Generation Evaluator classifies the input as unsafe. However, if the text instruction is beneficial, such as “How to prevent such crimes?”, the Post-Generation Evaluator can classify the response as “safe” if it provides a safe and helpful answer. Our criteria are designed such that the alignment phase is only triggered when both the Pre- and Post-Generation Evaluators classify the response as unsafe; otherwise, the model’s original reply is directly output. As analyzed in Section 4.1.2 and demonstrated in Table 10, this setup allows for accurate judgments to handle such cases effectively.

To further verify whether ETA might have potential side effects under the reviewer’s setting, we used the image in Fig. 1, which depicts “Defensive Driving” classified as an unsafe image by ETA’s Pre-Generation Evaluator. We input this image along with the prompt “How to prevent such crimes?”. ETA directly outputs the response generated by the baseline, and we report the response below.

  To prevent crimes like the one depicted in the image, where a car is being used to commit a crime, it is essential to implement various preventive measures. These can include:

 1. Strengthening law enforcement: Increasing the presence of police officers and patrolling areas with high crime rates can deter potential criminals.
 ...

• Weakness 4: Sentence-level Deep Alignment: since the proposed deep alignment selects the optimal sentence under the consideration of the safeness of the given sentence and its correlation with the input image in a greedy manner, the resultant sequence of sentences might not be ensured to be logically connected. This potential side-effect might rather hinder the reliability of the VLMs.

Thank you for your valuable questions. ETA does not cause semantic or logical incoherence. Our method does not generate k full responses and then select the best candidate sentence for each individual sentence. This approach could indeed affect the logical coherence of the generated response. Instead, we start by generating k candidate sentences. For the first sentence, we select the highest-scoring sentence out of k candidates. Then, following this chosen sentence, we generate k candidates as the second sentence and select the one with the highest score. At this point, the response consists of two sentences. The process continues in this manner, guiding the generation until it is complete.

It is important to note that in the sentence-level Best-of-N approach, each candidate in the i-th generation is generated based on the first i-1 sentences. This ensures that the context established by the previous sentences is taken into account when generating the next one, preserving the coherence and logical flow of the response. Some papers have verified that methods similar to beam search do not affect semantic and logical coherence, such as [2]. We have updated the manuscript with a more detailed explanation.

Reference

[1] Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.

[2] Zhanhui Zhou, Zhixuan Liu, Jie Liu, et al, Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. NeurIPS 2024

​Dear Reviewer,

Thank you again for your valuable feedback and questions. We have provided responses addressing your concerns and uploaded a new version of our manuscript.

As the discussion period draws to a close, we kindly encourage you to take a look at our responses and let us know if you have any remaining questions or concerns.

We sincerely appreciate your time and effort in helping us improve our work.

Sincerely,

Authors

Thanks for the detailed responses. Almost all of my concerns were resolved, except for the novelty issue (simple leverage of CLIP and reward models) as the other reviewers mentioned. I raised my score to 5.

Dear PCs, SACs, ACs, and Reviewer 8ihd,

We have noticed that the comments provided by Reviewer 8ihd appear to correspond to a different paper, as we did not propose DynVLA in submission 675. We would greatly appreciate it if you could kindly review this issue and have the reviewer resubmit the correct comments for our submission.

Thank you very much for your time and assistance.

Best,

Authors of submission 675

• Weakness 3: The proof of the Motivation in the MULTIMODAL EVALUATION (i.e. Continuous Visual Token Embeddings Bypass Safety Mechanisms) is not very rigorous: In the proof experiment, continuous visual embeddings were converted to the nearest discrete text embeddings, showing a 7% reduction in unsafe rates on one dataset. Testing this approach across more datasets and models could strengthen its reliability.

Thank you for the constructive suggestion. To further support our conclusion, we have additionally evaluated this approach on the SPA-VL Harm test set and VLSafe. The VLSafe dataset contains a total of 1,100 data samples. Due to the significant time required for the experiments, we randomly sampled 100 data points for testing. In the finalized version, we plan to include the complete results and we believe they will be similar. Additionally, we tested four baseline models on these two datasets, with the results presented in the table below. The decrease in USR after applying the mapping supports our motivation: "Continuous visual token embeddings bypass safety mechanisms (which are aligned on discrete text token embeddings)."

• Weakness 4: The effectiveness of the strategy is not well proven. The number of baselines (especially for inference-time baselines) is too small to prove the effectiveness of the strategy.

Thank you for the valuable comment. Considering that inference-time alignment strategies are still underexplored in the field of VLMs, ECSO is the only open-sourced method that does not require additional data or training, making it a plug-and-play solution similar to ETA. Some methods are categorized as inference-time approaches but require additional data and training phases, like [2] and [3]. Therefore, we believe ECSO is the most comparable method to ETA. To further demonstrate the effectiveness of ETA, we apply it to three more recent and advanced VLM baselines: LLaVA-NeXT-8B, LLaVA-OneVision-7B-Chat, and Llama3.2-11B-Vision-Instruct. The results, shown in the table below, indicate that even for Llama3.2-Vision-11B-Instruct, which has already been aligned via RLHF, our ETA successfully decreases the USR across five different test settings. This highlights the superiority of ETA in enhancing VLM safety.

In addition, we have summarized the extensive experiments conducted in our study, which include four baseline models, four safety-related datasets, and seven helpfulness-related datasets, all of which demonstrate the remarkable effectiveness of ETA:

(i) ETA Enhances the Safety Capability of VLMs: In Table 1, 6, and 7, we demonstrate that ETA not only significantly decreases the USR across all four baselines but also outperforms ECSO.

(ii) ETA Does Not Compromise the General Ability of VLMs: In Table 2, 3, and 8, we show that compared to ECSO, our ETA maintains the models' general abilities and provides more helpful suggestions in unsafe settings.

(iii) Both the Evaluation and Alignment Phases of ETA Outperform the ECSO: In Table 9, to further highlight the effectiveness of both the evaluating and aligning phases of our ETA, we replace our multimodal evaluator with the self-evaluation approach used in ECSO. Comparing "ECSO Eval. & ETA Align." with "ETA" we observe that the complete ETA process not only provides safer responses but also maintains the models' general capabilities.

Reference

[1] Yongting Zhang, Lu Chen, Guodong Zheng, and et al. Spa-vl: A comprehensive safety preference alignment dataset for vision language model. arXiv preprint arXiv:2406.12030, 2024c.

[2] Renjie Pi, Tianyang Han, Yueqi Xie, et al. Mllm-protector: Ensuring mllm’s safety without hurting performance. arXiv preprint arXiv:2401.02906, 2024.

[3] Yunhao Gou, Kai Chen, Zhili Liu, et al. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation. arXiv preprint arXiv:2403.09572, 2024.

We would like to thank the reviewer for the thoughtful and thorough comments on our paper as well as for recognizing our presentation, the effective and generalizable framework of our ETA, and the detailed and rich experiments. We answer your questions below.

• Weakness 1: The setting of the helpfulness experiment (Table 2) is not well rigorous. The experiment is conducted using general comprehensive benchmarks and VQA datasets, which is not fully consistent with the "usefulness" goal the authors aim to address—namely, ensuring helpful responses even with unsafe inputs. While Table 3 provides some helpfulness results, the range of benchmarks used is somewhat limited. Testing on more models and datasets can better verify the effectiveness of this strategy.

Thanks for the constructive comment. We evaluate helpfulness/usefulness from two aspects in this paper.

(i) Helpfulness/Usefulness under Unsafe Settings: To evaluate the helpfulness in unsafe scenarios, we followed the setup from [1], which is the only benchmark assessing helpfulness in unsafe settings. The results, shown in Table 3 and 4, demonstrate that ETA significantly enhances both the safety and helpfulness of models in such settings. Please note that the evaluation metrics in Table 3 and 4 are based on the GPT-4 API, which is extremely costly. Therefore, we did not conduct experiments on more models. We believe that the 96.6% Helpfulness Win-Tie rate compared to the baseline on LLaVA-1.5-7B is already strong evidence to demonstrate the superiority of our ETA. We have also provided examples of helpful responses by ETA under unsafe settings in the Appendix. For instance, as illustrated in the case study in Figure 10, ETA can offer helpful suggestions like: “Instead, I can share general advice to protect cars and their contents from potential thefts: ……”

(ii) Helpfulness/Usefulness under General Settings: We observed that existing methods, including fine-tuning-based and inference-time-based approaches, tend to degrade the general capabilities of models, as reflected in the comprehensive benchmarks and VQA datasets you mentioned. We provide the results of applying ETA to VLM baselines in Table 2 and Table 8, along with a comprehensive analysis in Section 5.2 that highlights the helpfulness of ETA in this setting.

• Weakness 2: The analysis of the ablation study results is inadequate: When the Utility Guide component is removed from the strategy, the unsafe rate actually decreases, yet this result is not fully analyzed. Does this suggest that the Utility Guide might be unnecessary? A deeper exploration is needed to clarify its role and significance.

Thank you for the insightful comment.

(i) It is worth noting that the scales and evaluation methods of USR and the Helpful Score in Table 5 are different. USR is calculated by $\text{USR}=\frac{|\{\text{unsafe responses}\}|}{|\{\text{all responses}\}|}$, while the Helpful Score represents the average score given by GPT-4 for all responses, with a maximum score of 10. If we normalize the Helpful Score to match the scale of USR, the improvement in Helpful Score from the fourth row to the fifth row, with the addition of the Utility Guide, corresponds to an increase of 1.20. However, the increase in USR is only 0.38. This suggests that the Utility Guide significantly enhances the helpfulness of responses but has only a slight impact on their safety capabilities.

(ii) We used the UnSafe Rate (USR) evaluated using MD-Judge-v0.2-internlm2_7B in our paper. To further verify the effect of the Utility Guide, we also report the Attack Success Rate (ASR) assessed through key-string matching and the USR evaluated by Llama-Guard-3-8B in the table below, which are another two common metrics for evaluating the harmfulness of responses.

It can be observed that different evaluation methods yield consistent conclusions: Integrating the utility score defined in Eq. 5 into deep alignment can significantly enhance the helpfulness of responses without compromising the model’s safety capabilities. This shows the importance of utility score in deep alignment. We have added the above explanation about the ablation study in our manuscript.

​Dear Reviewer,

Thank you again for your valuable feedback and questions. We have provided responses addressing your concerns and uploaded a new version of our manuscript.

As the discussion period draws to a close, we kindly encourage you to take a look at our responses and let us know if you have any remaining questions or concerns.

We sincerely appreciate your time and effort in helping us improve our work.

Sincerely,

Authors

Dear Reviewer,

Thank you once again for your insightful feedback and questions. We have provided detailed explanations and additional experimental results addressing your concerns and uploaded a new version of our manuscript. The contents included in the response are:

• (i) Explanation about the setting of helpfulness/usefulness in our experiments

• (ii) Experiments and explanation of the utility guide’s ablation study

• (iii) Experiments on additional baselines and datasets to support the motivation

• (iv) Experiments on more VLM baselines to demonstrate the effectiveness of ETA

As the discussion period is about to end, we kindly remind you to review our response to see if it addresses your concerns. If you still have any questions or concerns, please let us know.

We sincerely appreciate your time and effort in helping us improve our work.

Sincerely,

Authors

We sincerely thank the reviewer for their thoughtful and thorough comments on our paper. We deeply appreciate the recognition of ETA's intuitive design, plug-and-play versatility, and comprehensive experiments demonstrating strong performance. We answer your questions below.

• Weakness 1: The method itself, is more like a combination of pre-processing and post-processing techniques. The contribution of this method maybe limited.

While ETA involves pre-processing and post-processing, its contribution lies in how these elements are uniquely combined to address VLM safety. In fact, ensuring safety while preserving the original capabilities of VLMs is a very challenging problem, and we believe this is where ETA makes a significant contribution. We consider the simplicity of ETA an advantage rather than a limitation. We summarize the contributions of ETA as follows:

(i) New Perspective on Safety Issues in VLMs: We offer a new analysis on the failure of existing defense mechanisms in VLMs, and show that the primary factor contributing to safety problems in VLM systems is the continuous nature of visual token embeddings.

(ii) First to Introduce Additional Visual Safeguards: The usage of CLIP in our work differs significantly from its application in prior studies. Previous works have primarily leveraged the CLIP score during post-processing stages, such as inference and evaluation, to assess the similarity between output text and input images. In contrast, our paper innovatively utilizes the CLIP model during the pre-processing stage. We demonstrate that the CLIP score can be effectively generalized to evaluate the safety of continuous visual embeddings, which can easily bypass the existing safety mechanisms in VLMs. By utilizing predefined prompts with unsafe semantics, we show that it is generalizable to classify input images for safety directly, providing a novel application of CLIP in ensuring the safety of visual inputs.

(iii) An innovative Strategy to Improve Safety Evaluation Capability of Reward Models: We demonstrate that designing a safety-specific input format for general reward models can significantly enhance their ability to more effectively evaluate the safety of responses. This differs from RMs’ usage in prior studies.

(iv) Harmless and Helpful Responses, Rather Than Generic Refusals: We found that existing methods often provide generic refusals such as “I cannot help you, …”. However, an ideal response should be both harmless and helpful, offering helpful suggestions to users. By leveraging the multimodal evaluator (CLIP+RM) for sentence-level Best-of-N searching, we can effectively identify safer and more helpful generation trajectories. This is also a novel usage of CLIP and RM.

(v) Superior Performance of ETA as a Simple Plug-and-Play Framework: Extensive experiments demonstrate that our plug-and-play approach is applicable to common VLM baselines, significantly enhancing the model’s safety without affecting its general capabilities. Moreover, it improves both the safety and usefulness of the model’s responses in unsafe scenarios.

• Weakness 2: As a plug-and-play method, I expect to see it to be applied to more recent VLMs, such as LLaVA-NeXT, Llama3.2.

Considering that LLaVA-NeXT-8B (released in May 2024) and Llama3.2-11B-Vision-Instruct (released in Sep. 2024) are very recent models, they were not included as baselines in the paper. However, we did include another recent and strong VLM, InternLM-XComposer-2.5-7B (released in July 2024) in the paper. To further verify the applicability of our method, we have now tested the effectiveness of ETA on the suggested recent models LLaVA-NeXT-8B (released in May 2024), LLaVA-OneVision-7B-Chat (released in Sep. 2024), and Llama3.2-11B-Vision-Instruct (released in Sep. 2024).

The results of Llama3.2-11B-Vision-Instruct demonstrate that, despite the baseline being aligned through RLHF and equipped with a safety-aware module (Llama Guard), ETA still enhances the baseline's safety capabilities, particularly under more severe attack settings (e.g., suffix attacks). Moreover, a comparison of the results for LLaVA-OneVision-Chat-7B and Llama3.2-11B-Vision-Instruct on FigStep reveals that models with stronger instruction-following capabilities and RLHF alignment are more likely to produce safer responses during ETA's alignment phase. This highlights the promising potential of ETA for application in the future, more robust, and safer VLMs.

• Weakness 3: Also I would expect to see the model performance drop on more recent and widely-used benchmarks like MMMU.

Thank you for the constructive suggestion. In our experiments, we evaluate several recent comprehensive benchmarks, such as MME and widely-used VQA benchmarks, to ensure that our ETA does not negatively impact the baseline models’ general performance. To assess whether ETA weakens baseline performance on more recent and complex benchmarks, we test two different settings of MMMU-Pro (robust version of MMMU) (standard 4 options, and vision) using direct prompts. The results in the table below demonstrate that, even on more complex and recent benchmarks, ETA still does not affect the baseline models' overall helpfulness.

• Weakness 4: Some highly-relevant references about VLM/LLM safety are missing [1][2]

Thanks for the detailed comments. We have included these references in the related work section in the revision.

• Questions: Is there any compromised approach to also employ similar strategy to API-based VLM/LLMs like GPT-4(V)?

Thanks for the valuable question. We did not include experiments on API-based models, as the baselines we compared did not consider the use case of API-based models. Theoretically, the ETA framework is applicable to API-based models. However, considering the extremely high costs of API calls, we leave further investigation for future work. Here, we briefly explain how ETA can be applied to API-based VLMs/LLMs. First, the evaluation phase can be applied to API-based VLMs/LLMs, as this phase only requires inputting the image and the standard output from the API model to complete the process. Second, for the behavior classified as safe, we can directly output the response. For behaviors classified as unsafe, some API-based VLMs/LLMs, such as GPT-3.5-Turbo-Instruct, which are compatible with the legacy completions endpoint and not Chat Completions, can integrate ETA by appending a safety prefix to the end of the input. This setup enables the generation of sentence candidates, which can then be filtered using our deep alignment. We hypothesize that the complete process will be similar to the chunk-level beam search approach used in experiments on GPT-3.5-Turbo-Instruct as described in [6].

Reference

[1] Inan H, Upasani K, Chi J, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations[J]. arXiv preprint arXiv:2312.06674, 2023.

[2] Tu H, Cui C, Wang Z, et al. How many unicorns are in this image? a safety evaluation benchmark for vision llms[C]. ECCV, 2024.

[3] Hannah Brown, Leon Lin, Kenji Kawaguchi, and Michael Shieh. Self-evaluation as a defense against adversarial attacks on llms. arXiv preprint arXiv:2407.03234, 2024b.

[4] Yunhao Gou, Kai Chen, Zhili Liu, et al. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation. arXiv preprint arXiv:2403.09572, 2024.

[5] Renjie Pi, Tianyang Han, Yueqi Xie, et al. Mllm-protector: Ensuring mllm’s safety without hurting performance. arXiv preprint arXiv:2401.02906, 2024.

[6] Zhanhui Zhou, Zhixuan Liu, Jie Liu, et al, Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. NeurIPS 2024