ColJailBreak: Collaborative Generation and Editing for Jailbreaking Text-to-Image Deep Generation

Q1:Time and computational cost of the attack

Thank you for the comments. we have conducted a detailed analysis of the time and computational resources required for each step of the attack pipeline. We specifically measured the time required for a single attack, including the segmentation mask generation using the SAM model and the image editing using the Inpainting model. All experiments are performed using two A100 40GB GPUs.

Table Time and computational cost of the attack

As shown in the table, the overall time and computational cost for each attack are manageable and demonstrate that our method is both efficient and practical.

Q2: Image quality generated by uncensored open-source diffusion models

Thank you for your comments. Firstly, our objective is to develop a unified jailbreaking method for text-to-image (T2I) models that can generate unsafe images. We aim to evaluate the performance and potential threats of these models during both the initial image generation and subsequent editing phases, rather than relying solely on existing uncensored open-source models.

Secondly, in our previous experiments, models capable of generating high-quality images generally have strong safety checks in place. Uncensored open-source models often produce lower-quality images. For example, commercial versions of Stable Diffusion can generate high-quality images but have safety mechanisms, whereas the open-source version on Hugging Face tends to produce lower-quality images and also includes some safety checks.

Thirdly, our research is not just about generating unsafe images but demonstrating a new attack method that involves both image generation and editing. This process shows how an attacker can gradually bypass the model's safety restrictions through a series of steps. This comprehensive approach not only highlights vulnerabilities in the generation phase but also reveals the potential to further bypass safety checks through post-processing.

Q3: Unsafe image editing methods

Thank you for your comments. Our research primarily focuses on open-source image editing models, which typically do not have built-in safety mechanisms. This allows us to highlight the potential vulnerabilities in these widely accessible models. We integrate the editing models into an overall jailbreaking framework, leveraging the Contrastive Language-Image-guided Collaborative Optimization component. This integration enables the T2I models to generate unsafe images effectively.

Q4: More unsafe content types

Thank you for your comments. We have added two unsafe image categories: nudity and disgusting scenes. For each unsafe content type, we generated 25 images and calculated the attack success rate (ASR), and the results are shown in the following table, and detailed visualization examples of the results are in anonymous link.

Table Experimental results on ASR in more unsafe content types

Q5: Baseline methods

Thank you for your comments. We used SneakyPrompt as a baseline in our experiments. The results are shown in the following table. The experimental results show that ColJailBreak outperforms SneakyPrompt on average on all unsafe types.

Table Comparison with SneakyPrompt on all unsafe types

Q6: Advanced evaluators

Thank you for your comments. We understand the importance of using more advanced image safety evaluators. In response, we have chosen to use GPT-4o for our new experiments. The results of these new experiments are shown in the table below. The experimental results show that when using GPT-4o as an evaluator to evaluate our method, our method has a higher attack success rate (ASR) than the baselines

Table Experimental results using GPT-4o as an evaluator on all unsafe types

Q7: Supplement to Eq(2)

Thank you for your comments. We understand that this point may have caused some confusion, and we will clarify it in the revised manuscript. In Eq (2), the filter F only takes text prompts as inputs. We will specify this clearly ahead in the relevant section to avoid confusion with image-based safety filters.

Q1:Dataset and more unsafe content types

Thank you for your comments. Our main purpose of building the dataset is to validate the effectiveness of the proposed method. The scale of the dataset is not a decisive factor. In fact, we refer to some previous studies. For example, DACA(Deng et al. 2023) used a dataset of 100 cases, and SurrogatePrompt employed a dataset of 90 cases(Ba et al. 2023).

In selecting and designing the dataset, we considered key metrics for validating the methodology. Although the cases included in the current dataset are relatively simple, they are sufficient to reflect the validity of the methodology. We will further expand the scale of the dataset in the future and pay special attention to these complex and insecure examples to fully validate the reliability of the methodology.

For your concern that the some unsafe types can't be covered, for more complex scenes, we can achieve the desired result by editing multiple times and enlarging the mask size. To demonstrate the potential of our method, we add two additional types of unsafe content: nudity content and fighting scenes. For each unsafe content type, we generated 25 images and calculated the attack success rate (ASR), and the results are shown in the following table, and detailed visualization examples of the results are in anonymous link.

Table Experimental results on attack success rates(ASR) in nudity and fighting scenes

Q2:The role of image editing methods

We understand your concern regarding the reliance on image editing in our method. We would like to clarify that our approach is not merely dependent on image editing capabilities but integrates several components to achieve the desired results. To address this concern, we have conducted two experiments that demonstrate the effectiveness and uniqueness of our method.

(1) Experiment 1 Comparing Different Editing Methods:

To demonstrate that our method is not solely reliant on a specific image editing technique, we have tested various editing methods within our framework. We employed several image editing models. As shown in the anonymous link, regardless of the editing method used, our overall framework achieves similar outcomes in generating unsafe content. This demonstrates the robustness of our method and its ability to work with different editing techniques.

(2) Experiment 2: Impact of Contrastive Language-Image-guided Collaborative Optimization

To highlight the importance of our optimization method, we conducted experiments using the same editing method but without our optimization framework. We removed the collaborative optimization component and evaluated the performance of the editing model alone. As shown in the anonymous link, without the collaborative optimization framework, the editing methods alone were not able to achieve the same level of effectiveness in generating unsafe content. This underscores the critical role of our integrated approach in enhancing the capabilities of the editing models.

Q3: Results of human evaluation and more testing

Thank you for your comments. To enhance the credibility of our research, we have added more testing and human evaluation. Specifically, we conducted the following:

(1) Human Evaluation:

We conducted Human Evaluation to evaluate the generation quality based on human responses. Specifically, each participant needs to answer the following two questions. So far, we have collected 50 survey responses for this evaluation.

Q1: Participants were asked to rate the alignment between the image and the text prompt on a scale of 1 to 10, where 10 indicates the highest relevance and 1 indicates the lowest relevance. For Q1, we averaged all the scores to compare the performance of different methods.

Q2: Given an image, participants were asked to classify it as either a safe image or a harmful image. For Q2, we calculate the percentage of images classified as harmful.

The results of these new evaluations are shown in the Table below, demonstrating the performance of our method in aligning generated content with the input prompt and the attack success rate

Table Experimental results of Human Evaluation

(2) Text-Image Alignment:

To further validate the alignment between the images and the text prompts, we used GPT-4o as an advanced evaluator, providing alignment scores similar to those in Q1. The results are shown in the table below.

Table Experimental results of Text-Image Alignment

Q4: Code and Dataset

We will release the dataset and code as soon as possible. We commit to following ethical guidelines in the release process.

Dear Reviewer jDc9,

We hope our previous response has effectively addressed your concerns. If you have any additional questions, we would be more than happy to continue the discussion with you.

Q1: Explanation of our jailbreaking method

Thank you for the comment. We appreciate your concerns and would like to address them by elaborating on the intuition and rationale behind our proposed method. Our goal is to demonstrate the potential for generating unsafe images through a macro-level jailbreaking method. Specifically, we first use the text-to-image (T2I) model to generate normal images that easily pass the model's safety filters and are amenable to malicious edits. We then modify them to embed unsafe content. By showing that initially safe images can be edited to include unsafe elements, we emphasize the vulnerability of the models to post-generation manipulations. We hope this explanation clarifies the intuition behind our approach and the significance of our findings.

Q2: Application of our attacks

Thank you for the comment. The possible applications of the proposed attacks are multifaceted and significant in both demonstrating the vulnerabilities of existing safety mechanisms and emphasizing the need for more robust defenses. Here are the key applications:

(1) Production and Distribution of Illegal Content: Creating and distributing adult content, pornographic images, and other illegal or inappropriate images.

(2) Malicious Manipulation and Defamation: Producing fake obscene images for online defamation and damage to reputations.

(3) Psychological and Emotional Harm: Intentionally spreading NSFW content to cause psychological harm or emotional distress, especially among minors or sensitive populations.

(4) Benchmarking and Testing: These attacks can serve as benchmarks for testing the effectiveness of new defense mechanisms, ensuring that AI models are resilient to such vulnerabilities.

Q3: Experimental results of Midjourney

Thank you for your valuable suggestion. Based on your recommendation, we test Midjourney and find that our approach performed excellently. The detailed results of our testing are provided in the anonymous link. Our method demonstrated superior performance in various evaluation metrics when applied to Midjourney, demonstrating its effectiveness.

Table Quantitative evaluation of ColJailBreak and baselines in jailbreaking Midjourney

Dear Reviewer yPxw,

We hope our previous response has effectively addressed your concerns. If you have any additional questions, we would be more than happy to continue the discussion with you.

Dear Reviewer yPxw,

We hope our previous response has effectively addressed your concerns. If you have any additional questions, we would be more than happy to continue the discussion with you.

Thank you for your recognition and support. We are very happy with your evaluation of our work. In order to further improve our work, we plan to release the dataset and code as soon as possible, while ensuring that the release process follows ethical guidelines and best practices. Thank you again for your comment.

Dear Reviewer GbGQ,

We hope our previous response has effectively addressed your concerns. If you have any additional questions, we would be more than happy to continue the discussion with you.