Automatic Jailbreaking of Text-to-Image Generative AI Systems for Copyright Infringement

Thank you for your time and constructive feedback. We appreciate your comments and have made every effort to address them in the revision. If you have any further concerns, please feel free to reach out for discussion.

[W1. Editorial comments-1. Figure 1a] Figure 1a doesn’t seem necessary.

• We removed Figure 1a and updated Figure 1 and 2.

[W2. overclaim updates instruction] Figure 2 is hard to interpret; it’s unclear why weight updates or gradient updates aren’t needed, and how to update the instructions isn’t explained well. maybe it's an overclaim because you still need to update the instructions.

• We believe this may be a misunderstanding of our work. Our approach does not require any weight or gradient updates, as we adjust the instructions or descriptions solely based on language model inference using the given prompt as [1,2] (Lines 773-897).
• Specifically, we provide the language model with a previous output and its corresponding score, then ask the LLM to revise the output (instruction/description) to achieve a higher score. We refer to this process as an 'update.' The detailed pipeline can be found in Figure 14.

[1] Large Language Models as Optimizers
[2] Improving Text-to-Image Consistency via Automatic Prompt Optimization

[W3. Editorial comments-2. Formula] There are too many symbols in the equations that aren’t clearly defined.

• We reflect all your comments.
  • What does "m" refer to in line 210?
    • → It is an index of question and answer pairs (line 200).
  • In line 210, "v" is used to represent LLM, but then in line 231, "v" represents the encoder.
    • → Sorry for misuse of the term. We revise the vision encoder to “e_v”.
  • What’s the expression for ( S_k )?
    • → It is a keyword penalty in the score function (Line 190).
  • "f2" isn’t defined in line 301.
    • → It is a LLM model that refines the description (Line 186)
  • There are reference errors for Figure B.2 in lines 344 and 351, and the image formatting at the bottom of page seven is clearly off.
    • → We revised the reference error to Section B.2 Figure 17.
  • Additionally, there are reference errors for Figure 9 on line 407.
    • → We revise the reference error to Table 7.

[W4. Ablation study in Table 7] The ablation study in Table 7 is incomplete.

• We conducted additional ablation experiments in the following Table.
• However, as shown in the following table, the block rate trend is consistent with Table 7, where keyword penalties contribute the most to the block rate.
• Additionally, leaving the keyword penalty while removing the QA score allows prompts to bypass block censorship more easily; however, this results in the generation of general outputs that do not infringe on copyright, as illustrated in Figure 5.
• As each score component is independent, we believe Table 7 demonstrates enough ablation study to see which component plays important roles in the block rate.

[Q1. unlearning experiment] For the unlearning experiments, it seems that the attack is not directly targeting GPT, but rather the diffusion model with unlearning, right? This may not be very convincing; maybe the unlearning algorithm is just bad on this model.

• No, this is a misunderstanding of our experimental setting. We employ an open-sourced unlearning checkpoint of the diffusion model that is trained by authors and tested our APGP generated prompt on that diffusion model. This experiment demonstrates not only ineffectiveness of unlearning but also shown the transferability of our high-risk prompt to diffusion models.

This is a gentle reminder regarding your initial concerns.

We have addressed them through additional explanations and experiments, and we kindly invite you to review our response. If you have any remaining concerns or suggestions, we would be delighted to engage in a final discussion to ensure a fair review process.

• The manuscript has been revised and thoroughly proofread.
• All editorial comments have been addressed, and the figures have been revised for better clarity and understanding.
• An additional score function ablation experiment is provided.

We greatly appreciate your time and effort and look forward to hearing from you.

[Q1. Single scalar in score function] Why is a single scalar score given to LLM. Why is scoring not given separately and the LLM is asked to improve each of the scores.

• We selected the single scalar score given to the LLM, following the approach of previous work [1,2] in the instruction-following task.
• However, your comments were very insightful. As a result, we conducted an additional experiment using an independent score set rather than a single scalar. This approach is also optimized well for finding high-risk prompts, as shown in the table below. However, we think we need additional reasoning ability to fully leverage each independent score terms.

[1] Large Language Models as Optimizers
[2] Improving Text-to-Image Consistency via Automatic Prompt Optimization

[Q2. Why isn't a single VLM not enough?] Why is not a single VLM enough for our approach? What I mean is, what if we provide the score and the system prompt (to make an VLM act like an optimizer) and ask the VLM itself to generate descriptions such that improves the score ? Why is reason for not trying that and introducing a LLM separately ?

• Thanks for your insightful comments. When we use VLM, it could make a more powerful jailbreaking approach. However, we might need a slightly different score function and prompt template for the VLM since we give an image as a condition.
• It is possible to replace the LLM with a VLM in step 2, but we believe strong language performance is more relevant to update the prompts adequately based on the given score. Furthermore, using a VLM is computationally more expensive, as it requires processing images as an additional input.

[Q3. Why the gradient based methods to optimizing prompts for jailbreaking LLM have not been tried for T2I models?] Why the gradient based methods to optimizing prompts for jailbreaking LLM have not been tried for T2I models? (It could have helped in developing a better agentic framework)

• First, since we are targeting black box T2I systems, ChatGPT or Copilot, not single T2I models like stable diffusion or Dalle, which do not provide a whole system as a single API which makes it difficult to collect their feedbacks during the optimizing.
• Furthermore, we found gradient-based methods to be computationally inefficient compared to our approach, which needs training and weight updates of all models.
• Additionally, many gradient-based jailbreaks [1,2] tend to produce unnatural prompts that can be easily filtered inside the system.

[1] Universal and transferable adversarial attacks on aligned language models
[2] Autodan: Automatic and interpretable adversarial attacks on large language models.

Thank you for your time and constructive feedback. We appreciate your insightful comments and have done our best to address the initial concerns. If you have any further questions or suggestions, please feel free to reach out for discussion.

[W1. Formatting] The figures and text should be arranged properly.

• We apologize for the inconvenience. We've conducted another round of proofreading to improve the writing quality across the entire manuscript. As ICLR allows updating the pdf, we've updated the final PDF for better clarity. Please review the updated version.

[W2. [1] difference] The idea is similar to treating VLM and LLM as two agents helping to jailbreak the T2I diffusion model. How is the approach different from [1].

• While the high-level concept of using LLMs or VLMs as agents is similar, the jailbreaking task, target models and method differ significantly.
• [1] focuses on generating harmful images in T2I models with a single-layer safety filter, such as Stable Diffusion or DALLE. In contrast, our approach targets generating copyrighted content in T2I services with multi-layered filtering systems, like ChatGPT or Copilot.
• Additionally, the methods differ: [1] uses chain-of-thought reasoning and in-context learning to iteratively learn from previous attempts on the target model, whereas our approach relies on a single attempt guided by predefined score functions.
• We cited this concurrent work in the related work section.

[W3. Lack of dataset] However I feel the number of images is small to text the validity of the approach.

• We conducted an additional 70 images, bringing the total to 140 images. As shown in the following table, the block rate remains significantly low even with an enlarged dataset, demonstrating consistent risks in T2I systems.
• We believe that 140 images are sufficient to highlight the risks of current T2I systems. However, if you still have concerns about the dataset size and its ability to demonstrate our effectiveness, we are willing to collect additional data. Please let us know the sufficient size.
• Additional data can be found in the supplementary folder. The images are listed at the bottom. ||Product|Logo|Character|Art|Average| |-|-|-|-|-|-| |70 images|2.67$\pm$2.89|25.00$\pm$8.66|38.34$\pm$25.16|33.34$\pm$12.58|24.84$\pm$12.32| |140 images|5.71$\pm$2.86|15.24$\pm$5.95|26.67$\pm$22.19|24.00$\pm$12.00|17.90$\pm$10.75|

[W4. scoring function ablation] Lack of scoring function ablation details to understand each of its contributions. Why is there a linear addition? Is there no normalization of the values?

• It is linearly added because we believe each component is independently important to the final score. There is no normalization of the values.
• In the scoring function ablation, we simply remove each score term in Table 7.
• To demonstrate the contribution of each score function, we conducted a score ablation in Table 7. As seen in Table 7, the keyword penalty (S_k) plays the most significant role in bypassing the safeguard, while the QA score term contributes to generating precise images, as illustrated in Figure 5.

Thank you for your time and constructive feedback. We greatly appreciate your insightful comments and questions. We have done our best to address them, and if you have any further concerns, please feel free to reach out for further discussion.

[W1. Relation between Glaze and Ours] Can you talk about the relationship between your work and the Glaze tool [1]? The Glaze tool also aims to protect the copyrighted and private images created.

• The Glaze tool aims to protect copyrighted content during the training data collection phase (Line 135), preventing its features by adding adversarial noises into the content from being learned by T2I models.
• However, current T2I systems did not use Glaze when training the model. Instead, they employed alignment learning to refuse responses to copyright infringement requests.
• Our work demonstrates that current alignment does not fully protect against copyright infringement.

[W2. Editorial comment-Figure] Suggest to recreate Fig. 1 and consider combining Fig. 2 with Fig. 1.

• We revised the figure 1 and 2 in the pdf.

[W3. specific tool] Why is this tool copyright specific? I feel like the prompt generation pipeline is agnostic to the type of the attack?

• Our approach is not limited to a specific type of attack but can be applied to various scenarios that involve safeguarding specific content (like copyrighted content, or celebrities’ image).
• The APGP technique leverages the characteristics of T2I models to revoke specific images without relying on directly linked keywords. This allows it to bypass safeguards, which are typically aligned based on the keyword, thereby revealing the original content.
• As the reviewer pointed out, this tool can also be adapted to other types of attacks, such as those involving publicity rights leakage, as shown in Figure 11.
• As a result, while the prompt generation pipeline itself is agnostic to the type of attack, copyright protection serves as a practical demonstration of the tool's effectiveness.

[W4. Potential defense/solution] What is a potential solution for/defense against such type of prompt attack?

• Glaze could indeed serve as an effective initial protection for copyrighted content.
• Additionally, implementing extra layers of filtering systems or utilizing our high-risk APGP prompts for alignment learning could further enhance safeguard measures.
• If you find this potential defense and solution reasonable, we can include these points in the Ethical Statement section as well.

Thank you for your time and constructive feedback. We appreciate your acknowledgment of the originality, clarity, experimental quality, and significance of our research. If you have any further concerns or suggestions, please feel free to reach out, and we would be happy to discuss them further.

[MLLM detection] The authors counter argue the idea of using a copyright detection model, suggesting that since there's no open sourced one then it's not practical. Would it be possible to just use MLLMs themselves as filters of copyrighted contents? An LLM could probably guess a good list of entities that are copyrighted (or find them in a catalogue) given the prompt and then an MLLM can simply verify the presence or absence of the copyrighted content in the image.

• Thank you for the suggestion. While using MLLMs to filter copyrighted content is an interesting idea, copyright detection remains complex. MLLMs might identify some copyrighted elements, but they can still miss nuanced cases or produce false positives. Therefore, applying MLLMs to copyright detection requires additional prompting or reasoning approaches to create a reliable model.
• The lack of reliable, open-source copyright detection models underscores the limitations of simply applying previous harmful image T2I jailbreaking approaches to our copyright jailbreaking problem.

Minor observation, The figure after Figure 4 is obviously misplaced.

• We revised the manuscripts. Thanks for the comment.

Thank you for your time and constructive feedback. We appreciate your acknowledgment of our interesting initial observations, well-motivated approach, clear presentation and strong experimental evaluation. If you have any further concerns, let us know to discuss.

[W1. Latency of APGP] The authors should provide a comparison of the latency of their approach against other jailbreak methods

• The average latency for each image using APGP is 3.71 minutes. In comparison, the average latency for the most popular language jailbreaking using PAIR [1] and GCG [2] is 0.5 minutes and 108 minutes per prompt, respectively. This demonstrates that our approach is reasonable for text-to-image models.
• Additionally, please note that there are no other known jailbreak methods for copyright contents.
• For specific latency, step 1 takes 18.29 seconds and step 2 takes 33.49 seconds, with 2.89 seconds for updating prompts and 6.12 seconds for scoring all previous top five prompts. Overall, three iterations of step 1 and five of step 2 result in a total runtime of 3.71 minutes. Using fast inference methods can significantly reduce this latency even more.

[1] Jailbreaking Black Box Large Language Models in Twenty Queries
[2] Universal and transferable adversarial attacks on aligned language models

[W2. Small dataset] Based on Figure 13 (Appendix A.1), the VioT dataset has only 70 images. I think the evaluation of the proposed framework on only 70 images is not very convincing to demonstrate its effectiveness.

• We conducted an additional 70 images, bringing the total to 140 images. As shown in the following table, the block rate remains significantly low even with an enlarged dataset, demonstrating consistent risks in T2I systems.
• We believe that 140 images are sufficient to highlight the risks of current T2I systems. However, if you still have concerns about the dataset size and its ability to demonstrate our effectiveness, we are willing to collect additional data. Please let us know the sufficient size.
• Additional data can be found in the supplementary folder. The images are listed at the bottom.

[W3. Other LLM] [Minor] Although the paper has ablations on each component, it would further strengthen the draft if the authors could include an ablation on the LLM used for optimizing the prompt, which has been currently set as GPT-3.5-Turbo.

• We additionally employ GPT-4o-mini and the prompt risks even increased as shown in the Table.
• Since our approach is a black-box attack that does not rely on any weights or feedback from the target model, it remains applicable as long as the LLM demonstrates sufficient language performance.

Thank you for your valuable feedback and constructive comments. We have carefully addressed all your concerns through additional proofreading and supplementary experiments. Please let us know if you have any further questions or suggestions; we are happy to discuss them.

[W1. Q3. multiple iterations / small dataset] Could the authors report block rates and evaluation scores based on averages across multiple iterations?

• We conducted three trials and averaged the block rates shown in the table below. The results indicate consistent vulnerabilities in the T2I systems across multiple runs.
• Additionally, we experimented with 140 images to verify our approach, as shown in the following table. Additional data can be found in the supplementary folder. The images are listed at the bottom.

• Despite the variance, our approach still has a significant success rate in bypassing safeguards, and T2I systems continue to exhibit copyright infringement using our prompts. We believe these results validate the effectiveness of our work and highlight the importance of addressing this real-world problem.

[W2. Formatting, style] The paper has a high number of formatting, stylistic, and wrong element reference issues.

• We apologize for the inconvenience. We've conducted another round of proofreading to improve the writing quality across the entire manuscript, incorporating your comments. As ICLR allows updating the pdf, we've updated the final PDF for better clarity. Please review the updated version.

[W3. Human study] The human study provides more insight around participant perception of copyright infringement, than actual determination of violation determination.

• We didn’t train participants to make legal judgments about copyright infringement because it's too complex for non-experts. Instead, participants identified identical or similar images, as shown in Figure 15. While actual violation determinations may differ, our goal is to highlight potential copyright infringement risks that can be easily identified by non-professionals (Lines 491-494).

[W4. Release date or version for T2I models] Commercial T2I models are updated continuously. Please include the release date or version of each T2I model.

• Original experiment was tested on a ChatGPT (ChatGPT-4 version) (Line 754). We employ gpt-4-0125-preview, gpt-3.5-turbo-0125, and dall-e-3 version for generating the prompt. We add this in the revised manuscript.
• All rebuttal experiments were tested with a version of ChatGPT (ChatGPT-4o version).

[W5. Dataset release] One of the paper's contributions is a dataset, though the dataset was not included as an anonymized link for review.

• Thanks to your comment, we additionally provide the dataset in the supplementary file.

[Q1: "Identical violations" was mentioned twice in the paper but not defined.] In lines 359-360, authors state that "Upon examining the images classified as identical violations, it was found that over 50% were deemed to be cases of copyright infringement in product and logo." If identical violations refer to generating nearly identical images, the number of over 50% being deemed as infringement seems low.

• We believe that a 50% rate of identical violations is alarmingly high from our perspective.
• This issue's severity depends on the perceived importance of copyright infringement. Since copyright is legally protected, we believe T2I services—especially commercial ones—should ensure 100% compliance. However, current systems violate copyright in over 80% of cases, with 50% involving severe infringements. We find this deeply problematic.

[Q2: Lines 371-372: "In Figure 4, 42.19% of the generated images correctly answer more than seven questions."] What are the seven questions?

• 10 questions are generated with VLM based on given target images (Line 968). The LLM answers more than seven questions correctly for 42.19% of the generated images.
• Seven questions vary depending on the target image.

W1. Thank you for experimenting with more iterations and more images. For these two rows, do you have baselines for simple prompt? The current baseline for 70 images for copilot is 5% block rate (Table 1 and 3). Here, the new results for block rate after attack, for 140 images for copilot has the average of 11.38%. Is the copilot baseline for 140 images much higher than 11.38%?

• Yes, Copilot has updated its copyright protection, as shown in the following table. We have also included additional baselines using simple prompts for both Copilot and ChatGPT. With these updates, the block rates for simple prompts have improved compared to the original baselines. However, both systems remain vulnerable to our APGP prompts. The results provided represent the averaged block rates over three trials for consistency.

Q1. Thank you for the feedback. We will clarify the sentence as follows:

• Over 50% of the images classified as identical violations were deemed cases of nearly identical copyright infringement, while the remainder were categorized as similar infringements, particularly in product and logo categories.

Q2. I see that in table 4, around 10% target images (ground truth copyrighted images) has 4 questions matched out of 10 via automatic QA evaluation. 4/10 seems low for the target images, and there's a range for the numbers of QA question match. Is there any analysis on how correlated human evaluation and QA evaluation are? Or perhaps comparison of automatic QA score for images generated by your method, and images generated by simple prompt/ablated prompts?

• The range in QA matches arises from variations in question and answer quality, influenced by the capabilities of the LLM models. The correlation between human evaluation and automatic QA evaluation is 0.422, indicating a positive relationship. This supports the use of automatic evaluation as a cost-efficient and valid approach for preliminarily identifying copyright infringement (Line 364).
• Additionally, the average QA score for our method is 5.63, compared to 4.81 for methods without the $S_{qa}$ score function, further demonstrating that QA evaluation effectively reflects differences in copyright infringement quality.

If you have any further questions, please feel free to reach out. If our responses have addressed all your concerns, we kindly hope you will consider reflecting this in the final scores.