Evaluating the Robustness of Text-to-image Diffusion Models against Real-world Attacks

Thank you for the supportive comments. Below we address the detailed comments and hope that you may find our response satisfactory.

Question 1: The method limits the “imperceptability” of the attacks.

Thank you for kindly pointing it out. We make the following clarification in two perspectives. On one hand, for users, the mistakes made by themselves are imperceptible for the users because they did not notice these mistakes. Some kinds of perturbations (e.g., glyph and phonetic) are more imperceptible but can lead to significantly different images, as shown in Figures 1, 6, and 7. On the other hand, since the diffusion models typically rely on context to generate images, minor perturbations usually do not impact the model's understanding of the semantic meaning of the sentence. As demonstrated in Figure 9 in the Appendix, the diffusion model can still generate realistic and aligned images despite perturbing many words in the text prompt. Therefore, for smaller perturbations like those in our case, diffusion models are expected to produce images that resemble the original ones. However, our perturbed prompts successfully induced diffusion models to generate mismatched images.

Besides, we generally adopt the perturbation space introduced in the field of NLP attacks. These perturbations are deemed more imperceptible for text than other kinds of perturbations (word-level, sentence-level). Nevertheless, as our proposed method is generally applicable to any perturbation space, we could further extend it to generate more imperceptible perturbations if a more appropriate perturbation space is developed.

Question 2: Modifying not the most significant words and still being able to perturb the image would be another interesting track.

Thank you for your thoughtful suggestion. We have indeed taken this into consideration. The important words identified by our method may not universally align with what is considered significant to humans. Sometimes our method also selects words that are not significant to human understanding. We also conduct the human attack experiment in Appendix D, manually selecting words considered to be of high importance by the human. However, the actual outcomes were not entirely satisfactory. We will consider your suggestion and explore methods for perturbing sentences while adhering to constraints on words deemed important by human evaluation in future work.

Question 3: Are there any scenarios where a malicious attacker can really cause any harm with these attacks. An explicit analysis of this issue would provide additional context for the results and broader impact of this work.

Thank you for your kind advice. As described in the introduction, our proposed attack aims to evaluate the robustness of text-to-image diffusion models under real-world perturbations rather than generating harmful images. Therefore, the attacker is more likely to be the model developer rather than a malicious attacker. Evaluating the robustness of diffusion models in our setting is also crucial for real-world deployment, which can not only improve user efficiency but also cut down on user costs by eliminating the need to revisit and rectify mistakes in prompts, especially when dealing with bulk operations. We will further extend our method to generate harmful content in future work.

Thank you for your follow-up comment. Below we address the detailed concerns. We hope that you may find our response satisfactory.

Question 1: T2I models being not robust does not seem like a critical issue.

Thanks for the comments and we make the following clarifications. As described in the introduction, our paper aims to evaluate the robustness of text-to-image diffusion models in the face of real-world perturbations. Such perturbations may arise from human errors and could lead to the generation of misaligned images. Consequently, the effectiveness of the model for end-users may be considerably compromised. The robustness of T2I diffusion models is a crucial aspect for real-world deployment to ensure reliable performance across variations in inputs [1]. Although this is not a security issue, it remains a critical factor influencing the practical utility of a diffusion model.

Question 2: How evaluating robustness can help improve user efficiency.

We are sorry for the potential confusion and make the following clarification. As discussed above, evaluating robustness can be used for model developers to evaluate the robustness of their T2I models before being deployed by simulating real-world input errors from humans to ensure utility for users. If the robustness has been improved during deployment phase, the final T2I models for users could not only improve user experience and enhance efficiency of use, but also reduce user costs by avoiding the need to go back and check mistakes in the prompts and make corrections after generating erroneous images.

Further discussion is welcome if you have any other questions. Thank you again.

Reference: [1] Lee et al., Holistic Evaluation of Text-to-Image Models, NeurIPS 2023 Track on Datasets and Benchmarks.

Question 6: The paper can be improved by developing methods for generating images based on the adversary’s chosen semantics.

Thank you for your kind suggestion. Since our attacks simulate human mistakes in real-world scenarios, they are more effective for non-targeted attacks. In the introduction, we emphasized the distinction between our approach and malicious attacks, highlighting our focus on a more realistic and objective form of attack, devoid of subjective chosen targeted semantics. We will further study the application of our method to targeted attacks in future work.

Question 7: Improve the paper by including a discussion of the ability to undo the adversarial modifications with different tokenization, with hand-written rules, or with Large Language Models that undo typos and glyph and phonetic replacements.

Thank you for your thoughtful suggestions. Given that the targeted model is a black-box model, modifying tokenization is not within our control. Regarding the typo correction, as outlined in Appendix C.2 of our paper, we employed two typo correction tools with different rules. We found that some of our texts can be maintained as adversarial samples.

Additionally, large language models are indeed effective in mitigating character-level attacks. We employed ChatGPT to repair samples generated by three attack methods: typo, glyph, and phonetic. With 50 samples for each method, the success rate of repairs exceeded 90%. We add this discussion in Appendix C.3.

Thank you for the valuable and comprehensive review. Below we address the detailed comments and hope that you may find our response satisfactory.

Question 1: The example prompt and image pairs that are selected can reasonably be challenged on the grounds that many modifications, although minor in Levenstein distance, actually alter the semantics.

We are sorry for the potential confusion and make the following clarification in two perspectives. On the one hand, for users, the mistakes made by themselves won't affect the semantics as perceived by the users because they did not notice these mistakes. On the other hand, since the diffusion models typically rely on context to generate images, minor perturbations usually do not impact the model's understanding of the semantic meaning of the sentence. As demonstrated in Figure 9 in the Appendix, the diffusion model can still generate realistic and aligned images despite perturbing many words in the text prompt. Therefore, for smaller perturbations like those in our case, diffusion models are expected to produce images that resemble the original ones. However, our perturbed prompts successfully induced diffusion models to generate mismatched images. Besides, other kinds of perturbations (glyph and phonetic) are more imperceptible but can lead to significantly different images, as shown in Figures 1, 6, and 7.

Question 2: Can you define exactly how SI2T and ST2I and their adversarial counterparts are computed?

As described in the Appendix B.1, \text{S_{I2T}}= \frac{1}{N} \sum_{i=1}^N \max(0, 100\cdot g_{\phi}(c)^\top h_{\phi}(x)) calculates the cosine similarity between the embeddings of images and text, no matter whether the images are generated by the original text or the adversarial text. \text{S_{T2T}}= \max(0, 100\cdot g_{\phi}(c)^\top g_{\phi}(c')) calculates the cosine similarity between the embeddings of original text and the adversarial text.

Question 3: Can you define the human-provided consistency scores described in Appendix D (N1 and N2)?

The Description Consistency Rate (DCR) in Appendix D differs from $N_1$ and $N_2$ in Appendix B.2. The former evaluates the consistency between sentences, while the latter evaluates the consistency between sentence and image. As described in Appendix D.2, we sampled 100 texts from successfully attacked texts for each attack method and evaluated the description consistency between these adversarial texts and their corresponding original texts by humans. To avoid bias, we evaluated each text with three people and took the plural of the three people’s opinions as the final decision. This is how we calculate the Description Consistency Rate (DCR). As described in Appendix B.2, $N_1$ represents the consistency between the original text and the original image, and $N_2$ represents the consistency between the original text and the adversarial image, which only needs human annotators to evaluate directly.

Question 4: Why is a difference greater than 1 (as opposed to 0) the “pivotal” difference determining attack success rate?

Thank you for kindly pointing it out. Due to the inherent randomness in image generation, we believe that solely relying on $N_1$ not equal to $N_2$ (i.e., $N_1 - N_2 > 0$) is not a sufficiently stringent condition to define a successful attack. Therefore, we impose a more stringent criterion, requiring ($N_1 - N_2$) > 1 as a condition for a successful attack. Similarly, we consider $N_1 - N_2 < 1$ as the condition for filtering out meaningless prompts.

Question 5: Is there a way to normalize for the strength of the attack?

Thanks for the comment. Perturbation rate can be used for measuring the strength of an attack. Figure 3 illustrates the generation performance of diffusion models under various attack strengths. The horizontal axis represents the perturbation rate, serving as a measure of attack strength, where a higher perturbation rate indicates a stronger attack. The vertical axis depicts the consistency between the images generated by diffusion models and the corresponding text at different attack strengths, representing the overall generation effectiveness of diffusion models.

Question 4: Does this "mismatch" only happen in the noun words?

This ”mismatch“ typically occurs in nouns, adjectives, and verbs. Since the cases we provide are descriptive sentences about objects, there are more instances of mismatches in nouns. It is important to note that our approach aims to identify words that are most important for the model, rather than those deemed most important by humans. Therefore, in theory, it is not limited to a specific part of speech. We add some cases without noun modification in Figure 10 in Appendix D.4.

Question 5: The details of the human evaluation are not clear. Can you provide the human evaluation on the original task as the baseline?

We agree that the mismatch also exists for clean inputs. We deal with this problem by measuring the percentage of samples that leads to mismatch while the clean inputs do not. As described in Appendix B.2, $N_1$ represents the number of images generated by the original text that are consistent with the original text, and $N_2$ represents the number of images generated by the adversarial text that are consistent with original text. If ($N_2 - N_1$) > 1, the attack on that particular prompt text is deemed meaningless. Let's assume the frequency of samples where ($N_2 - N_1$) > 1 as $N_u$, and the effective total number of samples should be $N_{\text{total}} - N_u$. If $(N_1 - N_2 > 1)$, it indicates a successful attack. We use $N_c$ to represent the number of samples where the attack is successful. Thus, the final score for each evaluator is given by $N_c / (N_{\text{total}} - N_u)$. The average of three human annotators represents the overall human evaluation score.

Question 6: What is the possible application of these real-world attacks?

As we discussed in Question 1, the goal of this paper is to evaluate the robustness of text-to-image diffusion models under real-world perturbations. To evaluate model robustness, we adopt adversarial attacks as an indispensable method. Therefore, the possible application of our proposed real-world attack approach is for model developers to evaluate the robustness of their T2I models before being deployed by simulating real-world input errors from humans to ensure utility for users.

Reference: [1] Lee et al., Holistic Evaluation of Text-to-Image Models, NeurIPS 2023 Track on Datasets and Benchmarks.

Thank you for the valuable and comprehensive review. Below we address the detailed comments. We hope that you may find our response satisfactory and could kindly raise your score.

Question 1: It is hard to understand the purpose of these "adversarial attacks" in this scenario.

We are sorry for the potential confusion and make the following clarification. As described in the introduction, the goal of this paper is to evaluate the robustness of text-to-image diffusion models under real-world perturbations. Such perturbations can stem from human mistakes and have the potential to induce misaligned images. Thus, the model's efficacy for users can be significantly diminished. The robustness of T2I diffusion models is a crucial aspect for real-world deployment to ensure reliable performance across variations in inputs [1].

To evaluate model robustness, we adopt adversarial attacks as an indispensable method. Our attack method can be used by the model developers to evaluate the robustness of their T2I models before being deployed to ensure utility for users. In other words, the attacker is more likely to be the model developer rather than a malicious user. Robust T2I models not only enhance user efficiency but also reduce user costs by avoiding the need to go back and check mistakes in the prompts and make corrections after generating erroneous images. Overall, our work can be significant for studying the vulnerabilities of T2I models, as agreed by Reviewer BYdd.

Question 2: Whether the robustness of the diffusion model to the text input is good or bad is another important question.

Thanks for the comments and we make the following clarifications. We believe that the robustness of diffusion models to input variations is good and crucial for real-world deployment. As illustrated in Figure 9 in the Appendix, the diffusion model demonstrates an ability to generate realistic and aligned images even when most words in the image are randomly perturbed. This is a desirable ability of diffusion models because the generations can better conform to the user’s real intention instead of the potential bias in the provided prompts [1]. This work evaluates such an ability of diffusion models by adversarially perturbing the prompt and checking to what extent the mismatched generations occur.

Question 3: Is the human evaluation based on the difference between the image content and the original meaning of the text or the actual meaning of the text?

As we discussed above, generating an image that aligns with the original text should be deemed as 'good.' Therefore, human evaluation is based on evaluating the disparity between the image content and the original meaning of the text.

To prove that human evaluation won’t be affected by the adversarial noise of text. We also added an experiment to evaluate the difference between the image content and the actual meaning of the adversarial text in revision. We let $N_1$ represents the number of images generated by the original text that are consistent with the original text, while $N_2$ represents the number of images generated by the adversarial text that are consistent with the adversarial text. The new overall human evaluation score is calculated in the same way as the original paper (also see Question 5 for details). Table 8 in Appendix D.3 shows the new overall human evaluation on the same adversarial dataset generated under real-world attack with the 2ST attack objective in Section 5. We found that although the new overall human evaluation score will decrease to some extent compared to the old score, the change is not significant, indicating that human evaluations are not affected by the adversarial text. We also show this table below：

Thank you for your response. I believe there might be some misunderstanding here. The attribute you mentioned, which makes the model more responsive to inputs, is commonly referred to as model sensitivity[1]. It is a concept opposite to robustness. Our focus here is primarily on exploring model robustness, which is typically defined as the ability to maintain the original output after applying imperceptible perturbations[2,3], as opposed to sensitivity, which involves generating different outputs in response to perturbations.

Balancing the exploration of robustness and sensitivity is fundamentally an important topic[1]. However, in our work, we are specifically considering robustness, which is clearly and solely defined as the capability to preserve the original output under external perturbations. Therefore, there is no ambiguity in the definition.

References:

[1] Kim et al. Balancing Robustness and Sensitivity using Feature Contrastive Learning. arXiv:2105.09394 (2021).

[2] Chen et al. From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework. Findings of ACL 2023.

[3] Lee et al., Holistic Evaluation of Text-to-Image Models, NeurIPS 2023 Track on Datasets and Benchmarks.

Thank you for the valuable review. Below we address the detailed comments and hope that you may find our response satisfactory.

Question 1: The authors need to provide enough details to guarantee its reproducibility.

Thanks for the comments and we make the following clarifications. Firstly, Section 3.3 contains detailed information about our method. In addition, the code we submitted includes sufficient details to ensure the reproducibility of our method, as agreed by Reviewer BYdd.

Question 2: Based on the examples provided in the paper, successful attacks have led to changes in the subject of prompts. The authors need to demonstrate the generation of significantly different images when the perturbed prompt and the original prompt should result in similar images.

We appreciate the comments and would like to provide the following clarifications. As illustrated in Figure 9 in the Appendix, the diffusion model has good robustness to random text perturbations, i.e., it can generate realistic and aligned images even when most words in the text prompt are perturbed. As a comparison, our method only modifies one or two words in the prompt to generate semantically different images. Although the typo may lead to nonsense words (e.g., “pamda”), they are realistic errors of humans. The other kinds of perturbations (glyph and phonetic) are more imperceptible but can lead to significantly different images, as shown in Figures 1, 6, and 7.

Question 3: The authors should include experimental results for single image guidance in the 'real-world attack experiment' section.

Thank you for the kind suggestion. We are delighted to add the experimental results for single-image guidance in the real-world attack experiment in the revision. Due to the extensive workload in this experiment, we have temporarily only added results for single image guidance on the ChatGPT-GP dataset in the real-world attack experiment with the 2ST attack objective. Once other results have been obtained, we will update the results to the revision as soon as possible. Table 5 in Appendix C.3 shows that the $Adv. S_{I2T}$ and human evaluation scores associated with this objective are relatively worse. This observation suggests that the attack with the DI objective(i.e. single-image guidance attack) may be susceptible to overfitting on a single image. We also show this table below：