---

---

Summary of the Review & Response

---

---

We greatly appreciate the reviewer's positive assessment of our work. The reviewer finds our approach novel (S1), our paper well written, well motivated (S2), and our work to be novel and of high quality in general (Q). We are glad that the reviewer identifies our usage of multiple SID and T2I methods in evaluation (S4), and the overall high success rate of our black-box method (S3) as strengths.

Primary concerns by the reviewer include:

1) extension to other text-description datasets (W1)

2) ablation on the number of samples needed to compute PolyJuice directions (W2 & W3)

3) potential defense mechanisms (W4 & W5 and Lm1 &Lm2)

We address:

1) by extending the evaluation to a new text description dataset (PartiPrompts) in Table R5

2) by providing ablation results on the effect of the number of samples on the success rate and clarifying the pipeline for selecting these samples (Table R6)

3) by discussing the potential defense mechanism that will be added to the final version of the manuscript (Table R7)

In the following, we address each of the concerns raised by the reviewer in detail.

---

---

Response to Weaknesses

---

---

W1. The authors only make use of the COCO datasets. Despite them providing reasons for only using this dataset, it would still be useful to make use of other datasets, or even using text prompt outside the COCO dataset textual descriptions for comparison.

We thank the reviewer for their suggestion, which improves the breadth of the paper. As noted in Sec. 4, we use COCO as it contains diverse text prompts that cover different domains, such as humans, animals, natural scenery, and common objects. This ensures the training data is not domain-specific. Per the reviewer’s suggestion, we evaluate PolyJuice attacks (directions learned from COCO) on a subset of text prompts from the PartiPrompts dataset (Yu et al, 2022) and present the results in Table R5. The results demonstrate the generalizability of the PolyJuice attacks to text prompts outside COCO.

Table R5. Attack success rate (%) of PolyJuice on text descriptions from the PartiPrompts dataset.

---

W2. …The authors do not conduct any ablation study on the impact of the number of TP & FN has on the performance of PolyJuice.

To address this concern, we estimate the shift directions using 16K, 40K, and 52K data samples from FLUX[dev] and use them to perform attacks against UFD. The results presented in Table R6 demonstrate that approximating the steering directions with 16K data points yields the same result as the 40K we use in the paper (SR of 96.3%). Using more data points (52K) leads to a very small increase in success rate (SR of 96.5%), which implies the low sensitivity of PolyJuice SR to the number of training samples. This discussion and results will be added to the revised paper.

Table R6. Effect of the number of samples on PolyJuice's success.

---

W3 Further to the above point, could the authors give more detail of how they constructed the 20k TP and 20k FN images? …

First, using a T2I model (e.g. FLUX), we generate a set of fake images using the text prompts available in the COCO training set. These fake images are then labeled by a given black-box SID as fake (TP) or real (FN). To have a balanced dataset, we sample an equal number (up to 20K) of FN and TPs from the labeled data. We will further clarify these details in the experiment settings subsection of Sec. 4 in the revised paper.

---

W4. The authors defence mechanism involves the use of PolyJuice images to calibrate SID models, could the authors also train a SID classifier using PolyJuice images within the dataset and evaluate its robustness.

We would like to note that the calibration strategy used in the paper is based on EER (Equal Error Rate), which is a widely accepted method in the literature. Per the reviewer’s suggestion, we take the RINE detector and fine-tune it (by training an MLP head) using the attacks generated by PolyJuice. We then compare the detection rate of the original RINE, PolyJuice-calibrated RINE (Sec 4.3, Table 3), and the PolyJuice-trained RINE. Table R7 shows the results of the comparison, which demonstrate the contribution of PolyJuice in improving the defense of the SID (lowering FNR). From the result, we also find that the calibration of SIDs leads to a better FNR than fine-tuning.

Table R7. Comparison of the FNR of the original RINE, PolyJuice-calibrated RINE, and PolyJuice-trained RINE.

---

W5. The paper lacks more discussion on the defence methods against this attack, which raises ethical questions.

We thank the reviewer for identifying this concern. First, PolyJuice requires a certain number of queries to the SID in order to construct the dataset and compute the steering direction. Therefore, commonly adopted security barriers, such as a rate limits, can raise the difficulty of applying PolyJuice maliciously. Apart from security barriers, a model can become robust to PolyJuice by learning a representation space that collapses the TP and FN distributions. For more details, please refer to the response to L1.

In summary, while we acknowledge the potential for misuse, we believe that raising awareness of this vulnerability ultimately benefits the community. By highlighting the issue, our work can inspire further research into developing effective mitigation strategies.

---

---

Response to Limitations

---

---

L1. …For instance, the authors visualise the CLIP embeddings of both PolyJuice and non-PolyJuice aided synthetic images, which show two clear clusters of images, could this be used to develop some mitigation strategy?

One possible mitigation strategy can be training a head on top of the CLIP feature space that collapses the FN and TP clusters. We want to find a subspace that is orthogonal to the direction of shift between these two clusters, while preserving the information of the target attribute (i.e., real vs. fake). Ideas from invariant representation learning, such as AIFL (Xie et al. 2017) or FairerCLIP (Dehdashtian et al. 2024), can be adopted for this purpose. Specifically, in the notation of FairerCLIP, the target $Y$ would be real vs. fake label, while the attribute to be removed, $S$, would be TP vs. FN. This discussion can enrich the paper, so it will be added to the revised version of the paper.

---

L2. Lastly, a discussion of how the result of this work can influence policy should be had, even if only brief. I.e. The authors mention that images generated by their method seem to consist of warmer colours? Could such aspects of images help identify synthetic images which could be written into a policy?

We would like to clarify that the common pattern analyzed in Section B.1 of the supplementary material is SID-specific and is not generalizable to other SIDs, since the vulnerability of each model is different. As a result, PolyJuice needs to be applied on each SID to find the vulnerability of the model, and this is not necessarily the same for other detectors. However, a per-SID policy can be written based on the common patterns.

---

---

Please consider raising the scores if we address your concerns. If you have more questions, we would be happy to answer them.

---

---

Reference

Yu et al. "Scaling autoregressive models for content-rich text-to-image generation." TMLR (2022).

Xie, Qizhe, et al. "Controllable invariance through adversarial feature learning." NeurIPS (2017).

Dehdashtian et al. "Fairerclip: Debiasing clip's zero-shot predictions using functions in rkhss." ICLR (2024).

---

---

Summary of the Review & Response

---

---

We thank the reviewer for their thoughtful comments. We are encouraged that the reviewer finds our empirical results strong (S1), our problem domain to be of critical importance (S2), and our black-box threat model practical in the context of real-world settings (S3).

Concerns by the reviewer include:

1) certain stylistic choices made in the paper (W1)

2) lack of a formal definition of the threat model (W2)

3) Potential confusion about the latent space and the computation of the steering directions (W1 & W3)

We address

1) by explaining our rationale behind our stylistic choices

2) by formally defining our threat model

3) by providing clarifying details

---

---

Response to Weaknesses

---

---

W1a. The writing and the general structure of this paper needs to be improved, including how the latent space shifting helps to evade the detection.

We explain in Section 3.1 how PolyJuice identifies a direction in the T2I latent space that statistically correlates with the SID’s prediction of “realness”. This is based on a distribution shift between the latents of samples predicted as real versus fake (see Fig. 1b). By universally steering the latent in this direction during the image generation process (Fig. 2), we create samples that are more likely to land in SID failure regions (as further visualized in Fig. 5 and Fig. 7).

---

W1b. I saw too many unnecessary bolded text in the paper, which may reduce the readability of the paper.

We put these short, bold sentences at the beginning of the paragraphs so that a first-time reader can quickly skim the whole paper and get a general idea of the paper. We would appreciate it if the reviewer could point to the exact places where the bold text is reducing the readability of the paper, so we can address them in the revised version.

We hope these clarifications make the core mechanism of PolyJuice more accessible and improve the overall readability of the paper.

---

W2. Threat Model is Missing: I didn't find the threat model for the proposed attacks in this paper, including the description to the attacker's capability, goal and attack scenarios.

We are encouraged that in item 3 of the strength section, the reviewer finds our threat model practical and applicable to real-world scenarios. We adopt the suggestion of the reviewer and formally define the threat model for our method in the following.

We extend the notation defined in Sec. 2 of the paper to our threat model.

Threat Model:

• Attacker’s Goal: Given a text prompt $c$ and a latent $z$, the attacker aims to generate synthetic images using a text-to-image (T2I) generative model $G:\mathcal{Z}\times\mathcal{C}\rightarrow\mathcal{X}$ that deceives a target synthetic image detector (SID) $f:\mathcal{X}\rightarrow[0,1]$ into misclassifying them as real (class $0$).
• Attacker’s Capability:
  • Black-box access to the SID: The attacker can only query the SID and observe hard labels $Y$ (real/fake), without access to model weights or gradients.
  • Full access to the T2I generative model: The attacker can manipulate the latent space $Z$ and control the generation process of a text-to-image (T2I) model.
  • Sufficient number of queries: The attacker can generate a dataset of image-latent-label triplets to analyze the SID's behavior in response to various inputs.
• Attack Scenario:
  • Step 1: The attacker queries the black-box SID with fake images and obtains hard labels, constructing a dataset of TP and FN samples.
  • Step 2: The attacker pre-computes steering directions in the T2I latent space that correlate with increased probability of being misclassified as real by the SID (Eq. 3).
  • Step 3: At test time, the attacker applies this universal direction to arbitrary prompts, producing images that evade detection by the SID (Eq. 6).

We will add this formal threat model to the revised version of the paper.

---

W3a. …Is the latent space of vae or just the embedding space in CLIP?

The latent space $Z$ mentioned in the paper is the latent space of the VAE used by a latent diffusion/flow-based T2I model, and not the embedding space of a CLIP model.

---

W3b. …how PolyJuice find the optimization direction of the images?

We first form a set of true positive (TP) and false negative (FN) images using hard labels $Y$ obtained from the black box SID. Our goal is to find the direction of the distribution shift from TP to FN (see Fig. 1b) in each image generation step. These directions are estimated by maximizing the statistical dependence (measured through HSIC) between the latents $Z$ and the labels $Y$, i.e., calculating SPCA. This optimization has a closed-form solution (L121 - L123). At inference time, these directions are used to steer the trajectory of the T2I model (Eq. 6) such that the final generated image is guided towards high error rate regions (see Fig. 2).

---

---

Please consider raising your score if we have addressed your concerns. If you have additional questions, we would be happy to answer them.

---

---

---

---

Summary of Reviews & Response

---

---

We are encouraged that the reviewer highlights the novelty (S1), efficiency (S2), effectiveness (S3), and practical utility (S4) of our red-teaming approach as well as the clarity of our paper (S5) and its broader applicability beyond SID (in “comment on scope”).

Reviewer’s primary concerns are:

1) literature review (W1 & 2, Q2, L2)

2) paper’s claims (W4, 5, 7 & 8)

3) image quality evaluation (W3)

We address:

1) by demonstrating PolyJuice’s effectiveness on four additional SID models in Table R3, and by comparing it to a transfer-based black-box attack (Q1) in Table R4. We point out that the paper already evaluates against RINE (ECCV 2024), a recent SID that is as recent as or newer than the reviewer’s suggestions (W2) and has a lower FNR (Table R3).

2) by clarifying misunderstandings on latent space size, data requirements, and SPCA components. We refer to Table 3, where we empirically show that PolyJuice reduces SID error (and also in Table R7 reviewer 9sRv-W4), directly supporting our red-teaming claims. Although we discuss PolyJuice’s applicability to commercial SIDs, we make no performance claims, as access to those models was not granted.

3) by providing image quality score (CLIP IQA) and prompt alignment score in Table R2.

---

---

Response to Weaknesses

---

---

W1a. Comparison to Kotyan et al. (EvoSeed)

EvoSeed is not scalable to modern T2I models. It needs up to $\tau \times (4 + 3 \times \log(n))$ = 16,500 imgs / 1 atk for SD. To match our experiments (SR(%) on 1000 atks), it needs up to 16.5M images, which is practically infeasible.

---

W1b. Comparison to transfer-based attack

We perform a transfer attack on the RINE detector with DiffPGD and a surrogate (Q2). Table R4 shows that PolyJuice achieves a better SR(%) than the transfer attacks.

Table R4. SR(%) of PolyJuice vs. regular ($x^n$) & realistic ($x^n_0$) attack from transferred DiffPGD$^t$.

---

W1c. Comparison to white-box attack

We believe such a comparison is not informative. A black-box attack with 40% SR appears weak vs. a white-box attack with 90%, but if the theoretical max. for black-box attacks is 41%, then it's very strong. So, comparing against white-box is not enough to judge a black-box attack.

---

W2. Not SOTA / Limited SID

We believe there is a misunderstanding. While the reviewer questions UFD’s relevance, they overlook our inclusion of RINE (ECCV 2024), a recent and competitive SID published alongside CoDE and after DRCT, which qualifies it as a SOTA academic SID. UFD remains a critical baseline, widely used or built upon by DRCT, CoDE, NPR, FatFormer, and RINE.

To address the reviewers’ suggestions (2n7t & 719g), we evaluate PolyJuice against four new SIDs and report strong improvements in Table R3.

Table R3. SR(%) of unsteered SD3.5 samples vs. PolyJuice.

---

W3a. Image Quality Metrics

Table R2 reports (i) CLIP-IQA scores for image quality and (ii) CLIP alignment scores for prompt consistency. The results show that PolyJuice significantly boosts attack success while preserving both quality and alignment, confirming it targets SID blind spots rather than exploiting degradation.

Table R2. Quality & alignment scores vs. SR (%) on UFD & RINE, using FLUX[dev].

---

W3b. Regarding Zhong et al.

Zhong et al. show that UFD has up to 8% drop when applied to degraded images, compared to clean images. In stark contrast, PolyJuice improves the attack success rate against UFD by up to 67%. This substantial gap shows that the improved attack success is not caused by degradation (supported by Table R2).

---

W3c. Stance on slight degradation

Although Table R2 shows that PolyJuice preserves image quality, it’s worth noting that many recent SIDs are explicitly trained to be robust against degradations like blurring, pixelization, etc. (e.g. CoDE, Sec. 3-2, pg. 5). Therefore, if a slightly degraded image still bypasses such defenses, it exposes a critical failure mode that needs to be addressed.

---

W4a. “large-p, small-n” assumption

The reviewer’s assumption is incorrect. For 256x256 images, the latent size in the FLUX/SD3.5 is p=16,384 (i.e. 16x32x32), while we use n=40,000 data points. Moreover, due to the resolution transferability of PolyJuice (Sec 4.4), we do not need to estimate this direction again for higher resolutions. Table R6 (reviewer 9sRv-W2) also shows that the eigenvector is not sensitive to the number of data points.

---

W4b. Selection of SPCA eigenvectors

For $Y$ with $m$ classes, subspace’s dimensionality (num. of non-zero $\sigma_i$) is $d=m-1$. In our binary case, $d=1$ and Eq. (4) becomes $\delta=\sigma_0U_0$ as is expected by the reviewer. We will clarify this in the revised paper.

---

W5. Samuel et al. & Reverse Process Validity

Samuel et al. is only concerned with the initial Gaussian noise prior, $x_T \sim \mathcal{N}(0, \mathbf{I})$, but not the trajectory. They show that if $x_T$ is perturbed such the norm deviates from $\sqrt{d}$, the image quality significantly drops. There is no contradiction with our findings, as we do not apply any perturbation on the initial noise; we only apply the direction starting from step 1 (L134 & Eq 6).

Previous papers (Universal Guidance) justify empirically choosing steering strength. Table R2 shows that PolyJuice’s success is not due to image degradation.

---

W6. Ablation on Hyperparameters

We ablate w.r.t the number of samples by using n = {16K, 40K, 52K}. The results, shown in Table R6 (reviewer 9sRv, W2), indicate that PolyJuice directions are not sensitive to n. We also visualize the correlation between successful attacks and (i) steering strength $\lambda$ and (ii) steering timesteps in Fig. 9 and Fig. 10, respectively.

---

W7. Overarching Red-Teaming Claims

We show in Sec. 4.3 that using PolyJuice attacks to calibrate the detectors improves their detection rate by up to 30%. In fact, this is acknowledged as a strength by the reviewer (S4. Practical utility), and also by reviewer 71g9 (S4). Table R7 (rev. 9sRv, W4) also supports this.

---

W8. Untested Commercial Claims

We clarify that PolyJuice does not claim to demonstrate empirical performance on commercial models but rather enables the possibility of red teaming them due to its black-box design.

During the rebuttal period, we were unable to access commercial SIDs due to either their ToS or the companies’ disagreement.

---

Comment on Scope

We appreciate the reviewer’s recognition of the method’s broad applicability. Our work focuses on red teaming SIDs due to the urgent need to address risks posed by AI-generated content (AIGC), making it a timely and critical problem. While this paper targets SIDs, we hope future works explore its applications to other domains.

---

---

Response to Questions

---

---

Q1. …approximate the steering direction within 5 pp …?

As shown in Table R6 (reviewer 9sRv-W2), approximating the steering directions with 16K points yields the same result as the 40K we use in the paper (SR of 96.3%). Using more data points, 52K, leads to a very small increase (96.5%), which alludes to the low sensitivity of directions to n.

---

Q2. …DiffPGD trained on a surrogate SID?

Addressed in W1 & Table R4; we apply DiffPGD (with 2 T2Is) on surrogate SID (ResNet50) and transfer to black-box RINE. We find that PolyJuice has better attack performance.

---

Q3. Non-COCO domains

PolyJuice's primary focus is on red teaming general AI-gen. image detectors, rather than domain-specific ones. As a dataset beyond COCO, we also evaluate on PartiPrompts (Table R5, reviewer 9sRv-W1).

---

Q4. Sensitivity to $d$ and $\lambda$

The dimensionality of the subspace $d$ is determined by the number of classes of $Y$ and in the binary case is equal to one; therefore there is no variability here wrt $d$. For $\lambda$ sensitivity, please refer to Sec. A.3 supplementary material.

---

Q5. Possible usage of non-linear SPCA

Thank you for the thoughtful question. As noted in L320, while a non-linear kernel may offer a boost in success rate by improving separability, it adds significant complexity due to the need for solving a pre-image problem.

---

---

Response to Limitations

---

---

We address (L1) in response to W5 of the reviewer 9sRv, (L2) in W1, W2, (L3) in Sec 6 & W4 5 of the reviewer 9sRv. We would appreciate clarification on the noisy TP+FN scenario described in (L4).

---

---

Please consider raising your score if we addressed your concerns. We are happy to answer more questions.

---

---

Thank you for detailed clarifications. I am partially convinced by the arguments made by authors.

Baseline Coverage

I appreciate the author's experiment on DiffPGD. That resolves my concern about baselines, since now there is atleast one attack compared quantitavely.

Though I am not convinced by the author's argument against comparision with white-box attacks. It is true thay black-box attack will be less potent than a white-box attack. However, I feel strongly, that we need to close the gap with better optimizations. I am not convinced that black-box attacks have a theoretical max. Black-box performance depends on the optimizations they used and such a commenting on the gap for this task would lead better to development of better black-box optimizations to close the gap. If the authors, really feel that there is a theoretical max, they need to demonstrate it.

Limited Detectors

I appreciate the author's clatification about RINE and additional of experiments using CoDE and DRCT. This resolves my concerns about limited evaluation on SID Detectors.

Image Quality Metrics

I appreicate the author's addition of CLIP-based Image Metrics that resolve my concern to some extent. I'd like to note that I share authors opinion here that PolyJuice doesnot degrade image quality. As such I like to see concrete irrrefutable evidence for it. Otherwise it's quite easy to misunderstand that vulnervaility against image quality is being exploited.

Therefore, I would urge the authors, to also use some non-CLIP (or in general model) based Image Quality Metrics like FID to measure image degradation. Since it's possible for CLIP to have some robustness against Image degradation and as such these degradations will not be measured using CLIP-based or any general model based IQM.

Theoretrical Claims

I appreciate the author's clarification on large-p, small-n assumption of mine. While I agree that authors have used large n, small p cases in the experiments. I think it's pretty clear that this is a theoretical limitation and as such should be mentioned explictly in the article to prevent the cases of large-p and small-n experiments by people experimenting.

Reverse Process Validity

I am not convinced with the argument made against reverse process validity. As I mentioned in my earlier comment, I share the opinion that image degradation is not happening, but I am not throughly convinced theoretically or emprically. Authors have to understand image degradation goes beyond perceivable degradation like blurring. For example, JPEG compression at 85% since results the image visually similar but some information is lost. Thererfore, while exemplar images can demonstrate perceivable degradations are not happening, I am not convinced whether non-perceviable degradations are happening or not.

Ablation Studies

Thank you for the experiments, I have no further concerns about ablations.

Overarching Claims

Thank you for the argument and directing me towards section 4.3. I misjuuged the Table 3.

Commerical Claims

I stand by my point that unless any commerical system is attacked, it should not be mentioned as a contribution. It would be misleading to say that your attack can be tried on commercial systems, but there's always a chance that the attack might fail. Unless you demonstrate explictly, that the attack doesnot fail for commerical systems, it should not be listed as a contribution. Its more like a future work or so, maybe in conclusions (But I would argue even against it).

Scope

I highly feel the work is much more beyond than attack against SID.

Questions

Thank you for the answers.

Final Comments: Once again thank you for your rebuttal. All of my concerns except concrete evidence against image degradation (Emperical: Point on Image Quality Metrics, Theoretical: Reverse Process Validity) is addressed , as such I am willing to raise my score. I again urge the authors to provide concrete evidence that cannot be argued to show that any form of vulnervaility against image degradation is not being exploited by the Polyjuice.

---

---

Summary of the Review & Response

---

---

We thank the reviewer for their positive feedback. We are pleased that the reviewer finds the manuscript to be well-written, clearly motivated, and easy to follow (S1), and we view this as a key strength of our presentation. The reviewer identifies that our paper addresses an urgent and practical challenge regarding AI-generated image detection (S2) through a novel attack strategy (S3). Further, the reviewer highlights our positive contribution to improving detection methods (S4).

Primary concerns and questions are regarding:

1) whether the attack directions are reusable between detectors (W1, L1)

2) whether the method generalizes over diverse prompts (W2)

3) prompt faithfulness and quality of the generated images (W3)

4) experiments on additional SID models (W4)

We address:

1) by clarifying that the attack directions are SID-specific, and that transfer attacks are more relevant for white-box approaches

2) by highlighting the diversity of the prompts used in PolyJuice evaluation (Table R1); we also extend PolyJuice to captions beyond COCO using PartiPrompts (Table R5, response to reviewer 9sRv, W1)

3) by computing CLIP alignment score (prompt faithfulness) and CLIP-IQA score (quality) in Table R2

4) by evaluating the effectiveness of PolyJuice on 4 additional SID models (Table R3)

---

---

Response to Weaknesses

---

---

W1. It is unclear whether the observed latent space distribution shift is consistent across different detectors. If the distribution shift is detector-specific, how transferable or generalizable is the proposed PolyJuice approach?

We first clarify that the distribution shift is detector-specific, as different detectors have distinct vulnerabilities. The shifts are not consistent unless the models share similar weaknesses. As shown in Sec B.1 of the supplementary material, the common patterns differ across models, reflecting variations in training data and techniques. In general, transferability is more relevant in the context of white-box attacks where one must craft an attack on a surrogate detector in order to apply a transfer attack to a target black-box model. In contrast, we focus on black-box attacks where we are able to target each model directly.

---

W2. The method assumes that a linear direction in latent space can universally guide images towards detector failure modes. Is this assumption valid across diverse prompts and image categories? A discussion or ablation study would help clarify.

Thank you for your suggestion, we agree that such an ablation can be useful to the paper. We first inspect the successful attacks generated by PolyJuice using COCO validation prompts and categorize the attacks according to available meta-labels. Table R1 shows the fraction of successful attacks per prompt category. We observe that PolyJuice improves the success rate across all categories, demonstrating that the direction is universally valid across diverse prompts and image categories. We also extend PolyJuice to another diverse dataset, i.e., PartiPrompts (see Table R5, in response to reviewer 9sRv-W1).

Table R1. Success rate per prompt category in unsteered vs. PolyJuice (on COCO).

---

W3. While the method improves attack success rates, it is important to analyze whether the manipulated/generated images remain visually realistic and prompt-faithful. Otherwise, there’s a risk that PolyJuice may just generate degraded or unrealistic images that trivially evade detection.

First, we want to clarify our stance on how an SID should process an “unrealistic” image versus a “degraded” one. Since the goal of red teaming is to discover and mitigate the failure modes, finding attacks that are not necessarily realistic (e.g., cartoonish features that fool the SID) is also important to us, since it reveals the failure mode of the target SID. An ideal SID must trivially detect these kinds of unrealistic images as fake.

We agree with the reviewer’s remark that it is important to generate attacks that are prompt faithful, and of reasonable image quality. To measure prompt faithfulness, we compute the CLIP alignment score between the generated attacks and the input prompts; to measure image quality, we use the CLIP IQA score (Wang et al. 2023). We present the alignment and image quality score alongside the success rate for FLUX[dev] in Table R2. We find that the CLIP scores and CLIP-IQA scores are comparable to those of the unsteered models, suggesting that PolyJuice is not trivially evading detection by degrading the images.

Table R2. Quality & alignment scores vs. SR (%) on UFD and RINE, using FLUX[dev]

---

W4. The authors evaluate PolyJuice primarily on UFD and RINE as target synthetic image detectors. However, it remains unclear how well the method generalizes to more recent and potentially more robust detectors such as FatFormer (CVPR 2024) and NPR (CVPR 2024).

First, we note that RINE (ECCV 2024), which is evaluated in our paper, is a recent and concurrent work to the detectors noted by the reviewer. We thank the reviewer for suggesting these SID models that can enrich the empirical results of the paper. As such, we adopt additional SID models: NPR and FatFormer (suggested by reviewer 71g9), as well as DRCT and CoDE (suggested by reviewer 2n7t) for further evaluating the effectiveness of PolyJuice. The additional results are shown in Table R3. These results show the effectiveness of PolyJuice in improving the success rate of the attacks by 56.7% on CoDE, 75% on DRCT, 82% on NPR, and 56% on FatFormer. These results will be added to Table 1 of the paper in the final revision.

Table R3. SR(%) of unsteered SD3.5 samples vs. PolyJuice.

---

---

Response to Questions

---

---

Q2. Is the observed latent space distribution shift consistent across different synthetic image detectors? Specifically, does PolyJuice need to re-estimate the steering direction for each SID individually, or can the same direction be transferred between detectors?

As mentioned in the response to W1, different models have different vulnerabilities, so the directions are different. As such, the directions need to be estimated for each SID. However, estimating the steering directions requires a negligible computational cost since once we generate the images and latents for the first SID, we need to 1) predict the labels using the new SID, 2) calculate the eigenvalue problem in Eq. 3.

---

Q3. How does the latent steering affect the visual quality and semantic consistency of the generated images? Are there metrics or human evaluations to ensure the attack does not simply degrade image realism?

The latent steering performed by PolyJuice has a negligible effect on the visual quality and semantic consistency of the generated images, as shown in qualitative analysis in Sec B of the supplementary material. Further, we employ CLIP IQA score and CLIP alignment score to measure image quality and prompt faithfulness of the generated attacks compared to unsteered images, and the results are provided in Table R2. The results support our remark that the quality and consistency are practically unchanged by PolyJuice while the attack success rate (SR) is improved significantly.

---

---

Response to Limitations

---

---

L. The proposed method relies on identifying a latent space shift that separates correctly and incorrectly classified samples for a specific detector. This shift may not generalize across detectors, potentially limiting the reusability of the learned steering direction.

We mentioned that the directions are not necessarily transferable across SID due to the difference in their vulnerabilities. However, if two SIDs’ predictions have a significant overlap with each other, then the directions from one SID can potentially be useful in attacking the other SID.

---

---

Please consider updating your scores if we have addressed your concerns. If you have more questions, we would be happy to answer them.

---

---

References

Wang et al. "Exploring clip for assessing the look and feel of images." AAAI (2023).